REVIEW 9 cited by
FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The ability to follow instructions is crucial for Large Language Models (LLMs) to handle various real-world applications. Existing benchmarks primarily focus on evaluating pure response quality, rather than assessing whether the response follows constraints stated in the instruction. To fill this research gap, in this paper, we propose FollowBench, a Multi-level Fine-grained Constraints Following Benchmark for LLMs. FollowBench comprehensively includes five different types (i.e., Content, Situation, Style, Format, and Example) of fine-grained constraints. To enable a precise constraint following estimation on diverse difficulties, we introduce a Multi-level mechanism that incrementally adds a single constraint to the initial instruction at each increased level. To assess whether LLMs' outputs have satisfied every individual constraint, we propose to prompt strong LLMs with constraint-evolution paths to handle challenging open-ended instructions. By evaluating 13 closed-source and open-source popular LLMs on FollowBench, we highlight the weaknesses of LLMs in instruction following and point towards potential avenues for future work. The data and code are publicly available at https://github.com/YJiangcm/FollowBench.
Forward citations
Cited by 9 Pith papers
-
AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios
AgentIF introduces a realistic, long-form instruction-following benchmark for agentic scenarios and shows that current LLMs follow fewer than 30% of such instructions perfectly.
-
HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following
A new 65-task benchmark measures whether AI agents obey long company handbooks across multi-tool workflows; the best model passes 36.2% under strict grading.
-
How Many Instructions Can LLMs Follow at Once?
IFScale measures instruction-following at densities from 10 to 500 constraints and finds that even top frontier models satisfy only about two-thirds of 500 simultaneous keyword instructions.
-
A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedback
MultiCodeIF introduces a 2,021-task, 14-language benchmark with 27 constraint types to evaluate code instruction following, finding that multi-level constraints sharply reduce model success and iterative feedback subs...
-
Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models
Across 23 reasoning models on 420 constrained math problems, stronger reasoning-oriented training and longer chains of thought are associated with worse adherence to user-specified constraints.
-
IHEval: Evaluating Language Models on Following the Instruction Hierarchy
IHEval shows that current language models often follow lower-priority instructions over system messages, and simple prompting does not fix the problem.
-
QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewriting
A contextual bandit that chooses among five query-rewrite strategies, conditioned on 17 linguistic features, reduces LLM hallucination on QA benchmarks and beats static prompting and no-rewrite baselines.
-
Evaluating LLM Agent Adherence to Hierarchical Safety Principles: A Lightweight Benchmark for Probing Foundational Controllability Components
A lightweight grid-world benchmark shows that LLM agents pay a task-performance cost for following safety principles and that high adherence can mask inability rather than principled choice.
-
Verifiable Format Control for Large Language Model Generations
A fully verifiable format-following dataset and a progressive SFT-plus-DPO self-improvement pipeline improve 7B LLMs' format control, with mixed out-of-domain transfer.
Discussion (0). Continue with ORCID to comment.