Pith. sign in

REVIEW 16 cited by

Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.15553 v2 pith:3TILJDF3 submitted 2024-10-21 cs.CL

Yun He , Di Jin , Chaoqi Wang , Chloe Bi , Karishma Mandyam , Hejia Zhang , Chen Zhu , Ning Li
show 11 more authors
Tengyu Xu Hongjiang Lv Shruti Bhosale Chenguang Zhu Karthik Abinav Sankararaman Eryk Helenowski Melanie Kambadur Aditya Tayade Hao Ma Han Fang Sinong Wang
This is my paper · ORCID
classification cs.CL
keywords instructionsllmsmulti-ifmultilingualmulti-turnfollowinglanguagesmodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in various tasks, including instruction following, which is crucial for aligning model outputs with user expectations. However, evaluating LLMs' ability to follow instructions remains challenging due to the complexity and subjectivity of human language. Current benchmarks primarily focus on single-turn, monolingual instructions, which do not adequately reflect the complexities of real-world applications that require handling multi-turn and multilingual interactions. To address this gap, we introduce Multi-IF, a new benchmark designed to assess LLMs' proficiency in following multi-turn and multilingual instructions. Multi-IF, which utilizes a hybrid framework combining LLM and human annotators, expands upon the IFEval by incorporating multi-turn sequences and translating the English prompts into another 7 languages, resulting in a dataset of 4,501 multilingual conversations, where each has three turns. Our evaluation of 14 state-of-the-art LLMs on Multi-IF reveals that it presents a significantly more challenging task than existing benchmarks. All the models tested showed a higher rate of failure in executing instructions correctly with each additional turn. For example, o1-preview drops from 0.877 at the first turn to 0.707 at the third turn in terms of average accuracy over all languages. Moreover, languages with non-Latin scripts (Hindi, Russian, and Chinese) generally exhibit higher error rates, suggesting potential limitations in the models' multilingual capabilities. We release Multi-IF prompts and the evaluation code base to encourage further research in this critical area.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+

    cs.CL 2026-08 conditional novelty 7.0 of 10

    Across 30 languages, commercial LLMs outscore all open-weight models in every EU language, and non-English service costs more and scores lower, suggesting equality requires resources beyond public web crawls.

  2. Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understanding

    cs.CL 2026-07 conditional novelty 7.0 of 10

    A 209-task Chinese benchmark across six dialogue-failure modes shows that even the top frontier model fully satisfies only 41.1% of long multi-turn requests.

  3. AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios

    cs.AI 2025-05 conditional novelty 7.0 of 10

    AgentIF introduces a realistic, long-form instruction-following benchmark for agentic scenarios and shows that current LLMs follow fewer than 30% of such instructions perfectly.

  4. Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Skill-Use is a 79-skill, 177-task benchmark showing that LLM agents fail to reliably retrieve, follow, and respect the boundaries of skills under progressive disclosure, with harness choice shifting model rankings.

  5. HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A new 65-task benchmark measures whether AI agents obey long company handbooks across multi-tool workflows; the best model passes 36.2% under strict grading.

  6. In-Place Tokenizer Expansion for Pre-trained LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Continuing a model's own BPE merges and training only new embedding rows preserves quality while cutting token counts 2.4–4× for previously under-tokenized languages.

  7. TextQuests: How Good are LLMs at Text-Based Video Games?

    cs.AI 2025-07 conditional novelty 6.0 of 10

    Frontier LLMs complete few of 25 Infocom text adventures even when given the official hint booklets, revealing a weakness in sustained long-context reasoning.

  8. Marco-Bench-MIF: On Multilingual Instruction-Following Capability of Large Language Models

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Marco-Bench-MIF localizes IFEval into 30 languages with cultural adaptation and finds large resource gaps and scale effects in multilingual instruction following.

  9. How Many Instructions Can LLMs Follow at Once?

    cs.AI 2025-07 conditional novelty 6.0 of 10

    IFScale measures instruction-following at densities from 10 to 500 constraints and finds that even top frontier models satisfy only about two-thirds of 500 simultaneous keyword instructions.

  10. VerIF: Verification Engineering for Reinforcement Learning in Instruction Following

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A hybrid verifier that combines rule-based code checks and a reasoning-LLM judge enables reinforcement learning to improve LLM instruction following on several benchmarks.

  11. Evaluating the Sensitivity of LLMs to Prior Context

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Prior conversational context, especially from a different knowledge domain, can sharply reduce LLM multiple-choice accuracy, and repeating the task near the query mitigates the drop.

  12. LIFEBench: Evaluating Length Instruction Following in Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    LIFEBench's evaluation of 26 LLMs shows most follow short length instructions but degrade sharply beyond a few hundred words, and none reliably hit vendor-claimed maximum output lengths.

  13. Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Across 23 reasoning models on 420 constrained math problems, stronger reasoning-oriented training and longer chains of thought are associated with worse adherence to user-specified constraints.

  14. Mind the Gap! Choice Independence in Using Multilingual LLMs for Persuasive Co-Writing Tasks in Different Languages

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Users who first used a Spanish AI writing assistant subsequently used the English AI writing assistant less, suggesting a spillover that violates choice independence.

  15. SEADialogues: A Multilingual Culturally Grounded Multi-turn Dialogue Dataset on Southeast Asian Languages

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    SEADialogues is a culturally grounded multi-turn dialogue dataset covering eight Southeast Asian languages.

  16. AgentScope 1.0: A Developer-Centric Framework for Building Agentic Applications

    cs.AI 2025-08 unverdicted novelty 4.0 of 10

    AgentScope 1.0 packages the components needed to build, evaluate, and deploy LLM agent applications into one developer framework.

Pith tools