Pith. sign in

REVIEW 2 major objections 6 minor 6 cited by

AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AgentIF benchmark shows even the best LLM follows under 30% of real agent instructions perfectly.

desk verdict A substantial benchmark artifact whose headline failure rates still rest on an unvalidated GPT-4o judge; accept conditionally and demand evaluator checks. read the letter →

arxiv 2505.16944 v1 pith:WTGS2XBT submitted 2025-05-22 cs.AI cs.CL

Yunjia Qi , Hao Peng , Xiaozhi Wang , Amy Xin , Youfeng Liu , Bin Xu , Lei Hou , Juanzi Li This is my paper · ORCID
classification cs.AIcs.CL
keywords instructionfollowingLLMagentsbenchmarkconstraintsatisfactiontoolconstraintsconditionalsystempromptsevaluationmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces AgentIF, a benchmark that measures instruction following in the long, constraint-dense settings where LLM agents actually operate, rather than in short synthetic prompts. AgentIF consists of 707 human-annotated instructions drawn from 50 real agent applications, averaging 1,723 words and 11.9 constraints each, with a constraint taxonomy covering formatting, semantic, and tool requirements in vanilla, conditional, and example forms. Evaluating 15 current models, the authors report that the best performer, o1-mini, satisfies only 59.8% of constraints and completes just 27.2% of instructions with every constraint met. The paper argues that this gap—especially on tool specifications and condition-triggered constraints—is a fundamental bottleneck for building reliable LLM agents and that existing benchmarks have hidden it.

What carries the argument

The load-bearing object is the AgentIF dataset and its constraint annotation pipeline: instructions are split into semantic blocks, constraints are extracted block-wise by an LLM and then cross-validated across blocks, and human annotators verify every constraint. Each constraint is assigned an evaluation method—deterministic Python code, LLM judgment, or a hybrid that uses an LLM to extract a span and code to check it—and conditional constraints are first checked for whether their trigger fires. The two metrics, CSR and ISR, convert this machinery into a single difficulty measure: CSR reports individual constraint compliance while ISR reports how often a model follows the entire instruction.

What would settle it

Take a sample of unmodified user requests logged by deployed agent systems, pair them with the production system prompts used at the time, and run the same model suite with the same scoring protocol; if models satisfy a much higher fraction of these real constraints than the roughly 60% CSR and 27% ISR reported on AgentIF, the central difficulty claim would not transfer. A cheaper version is to measure constraint coverage: verify that AgentIF's constraint taxonomy covers the constraints human raters extract from those real logs.

Watch

Extended reading notes

Core claim

The central claim is that current LLMs are not reliably instruction-following in realistic agentic scenarios, and that the failure is systematic rather than due to a few hard prompts. The evidence comes from AgentIF, where every instruction is decomposed into atomic constraints, each scored by a deterministic or LLM-based check, and aggregated into constraint success rate (CSR) and instruction success rate (ISR). Across the model suite, the best CSR is 59.8 and the best ISR is 27.2, with GPT-4o dropping from 87.0 CSR on IFEval to 58.5 on AgentIF. Error analysis attributes the largest losses to conditional constraints (models misjudging whether a condition fires) and tool constraints (omitted required tools, disallowed tool use, wrong names or parameters), while instruction-level success collapses as length passes 6,000 words.

Load-bearing premise

The benchmark's claim to realism rests on the assumption that GPT-4o-generated user queries, rewritten by human annotators, faithfully represent the requests real users make to agent systems, and that the extracted constraints capture what those instructions actually require; if either fails, the reported difficulty rankings may not transfer to deployed agents.

Editorial extensions

If this is right

  • Agent developers should treat instruction following, not just tool-calling skill, as a prerequisite bottleneck: the best model in the suite still fails one in four constraints overall and misses at least one constraint in nearly three of every four instructions.
  • Tool constraints are the weakest category for every model tested, so system prompts that restrict or prescribe tools need dedicated evaluation and likely dedicated post-training data.
  • Conditional constraints fail in part because models misjudge whether the stated condition has fired; prompting or training that makes condition checks explicit could recover a measurable share of errors.
  • Performance degrades smoothly with instruction length and constraint count, and near zero ISR appears above 6,000-word instructions, suggesting long agent prompts should be split into shorter sub-instructions.
  • Meta constraints—selection, detailing, and prioritization among constraints—are present in about 25% of instructions and are handled worst, indicating conflicts between constraints are a distinct failure mode.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The realism assumption is testable: comparing AgentIF failure rates against a sample of unmodified production user queries would show whether the reported difficulty ranking holds outside the benchmark.
  • The evaluation machinery doubles as a data-generation recipe for post-training, since condition-check questions and stand-alone constraints are exactly the pairs an instruction-following fine-tuning set needs.
  • The meta-constraint results point toward a concrete intervention worth testing: giving meta constraints explicit higher priority, as instruction-hierarchy methods propose, may resolve many selection and prioritization errors.
  • By construction the benchmark does not cover prompt engineering, so it remains open whether simple restatements, such as repeating tool signatures near the call site, close part of the tool-constraint gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces AgentIF, a benchmark for evaluating instruction following in agentic scenarios. It is constructed from 50 real-world agents (40 open-source and 10 industrial), containing 707 instructions that average 1,723 words and 11.9 constraints each. Constraints are classified along two dimensions: type (formatting, semantic, tool) and presentation (vanilla, conditional, example); each constraint is paired with a code, LLM, or hybrid verification method. The paper evaluates 15 proprietary and open-source LLMs with greedy decoding and reports constraint success rate (CSR) and instruction success rate (ISR). The best model (o1-mini) achieves a CSR of 59.8 and an ISR of 27.2, and the paper identifies tool and condition constraints as the most difficult. Additional analyses examine error causes on conditional and tool constraints, performance versus instruction length and constraint count, and a newly proposed category of meta constraints.

Significance. If the evaluation pipeline is trustworthy, AgentIF would be a valuable contribution: it addresses a genuine gap left by short synthetic benchmarks such as IFEval and SysBench, and it demonstrates that long, tool-heavy agentic prompts are far harder to follow than the short instructions used in prior evaluations. The dataset is human-annotated, code and data are released, the evaluation mixes deterministic code checks with LLM judgments, and the main experiments use temperature 0, which aids reproducibility. The paper also provides a useful error analysis that separates condition-check failures from constraint-following failures. The significance is conditional, however, on validating the LLM-judge component, because the headline numbers cannot be interpreted until the evaluator's agreement with human judgments is quantified.

major comments (2)
  1. [§3.4, Appendix E] All LLM and hybrid evaluations are scored by gpt-4o-2024-11-20, and for conditional constraints the same model decides whether the condition is triggered, yet the paper reports no human-validated accuracy, inter-annotator agreement, or error analysis of the evaluator judgments. The manual review described in §3.2 covers the generated evaluation scripts and questions, not the evaluator's decisions on model responses. Because ISR requires every constraint to pass, a single incorrect judgment reverses an instruction-level outcome; for conditional constraints, errors in the trigger decision compound before constraint checking. This unvalidated component is load-bearing for Table 2's absolute rates and for Figure 1(b)'s difficulty ordering (tool/condition hardest). I request a validation study on a stratified sample across constraint types and evaluation modes, reporting agreement between gpt-4o and human judgments, or an explicit argument and supporting evidence that such validation is unnecessary.
  2. [§3.2] The claim that AgentIF reflects real-world agentic usage rests on GPT-4o-generated user queries that are then rewritten by human annotators; the paper provides no evidence that the generated queries are representative of actual user requests (e.g., no comparison with real usage logs, and the industrial agents' real queries are deliberately excluded). Without such evidence the transferability of the difficulty ranking to deployed systems is uncertain. Please report the annotation instructions, the number and degree of query rewrites, and any plausibility checks, or temper the 'Realistic' claim accordingly.
minor comments (6)
  1. [References] References [36] and [37] are the same IFEval paper listed twice with different numbers, and the citation markers in the text are inconsistent; please merge them or correct the numbering.
  2. [§2, §3.3, §5] The average instruction length is given as 1,723 words in the abstract and §3.3, as 1,700 tokens in §2, and as 1,717 tokens in the conclusion, while the average constraint count is reported as 14 in §2 but 11.9 elsewhere; please standardize these statistics and state the units explicitly.
  3. [Figure 5 caption] The caption says 'Gray lines show results of the top 6 models in Figure 2,' but Figure 2 is an example instruction; the reference should be to Table 2 or Figure 1(b).
  4. [§4.2] The claim that 'GPT-4o drops from 87.0 to 58.5' relative to IFEval is not supported by any evaluation reported in this paper or by a citation to the source of the 87.0 figure; please provide the reference or remove the comparison.
  5. [§4.5] The meta-constraint analysis introduces a new category without defining its place in the annotation and evaluation protocol; please clarify whether meta constraints are included in the 11.9 per-instruction constraint count and how the success rates in Figure 6(b) are computed.
  6. [Throughout] Minor typographical issues include 'exclued' in §3.4 and 'Pequirements' in Appendix C.1, and Figure 1(b)'s caption directs readers to §3.2 for constraint descriptions when the taxonomy is actually presented in §3.1.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the benchmark's scores are independently measured, the self-citations are interpretive rather than load-bearing, and the GPT-4o-as-judge loop is a validity risk with no by-construction reduction.

full rationale

AgentIF's central empirical claims — best model o1-mini reaches CSR 59.8 / ISR 27.2, and tool/condition constraints are the hardest (Table 2, Fig. 1b) — are measured outcomes of an executable evaluation protocol, not quantities derived from fitted parameters or presupposed by the construction. The chain query-generation → constraint-extraction → evaluation-method generation → human review → per-response scoring (Sections 3.2–3.4) contains no equation-level reduction: the code-based check_following functions and the LLM-judge validation questions must actually inspect each response, and the judge renders negative verdicts even for GPT-4o itself (e.g., 26.4 tool-constraint and 35.1 condition-constraint success), so the instrument discriminates. The self-citations are real but not load-bearing. Reference [20] (Peng et al. 2023, overlapping authors) is invoked only to interpret the headline difficulty finding ('The primary reason may be that the models fails in handling specification-heavy tasks [20, 2]'), hedged with 'may be', while the failure rates and error proportions it explains come from this paper's own Table 2 and Figures 4–5; reference [21] supplies the Crab-DPO baseline, which scores lowest — a self-critical, not self-serving, citation. The genuine risk — gpt-4o-2024-11-20 is simultaneously the query/constraint/evaluation-method generator, the LLM judge for LLM-based and hybrid constraints, and an evaluated model, with no human-accuracy or inter-annotator validation of evaluator judgments reported (Section 3.4, Appendix E) — is a measurement-validity and correctness concern (possible judge leniency, condition-trigger disagreement, and a confound between the tool/condition-vs-semantic difficulty ranking and evaluator strictness), not a circular derivation: nothing forces the judge's verdict by construction. Accordingly, no circular step meets the evidentiary bar, and the minor self-citation usage keeps this at 'no significant circularity' rather than higher.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

The benchmark's central claims rest on the quality of human annotation, the representativeness of synthetic queries, and the reliability of the GPT-4o judge. No numerical parameters are fitted to data; the constraint taxonomy and meta-constraint category are conceptual constructs rather than fitted quantities.

assumptions (3)
  • domain assumption Human-annotated constraints and evaluation methods faithfully capture the requirements in each agentic instruction.
    The benchmark's validity and all CSR/ISR scores depend on the accuracy of the human annotation step described in Section 3.2 and Appendix D. The paper does not report inter-annotator agreement; it mentions only three rounds of sampled review.
  • domain assumption GPT-4o-generated user queries, after human rewriting, are representative of real agentic user requests.
    Instructions are created by generating about 20 queries per agent from system prompts and rewriting them (Section 3.2). If real users ask differently, the benchmark's 'realistic' property and model failure rates may not transfer to deployed systems.
  • domain assumption GPT-4o is a reliable judge of constraint satisfaction in LLM-based and hybrid evaluations.
    Section 3.4 uses gpt-4o-2024-11-20 for semantic evaluation. Since GPT-4o is itself one of the evaluated models, judge bias is a possible confound for the headline comparison.
invented entities (1)
  • Meta constraints
    purpose: A taxonomy category for constraints that govern other constraints (selection, detailing, prioritization), introduced to explain a class of failures in agentic instruction following.
    Introduced in Section 4.5 based on qualitative observation that about 25% of instructions contain such constraints. No independent falsifiable prediction is made; it is a descriptive schema.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios." pith.science (2026). https://pith.science/paper/WTGS2XBT

@misc{pith2026250516944,
  author       = {Pith},
  title        = {Pith review of: AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WTGS2XBT}},
  note         = {Machine review of arXiv:2505.16944}
}
read the original abstract

Large Language Models (LLMs) have demonstrated advanced capabilities in real-world agentic applications. Growing research efforts aim to develop LLM-based agents to address practical demands, introducing a new challenge: agentic scenarios often involve lengthy instructions with complex constraints, such as extended system prompts and detailed tool specifications. While adherence to such instructions is crucial for agentic applications, whether LLMs can reliably follow them remains underexplored. In this paper, we introduce AgentIF, the first benchmark for systematically evaluating LLM instruction following ability in agentic scenarios. AgentIF features three key characteristics: (1) Realistic, constructed from 50 real-world agentic applications. (2) Long, averaging 1,723 words with a maximum of 15,630 words. (3) Complex, averaging 11.9 constraints per instruction, covering diverse constraint types, such as tool specifications and condition constraints. To construct AgentIF, we collect 707 human-annotated instructions across 50 agentic tasks from industrial application agents and open-source agentic systems. For each instruction, we annotate the associated constraints and corresponding evaluation metrics, including code-based evaluation, LLM-based evaluation, and hybrid code-LLM evaluation. We use AgentIF to systematically evaluate existing advanced LLMs. We observe that current models generally perform poorly, especially in handling complex constraint structures and tool specifications. We further conduct error analysis and analytical experiments on instruction length and meta constraints, providing some findings about the failure modes of existing LLMs. We have released the code and data to facilitate future research.

Figures

Figures reproduced from arXiv: 2505.16944 by the authors.

Figure 1
Figure 1. (a) The length distribution of instructions across AGENTIF (log-scale). (b) Success rates of several representative LLMs on different constraint dimensions (detailed descriptions are in § 3.2). benchmark IFEval [36] is synthetically constructed with various constraint types, such as formatting, and has an average instruction length of only 45 words. Subsequent studies have expanded the instruction scope to include m… view at source ↗
Figure 2
Figure 2. An example instruction of AGENTIF. We also find the performance declines as instruction length increases in AGENTIF. Additionally, we identify a novel category of constraints, meta constraints, which reflects underlying prioritization issues. In conclusion, advanced LLMs still struggle to follow real-world agentic instructions. 2 Related Work Instruction following is a fundamental capability of LLMs, referring to fo… view at source ↗
Figure 3
Figure 3. The data construction process and evaluation workflow of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Error proportions (%) on condition and tool constraints. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Success rates on instructions with varying length or constraint counts. Gray lines show [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Figure (a) illustrates three types of meta constraints and examples. Most meta constraints fall within the Constraint Selection category, which requires models to follow one specific constraint. Figure (b) presents the success rates of different investigated models on …
Figure 7
Figure 7. Figure 7: Distribution of constraint types. The inner ring shows the breakdown of con￾straint types into Semantic, Formatting, and Tool categories. The outer ring further catego￾rizes constraints based on their presentation type, including Vanilla, Condition, and Exam￾ple. Seman…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse

    cs.SE 2026-07 conditional novelty 6.5 of 10

    Code LLMs correctly label incorrect repair instructions as wrong, then follow them anyway, creating compounding Ghost Errors that self-guided iterative repair usually cannot reverse.

  2. OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agents

    cs.CL 2026-08 conditional novelty 6.0 of 10

    OneDayAgent, a decompose-memory-verify harness, reaches 0.821 on AgentIF-OneDay with a GLM-5.2 backend and runs across five LLMs without tuning.

  3. E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A controllable synthetic multi-step tool-use benchmark shows frontier LLMs remain unreliable on product-state changes, with Pass^3 below 60% (below 70% even with code).

  4. UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization and Distillation

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    UniComp finds that LLM compression preserves factual recall but degrades multi-step reasoning, multilingual ability, and reliability, while task-specific calibration recovers up to 50% of lost reasoning performance in...

  5. Controllable LLM Reasoning via Sparse Autoencoder-Based Steering

    cs.AI 2026-01 conditional novelty 6.0 of 10

    SAE-Steering finds, via keyword-logit recall plus effectiveness ranking, sparse-autoencoder features that steer a reasoning model into a chosen reasoning strategy, beating baseline steering by ~15% on a judge-based me...

  6. Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI

    cs.SE 2025-05 conditional novelty 3.0 of 10

    A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.

Reference graph

Works this paper leans on

47 extracted references · 22 canonical work pages · cited by 6 Pith papers

  1. [1]

    Introducing claude 3.5 sonnet, October 2024

    Anthropic. Introducing claude 3.5 sonnet, October 2024. Accessed: 2025-05-15

    work page 2024

  2. [2]

    Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks.arXiv preprint arXiv:2412.15204, 2024

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, et al. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks.arXiv preprint arXiv:2412.15204, 2024

    arXiv 2024

  3. [3]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. InProceedings of NeurIPs, pages 1877–1901, 2020

    work page 1901

  4. [4]

    Llm agents for education: Advances and applications

    Zhendong Chu, Shen Wang, Jian Xie, Tinghui Zhu, Yibo Yan, Jinheng Ye, Aoxiao Zhong, Xuming Hu, Jing Liang, Philip S Yu, et al. Llm agents for education: Advances and applications. arXiv preprint arXiv:2503.11733, 2025

    arXiv 2025

  5. [5]

    The llama 3 herd of models.arXiv e-prints, pages arXiv–2407, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv e-prints, pages arXiv–2407, 2024

    2024

  6. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  7. [7]

    A real-world webagent with planning, long context understanding, and program synthesis

    Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. InProceedings of ICLR, 2024

    work page 2024

  8. [8]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. InProceedings of ACL, pages 6864–6890, 2024

    work page 2024

Show all 47 references
  1. [9]

    Can large language models understand real-world complex instruc- tions? InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18188–18196, 2024

    Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Jiaqing Liang, and Yanghua Xiao. Can large language models understand real-world complex instruc- tions? InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18188–18196, 2024

    2024

  2. [10]

    Multi-if: Benchmarking llms on multi-turn and multilingual instructions following.arXiv preprint arXiv:2410.15553, 2024

    Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, et al. Multi-if: Benchmarking llms on multi-turn and multilingual instructions following.arXiv preprint arXiv:2410.15553, 2024

    arXiv

  3. [11]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  4. [12]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  5. [13]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023

  6. [14]

    Followbench: A multi-level fine-grained constraints following benchmark for large language models.arXiv preprint arXiv:2310.20410, 2023

    Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. Followbench: A multi-level fine-grained constraints following benchmark for large language models.arXiv preprint arXiv:2310.20410, 2023

    arXiv

  7. [15]

    T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024. 10

    2024 arXiv

  8. [16]

    Agentorca: A dual-system framework to evaluate language agents on operational routine and constraint adherence.arXiv preprint arXiv:2503.08669, 2025

    Zekun Li, Shinda Huang, Jiangtian Wang, Nathan Zhang, Antonis Antoniades, Wenyue Hua, Kaijie Zhu, Sirui Zeng, William Yang Wang, and Xifeng Yan. Agentorca: A dual-system framework to evaluate language agents on operational routine and constraint adherence.arXiv preprint arXiv:...

    2025 arXiv

  9. [17]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  10. [18]

    Prompt injection attack against llm-integrated applications.arXiv preprint arXiv:2306.05499, 2023

    Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, et al. Prompt injection attack against llm-integrated applications.arXiv preprint arXiv:2306.05499, 2023

    2023 arXiv

  11. [19]

    Large language model agent: A survey on methodology, applications and challenges.arXiv preprint arXiv:2503.21460, 2025

    Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. Large language model agent: A survey on methodology, applications and challenges.arXiv preprint arXiv:2503.21460, 2025

    2025 arXiv

  12. [20]

    When does in-context learning fall short and why? a study on specification-heavy tasks.arXiv preprint arXiv:2311.08993, 2023

    Hao Peng, Xiaozhi Wang, Jianhui Chen, Weikai Li, Yunjia Qi, Zimu Wang, Zhili Wu, Kaisheng Zeng, Bin Xu, Lei Hou, et al. When does in-context learning fall short and why? a study on specification-heavy tasks.arXiv preprint arXiv:2311.08993, 2023

    2023 arXiv

  13. [21]

    Constraint back- translation improves complex instruction following of large language models.arXiv preprint arXiv:2410.24175, 2024

    Yunjia Qi, Hao Peng, Xiaozhi Wang, Bin Xu, Lei Hou, and Juanzi Li. Constraint back- translation improves complex instruction following of large language models.arXiv preprint arXiv:2410.24175, 2024

    2024 arXiv

  14. [22]

    Sysbench: Can large language models follow system messages?arXiv preprint arXiv:2408.10943, 2024

    Yanzhao Qin, Tao Zhang, Yanjun Shen, Wenjing Luo, Haoze Sun, Yan Zhang, Yujing Qiao, Weipeng Chen, Zenan Zhou, Wentao Zhang, et al. Sysbench: Can large language models follow system messages?arXiv preprint arXiv:2408.10943, 2024

    2024

  15. [23]

    Infobench: Evaluating instruction following ability in large language models.arXiv preprint arXiv:2401.03601, 2024

    Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuan- sheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models.arXiv preprint arXiv:2401.03601, 2024

    2024 arXiv

  16. [24]

    Conifer: Improving complex constrained instruction-following ability of large language models

    Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang. Conifer: Improving complex constrained instruction-following ability of large language models. arXiv preprint arXiv:2404.02823, 2024

    2024 arXiv

  17. [25]

    Qwq: Reflect deeply on the boundaries of the unknown, November 2024

    Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. Accessed: 2025-05-15

    2024

  18. [26]

    Qwen3: Think deeper, act faster, April 2025

    Qwen Team. Qwen3: Think deeper, act faster, April 2025. Accessed: 2025-05-15

    2025

  19. [27]

    Glm-z1-32b-0414: A deep reasoning model, April 2025

    THUDM. Glm-z1-32b-0414: A deep reasoning model, April 2025. Accessed: 2025-05-15

    2025

  20. [28]

    The instruction hierarchy: Training llms to prioritize privileged instructions.arXiv preprint arXiv:2404.13208, 2024

    Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training llms to prioritize privileged instructions.arXiv preprint arXiv:2404.13208, 2024

    2024 arXiv

  21. [29]

    Benchmarking complex instruction-following with multiple constraints composition.Advances in Neural Information Processing Systems, 37:137610– 137645, 2024

    Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxing Xu, et al. Benchmarking complex instruction-following with multiple constraints composition.Advances in Neural Information Processing Systems, 37:137610– 137645, 2024

    2024

  22. [30]

    Codeif: Benchmarking the instruction-following capabilities of large language models for code genera- tion.arXiv preprint arXiv:2502.19166, 2025

    Kaiwen Yan, Hongcheng Guo, Xuanqing Shi, Jingyi Xu, Yaonan Gu, and Zhoujun Li. Codeif: Benchmarking the instruction-following capabilities of large language models for code genera- tion.arXiv preprint arXiv:2502.19166, 2025

    2025 arXiv

  23. [31]

    Large language model-brained gui agents: A survey.arXiv preprint arXiv:2411.18279, 2024

    Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, et al. Large language model-brained gui agents: A survey.arXiv preprint arXiv:2411.18279, 2024

    2024 arXiv

  24. [32]

    Appagent: Multimodal agents as smartphone users

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–20, 2025. 11

    2025

  25. [33]

    Cfbench: A comprehensive constraints-following benchmark for llms.arXiv preprint arXiv:2408.01122, 2024

    Tao Zhang, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, et al. Cfbench: A comprehensive constraints-following benchmark for llms.arXiv preprint arXiv:2408.01122, 2024

    2024

  26. [34]

    Pptagent: Generating and evaluating presentations beyond text-to-slides.arXiv preprint arXiv:2501.03936, 2025

    Hao Zheng, Xinyan Guan, Hao Kong, Jia Zheng, Weixiang Zhou, Hongyu Lin, Yaojie Lu, Ben He, Xianpei Han, and Le Sun. Pptagent: Generating and evaluating presentations beyond text-to-slides.arXiv preprint arXiv:2501.03936, 2025

    2025 arXiv

  27. [35]

    Least-to-most prompting enables complex reasoning in large language models

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, et al. Least-to-most prompting enables complex reasoning in large language models. InProceedings of ICLR, 2023

    2023

  28. [37]

    explain the concept in three steps

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023. 12 Appendices A Limitations We discuss the limitations as follows: (1) A...

    2023 arXiv

  29. [38]

    variable_name_1

    ATask Descriptionsummarizing what the agent is designed to do. 2.Input Variable Annotationslisting the input variables, each with a brief explanation. Your task is to: • Generate multiple sets of variable content that are rich, detailed, and expanded. Each set must include spe...

    2024

  30. [39]

    Ayes/no questionthat can be used to verify whether theconditionis satisfied

  31. [40]

    If...” or “When

    Themain constraintthat should be enforced if the condition is true. This should exclude the conditional part and be expressed as a standalone, unconditional constraint. Please follow these instructions: • If the condition refers to theinput query, the question should focus on ...

  32. [41]

    yes” or “no

    The question should beclear, specific, and binary— it should be answerable with “yes” or “no”

  33. [42]

    It must refer explicitly to what the constraint is checking (e.g., structure, length, tone, factuality)

  34. [43]

    intro”, “step 1

    If the constraint refers to a specific section (e.g., “intro”, “step 1”, “conclusion”), include that in the question. Return your answer in the following format: { "validation_question": "{Your full yes/no question here}? Please answer YES/NO directly and do not enter anything...

  35. [44]

    Extract the introduction part

    The instruction should clearly specifywhatto extract from the response (e.g., “Extract the introduction part”, “Extract the function used”, “Extract the final answer sentence”)

  36. [45]

    Base your output on the constraint provided below

  37. [46]

    You may refer to the instruction paragraphonly when the constraint is ambiguousand requires context

  38. [47]

    extraction_instruction

    Return your output as a JSON dictionary in the following format: { "extraction_instruction": "{your generated extraction instruction}. Return the extracted content verbatim from the response. If multiple segments are found, return them as a Python-style list of strings. If not...

  39. [48]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.