Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

Inverse IFEval tests whether LLMs can override training habits, and most models fail.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A new 1,012-question benchmark shows LLMs often fail instructions that deliberately invert common training conventions, revealing a measurable gap in counterintuitive instruction following.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A genuinely useful new benchmark with a real confound: two of its eight categories reward false statements, so the headline 'cognitive inertia' story is only partially supported. the 4 major comments →

arxiv 2509.04292 v1 pith:RVRR7ZZM submitted 2025-09-04 cs.CL

Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?

Qinyan Zhang , Xinping Lei , Ruijie Miao , Yu Fu , Haojie Fan , Le Chang , Jiafan Hou , Dingling Zhang
show 13 more authors
Zhongfei Hou Ziqiang Yang Changxin Pu Fei Hu Jingkai Liu Mengyun Liu Yang Liu Xiang Gao Jiaheng Liu Tong Yang Zaiyuan Wang Ge Zhang Wenhao Huang
This is my paper
classification cs.CL
keywords cognitive inertiacounterintuitive instructionsinstruction followingLLM evaluationbenchmarkout-of-distribution instructionsthinking mechanismsLLM-as-a-judge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models are trained on clean, standardized responses—correct facts, readable formatting, polished text—and this paper argues that the same training makes them stumble when an instruction asks for the opposite. To test that, the authors build Inverse IFEval, a 1,012-item Chinese/English benchmark with eight 'inverted' instruction types: flawed questions, deliberate typos, comment-free code, unformatted prose, intentionally wrong answers, reframed classic problems, mid-prompt instruction changes, and counterfactual passages. On this benchmark, leading models score far lower than on the original IFEval, and models tuned specifically to follow instructions drop the most. The authors conclude that current alignment overfits to training conventions and that a separate 'counterintuitive ability' deserves a place in evaluation.

Core claim

The central claim is that Inverse IFEval reveals a real gap: LLMs perform much worse on instructions that contradict training conventions—correct facts, readable formatting, commented code, stable instructions—than on ordinary instructions. The eight categories invert the idealized response patterns used in SFT annotation. Across 1,012 Chinese and English questions judged by an LLM-as-a-Judge validated at 98% agreement with human graders, the best model scores about 76, several fine-tuned models score in the 30s, and Qwen3-235B-A22B-Instruct drops from 5th on IFEval to 15th on Inverse IFEval. The authors conclude this measures a distinct counterintuitive ability that current alignment leaves

What carries the argument

The load-bearing object is a set of eight 'inverse instruction' templates, each obtained by negating a standard SFT annotation convention: flawed questions instead of clean stems, deliberate typos instead of polished text, comment-free code instead of best-practice code, unformatted prose instead of lists, intentionally wrong answers instead of correct ones, reframed classic problems instead of memorized solution paths, mid-prompt instruction reversals instead of one-shot instructions, and counterfactual passages instead of factual references. The benchmark then operationalizes these templates through a human-in-the-loop pipeline and an optimized LLM-as-a-Judge matrix with per-type judge mod

Load-bearing premise

The benchmark's practical force rests on treating deliberately nonsensical reversed instructions as a proxy for unusual-but-real user requests; if failing the proxy does not predict failing genuine long-tail instructions, the central conclusion about real-world robustness does not follow.

What would settle it

Take a set of unconventional but meaningful user requests, score model compliance by human raters, and correlate with Inverse IFEval scores. Weak correlation would refute the construct-validity claim. Alternative: fine-tune a model to pass the eight inverse templates and test on novel non-template instructions; no transfer would show the benchmark rewards narrow pattern-matching, not general adaptability.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Current IFEval-style leaderboards overstate real-world instruction-following flexibility; rankings shift substantially on inverse instructions.
  • Alignment work should treat adaptability under unconventional instructions as a target on par with fluency and factuality.
  • Thinking/reasoning modes help: non-thinking variants of the same models score consistently lower, so inference-time reasoning is part of the fix.
  • Test-time compute scaling (Best-of-N at N=32) pushes several models near 90, suggesting the capability is present but not reliably selected.
  • The benchmark is bilingual; models trained mainly on Chinese do better on the Chinese half, so language-specific training conventions matter.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the benchmark's items are deliberately meaningless (the authors say so), the practical payoff depends on a correlation the paper does not measure: that failure on 'answer with three typos' predicts failure on uncommon but meaningful user requests. A field study could settle this.
  • The same inversion logic could be applied to other modalities—vision-language instruction following, agentic tool-use conventions, or code-generation style constraints—to look for similar inertia.
  • If the diagnosis holds, one cheap intervention suggests itself: include a small fraction of anti-paradigm instructions in SFT/RLHF data so models learn to suppress default formats on demand.
  • The Best-of-N result hints that some failures are selection problems rather than capability gaps, meaning decoding-time or verifier-based methods might recover much of the lost performance without further training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Inverse IFEval, a bilingual (English/Chinese) benchmark of 1,012 items across eight 'inverse instruction' types that deliberately invert standard SFT response conventions (e.g., intentional typos, counterfactual answering, deliberately incorrect answers). The items are generated with a human-in-the-loop pipeline, reviewed by experts, and scored by an LLM-as-a-judge whose prompt and model are selected per instruction type, reaching a reported 98% accuracy on a validation set. Experiments on a wide range of closed- and open-source LLMs show best overall scores around 75%, with fine-tuned non-thinking models often scoring much lower, and the paper interprets this as evidence of 'cognitive inertia' — a distinct weakness in following unconventional instructions not captured by IFEval.

Significance. If the validity concerns can be addressed, Inverse IFEval would fill a real gap: existing instruction-following benchmarks focus on conventional, verifiable instructions, while long-tail user requests that conflict with training conventions are under-tested. The paper's strengths are the public release of the dataset, the systematic construction of eight well-differentiated instruction types, the bilingual coverage, the broad model evaluation, and the explicit comparison with IFEval showing rank reversals. The central claim, however, rests on the assumption that low scores on inverted instructions measure a coherent 'counterintuitive ability' rather than other, better-understood behaviors such as truthfulness, safety refusal, or failure to parse deliberately meaningless requests. The current evidence does not yet uniquely support that interpretation.

major comments (4)
  1. [§3.1, §2.2, Appendix C Figure 19] The two categories 'Deliberately Incorrect Answers' (186 items) and 'Counterfactual Answering' (108 items) constitute roughly 29% of the benchmark and reward outputs that are factually wrong. Appendix C, Figure 19, explicitly labels as an error Doubao-1.6-Thinking's factually correct answers to a 'only one correct' prompt. A low score on these categories can therefore reflect a model's truthfulness or safety training, not 'cognitive inertia' in the sense of overfitting to SFT formats. Since these categories are included in the aggregate scores of Table 3, the central inference that 'fine-tuned models perform poorly' is not uniquely supported. Please report per-category analyses that exclude or separate DIA and CA, and provide a failure-mode breakdown (refusal, adherence, partial execution) to distinguish truthfulness effects from format-inertia effects.
  2. [§3.1, §2.3, §2.5] The paper reports overall scores between roughly 30% and 75% and concludes that models 'fail substantially' and that 'the dataset effectively meets its intended purpose.' However, no random baseline, trivial baseline, or human performance is reported for the 1,012 items, and no inter-annotator agreement is reported for the final expert review step described in §2.3. Without a lower bound, it is unclear whether a score of 40% is low relative to chance or relative to what a careful human could achieve. I request random/trivial baselines for each instruction type (e.g., always follow the formatting constraint; always answer factually) and human expert scores on a representative sample, plus inter-annotator agreement metrics such as Cohen's kappa.
  3. [§2.5] The judge model and judging template are selected per instruction type on the validation set, and the reported 98% judge accuracy is presumably computed on the same validation set. This creates a risk of overfitting the judge to the validation items, so the 98% figure may not reflect accuracy on the final benchmark. Please either perform a held-out evaluation, use cross-validation, or at minimum report per-type judge accuracy on a disjoint test set and compare against a non-optimized judge baseline. This is load-bearing because the benchmark's conclusions depend on the reliability of the automated judge.
  4. [§2.2 Note, §5] The paper explicitly states that the instructions are not 'inherently meaningful in a practical sense' and draws an analogy to IQ tests, while also claiming the benchmark captures 'real instructions' and OOD robustness. This is an assertion of construct validity, not a demonstration. The leap from deliberately inverted, often meaningless instructions to real-world unconventional user requests needs empirical support. A concrete test would be to correlate Inverse IFEval scores with performance on naturally occurring but unconventional instruction sets (e.g., requests that contradict formatting defaults but remain semantically meaningful), or to show that Inverse IFEval predicts residual variance in an independent instruction-following robustness measure beyond IFEval.
minor comments (4)
  1. [§4] The related work section refers to 'INSTURCTIR' instead of 'InstructIR' (reference [19]); please correct this typo.
  2. [§3.2.1, Figure 4b] The text discusses AdaCoT, while Figure 4(b) is labeled 'AutoCoT'; also the legend 'w.o. CoT' and 'w. CoT' is unclear (probably 'without CoT' and 'with CoT'). Please align the notation.
  3. [§2.4, Table 2] The 'Q Length' value of 22.7 for Counter-Conventional Formatting appears anomalously low relative to the other types and to the example in Appendix A; please clarify the unit (characters, tokens, words) and verify the statistic.
  4. [Throughout] The terms 'Counter-intuitive Ability', 'Counter-Cognitive Ability', and 'counterintuitive ability' are used inconsistently; choose one term for the central construct and use it consistently.

Circularity Check

0 steps flagged

No significant circularity: the benchmark construction and evaluation are self-contained, with minor construct-validity limitations but no derivation-level circularity.

full rationale

The paper is a benchmark-construction and empirical-evaluation paper, not a mathematical or predictive derivation. There is no equation-level circularity, no fitted parameter renamed as a prediction, and no load-bearing self-citation chain. The dataset items are produced through a human-in-the-loop pipeline with expert review, and the LLM-as-a-Judge is calibrated against human labels to 98% agreement, providing independent grounding for the scoring. The central observation that fine-tuned models perform poorly is an empirical result, not an input forced by construction; Appendix B.1 shows that with Best-of-N sampling models can approach 90, indicating the benchmark is not trivially impossible. The authors explicitly disclaim in Section 2.2 (and Appendix A) that the instructions are "not inherently meaningful in a practical sense," which is a valid concern about external construct validity but not a circularity. The cited prior works by overlapping authors (e.g., IFEvalCode, CIF-Bench) are used only in benchmark comparisons, not as evidence for the central claim. Confounds such as truthfulness pressure in the DIA and CA categories are correctness/validity risks, not circular reasoning. Therefore no significant circularity is present.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 1 invented entities

The central claim rests primarily on the construct of cognitive inertia and on trusting the human-in-the-loop quality control and LLM judge. No numerical free parameters are fitted; the main design choices are the eight instruction types, per-type judge selection, and human review thresholds.

axioms (4)
  • domain assumption LLM-as-a-judge scores, validated at 98% accuracy on a human-labeled set, are a valid proxy for true instruction-following compliance on all 1,012 items.
    Section 2.5 establishes judge accuracy against human labels but assumes this accuracy generalizes across instruction types and held-out questions.
  • domain assumption SFT and post-training instill a robust 'idealized paradigm' that creates cognitive inertia.
    Sections 2.1 and 2.2 present this as an observation from annotation experience, not a causally demonstrated mechanism.
  • ad hoc to paper The eight reversed paradigms capture a single coherent construct of counterintuitive ability.
    Section 2.2 defines the eight types; no factor analysis or external validity evidence supports treating them as one construct.
  • domain assumption Human verification removes ambiguity and ensures each item has an unambiguous correct behavior.
    Section 2.3 describes expert review but reports no inter-annotator agreement for the final 1,012-item dataset.
invented entities (1)
  • Cognitive inertia (Counter-Cognitive Ability) no independent evidence
    purpose: Explanatory construct for why LLMs fail instructions that invert training conventions.
    Term introduced in Sections 1 and 2.1; the only evidence is performance on the benchmark itself, with no external predictive test.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?." pith.science (2026). https://pith.science/paper/RVRR7ZZM

@misc{pith2026250904292,
  author       = {Pith},
  title        = {Pith review of: Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RVRR7ZZM}},
  note         = {Machine review of arXiv:2509.04292}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs) achieve strong performance on diverse tasks but often exhibit cognitive inertia, struggling to follow instructions that conflict with the standardized patterns learned during supervised fine-tuning (SFT). To evaluate this limitation, we propose Inverse IFEval, a benchmark that measures models Counter-intuitive Abilitytheir capacity to override training-induced biases and comply with adversarial instructions. Inverse IFEval introduces eight types of such challenges, including Question Correction, Intentional Textual Flaws, Code without Comments, and Counterfactual Answering. Using a human-in-the-loop pipeline, we construct a dataset of 1012 high-quality Chinese and English questions across 23 domains, evaluated under an optimized LLM-as-a-Judge framework. Experiments on existing leading LLMs demonstrate the necessity of our proposed Inverse IFEval benchmark. Our findings emphasize that future alignment efforts should not only pursue fluency and factual correctness but also account for adaptability under unconventional contexts. We hope that Inverse IFEval serves as both a diagnostic tool and a foundation for developing methods that mitigate cognitive inertia, reduce overfitting to narrow patterns, and ultimately enhance the instruction-following reliability of LLMs in diverse and unpredictable real-world scenarios.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios

    cs.AI 2026-07 conditional novelty 6.0

    A controllable synthetic multi-step tool-use benchmark shows frontier LLMs remain unreliable on product-state changes, with Pass^3 below 60% (below 70% even with code).

  2. Seed1.8 Model Card: Towards Generalized Real-World Agency

    cs.AI 2026-03 unverdicted novelty 5.0

    Seed1.8 is a new foundation model that adds unified agentic capabilities for search, code execution, and GUI interaction to existing LLM and vision strengths.

  3. Seed2.0 Model Card: Towards Intelligence Frontier for Real-World Complexity

    cs.AI 2026-06 unverdicted novelty 2.0

    Seed2.0 model series reports gains in reasoning, visual understanding, search, and reliability on intricate long-horizon tasks via an internal evaluation system.

Reference graph

Works this paper leans on

44 extracted references · 16 canonical work pages · cited by 3 Pith papers · 2 internal anchors

  1. [1]

    gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025

    Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Visit-bench: A benchmark for vision-language instruction following inspired by real-world use

    Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schmidt. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use. arXiv preprint arXiv:2308.06595, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Training verifiers to solve math word problems.arXiv preprint arXiv: Arxiv-2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv: Arxiv-2110.14168, 2021

    Pith/arXiv arXiv 2021

  4. [4]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Instructblip: Towards general-purpose vision-language models with instruction tuning.Advances in neural information processing systems, 36:49250–49267, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning.Advances in neural information processing systems, 36:49250–49267, 2023

    2023

  6. [6]

    M-IFEval: Multilingual Instruction-Following Evaluation

    Antoine Dussolle, Andrea Cardeña Díaz, Shota Sato, and Peter Devine. M-ifeval: Multilingual instruction-following evaluation. arXiv preprint arXiv:2502.04688, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    Ifeval-audio: Benchmarking instruction-following capability in audio-based large language models, 2025

    Yiming Gao, Bin Wang, Chengwei Wei, Shuo Sun, and AiTi Aw. Ifeval-audio: Benchmarking instruction-following capability in audio-based large language models, 2025. URLhttps://arxiv.org/abs/2505.16774

    arXiv 2025

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Glore: When, where, and how to improve llm reasoning via global and local refinements

    Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Raileanu. Glore: When, where, and how to improve llm reasoning via global and local refinements. arXiv preprint arXiv:2402.10963, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  11. [11]

    Ciem: Contrastive instruction evaluation method for better instruction tuning.arXiv preprint arXiv:2309.02301, 2023

    Hongyu Hu, Jiyuan Zhang, Minyi Zhao, and Zhenbang Sun. Ciem: Contrastive instruction evaluation method for better instruction tuning.arXiv preprint arXiv:2309.02301, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Followeval: A multi-dimensional benchmark for assessing the instruction-following capability of large language models.arXiv preprint arXiv:2311.09829, 2023

    Yimin Jing, Renren Jin, Jiahao Hu, Huishi Qiu, Xiaohua Wang, Peng Wang, and Deyi Xiong. Followeval: A multi-dimensional benchmark for assessing the instruction-following capability of large language models.arXiv preprint arXiv:2311.09829, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Gonzalez, and Ion Stoica

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.ArXiv, abs/2406.11939, 2024. URL https://api.semanticscholar.org/CorpusID:270562889

    Pith/arXiv arXiv 2024

  14. [14]

    CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models

    Yizhi Li, Ge Zhang, Xingwei Qu, Jiali Li, Zhaoqun Li, Zekun Wang, Hao Li, Ruibin Yuan, Yinghao Ma, Kai Zhang, et al. Cif-bench: A chinese instruction-following benchmark for evaluating the generalizability of large language models. arXiv preprint arXiv:2402.13109, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  15. [15]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Exploring and evaluating hallucinations in llm-powered code generation.arXiv preprint arXiv:2404.00971, 2024

    Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, Li Zhang, Zhongqi Li, and Yuchi Ma. Exploring and evaluating hallucinations in llm-powered code generation.arXiv preprint arXiv:2404.00971, 2024

    arXiv 2024

  17. [17]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  18. [18]

    Adacot: Pareto-optimal adaptive chain-of-thought triggering via reinforcement learning.arXiv preprint arXiv:2505.11896, 2025

    Chenwei Lou, Zewei Sun, Xinnian Liang, Meng Qu, Wei Shen, Wenqi Wang, Yuntao Li, Qingping Yang, and Shuangzhi Wu. Adacot: Pareto-optimal adaptive chain-of-thought triggering via reinforcement learning.arXiv preprint arXiv:2505.11896, 2025. 13

    Pith/arXiv arXiv 2025

  19. [19]

    Instructir: A benchmark for instruction following of information retrieval models.arXiv preprint arXiv:2402.14334, 2024

    Hanseok Oh, Hyunji Lee, Seonghyeon Ye, Haebin Shin, Hansol Jang, Changwook Jun, and Minjoon Seo. Instructir: A benchmark for instruction following of information retrieval models.arXiv preprint arXiv:2402.14334, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022

    Pith/arXiv arXiv 2022

  21. [21]

    Instruction tuning with gpt-4.arXiv preprint arXiv:2304.03277, 2023

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4.arXiv preprint arXiv:2304.03277, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    Instruction tuning with loss over instructions.Advances in Neural Information Processing Systems, 37:69176–69205, 2024

    Zhengyan Shi, Adam X Yang, Bin Wu, Laurence Aitchison, Emine Yilmaz, and Aldo Lipani. Instruction tuning with loss over instructions.Advances in Neural Information Processing Systems, 37:69176–69205, 2024

    2024

  23. [23]

    Can chatgpt replace traditional kbqa models? an in-depth analysis of the question answering performance of the gpt llm family

    Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. Can chatgpt replace traditional kbqa models? an in-depth analysis of the question answering performance of the gpt llm family. InInternational Semantic Web Conference, pages 348–367. Springer, 2023

    work page 2023

  24. [24]

    Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Improving llm code generation with grammar augmentation.CoRR, 2024

    Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. Improving llm code generation with grammar augmentation.CoRR, 2024

    work page 2024

  26. [26]

    Can chatgpt defend its belief in truth? evaluating llm reasoning via debate

    Boshi Wang, Xiang Yue, and Huan Sun. Can chatgpt defend its belief in truth? evaluating llm reasoning via debate. arXiv preprint arXiv:2305.13160, 2023

    Pith/arXiv arXiv 2023

  27. [27]

    A review on code generation with llms: Application and evaluation

    Jianxun Wang and Yixiang Chen. A review on code generation with llms: Application and evaluation. In2023 IEEE International Conference on Medical Artificial Intelligence (MedAI), pages 284–289. IEEE, 2023

    2023

  28. [28]

    Measuring short-form factuality in large language models

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. 2024. URL https://api. semanticscholar.org/CorpusID:273877483

    work page 2024

  29. [29]

    Benchmarking complex instruction-following with multiple constraints composition.Advances in Neural Information Processing Systems, 37:137610–137645, 2024

    Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxing Xu, et al. Benchmarking complex instruction-following with multiple constraints composition.Advances in Neural Information Processing Systems, 37:137610–137645, 2024

    work page 2024

  30. [30]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Ifevalcode: Controlled code generation, 2025

    Jian Yang, Wei Zhang, Shukai Liu, Linzheng Chai, Yingshui Tan, Jiaheng Liu, Ge Zhang, Wangchunshu Zhou, Guanglin Niu, Zhoujun Li, Binyuan Hui, and Junyang Lin. Ifevalcode: Controlled code generation, 2025. URL https://arxiv.org/abs/2507.22462

    Pith/arXiv arXiv 2025

  32. [32]

    Glm-4.5: Agentic, reasoning, and coding (arc) foundation models.arXiv preprint arXiv:2508.06471, 2025

    Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models.arXiv preprint arXiv:2508.06471, 2025

    Pith/arXiv arXiv 2025

  33. [33]

    A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods.arXiv preprint arXiv:2403.02901, 2024

    Yang Zhang, Hanlei Jin, Dan Meng, Jun Wang, and Jinghua Tan. A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods.arXiv preprint arXiv:2403.02901, 2024

    arXiv 2024

  34. [34]

    Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    problem trap

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for llm question answering with external tools.Advances in Neural Information Processing Systems, 36:50117–50143, 2023. 14 Appendix A Demonstration of Adversarial Instructions This section shows the types of these eight instructions, their corresponding regular training parad...

    work page 2023

  36. [36]

    The combination aids in bone strength, gut health, and sustained satiety

    Greek yogurt – Thick, strained yogurt containing roughly 10 grams of protein per 100g, along with probiotics, calcium, and vitamin B12. The combination aids in bone strength, gut health, and sustained satiety

    work page

  37. [37]

    These nutrients promote brain development, eye health, and efficient metabolism

    Eggs – One large egg provides around 6 grams of high-quality protein together with choline, vitamin D, and lutein. These nutrients promote brain development, eye health, and efficient metabolism

    work page

  38. [38]

    avoidance list

    Tuna (canned in water) – Offers about 30 grams of protein per 100g and is a notable source of omega-3 3 fatty acids, vitamin D, and selenium. This profile helps lower inflammation, supports cardiovascular health, and improves cognitive function. Error Analysis: The formatting requirements for the “avoidance list” are not met. Figure 18 Error Analysis For ...

    work page

  39. [39]

    Which celestial body does the Earth revolve around?

    work page

  40. [40]

    What is 9 multiplied by 3?

    work page

  41. [41]

    What is the capital city of Japan?

    work page

  42. [42]

    Which country has the largest land area in the world?

    work page

  43. [43]

    What is the date of France’s Bastille Day? Answer:

    work page

  44. [44]

    only one correct answer

    July 14 Error Analysis: The model is answering questions normally, completely ignoring (not following) the instruction of “only one correct answer”. Error Instruction Type: Counter-Conventional Formatting Prompt: Outline the detailed process and key precautions for planning a one-week trip. Do not format your answer as a list or as paragraph breaks. Answe...

    work page

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.