Pith. sign in

REVIEW 27 references

DecIF: Improving Instruction-Following through Meta-Decomposition

T0 review · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DecIF generates high-quality instruction-following training data from scratch with meta-decomposition and response filtering, and SFT with it improves IFEval, Multi-IF, FollowBench, and LiveBench scores over prior synthetic data methods.

arxiv 2505.13990 v2 pith:F6GQR6O3 submitted 2025-05-20 cs.CL

Tingfeng Hui , Pengyu Zhu , Bowen Ping , Ling Tang , Guanting Dong , Yaqi Zhang , Sen Su This is my paper · ORCID
classification cs.CL
keywords instruction-followingdecifllmsdatainstructionflexibilityfurthergeneralizability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most instruction-following data is made by taking existing documents or conversations and rewriting them into prompts. DecIF instead starts with nothing but a large language model. The model first names broad real-world domains, such as sports, education, or technology, then names abstract requests inside each domain, and then turns each request into concrete scenarios with people, places, and conditions. This three-level decomposition is meant to keep the final prompts diverse and controllable.

After a scenario exists, DecIF attaches one to five verifiable constraints, such as word limits, required keywords, or formatting rules. The model checks the assembled instruction for internal conflicts and rewrites it if needed. Next, an LLM writes an answer, and then the instruction is broken back down into atomic yes or no evaluation questions, for example, does the response include a title in angle brackets. Only responses that pass every yes or no check are kept for training.

The authors fine-tune smaller models such as LLaMA-3.1-8B and Qwen-3-8B on this generated data and report consistent gains over existing synthetic datasets on four instruction-following benchmarks. They also show the data can be mixed into Tulu-3's training mixture without hurting other skills. The main caveats are that the constraint list is taken from the IFEval benchmark used for evaluation, and that the model acting as judge is the same model that wrote the answers, so some of the reported gains may be specific to this setup.

Extended reading notes

Core claim

Table 1 reports that on LLaMA-3.1-8B, DecIF with 10k synthesized pairs reaches 70.98 prompt-level strict IFEval and 64.75 Multi-IF turn 1, versus 53.97 and 52.55 for UltraIF 10k; with 30k pairs it exceeds Tulu-3-IF 30k on IFEval, FollowBench, and LiveBench while staying close on Multi-IF. The paper claims DecIF is a fully autonomous framework that constructs high-quality instruction-following data without relying on any external documents, datasets, or human-annotated resources.

Load-bearing premise

DecIF assumes the supervising LLM is a reliable oracle for every stage: generating diverse domains, requests, scenarios, and constraints, detecting conflicts, decomposing instructions into atomic evaluation criteria, and correctly labeling its own responses as YES or NO. If this self-verification is biased, for example if the judge favors short or generic answers, the all-YES filter would systematically select easy data rather than correct data. The authors acknowledge this dependence in the Limitations section: 'Since DecIF relies entirely on LLMs for data synthesis, it exhibits a strong dependence on the capabilities of the underlying models.'

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on hand-set generation and filtering choices and on unverified assumptions about the supervising model's self-generation and self-judgment abilities. No physical or external entities are introduced.

free parameters (5)
  • constraint-count probability vector p = [0.2, 0.3, 0.3, 0.1, 0.1]
    Hand-set distribution over sampling 1-5 constraints per instruction (Algorithm 1 line 15; Section E.1); changes complexity and diversity of generated data.
  • generation scales (D, R, S, iterations) = D=25 per iteration, 1000 iterations, R=30 per domain, S=20 per request
    Section E.1; these counts determine dataset size and diversity.
  • sampling temperature and top_p = temperature=0.6, top_p=0.95
    Section E.1; data synthesis sampling hyperparameters.
  • SFT learning rate, epochs, max length = lr=1e-5, 3 epochs, max_len=8192
    Section E.2; training settings for instruction-following data.
  • response retention rule = retain only if all atomic criteria are YES
    Section 2.3; hand-set filtering threshold, with no ablation on partial credit.
assumptions (5)
  • domain assumption A single LLM prompt can elicit a diverse, globally representative set of real-world domains, requests, and scenarios without external grounding.
    Section 2.2 meta-generation steps; if diversity collapses, DecIF data becomes homogeneous.
  • domain assumption The supervising LLM can reliably detect and fix conflicts among constraints in generated instructions.
    Section 2.2 consistency check; no independent conflict detector is used.
  • domain assumption Atomic yes/no evaluation questions completely capture an instruction's requirements, and the LLM judge answers them correctly.
    Section 2.3 and Appendix E.4 judge prompt; filtering quality depends on this.
  • domain assumption The IFEval-derived constraint pool is a sufficient ontology of verifiable constraints for general instruction-following.
    Appendix E.3 states constraint types are primarily derived from Zhou et al. 2023; all DecIF instructions are built from this pool.
  • domain assumption SFT on the filtered synthetic pairs transfers to unseen instruction-following benchmarks and does not degrade common capabilities.
    Tables 1-3 and 6-7 provide supporting evidence, but this is an empirical premise about distributional transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DecIF: Improving Instruction-Following through Meta-Decomposition." pith.science (2026). https://pith.science/paper/F6GQR6O3

@misc{pith2026250513990,
  author       = {Pith},
  title        = {Pith review of: DecIF: Improving Instruction-Following through Meta-Decomposition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F6GQR6O3}},
  note         = {Machine review of arXiv:2505.13990}
}
read the original abstract

Instruction-following has emerged as a crucial capability for large language models (LLMs). However, existing approaches often rely on pre-existing documents or external resources to synthesize instruction-following data, which limits their flexibility and generalizability. In this paper, we introduce DecIF, a fully autonomous, meta-decomposition guided framework that generates diverse and high-quality instruction-following data using only LLMs. DecIF is grounded in the principle of decomposition. For instruction generation, we guide LLMs to iteratively produce various types of meta-information, which are then combined with response constraints to form well-structured and semantically rich instructions. We further utilize LLMs to detect and resolve potential inconsistencies within the generated instructions. Regarding response generation, we decompose each instruction into atomic-level evaluation criteria, enabling rigorous validation and the elimination of inaccurate instruction-response pairs. Extensive experiments across a wide range of scenarios and settings demonstrate DecIF's superior performance on instruction-following tasks. Further analysis highlights its strong flexibility, scalability, and generalizability in automatically synthesizing high-quality instruction data.

Figures

Figures reproduced from arXiv: 2505.13990 by the authors.

Figure 1
Figure 1. The overall workflow of DecIF. tion into three distinct types of meta-information. The process begins by prompting LLMs to gen￾erate meta-domains, which represent high-level conceptual categories such as Sports, Technology or Health. These domains act as broad thematic boundaries, ensuring diversity and contextual rele￾vance in the resulting instructions. Based on these meta-domains, we then prompt the model to gen￾… view at source ↗
Figure 2
Figure 2. The results on Tulu-3-Mixture and Tulu-3-DecIF (Left) and the results of different large-scale general [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The results on four instruction-following [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 21 canonical work pages

  1. [1]

    Each instruction must be **less than 4 words**, specific, unique, realistic, common, and *model-solvable*

    work page

  2. [2]

    OpenCompass Contributors

    Training verifiers to solve math word prob- lems.Preprint, arXiv:2110.14168. OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/ opencompass. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat la...

    arXiv 2023

  3. [3]

    No duplicate instructions within the output

    work page

  4. [4]

    Do task"). *Example output for

    Instructions should be clear and actionable (avoid vague commands like "Do task"). *Example output for "Education" domain (Strictly follow this format and use lowercase letters)* - explain the math concept - grade student essays - create lesson plan - suggest teaching methods - recommend educational apps Now generate nearly {number of requests} diverse me...

    work page

  5. [5]

    Wei Liu, Yancheng He, Hui Huang, Chengwei Hu, Jia- heng Liu, Shilong Li, Wenbo Su, and Bo Zheng

    Let’s verify step by step.Preprint, arXiv:2305.20050. Wei Liu, Yancheng He, Hui Huang, Chengwei Hu, Jia- heng Liu, Shilong Li, Wenbo Su, and Bo Zheng. 2025. Air: Complex instruction generation via automatic iterative refinement.Preprint, arXiv:2502.17787. Ilya Loshchilov and Frank Hutter. 2019. De- coupled weight decay regularization.Preprint, arXiv:1711....

    arXiv 2025

  6. [7]

    C Detailed Description of Evaluation Benchmarks We utilize OpenCompass (Contributors, 2023) to evaluate most of the benchmarks

    as our baselines. C Detailed Description of Evaluation Benchmarks We utilize OpenCompass (Contributors, 2023) to evaluate most of the benchmarks. For the evaluation of instruction-following capa- bility, we utilize the following benchmarks: IFEval(Zhou et al., 2023) is an easily pro- ducible benchmark specifically designed to assess the instruction-follow...

    work page arXiv 2023

  7. [8]

    The maximum token length is set to 4096

    and perform full fine-tuning with a learning rate of 5e-6. The maximum token length is set to 4096. We use AdamW as the optimizer with a warmup ratio of 0.03 and train for 2 epochs. Addi- tionally, we employ a LinearLR scheduler through- out the entire training process. We utilize LLaMA- Factory (Zheng et al., 2024) framework for all the training process....

    work page 2024

  8. [10]

    All instructions must be relevant to the {domain} domain

    work page

Show all 27 references
  1. [13]

    Be specific with clear context (who, what, where, why)

  2. [14]

    Be from different domains (work, education, personal life, etc.)

  3. [15]

    Be 1-2 sentences maximum

  4. [16]

    Use hyphen formatting (- ...) for each scenario An Example: Meta request: create guide - A fitness trainer needs to create a workout guide for elderly clients at a local community center A software company wants to create an onboarding guide for new remote employees A parent n...

  5. [17]

    The above example is not tied to any particular persona, but you should create one that is unique and specific to the given persona

  6. [18]

    The instruction should contain all the following verifiable constraint(s): {the selected constraint(s)}

  7. [19]

    User instruction:

    Your output should start with "User instruction:". Your output should not include an answer to the instruction. Prompt Template for Consistency Judgement You are an expert in analyzing instructions for internal conflicts. Your task is to analyze the following instruction: {ins...

  8. [20]

    Check if there are any conflicting requirements (e.g., requiring both Chinese and English)

  9. [21]

    The refined instruction must be clear, concise, and free of any explanatory text

    If there is a conflict, refine the instruction to resolve it. The refined instruction must be clear, concise, and free of any explanatory text

  10. [22]

    If there is no conflict, return the original instruction unchanged

  11. [23]

    yes" or

    Format your response as follows: - Original: <original_instruction> - Conflict: True/False - Refined: <refined_instruction> Ensure that the ’Refined’ field contains ONLY the refined instruction without any additional explana- tions or context. 18 Prompt Template for Instructio...

  12. [24]

    Examine each criterion independently

  13. [25]

    Be strict but fair - only mark ’YES’ if the response fully satisfies the criterion

  14. [26]

    Ignore any stylistic preferences not explicitly listed in the criteria

  15. [27]

    Begin evaluation: 20

    Focus exclusively on the criteria provided Instruction: {instruction} Response to evaluate: {response} Evaluation criteria: {criteria} Your task: For each criterion above, output ONLY either ’YES’ or ’NO’ on its own line, in order. Begin evaluation: 20

  16. [2021]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E

    Evaluating large language models trained on code.Preprint, arXiv:2107.03374. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source ch...

    2023 arXiv

  17. [2023]

    Hui Huang, Jiaheng Liu, Yancheng He, Shilong Li, Bing Xu, Conghui Zhu, Muyun Yang, and Tiejun Zhao

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.Preprint, arXiv:2305.02301. Hui Huang, Jiaheng Liu, Yancheng He, Shilong Li, Bing Xu, Conghui Zhu, Muyun Yang, and Tiejun Zhao

    arXiv

  18. [2024]

    InProceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 3: System Demonstra- tions), Bangkok, Thailand

    Llamafactory: Unified efficient fine-tuning of 100+ language models. InProceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 3: System Demonstra- tions), Bangkok, Thailand. Association for Computa- tional Linguistics. Jeffrey Zhou, T...

    2023 arXiv

  19. [2025]

    Preprint, arXiv:2502.11541

    Musc: Improving complex instruction follow- ing with multi-granularity self-contrastive training. Preprint, arXiv:2502.11541. Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024. Follow- bench: A multi...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.