REVIEW 5 major objections 5 minor 1 cited by
Boosting LLM via Learning from Data Iteratively and Selectively
T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Training an LLM on only 5% of an instruction dataset per epoch, with scores recomputed as the model changes, out-performs full-data fine-tuning on average benchmark scores.
desk verdict A plausible new iterative data-selection method with good ablations, but the consistency claim overstates the evidence and the Alpaca-GPT4 b=0.0 choice looks post-hoc. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the composite score $S_i = S^{COM}_{i,\theta_t} \times S^{DIV}_i$, where $S^{COM}_{i,\theta_t}$ is the ratio of the perplexity of response $Y_i$ conditioned on instruction $X_i$ to the perplexity of $Y_i$ alone (the IFD score, a model-specific complexity measure), and $S^{DIV}_i$ is the sum of TF-IDF weights of n-grams in $Y_i$ with a decay factor $\alpha_g$ that multiplies by $b$ each time a sample containing n-gram $g$ is selected. The algorithm operates in epochs: at epoch $t$ it computes $S^{COM}$ for the current model over the retained candidate pool (the top $a \times M$ samples from the initial ranking), drops samples with $S^{COM} \ge 1$, and greedily fills the epoch's $M$-sample budget by repeatedly picking the largest $S_i$ and decaying $\alpha$ for the chosen n-grams. This turns data selection into a model-data feedback loop instead of a one-time filtering step.
What would settle it
A decisive test is to rerun IterIT on Alpaca-GPT4 with $b=0.1$ (the value the paper recommends for multi-task general data) and check whether the average score still surpasses Longest and GraphFilter. If the gain disappears or reverses, the Alpaca-GPT4 result is an artifact of choosing $b$ after seeing the test set.
Extended reading notes
Core claim
IterIT demonstrates that updating data-selection scores during fine-tuning improves instruction-tuned LLMs. The paper reports that re-ranking a coarse-to-fine candidate pool with the instruction-following-difficulty (IFD) score after every epoch — then greedily selecting samples that maximize the product of IFD complexity and a response-derived TF-IDF diversity score with decaying weights — beats Vanilla full-data SFT on average across seven benchmarks plus MixEval on four instruction datasets, and beats Longest, the strongest rule-based baseline, on most metrics, while using only 5% of the data at each epoch. The method also generalizes to CodeAlpaca code generation and to a different backbone, Qwen-2.5-7B.
Load-bearing premise
The load-bearing premise is that the weight-decay coefficient $b$ (set to 0.1 for most datasets and 0.0 for Alpaca-GPT4) is chosen by the stated rule for sharing of reserved words, not by looking at the test benchmarks; if $b=0.0$ was tuned on Alpaca-GPT4 results, the reported gains there may not replicate.
Editorial extensions
If this is right
- Instruction-tuned models improve when the selection metric is recomputed with the partially trained model, not a fixed pretrained scorer.
- Training on roughly 5% of the dataset per epoch can exceed full-data SFT on average multi-benchmark performance, cutting per-epoch gradient cost.
- Response-based diversity, not instruction-topic diversity, is what makes the diversity term effective in this pipeline.
- The approach carries over to code instruction tuning and to a second backbone model (Qwen-2.5-7B), suggesting the mechanism is not dataset- or architecture-specific.
- The 'long is more' heuristic is strong but incomplete: IterIT's selected sets overlap Longest's by only about 50% Jaccard similarity while outperforming it on most metrics.
Reading between the lines
- Editorially, the same iterative re-scoring loop could be applied to later post-training stages such as preference optimization, where the model's reward or policy changes across epochs and static preference data may quickly become stale.
- The finding that short-response samples remain valuable (the lower quartile of lengths is much lower than Longest's) suggests that sample value is relative to the model's current competence frontier, which shifts during training; a length-based static ranking cannot track that frontier.
- One testable extension is to swap the TF-IDF diversity score for an embedding-based semantic diversity measure inside the same loop; if gains persist, the method would be robust to the lexical proxy, and if not, the lexical coverage mechanism is doing the work.
- Also implicit is a recipe for noisy or multi-source synthetic data: iterative ranking with a model-specific difficulty filter (the $S^{COM} \ge 1$ cutoff) may serve as a general de-noising step for post-training corpora beyond instruction tuning.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes IterIT, an iterative data selection method for instruction tuning. It combines an IFD-based complexity score that is re-computed after each fine-tuning epoch on a coarse-to-fine candidate pool, with a response-level TF-IDF diversity score that is greedily updated via a weight-decay parameter b. On LLaMA-3-8B and Qwen-2.5-7B, using Alpaca, Alpaca-GPT4, WizardLM, Dolly, and CodeAlpaca, the authors report that selecting 5% of the data per epoch yields consistent improvements over full-data SFT (Vanilla) and over static baselines including Longest, Deita, Superfiltering, and GraphFilter, across seven benchmarks plus MixEval.
Significance. If the findings are robust, IterIT is a practically useful contribution: it provides a model-aware, iterative selection procedure that jointly optimizes complexity and response diversity, and it is evaluated under a unified re-implementation of baselines on multiple datasets and two backbone models. The paper includes useful ablations (iteration on/off, diversity component, hyper-parameter sensitivity) and promises to release code. The central premise—that model-specific complexity should be updated during fine-tuning rather than computed once—is interesting and supported by the reported 55.31% turnover of top samples after one epoch. However, the current empirical evidence contains several load-bearing gaps that must be addressed before the claims can be accepted.
major comments (5)
- [Sec. 3.4 / Sec. 5.4] The choice of b=0.0 for Alpaca-GPT4 is inconsistent with the paper's own recommendation. Sec. 3.4 sets b=0.0 for Alpaca-GPT4, while Sec. 5.4 states that b=0.0 is recommended only for task-specific scenarios where reserved words are undefined or shared. Alpaca-GPT4 is a general instruction dataset, so the stated rule does not cover the choice, and no dataset-specific justification is given. Because b controls the greedy diversity update in Eq. (7) and therefore directly affects the selected subset, the Alpaca-GPT4 results in Table 1 (AVG 62.20 vs. GraphFilter 61.45) could reflect per-dataset tuning rather than the algorithm's intrinsic behavior. Please provide an ablation of b on Alpaca-GPT4 (e.g., b=0.1) or a data-driven argument for why b=0.0 is appropriate for this dataset.
- [Table 3 / Appendix B] The Qwen-2.5-7B results for Alpaca-GPT4 and WizardLM appear to be swapped between Table 3 and Table 7 in Appendix B. For example, the Vanilla AVG/MixEval values for Alpaca-GPT4 are 68.96/40.15 in Table 3 but 69.17/38.20 in Table 7, and the corresponding WizardLM values are exchanged in the same way for all three methods. This is a factual inconsistency in a table that directly supports the generalization claim in Sec. 4.3. Please correct the tables and verify that the underlying experiments are reported consistently.
- [Section 4 results] The paper reports only single-run point estimates without error bars, multiple seeds, or statistical significance tests. Many of the reported gains are small—for instance, WizardLM AVG 60.29 vs. Longest 59.75 (Table 1), Dolly AVG 58.82 vs. Longest 59.14 (Table 6), and Qwen Alpaca AVG 66.43 vs. Longest 66.49 (Table 3), the latter being a loss on the average. Without variance estimates or repeated runs, the claimed 'consistent improvements' are not empirically established. Please report at least three seeds with mean and standard deviation, or otherwise justify that the differences exceed run-to-run noise.
- [Sec. 4.1] The statement that IterIT 'beats the strongest rule-based approach, Longest, on most of the metrics' is not supported by the tables across all datasets. On Alpaca-GPT4 (Table 1), IterIT loses to Longest on TruthfulQA, HumanEval, ARC, HellaSwag, and MixEval (5 of 9 columns). On Dolly (Table 6), IterIT loses on MMLU, TruthfulQA, BBH, HellaSwag, and the AVG itself. The claim should be qualified to 'on the aggregate average in most datasets' or should be reported per-dataset with exact win/loss counts.
- [Sec. 5.4] The text claims that setting b=0.1 improves GSM8K, BBH, and HumanEval by 2.50%, 2.59%, and 4.76%, respectively, compared with b=0.0. However, these per-benchmark numbers are not shown in Fig. 4(c) (which plots only the average score) or in any table. Because the choice of b is central to the method and the paper already reports per-task results elsewhere, please provide the full per-benchmark table for the b ablation on Alpaca, or explicitly state where these numbers come from.
minor comments (5)
- [Algorithm 1] The output line reads 'return M_{θ_{T−1}}'; after looping over T epochs, the final model should be M_{θ_T}. Please correct the subscript.
- [Tables 1, 6, 7] The column header 'A VG' appears with a space in several tables; this should be 'AVG'.
- [Figure 1] The caption writes 'I TER IT' with an extra space. Please fix the typo.
- [Sec. 4.1] The sentence 'data selection methods can easily achieve significant improvement in a single task when the selected data is closer to the distribution of a specific group of test data' is a speculative claim that is not substantiated by any controlled experiment in the paper. Please either remove it or support it with evidence.
- [Appendix B] The appendix table for Qwen-2.5-7B should be cross-checked with Table 3; in addition to the dataset-row swap noted above, several MixEval values are identical between datasets (e.g., 40.15 appears for both Alpaca-GPT4 Vanilla and WizardLM Vanilla in different tables), which is suspicious and should be verified.
Circularity Check
No significant circularity: the iterative data-selection method is evaluated on external benchmarks, and the unexplained b choice is a hyperparameter robustness concern rather than a self-referential derivation.
full rationale
The paper is an empirical data-selection method paper: no claimed derivation reduces a predicted quantity to a fitted input by construction. The complexity score in Eq. 3 is defined from IFD perplexity ratios, and the diversity score in Eq. 6 is defined from TF-IDF response features, both independently of the downstream evaluation benchmarks. The selection procedure in Algorithm 1 is a stated algorithmic procedure rather than a fitted mapping from test scores back to parameters. The central claim in Sec. 4.1 is supported by external benchmarks (GSM8K, MMLU, TruthfulQA, BBH, HumanEval, ARC, HellaSwag) plus MixEval. MixEval (Ni et al., 2024) overlaps with the present authors (Ni, Xue, You), but MixEval-hard-0601 is a fixed mixture of public benchmarks evaluated on held-out model responses, so citing it does not assume the paper's conclusion. The per-dataset choice b=0.0 for Alpaca-GPT4 in Sec. 3.4, while Sec. 5.4 recommends b=0.0 only for task-specific data, is an unexplained hyperparameter choice; it affects the selected subset, but the paper does not claim b is derived from first principles, and the difference between b=0.0 and b=0.1 is a benchmark-tuning robustness concern rather than circularity. No self-citation chain forces the result, no uniqueness theorem is invoked, and no known result is renamed. Accordingly, no circular step can be exhibited with the required quote-and-reduction evidence.
Assumptions & free parameters
free parameters (4)
- M (per-epoch selection size) =
5% of dataset (0.05N)
- a (candidate pool multiplier) =
3
- b (diversity weight decay) =
0.1 for Alpaca/WizardLM/Dolly, 0.0 for Alpaca-GPT4
- TF-IDF n-gram order
assumptions (4)
- domain assumption The IFD ratio S_COM (Eq. 3) is a valid measure of instruction-following difficulty and sample quality for SFT.
- ad hoc to paper Samples with S_COM >= 1 are unaligned and can be safely discarded.
- domain assumption TF-IDF of response n-grams measures response informativeness and redundancy.
- domain assumption Recomputing complexity scores with the partially trained model each epoch improves selection.
Cite this review
Pith. "Pith review of Boosting LLM via Learning from Data Iteratively and Selectively." pith.science (2026). https://pith.science/paper/XJ7EP3RF
@misc{pith2026241217365,
author = {Pith},
title = {Pith review of: Boosting LLM via Learning from Data Iteratively and Selectively},
year = {2026},
howpublished = {\url{https://pith.science/paper/XJ7EP3RF}},
note = {Machine review of arXiv:2412.17365}
}
read the original abstract
Datasets nowadays are generally constructed from multiple sources and using different synthetic techniques, making data de-noising and de-duplication crucial before being used for post-training. In this work, we propose to perform instruction tuning by iterative data selection (\ApproachName{}). We measure the quality of a sample from complexity and diversity simultaneously. Instead of calculating the complexity score once for all before fine-tuning, we highlight the importance of updating this model-specific score during fine-tuning to accurately accommodate the dynamic changes of the model. On the other hand, the diversity score is defined on top of the samples' responses under the consideration of their informativeness. IterIT integrates the strengths of both worlds by iteratively updating the complexity score for the top-ranked samples and greedily selecting the ones with the highest complexity-diversity score. Experiments on multiple instruction-tuning data demonstrate consistent improvements of IterIT over strong baselines. Moreover, our approach also generalizes well to domain-specific scenarios and different backbone models. All resources will be available at https://github.com/JiaQiSJTU/IterIT.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learning
An iterative data-optimization pipeline that simplifies, extends, and rewrites SFT examples based on the model's own loss, embedding sparsity, and self-scores reports up to 7.15 absolute points of average benchmark im...
Reference graph
Works this paper leans on
-
[3]
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,
-
[6]
Enhancing chat language models by scaling high-quality instructional conversations
9 Boosting LLM via Learning from Data Iteratively and Selectively Ding, N., Chen, Y ., Xu, B., Qin, Y ., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3029–3051,
work page 2023
-
[7]
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,
-
[8]
The trends are identical to that of Alpaca. We further extract the response representations of samples by NV-Embed (Lee et al., 2024), which achieves the state- of-the-art performance on MTEB benchmark (Muennighoff et al., 2022). The scatter plot visualized using t-SNE on response representations from the Alpaca dataset is shown in Figure
work page 2024
-
[9]
Selec- tive reflection-tuning: Student-selected data recycling for llm instruction-tuning
Li, M., Chen, L., Chen, J., He, S., Gu, J., and Zhou, T. Selec- tive reflection-tuning: Student-selected data recycling for llm instruction-tuning. arXiv preprint arXiv:2402.10110, 2024a. Li, M., Zhang, Y ., He, S., Li, Z., Zhao, H., Wang, J., Cheng, N., and Zhou, T. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning. In Ku, L.-W., ...
-
[10]
SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection
URL https://openreview.net/ forum?id=1qvx610Cu7. Liu, L., Liu, X., Wong, D. F., Li, D., Wang, Z., Hu, B., and Zhang, M. Selectit: Selective instruction tuning for large language models via uncertainty-aware self-reflection. arXiv preprint arXiv:2402.16705, 2024a. Liu, W., Zeng, W., He, K., Jiang, Y ., and He, J. What makes good data for alignment? a compr...
-
[11]
Mteb: Massive text embedding benchmark
Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316 ,
-
[13]
Instruc- tion tuning with gpt-4
Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruc- tion tuning with gpt-4. arXiv preprint arXiv:2304.03277,
Show all 19 references
-
[15]
W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al
10 Boosting LLM via Learning from Data Iteratively and Selectively Suzgun, M., Scales, N., Sch¨arli, N., Gehrmann, S., Tay, Y ., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al. Challenging big-bench tasks and whether chain-of- thought can solve them. InFindings ...
2023
-
[16]
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T
URL https://qwenlm.github.io/ blog/qwen2.5/. Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Inter- pretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845,
-
[17]
S., Naik, A., Stap, D., et al
Wang, Y ., Mishra, S., Alipoormolabashi, P., Kordi, Y ., Mirzaei, A., Arunkumar, A., Ashok, A., Dhanasekaran, A. S., Naik, A., Stap, D., et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07705,
-
[18]
The best of both worlds: Bridging quality and diversity in data selection with bipartite graph
Wu, M., Vu, T.-T., Qu, L., and Haffari, G. The best of both worlds: Bridging quality and diversity in data selection with bipartite graph. arXiv preprint arXiv:2410.12458, 2024a. Wu, M., Waheed, A., Zhang, C., Abdul-Mageed, M., and Aji, A. Lamini-lm: A diverse herd of distille...
-
[2006]
Instruction min- ing: Instruction data selection for tuning large language models
Cao, Y ., Kang, Y ., Wang, C., and Sun, L. Instruction min- ing: Instruction data selection for tuning large language models. arXiv preprint arXiv:2307.06290,
-
[2018]
Training verifiers to solve math word problems
Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,
-
[2019]
Rethinking data selection for supervised fine- tuning
Shen, M. Rethinking data selection for supervised fine- tuning. arXiv preprint arXiv:2402.06094,
-
[2021]
Think you have solved question answering? try arc, the ai2 reasoning challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,
-
[2022]
URL https://arxiv.org/ abs/2210.07316
doi: 10.48550/ ARXIV .2210.07316. URL https://arxiv.org/ abs/2210.07316. Ni, J., Xue, F., Yue, X., Deng, Y ., Shah, M., Jain, K., Neu- big, G., and You, Y . Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures. arXiv preprint arXiv:2406.06565,
-
[2023]
Alpagasus: Training a better alpaca with fewer data.arXiv preprint arXiv:2307.08701,
Chen, L., Li, S., Yan, J., Wang, H., Gunaratna, K., Yadav, V ., Tang, Z., Srinivasan, V ., Zhou, T., Huang, H., et al. Alpagasus: Training a better alpaca with fewer data.arXiv preprint arXiv:2307.08701,
-
[2024]
Measuring massive multitask language understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations. Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catan- zaro, B., and Ping, W...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.