Pith. sign in

REVIEW 4 major objections 6 minor 43 references

Learning Task Mixtures from Task Affinities: A Probabilistic Graphical Model for Supervised Fine-Tuning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TaskPGM claims that supervised fine-tuning mixtures can be learned in closed form from behavioral task affinities, beating uniform and size-based mixing and matching influence-based selection.

desk verdict Behavioral PMI/JSD affinities are a fresh signal for task mixing, but the weak-submodularity proof is wrong and the reported β/λ choice is missing from the sensitivity sweep. read the letter →

arxiv 2507.12612 v4 pith:V2TOX4R4 submitted 2025-07-16 cs.LG cs.AI

Prateek Chanda , Saral Sureka , Parth Pratim Chatterjee , Krishnateja Killamsetty , Nikhil Shivakumar Nayak , Ganesh Ramakrishnan This is my paper
classification cs.LGcs.AI MSC 68T0768T5090C2590C27
keywords taskmixturesupervisedfine-tuningMarkovrandomfieldaffinityJensen-Shannondivergencepointwisemutualinformationweaksubmodularityenergy-basedmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the mixture of tasks used for supervised fine-tuning can be chosen by modeling task interactions directly, rather than by dataset size or semantic similarity. It represents tasks as nodes of a Markov random field whose unary potentials capture how broadly representative each task is and whose pairwise potentials penalize redundancy, with both potentials built from a behavioral similarity matrix S computed from the predictive distributions of single-task fine-tuned models. The resulting optimization over the probability simplex has a closed-form KKT optimum that depends only on $S^{{-1}}$1 and one interpretable ratio $\beta$/$\lambda$, which trades representativeness against diversity. Across Qwen2-7B and Llama2-7B at 25K and 50K budgets, the paper reports consistent gains over uniform, random, and size-proportional mixing and competitive results against the task-specific influence-based selector LESS, while also providing a discrete variant with weak-submodularity guarantees.

What carries the argument

The energy-based Markov random field: E(p) = Σ_i βS_i p_i − (1/2) Σ_{i,j} λS_{ij} p_i p_j, where S_{ij} is a symmetric task affinity computed from Jensen-Shannon divergence or pointwise mutual information between predictive distributions of single-task fine-tuned models. The optimal mixture is p* = (β/λ)(1_n − ((β/λ)1_n^T 1_n − 1)/(1_n^T $S^{{-1}}$ 1_n) $S^{{-1}}$1_n), obtained via KKT conditions after spectral-shifting S to be positive semidefinite, so the mixture depends only on $S^{{-1}}$1 and the ratio β/λ. For budgeted discrete selection, the paper lifts this to f(A) = max_{p: supp(p) ⊆ A} E(p) and proves monotonicity and weak submodularity with ratio γ = c_Ω/C_Ω, justifying greedy task discovery.

What would settle it

Construct a small task pool with planted redundancy (two tasks drawn from the same underlying distribution) and a planted transfer pair (one task whose data improves another), compute S from single-task models, and check whether p* down-weights the duplicates and up-weights the transfer source. A stronger test compares the final SFT accuracy of the TaskPGM mixture against the best mixture found by brute-force search over β/λ for each benchmark; if energy-optimal mixtures show no correlation with held-out accuracy across random splits, or if a mixture built from a shuffled S achieves the same accuracy as the real one, the affinity signal is not carrying the argument.

Watch

Extended reading notes

Core claim

The central claim is that a single target-agnostic mixture, derived in closed form from a PSD-corrected task similarity matrix, outperforms heuristic mixing strategies and often matches a per-benchmark influence-based selector that builds a separate checkpoint for each benchmark. The key discovery is that the row-sum-inverse structure of the behavioral affinity matrix carries enough information about coverage and redundancy to set useful mixture proportions, without any gradient datastore or per-example influence computation.

Load-bearing premise

The load-bearing premise is that the behavioral affinity matrix S, computed from how single-task fine-tuned models score each other's examples, captures the inter-task transfer and redundancy that actually matters for the final multi-task fine-tune, and that the closed-form optimum built from S is a reliable proxy for downstream SFT performance.

Editorial extensions

If this is right

  • Once the behavioral affinity matrix is computed, optimal mixtures for any budget can be derived in seconds from the closed-form solution, avoiding per-benchmark gradient datastores.
  • The ratio β/λ acts as an interpretable knob that navigates the representativeness-diversity tradeoff, so practitioners can tune mixture behavior without retraining affinity models.
  • The discrete variant inherits approximation guarantees from weak submodularity, supporting efficient greedy construction of task subsets under budget constraints.
  • Interpretable structure emerges automatically: tasks with zero mixture weight are functionally redundant, and similarity heatmaps reveal asymmetric specialization and broad-attractor tasks.
  • The method is model-family-agnostic, with consistent improvements reported across Qwen2-7B and Llama2-7B on reasoning-intensive suites.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If behavioral affinities computed on smaller proxy models transfer faithfully to larger models, TaskPGM could scale to much larger task pools at a fraction of the affinity-computation cost; the paper only demonstrates this at the 7B scale.
  • The dependence of p* on S^{-1}1 suggests a testable regularity: the condition number or spectral decay of S may predict how much the learned mixture will beat uniform sampling, since PMI produces a sharper spectral decay than JSD.
  • The energy landscape naturally defines a task-pruning criterion: a task whose removal barely changes p* is functionally redundant, which could be used to compress instruction pools without re-running full-mixture fine-tuning.
  • Because PMI and JSD are computed from predictive distributions rather than text embeddings, the same machinery could apply to non-textual fine-tuning settings where task behavior is observable, a direction the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes TaskPGM, an energy-based model for selecting supervised fine-tuning task mixtures. Tasks are nodes in a dense MRF; unary potentials are β times the row sums of a behavioral similarity matrix S, and pairwise potentials are λS. The similarity matrix is computed from PMI or JSD between predictive distributions of single-task LoRA-finetuned models. The main continuous mixture is obtained by solving a simplex-constrained quadratic program, with a claimed closed-form KKT solution (Eq. 3). The paper further claims the induced discrete set function is weakly submodular, and reports experiments on Qwen2-7B and Llama2-7B at 25K/50K budgets against uniform, random, EPM, and LESS baselines.

Significance. If the empirical and theoretical claims held, TaskPGM would offer a cheap, target-agnostic alternative to influence-based data selection, with interpretable task clusters. The paper is commendable for reporting compute costs, providing an anonymized code link, and for the algebraic derivation of Eq. (3), which is internally consistent. However, the significance is substantially weakened by three load-bearing problems: the closed-form solution is a one-parameter interpolation between the uniform vector and S^{-1}1, so the 'learned mixture' is a very constrained function of S; the weak-submodularity proof in Appendix C.6 is invalid, so the stated approximation guarantee is not established; and the headline hyperparameters β=20, λ=10 are absent from the paper's own sensitivity sweep, with no protocol for their selection, making the reported gains not reproducible from the disclosed evidence.

major comments (4)
  1. [§3.1, Eq. (3)] The closed-form optimum is much more constrained than the paper's framing suggests. Because Ψun=βS1 and Ψpair=λS, Eq. (3) reduces to p*(τ) = τ(1 - (τ n - 1)/(1^T S^{-1} 1) S^{-1} 1), an affine function of the uniform vector and S^{-1}1 with a single scalar τ=β/λ. Thus the entire 'energy-based' optimization is a one-dimensional interpolation, and any two affinity matrices with the same S^{-1}1 give the same mixture. This undercuts the novelty claim of learning rich task interactions from an MRF; the method is essentially a reweighting by row sums of S^{-1}.
  2. [Appendix C.6, Theorem 3] The weak-submodularity proof is not valid. The line 'E(ζ(L∪S))-E(ζ(L)) ≤ ⟨∇E(ζ(L)), ζ(L∪S)-ζ(L)⟩ - (c/2)||ζ(L∪S)-ζ(L)||^2' has the wrong sign and wrong constant: the RSM upper bound in Lemma 3 gives an upper bound with + (C/2)||p-q||^2, not - (c/2)||p-q||^2. The subsequent lower bound on marginal gains introduces a different constant L and then swaps to C without derivation. Consequently, the claimed submodularity ratio γ=c/C is not derived, and the abstract's 'enabling approximation guarantees for discrete selection variants' is unsupported.
  3. [§5 and Tables 2, 4] The reported results fix β=20, λ=10 (Section 5), but the sensitivity sweep in Table 4 does not include this configuration, nor any configuration with the same ratio τ=β/λ=2; the smallest listed ratio is 19/5=3.8. The closest listed row, β=19, λ=5 at 25K PMI, gives BBH 0.3576, whereas Table 2 reports 0.4095 for 'Ours (PMI)' — a gap of about 8 reported standard errors. Because Eq. (3) depends only on τ (when the interior solution is valid) or on the same ratio under PSD shifting, this discrepancy either indicates extreme sensitivity in a range the paper calls robust, or that Tables 2 and 4 were produced under different protocols. The paper gives no description of how β=20, λ=10 were selected, so the headline improvement cannot be distinguished from hyperparameter selection.
  4. [Abstract and Table 2] The abstract's claim that TaskPGM 'improves over standard mixing strategies' is too strong. Table 2 contains counterexamples: on Llama2-7B at 25K, Ours (JSD) BBH is 0.3454, below Uniform's 0.3501; on Qwen2-7B at 25K, Ours (JSD) GPQA is 0.3180, below Uniform's 0.3314. Several other cells are within one standard error of a heuristic or below it. The claim should be qualified to specific benchmarks and variants, and the 'consistently' language removed.
minor comments (6)
  1. [§3.3, Eq. (7)] The quantity in Eq. (7) is not pointwise mutual information; it is an average log-ratio of next-token probabilities. Renaming it avoids a technical misnomer.
  2. [§3.3, Eq. (8)] S_JSD is called a 'similarity score', but JSD is a divergence for which lower values indicate similarity. The sign convention should be stated explicitly so that the pairwise penalty term -λ p^T S p can be interpreted correctly.
  3. [§4 and Appendix C] Theorem numbering is inconsistent: the main text calls the weak-submodularity result 'Theorem 3', while Appendix C.6 labels it 'Theorem 1'. Please harmonize the numbering.
  4. [§5, Figure 3] The Figure 3 caption refers to 'greedy step k (mixture size)', but the main experimental protocol uses the continuous closed-form mixture, not greedy discrete selection. The relation between Figure 3 and Tables 2/4 should be explained.
  5. [§4] The first sentence of Section 4 refers to 'Section 6', but Section 6 is the Limitations section; the intended reference is Section 3.
  6. [Tables 2 and 4] The error bars are reported as ± standard deviations, but the number of independent runs or seeds is not stated. Please specify how the error bars were computed.

Circularity Check

1 steps flagged · score 5.0 of 10

Headline gains over baselines are partly fitted through benchmark-driven selection of the single scalar β/λ, while the affinity-based mixture itself is not circular by construction.

  1. fitted input called prediction [Eq. (3), Section 5 (Baselines for Comparison), Appendix G Table 4 / Figure 3]
    "In our experiments, we fix the hyperparameters governing the balance between unary and pairwise terms: the unary potential weight is set to β=20, and the pairwise diversity penalty is set to λ=10. ... For each downstream benchmark, the surface plots evaluation accuracy as a function of the greedy step k (mixture size) and the weight ratio β/λ, with β and λ varied on a log scale. ... Table 4: Llama-2-7b: Instruction-tuning performance on Leaderboard subsets with varying β and λ."

    Eq. (3) makes the mixture p* a deterministic function of the affinity matrix S and the scalar ratio r=β/λ. The paper's own Figure 3 and Table 4 sweep r against the same downstream Leaderboard benchmarks and even identify an 'optimal configuration' per budget and task; nevertheless, the main tables are reported at β=20, λ=10, a setting absent from Table 4, while the closest listed row β=19, λ=5 gives 25K-PMI BBH 0.3576 versus 0.4095 in Table 2. Thus the headline improvement over uniform/EPM/LESS is not an out-of-sample prediction of the model at a pre-specified hyperparameter; it is, at least for the Llama-2-7B numbers, a selection along the benchmark response surface, or an undocumented choice with the same observable signature.

full rationale

The core method is not circular by construction: the affinity matrix S is estimated from single-task fine-tuned models using PMI/JSD on training-task data only, and the downstream benchmarks are not used to build S; the final fine-tuned checkpoints are genuinely new evaluations. The monotonicity and weak-submodularity results are standard consequences of quadratic RSC/RSM and do not import the paper's empirical conclusions. What compromises independence is the single scalar β/λ. The paper's own Figure 3 and Table 4 explore accuracy on the downstream benchmarks as a function of this ratio, and the main setting β=20, λ=10 is not reported in Table 4 while a nearby row differs by roughly 5 points on BBH. The 'improves over standard mixing strategies' claim therefore conflates model quality with benchmark-driven hyperparameter selection for at least the Llama-2-7B headline numbers. This warrants a partial-circularity score of 5 rather than 0, but not 8-10, because the mixture itself remains a non-trivial function of an independently estimated affinity matrix and the evaluation spans multiple benchmarks, budgets, and model families.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method rests on one empirical assumption (behavioral affinities capture transfer), one ad hoc objective form (energy with unary proportional to row-sums of S), and standard convex optimization. The free parameters beta and lambda are chosen by hand and swept on the evaluation benchmarks. No new physical or ontological entities are introduced.

free parameters (2)
  • beta (unary potential weight) = 20
    Controls strength of the representativeness term; swept in Figure 3 and Table 4, so the reported beta=20 is a chosen value, not learned from data.
  • lambda (pairwise diversity penalty) = 10
    Controls strength of the diversity penalty; jointly swept with beta on the evaluation benchmarks (Table 4), so the reported value is selected on the benchmarks, not on a separate validation set.
assumptions (4)
  • domain assumption Predictive distributions of single-task fine-tuned models encode transfer and redundancy structure relevant for multi-task fine-tuning
    The entire similarity matrix S is built from these distributions; if they do not reflect true transfer, the mixture is meaningless. Invoked in Section 3.3.
  • ad hoc to paper The energy objective E(p) = beta (S1)^T p - (lambda/2) p^T S p is a valid proxy for final SFT performance
    The unary term is defined as beta S1, exactly proportional to the pairwise matrix, which makes the optimum a simple function of S^{-1}1. This form is chosen, not derived from a data model. Eq. (1).
  • domain assumption After spectral shifting, Psi_pair is positive semi-definite, so the quadratic problem is convex and KKT applies
    If S is far from PSD, the spectral shift adds a uniform bias; the paper does not quantify how much this changes the optimum. Section 3.1.
  • standard math Standard convex optimization results (KKT, PGD, weak submodularity of concave maximization)
    Used to derive Eq. (3) and Theorem 3; however the proof of Theorem 3 contains errors, so this axiom is invoked but not correctly applied.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Task Mixtures from Task Affinities: A Probabilistic Graphical Model for Supervised Fine-Tuning." pith.science (2026). https://pith.science/paper/V2TOX4R4

@misc{pith2026250712612,
  author       = {Pith},
  title        = {Pith review of: Learning Task Mixtures from Task Affinities: A Probabilistic Graphical Model for Supervised Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V2TOX4R4}},
  note         = {Machine review of arXiv:2507.12612}
}
read the original abstract

Supervised fine-tuning performance for large language models depends strongly on how training budget is distributed across a heterogeneous set of tasks. In practice, mixtures are often fixed using simple heuristics (e.g., uniform or size-proportional sampling) that ignore task interactions, which can hurt transfer and waste budget on redundant sources. We introduce TaskPGM, a framework for learning continuous task mixtures via an energy-based model over tasks. Tasks form the nodes of a Markov random field: unary potentials capture per-task utility, and pairwise potentials encode inter-task relationships using behavioral divergences computed from predictive distributions of single-task fine-tuned models (e.g., Jensen--Shannon divergence and pointwise mutual information). Optimizing this objective yields mixtures that balance coverage against redundancy. We show that the resulting set function is weakly submodular under budget constraints, enabling approximation guarantees for discrete selection variants. Across multiple model families (LLaMA-7B, Qwen2-7B) and evaluation suites (BIG-Bench Hard), TaskPGM improves over standard mixing strategies and provides interpretable structure over task interactions.

Figures

Figures reproduced from arXiv: 2507.12612 by the authors.

Figure 1
Figure 1. Illustrative SST-2 instance highlighting compositional negation. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. MRF over tasks with similarities Sij (left) and the learned mixture probabilities p ∗ i from Eq. (2) (right; shown the corr. probabili￾ties). Unary Potentials: We define the unary potential as a function of the similarity matrix Si , denoted as Ψi = βSi = βS1n, where β is a hyperparam￾eter that controls the strength of the potential. Pairwise Potentials: Similarly, we define the pair￾wise potential as Ψij = λSij , w… view at source ↗
Figure 3
Figure 3. Accuracy surfaces over β/λ ratios during greedy mixture construction (LLama-7B) For each downstream benchmark, the surface plots evaluation accuracy as a function of the greedy step k (mixture size) and the weight ratio β/λ, with β and λ varied on a log scale, illustrating how performance curvature varies across tasks. domain of all instances from all sub-tasks combined; 4) LESS [Xia et al., 2024], an influence-func… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of task similarity metrics using cosine similarity (top), and PMI and JSD-based [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Task similarity heatmaps for Qwen models computed using Jensen–Shannon divergence [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Eigenvalue spectra of similarity matrices derived from (a) Jensen–Shannon divergence (JSD) and (b) pointwise mutual information (PMI). The PMI-based matrix exhibits a steeper spectral decay, indicating a lower effective rank and thus a more compact embedding of similar…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 26 canonical work pages

  1. [1]

    Achille, M

    A. Achille, M. Lam, R. Tewari, A. Ravichandran, S. Maji, C. C. Fowlkes, S. Soatto, and P. Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6430--6439, 2019

    work page 2019

  2. [2]

    Agarwal, K

    I. Agarwal, K. Killamsetty, L. Popa, and M. Danilevsky. DELIFT : Data efficient language model instruction fine-tuning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=Fty0wTcemV

    work page 2025

  3. [3]

    Alvarez-Melis and N

    D. Alvarez-Melis and N. Fusi. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems, 33: 0 21428--21439, 2020

    work page 2020

  4. [4]

    S. P. Boyd and L. Vandenberghe. Convex optimization. Cambridge university press, 2004

    2004

  5. [5]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  6. [6]

    Y. Chen, M. R. Gupta, and B. Recht. Learning kernels from indefinite similarities. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 145--152, 2009

    work page 2009

  7. [7]

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25 0 (70): 0 1--53, 2024

    work page 2024

  8. [8]

    Q. Dai, D. Zhang, J. W. Ma, and H. Peng. Improving influence-based instruction tuning data selection for balanced learning of diverse capabilities. arXiv preprint arXiv:2501.12147, 2025

    work page Pith review arXiv 2025

Show all 43 references
  1. [9]

    Duchi, S

    J. Duchi, S. Shalev-Shwartz, Y. Singer, and T. Chandra. Efficient projections onto the l 1-ball for learning in high dimensions. In Proceedings of the 25th international conference on Machine learning, pages 272--279, 2008

    2008

  2. [10]

    Hwang, Y

    M. Hwang, Y. Jeong, and W. Sung. Data distribution search to select core-set for machine learning. In The 9th International conference on smart media and applications, pages 172--176, 2020

    2020

  3. [11]

    Killamsetty, X

    K. Killamsetty, X. Zhao, F. Chen, and R. Iyer. Retrieve: Coreset selection for efficient and robust semi-supervised learning. Advances in neural information processing systems, 34: 0 14488--14501, 2021

    2021

  4. [12]

    H. Kim, S. Sasaki, S. Hoshino, and U. Honda. A single linear layer yields task-adapted low-rank matrices. In N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, and N. Xue, editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Languag...

    2024

  5. [13]

    Kindermann and J

    R. Kindermann and J. L. Snell. Markov random fields and their applications, volume 1. American Mathematical Society, 1980

    1980

  6. [14]

    P. W. Koh and P. Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885--1894. PMLR, 2017

    2017

  7. [15]

    Kuhn and A

    H. Kuhn and A. Tucker. Nonlinear programming in proceedings of 2nd berkeley symposium (pp. 481--492). Berkeley: University of California Press.[Google Scholar], 1951

    1951

  8. [16]

    Lambert, J

    N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  9. [17]

    Y. Li, Z. Liu, and E. Xing. Data mixing optimization for supervised fine-tuning of large language models. arXiv preprint arXiv:2508.11953, 2025

    2025 arXiv

  10. [18]

    Q. Liu, X. Zheng, N. Muennighoff, G. Zeng, L. Dou, T. Pang, J. Jiang, and M. Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492, 2024 a

    2024 arXiv

  11. [19]

    Z. Liu, A. Karbasi, and T. Rekatsinas. TSDS : Data selection for task-specific model finetuning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 b . URL https://openreview.net/forum?id=wjbTHLUSzU

    2024

  12. [20]

    Longpre, L

    S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International conference on machine learning, pages 22631--22648. PMLR, 2023

    2023

  13. [21]

    Maharana, P

    A. Maharana, P. Yadav, and M. Bansal. D2 pruning: Message passing for balancing diversity and difficulty in data pruning. arXiv preprint arXiv:2310.07931, 2023

    2023 arXiv

  14. [22]

    Ortiz-Jimenez, A

    G. Ortiz-Jimenez, A. Favero, and P. Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. arXiv preprint arXiv:2305.12827, 2023. doi:10.48550/arXiv.2305.12827. URL https://arxiv.org/abs/2305.12827

    2023 doi arXiv

  15. [23]

    M. Paul, S. Ganguli, and G. K. Dziugaite. Deep learning on a data diet: Finding important examples early in training. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?...

    2021

  16. [24]

    H. S. V. N. S. K. Renduchintala, S. Bhatia, and G. Ramakrishnan. SMART : Submodular data mixture strategy for instruction tuning. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 12916--12934, Bangkok...

    2024 doi

  17. [25]

    V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021

    arXiv

  18. [26]

    Sener and S

    O. Sener and S. Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=H1aIuk-RW

    2018

  19. [27]

    Toneva, A

    M. Toneva, A. Sordoni, R. T. des Combes, A. Trischler, Y. Bengio, and G. J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=BJlxm30cKm

    2019

  20. [28]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023. URL http://arxiv.org/abs/2302.13971

    2023 arXiv

  21. [29]

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, pages 353--355, 2018

    2018

  22. [30]

    A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019

    2019

  23. [31]

    Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 conference on empirical met...

    2022

  24. [32]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  25. [33]

    G. Wu, E. Y. Chang, and Z. Zhang. An analysis of transformation on non-positive semidefinite similarity matrix for kernel machines. In Proceedings of the 22nd international conference on machine learning, volume 8. Citeseer Cham, 2005

    2005

  26. [34]

    M. Xia, S. Malladi, S. Gururangan, S. Arora, and D. Chen. LESS : Selecting influential data for targeted instruction tuning. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=PG5fV50maR

    2024

  27. [35]

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Yang, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. ...

    2024 arXiv

  28. [36]

    Zhang, J

    X. Zhang, J. Zhai, S. Ma, C. Shen, T. Li, W. Jiang, and Y. Liu. Staff: Speculative coreset selection for task-specific fine-tuning. In The thirteenth international conference on learning representations, 2025

    2025

  29. [37]

    Zheng, R

    H. Zheng, R. Liu, F. Lai, and A. Prakash. Coverage-centric coreset selection for high pruning rates. In 11th International Conference on Learning Representations, ICLR 2023, 2023

    2023

  30. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  31. [39]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  32. [40]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  33. [41]

    , " * write output.state after.block = add.period write newline

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

  34. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  35. [43]

    sibling model

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.