Pith. sign in

REVIEW 5 major objections 5 minor 10 cited by

Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a two-phase dataset pipeline built from 100M open-source instructions lets a 70B open model beat GPT-4-0314 on instruction-following benchmarks by 8.6% while matching its foundational scores.

desk verdict A genuinely useful large instruction dataset from an integration of known methods; the evaluation is plausible but has enough benchmark-guided selection and internal inconsistencies that the specific numbers need revision before being trusted. read the letter →

arxiv 2506.11116 v1 pith:VM2FN2YZ submitted 2025-06-09 cs.CL cs.AI

Jijie Li , Li Du , Hanyu Zhao , Bo-wen Zhang , Liangdong Wang , Boyan Gao , Guang Liu , Yonghua Lin This is my paper · ORCID
classification cs.CLcs.AI
keywords instructiontuningdatasetcurationdataselectionsynthesisDSIRfollowingfoundationalcapabilitiesopen-sourceLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a single 8.9M-example instruction dataset, assembled by a two-phase pipeline of selection then synthesis, can lift open-weight language models above their official instruction-tuned counterparts on both foundational benchmarks (math, code, knowledge) and conversational benchmarks (AlpacaEval 2.0, Arena-Hard, MT-Bench). Trained on this dataset, a 70B Llama model scores 8.6% higher than GPT-4-0314 on instruction-following tasks while matching its foundational performance. If correct, the result would mean that dataset construction — not model architecture or preference tuning — is the main lever for closing the open/proprietary gap.

What carries the argument

The central object is the dataset itself plus the pipeline that builds it. The load-bearing mechanism is DSIR importance resampling for math and code (selecting data to match the prompt distribution of the evaluation benchmarks) combined with a two-layer instruction labeling system (26 first-level labels and tens of thousands of second-level labels) that drives diversity sampling, difficulty filtering via language-model loss, and feedback-based synthesis of weak-domain instructions.

What would settle it

Retrain the same models on InfInstruct-F variants selected with target distributions drawn from held-out problem sets (or with rephrased GSM8K/MATH/HumanEval prompts), then evaluate on the original benchmarks and on fresh problem distributions; if gains over official instruct models shrink to near zero on the held-out sets, the selection step is encoding benchmark formats rather than general capability.

Watch

Extended reading notes

Core claim

Infinity-Instruct is a two-phase instruction dataset built from over 100M open-source instructions. Phase 1 filters 7.4M 'foundational' instructions using hybrid selection: source filtering, rule-based cleaning, Flan-based knowledge curation, and DSIR importance resampling whose target distributions are the training prompts of GSM8K, MATH, and HumanEval. Phase 2 starts from 1.2M seeds selected from 9M by diversity and difficulty, then evolves them with WizardLM-style rewriting and adds instructions targeting weaknesses diagnosed by comparing several open models' failures, yielding 1.5M conversational instructions. Fine-tuning Mistral-7B, Llama-3.1-8B/70B, Qwen2-7B, and Yi-1.5-9B on this dataset yields consistent gains over the official instruct versions, with the 70B model beating GPT-4-0314 by 8.6% on instruction following and surpassing it on code benchmarks.

Load-bearing premise

The math and code selection aims the dataset at the exact benchmarks used later to measure improvement, so the foundational gains could be partly an artifact of learning benchmark-shaped problem formats instead of general competence.

Editorial extensions

If this is right

  • If correct, other teams can fine-tune open base models on the released dataset and expect gains over official instruct versions without preference tuning.
  • The 8.6% AlpacaEval 2.0 gain at 70B indicates data quality can substitute for RLHF-style alignment for instruction-following.
  • The observed positive correlation between foundational and conversational gains supports training on both rather than only chat data.
  • Two-stage training (foundational then conversational) beats one-stage mixing, favoring curriculum-style data ordering.
  • Scaling curves suggest continued gains from larger curated instruction sets, at least within the tested ranges.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be re-running the selection with held-out or rephrased target prompts to see whether the foundational gains persist on truly out-of-distribution math and code problems.
  • Because the DSIR targets are the same benchmarks used later for evaluation, the selection loop resembles tuning on a validation set; decoupling selection targets from evaluation targets could make data-selection reporting more honest.
  • The pipeline's reliance on Qwen1.5-72B for labeling and GPT-4 for diagnosis means the dataset's quality ceiling is set by those models; as open models improve, iterative self-labeling may reduce or remove that dependency.
  • The ablation suggests dialogue-only training captures most chat performance but foundational data still helps, so the optimal balance between foundational and conversational data may shift with model scale.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces Infinity-Instruct, a large-scale instruction dataset built from over 100 million open-source instructions through a two-phase pipeline: a 7.4M-sample foundational dataset (InfInstruct-F-7.4M) selected via filtering, clustering, and DSIR-based importance resampling, and a 1.5M-sample conversational dataset (InfInstruct-G-1.5M) produced by labeling, seed selection, instruction evolution, and diagnostic filtering. The authors fine-tune Mistral-7B, Llama3.1-8B/70B, Qwen2-7B, and Yi-1.5-9B on this dataset and report consistent improvements over official instruction-tuned counterparts on foundational benchmarks (MATH, GSM8K, HumanEval, MBPP, MMLU, C-Eval) and conversational benchmarks (AlpacaEval 2.0, Arena-Hard, MT-Bench), including the headline claim that InfInstruct-Llama3.1-70B outperforms GPT-4-0314 by 8.6% on instruction-following tasks while matching it on foundational tasks.

Significance. If the reported results hold, this is a valuable contribution: the public release of a large instruction dataset, a reproducible pipeline combining selection and synthesis, and consistent gains across several model families would be a useful resource for the community. The paper also includes scaling experiments and an ablation of two-stage training, which speak to questions of data volume and curriculum design. However, the foundational gains are potentially confounded by benchmark-guided selection, and the manuscript contains internal numerical inconsistencies that prevent the reader from verifying the central claim. The contribution is significant in scope, but the evidence as presented is not yet sufficient to support the general-capability interpretation.

major comments (5)
  1. [Section 2.3] The DSIR target distributions are constructed from the evaluation benchmarks themselves: the Math selection uses prompts from GSM8K and MATH training samples as targets, and the Code selection uses prompt distributions from HumanEval samples. The subsequent 'Evaluation and Weak-domain Instruction Supplement' relaxes the selection criteria until a fine-tuned Mistral-7B approaches GPT-3.5 on the same benchmark set that is later reported as foundational gains in Tables 4 and 7. This is benchmark-guided selection rather than direct contamination, but it means the training distribution is explicitly shaped by the evaluation tasks, so the foundational improvements may partly reflect learning benchmark-shaped problem formats rather than general mathematical or coding ability. The central claim of the paper requires held-out evaluation on benchmarks not used in any part of the selection process (for example, AIME or LiveCodeBench for reasoning/code, or other unseen knowledge benchmarks), or a control experiment with target distributions that are not derived from the evaluation suites.
  2. [Tables 3-5, 4-7] The same model and dataset combination appears with materially different scores in different tables, and the manuscript does not explain these discrepancies. For Llama3.1-8B, GSM8K is 70.2 in Table 4 but 78.6 in Table 7; HumanEval is 53.7 versus 59.8; MATH is 28.1 versus 26.0; Arena-Hard is 33.7 in Table 3 versus 38.9 in Table 5; AlpacaEval 2.0 is 33.9 versus 34.0. The official Llama3.1-8B-Instruct row also differs between Table 4 (GSM8K 55.2, MMLU 66.1, C-Eval 42.3) and Table 7 (GSM8K 67.8, MMLU 53.2, C-Eval 47.5). Because the paper's core claim of consistent gains across models depends on these tables, the authors must clarify whether these are different training runs, different evaluation versions, or different prompts, and report variance or error bars. As written, the numerical inconsistencies make the results impossible to reconcile.
  3. [Tables 3-5] The 'Overall*' score is defined only by the footnote 'To calculate the overall average score, we mapped MT-Bench scores to 0-100', but no mapping formula is given. The abstract's headline claim that InfInstruct-Llama3.1-70B outperforms GPT-4-0314 by 8.6% on instruction-following tasks is based on this Overall* aggregate, and the mapping is a free parameter that must be specified for the result to be reproducible. I recommend reporting the exact transformation and, ideally, also reporting the three component scores separately rather than relying on a single aggregate.
  4. [Section 2.5] The contamination filtering relies on a BGE cosine-similarity threshold of 0.3 that is described as 'manually confirmed', but the paper reports no analysis of how many near-duplicates remain between the training data and each evaluation benchmark. Given that the DSIR target distributions are built from the evaluation prompts themselves, the claim of 'clean generalization' requires per-benchmark contamination statistics, such as n-gram overlap rates or embedding-similarity distributions, to show that the selected data does not contain benchmark-shaped examples beyond the threshold. Without this, the decontamination step cannot be assessed.
  5. [Section 3.3, Figure 4] The scaling curves in Figure 4 are described as showing 'a continuous trend of increasing overall model performance', but the displayed deltas include negative values (e.g., -0.8%, -2.2%) for some tasks, and no error bars are provided. The figure does not make clear whether the annotations are differences between the smallest and largest data scales or between consecutive scales, and the 'Average' line is undefined. Since the scaling analysis is used to argue that large-scale instruction data is necessary, the authors should clarify the computation and report variance across seeds or evaluation runs.
minor comments (5)
  1. [Section 2.2] There are several typos and inconsistent notations: 'OpenHermers' should be 'OpenHermes' in the text and Table 2; 'BMPP' should be 'MBPP'; 'C-EV AL' appears with a space in Table 2 but as 'C-Eval' elsewhere; and 'GSM-8K' and 'GSM8K' are used inconsistently across tables.
  2. [Table 6] The title 'InfInstruct-F-7.4M and InfInstruct-F-1.5M Dialog Turns Statistics' appears to contain a typo: the second dataset should likely be 'InfInstruct-G-1.5M', since the paper's conversational dataset is denoted G-1.5M.
  3. [Section 2.4] In the Diagnosis paragraph, 'for theses ability types' should be 'for these ability types'.
  4. [Appendix A] The benchmark name is written both as 'Mt-Bench' in the appendix and 'MT-Bench' in the main text; please standardize.
  5. [Section 2.4] The two-layer labeling system is central to the pipeline, but the paper only says 'Details about the labeling system can be found in [28]'. Since the 26 first-level labels and 15,000+ second-level labels are used in the diversity and difficulty filters, the manuscript should at least summarize the label taxonomy or include a list in the appendix so the selection criteria are self-contained.

Circularity Check

3 steps flagged · score 6.0 of 10

Foundational gains are benchmark-guided: DSIR targets and the saturation loop use the exact evaluation benchmarks later reported as improvements; the conversational results provide some independent support.

  1. fitted input called prediction [Section 2.3, 'Math' and 'Code' paragraphs; Tables 2, 4, 7]
    "To improve the model’s mathematical abilities, we will use the prompts from GSM8K and MATH training samples as target distributions to guide subset selection from the pool of mathematical datasets. ... we base the importance resampling process on prompt distributions obtained from HumanEval samples."

    The math and code splits of InfInstruct-F-7.4M are selected by DSIR to match the prompt distributions of the exact benchmarks (GSM8K, MATH, HumanEval) on which the paper later reports foundational gains (Tables 4 and 7). The model is trained on data deliberately shaped to resemble these benchmarks, so the subsequent improvements on those benchmarks are an expected consequence of the selection target rather than an independent measure of general mathematical or coding ability. The BGE decontamination step (threshold 0.3) removes near-duplicate examples but does not remove the distribution-matching effect, so the reported gains are partly forced by the data-selection objective.

  2. fitted input called prediction [Section 2.3, 'Evaluation and Weak-domain Instruction Supplement']
    "we evaluate the saturation level of the current dataset for each task by fine-tuning experiments in Mistral-7B. When a performance gap is observed between the fine-tuned Mistral-7B on the current dataset version and a baseline model (e.g., GPT-3.5) for a specific task, we will relax the data selection criteria to incorporate additional weak-domain data."

    The data-selection criteria for each task are iteratively adjusted until a Mistral-7B fine-tune closes the gap to GPT-3.5 on the Table 2 benchmark set (GSM8K, MATH, HumanEval, MBPP, MMLU, C-Eval). The same set is then used to claim foundational improvements in Tables 4 and 7. Thus the dataset composition and the evaluation on these benchmarks are jointly optimized; the 'prediction' that fine-tuning on Infinity-Instruct improves foundational performance is not an independent test but a reflection of the selection loop's objective.

1 more flagged steps
  1. self citation load bearing [Section 2.3 'Math' paragraph; Section 2.4 'Instruction Labeling System']
    "Details can be found in [26]. ... The final labeling system we constructed contains 26 first-level labels and more than fifteen thousand second-level labels. Details about the labeling system can be found in [28]."

    The math-synthesis pipeline and the instruction-labeling system, both load-bearing components of the dataset construction, are deferred to companion papers [26] and [28] whose author lists overlap substantially with the present paper (e.g., Bo-Wen Zhang, Guang Liu, Hanyu Zhao, Li Du). The present paper does not provide independent verification of these components; if the companion papers are not externally validated, part of the construction chain reduces to self-citation. This is a weaker form of circularity than the benchmark-guided selection, but it is present.

full rationale

The central foundational claim is partially circular. Section 2.3 sets DSIR target distributions from GSM8K and MATH training prompts (math) and HumanEval prompt distributions (code), and the saturation loop relaxes data-selection criteria until Mistral-7B approaches GPT-3.5 on the Table 2 benchmark set. Tables 4 and 7 then report improvements on precisely those benchmarks as evidence of broad foundational gains. This is benchmark-guided selection: the reported math/code improvements are an expected consequence of the selection objective, not an independent measure. The BGE-based deduplication (threshold 0.3) addresses direct contamination but does not eliminate the distribution-matching effect. The conversational results (AlpacaEval 2.0, Arena-Hard, MT-Bench) were not used as DSIR targets and provide some independent support, which prevents the whole paper from being entirely circular. There are also self-citations to companion papers ([26] and [28]) for the math synthesis and labeling system; these are load-bearing but secondary. Internal inconsistencies in reported numbers (e.g., Llama3.1-8B GSM8K 70.2 in Table 4 vs 78.6 in Table 7; Arena-Hard 33.7 in Table 3 vs 38.9 in Table 5) further prevent the manuscript from resolving the concern. Overall, the foundational capability claim is largely forced by the benchmark-guided selection, while the conversational claim retains independent content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claim rests on several hand-chosen curation thresholds: the 0.3 cosine decontamination cutoff, the DSIR target distributions taken from evaluation benchmarks, and the label-frequency retention cutoffs. The aggregated chat score used for the headline 8.6% number depends on an unstated MT-Bench to 0-100 mapping. Standard assumptions about GPT-4 judges and public benchmarks are invoked throughout. The only invented entity is the two-layer labeling taxonomy, which is defined in the authors' own prior paper [28] and lacks a direct falsifiable handle. Overall, the paper's added structure is engineering heuristics on top of previously published components.

free parameters (4)
  • Cosine-similarity contamination threshold = 0.3
    Section 2.5: manually confirmed threshold for BGE-based deduplication and decontamination against benchmarks; it controls how much of the 100M+ pool is admitted into the 7.4M training set.
  • DSIR target distributions = GSM8K+MATH prompts (math); HumanEval prompts (code)
    Section 2.3: the target distributions for importance resampling are the evaluation benchmarks' own training prompts, so the curation is benchmark-guided.
  • Label-frequency retention thresholds = freq 20-200 fully retained; 1/3 of freq 200-500 retained
    Section 2.4: hand-chosen cutoffs for long-tail diversity in high-quality seed selection.
  • MT-Bench to 0-100 mapping for Overall* = unstated
    Footnote of Tables 3-5: the aggregated chat score used for the 8.6% headline is computed by mapping MT-Bench to 0-100, but the mapping formula is not given.
assumptions (5)
  • domain assumption GPT-4 and GPT-4-Turbo judges provide reliable preference judgments for AlpacaEval 2.0, Arena-Hard, and MT-Bench.
    Standard practice in the field; the paper relies on these judge-based scores for the headline chat claims.
  • domain assumption The public benchmarks (MMLU, C-Eval, MATH, GSM8K, HumanEval, MBPP) measure general foundational ability.
    Used throughout Tables 2, 4, and 7 to define 'foundational performance'.
  • ad hoc to paper The two-layer labeling system of [28] is a valid taxonomy for diversity and difficulty selection.
    Section 2.4 delegates the core labeling-system details to self-cited [28] by overlapping authors; its validity is assumed for seed selection and diagnosis.
  • domain assumption Evol-Instruct rewriting preserves semantic identity unless the rewriting model flags harm or divergence.
    Section 2.4 relies on the rewriting model's own judgment to keep evolved instructions on-task.
  • domain assumption The 100M+ open-source instruction pool is a representative sample of general instruction distributions.
    The whole pipeline selects from this pool; if the pool is biased, the resulting dataset inherits the bias.
invented entities (1)
  • Two-layer instruction labeling system (26 first-level, 15,000+ second-level labels)
    purpose: Taxonomy for diversity-based seed selection, multi-capability difficulty scoring, and weak-ability diagnosis in Section 2.4.
    Defined in detail only in self-cited [28]; it has no direct falsifiable handle, and its utility is measured only indirectly through downstream model performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models." pith.science (2026). https://pith.science/paper/VM2FN2YZ

@misc{pith2026250611116,
  author       = {Pith},
  title        = {Pith review of: Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VM2FN2YZ}},
  note         = {Machine review of arXiv:2506.11116}
}
read the original abstract

Large Language Models (LLMs) demonstrate strong performance in real-world applications, yet existing open-source instruction datasets often concentrate on narrow domains, such as mathematics or coding, limiting generalization and widening the gap with proprietary models. To bridge this gap, we introduce Infinity-Instruct, a high-quality instruction dataset designed to enhance both foundational and chat capabilities of LLMs through a two-phase pipeline. In Phase 1, we curate 7.4M high-quality foundational instructions (InfInstruct-F-7.4M) from over 100M samples using hybrid data selection techniques. In Phase 2, we synthesize 1.5M high-quality chat instructions (InfInstruct-G-1.5M) through a two-stage process involving instruction selection, evolution, and diagnostic filtering. We empirically evaluate Infinity-Instruct by fine-tuning several open-source models, including Mistral, LLaMA, Qwen, and Yi, and observe substantial performance gains across both foundational and instruction following benchmarks, consistently surpassing official instruction-tuned counterparts. Notably, InfInstruct-LLaMA3.1-70B outperforms GPT-4-0314 by 8.6\% on instruction following tasks while achieving comparable foundational performance. These results underscore the synergy between foundational and chat training and offer new insights into holistic LLM development. Our dataset\footnote{https://huggingface.co/datasets/BAAI/Infinity-Instruct} and codes\footnote{https://gitee.com/li-touch/infinity-instruct} have been publicly released.

Figures

Figures reproduced from arXiv: 2506.11116 by the authors.

Figure 1
Figure 1. The overall structure for building the Infinity-Instruct dataset. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overall Pipeline of Data Selection Pipeline [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overall Pipeline of Data Synthesis Pipeline [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Scaling curves on foundational and conversational tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: T-SNE visualization and analysis of the first-level label type distribution of instructions in [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs

    cs.CL 2026-05 conditional novelty 7.0 of 10

    K12-KGraph is a textbook-derived knowledge graph that powers a new benchmark revealing LLMs' poor curriculum cognition and a small training corpus that outperforms general instruction data on educational tasks.

  2. Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Spoken math models that emit a 40%-compressed reasoning trace between question and answer beat full-reasoning baselines by ~3 accuracy points while using roughly one third of the text tokens.

  3. CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning

    cs.SD 2026-07 conditional novelty 6.0 of 10

    Routing early CLAP stages to the audio projector and later stages to the LLM yields encoder-free captioning that beats LLM-only distillation by +12.2 CIDEr-D on AudioCaps.

  4. From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons

    cs.CL 2026-04 conditional novelty 6.0 of 10

    FLUID adapts AR LLMs into strictly causal diffusion models with entropy-driven elastic horizons, matching strong AR performance on reasoning and code while using only billions of adaptation tokens.

  5. LsrIF: Enhancing Logic-Structured Instruction Following of Large Language Models

    cs.AI 2026-01 conditional novelty 6.0 of 10

    Logic-structured rewards—averaging parallel constraints, decaying rewards after sequential failures, rewarding only the active conditional branch—improve instruction-following and transfer to reasoning.

  6. Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Non-reasoning LLMs fail to correct their own errors (64.5% blind spot) while correcting identical external errors, and appending 'Wait' cuts the gap by 89.3%.

  7. Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs

    cs.CL 2026-07 conditional novelty 5.0 of 10

    AdaDSF uses per-layer cosine similarity to decide which tokens skip which layers, then distills the sparse model back toward the dense one, cutting FLOPs while holding accuracy close to dense.

  8. DataPrep-Bench: Benchmarking LLMs as Training Data Preparators

    cs.LG 2026-05 conditional novelty 5.0 of 10

    DataPrep-Bench jointly benchmarks data construction and data-quality evaluation for LLMs across six domains with downstream fine-tuning performance as ground truth.

  9. LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding

    cs.LG 2026-02 conditional novelty 5.0 of 10

    Replacing KL-divergence draft training with LK losses—negative log-acceptance or an adaptive KL–TV hybrid—raises average accepted-token length by up to about 8% at temperature 1 across six target LLMs.

  10. Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs

    cs.CV 2026-02 conditional novelty 5.0 of 10

    Visual token compression (4x fewer tokens) plus a three-stage generative/contrastive/judge-curated training pipeline yields state-of-the-art MLLM-based retrieval accuracy at lower inference cost.

Reference graph

Works this paper leans on

31 extracted references · 6 canonical work pages · cited by 10 Pith papers

  1. [1]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  2. [2]

    Curriculum learning

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. InProceedings of the 26th annual international conference on machine learning, pp. 41–48, 2009

    work page 2009

  3. [3]

    Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020

    Tom B Brown. Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  4. [4]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  5. [5]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  6. [6]

    Opencompass: A universal evaluation platform for foundation models.https://github.com/open-compass/opencompass, 2023

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models.https://github.com/open-compass/opencompass, 2023

    2023

  7. [7]

    Enhancing chat language models by scaling high-quality instructional conversations.arXiv preprint arXiv:2305.14233, 2023

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations.arXiv preprint arXiv:2305.14233, 2023

    arXiv 2023

  8. [8]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    arXiv 2009

Show all 31 references
  1. [9]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  2. [10]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. InAdvances in Neural I...

    2023

  3. [11]

    From quantity to quality: Boosting llm performance with self- guided data selection for instruction tuning.arXiv preprint arXiv:2308.12032, 2023

    Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting llm performance with self- guided data selection for instruction tuning.arXiv preprint arXiv:2308.12032, 2023

    2023 arXiv

  4. [12]

    Superfiltering: Weak-to-strong data filtering for fast instruction-tuning.arXiv preprint arXiv:2402.00530, 2024

    Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning.arXiv preprint arXiv:2402.00530, 2024

    2024 arXiv

  5. [13]

    From live data to high-quality benchmarks: The arena-hard pipeline, April 2024

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, et al. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024. URL https://lmsys.org/ blog/2024-04-19-arena-hard/

    2024

  6. [14]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models.https://github.com/tatsu-lab/alpaca_eval, 5 2023

    2023

  7. [15]

    The flan collection: Designing data and methods for effective instruction tuning.arXiv preprint arXiv:2301.13688, 2023

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning.arXiv preprint arXiv:2301.13688, 2023

    2023 arXiv

  8. [16]

    # instag: Instruction tagging for analyzing supervised fine-tuning of large language models

    Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. # instag: Instruction tagging for analyzing supervised fine-tuning of large language models. InThe Twelfth International Conference on Learning Representations, 2023. 10

    2023

  9. [17]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  10. [18]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  11. [19]

    Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023

    Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URLhttps://huggingface.co/datasets/teknium/OpenHermes-2.5

    2023

  12. [20]

    C-pack: Packaged resources to advance general chinese embedding, 2023

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023

    2023

  13. [21]

    Data selection for language models via importance resampling, 2023

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling, 2023. URLhttps://arxiv.org/abs/2302.03169

    2023 arXiv

  14. [22]

    Wizardlm: Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  15. [23]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing.arXiv preprint arXiv:2406.08464, 2024

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing.arXiv preprint arXiv:2406.08464, 2024

    2024 arXiv

  16. [24]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  17. [25]

    Aquila2 technical report.arXiv preprint arXiv:2408.07410, 2024

    Bo-Wen Zhang, Liangdong Wang, Jijie Li, Shuhao Gu, Xinya Wu, Zhengduo Zhang, Boyan Gao, Yulong Ao, and Guang Liu. Aquila2 technical report.arXiv preprint arXiv:2408.07410, 2024

    2024 arXiv

  18. [26]

    Infinitymath: A scalable instruction tuning dataset in programmatic mathematical reasoning

    Bo-Wen Zhang, Yan Yan, Lin Li, and Guang Liu. Infinitymath: A scalable instruction tuning dataset in programmatic mathematical reasoning. InProceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 5405–5409, 2024

    2024

  19. [27]

    Map-neo: Highly capable and transparent bilingual large language model series.arXiv preprint arXiv:2405.19327, 2024

    Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, et al. Map-neo: Highly capable and transparent bilingual large language model series.arXiv preprint arXiv:2405.19327, 2024

    2024 arXiv

  20. [28]

    Beyond iid: Optimizing instruction learning from the perspective of instruction interaction and dependency

    Hanyu Zhao, Li Du, Yiming Ju, Chengwei Wu, and Tengfei Pan. Beyond iid: Optimizing instruction learning from the perspective of instruction interaction and dependency. 2024. URL https://arxiv.org/abs/2409.07045

    2024 arXiv

  21. [29]

    Wildchat: 1m chatgpt interaction logs in the wild.arXiv preprint arXiv:2405.01470, 2024

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatgpt interaction logs in the wild.arXiv preprint arXiv:2405.01470, 2024

    2024 arXiv

  22. [30]

    P Xing, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat-1m: A large-scale real-world llm conversation dataset, 2023

    2023

  23. [31]

    Infinity Instruct

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 11 A Evaluat...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.