Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

Not All Correct Answers Are Equal: Why Your Distillation Source Matters

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Distillation source determines student reasoning quality: training Qwen2.5-32B on AM-Thinking-v1 distilled data beats training on Qwen3-235B-A22B or DeepSeek-R1 distilled data on AIME2024, AIME2025, MATH500, and LiveCodeBench under the…

desk verdict Large parallel distillation comparison with released data, but teacher attribution is confounded by per-teacher filtering. read the letter →

arxiv 2505.14464 v2 pith:XCMOLLDW submitted 2025-05-20 cs.CL

Xiaoyu Tian , Yunjie Ji , Haotian Wang , Shuaiting Chen , Sitong Zhao , Yiping Peng , Han Zhao , Xiangang Li This is my paper · ORCID
classification cs.CL
keywords knowledgedistillationreasoningdatateachermodelcomparisonchain-of-thoughtAIMEbenchmarksLiveCodeBenchperplexityadaptivegenerationlength
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the teacher model used to distill reasoning data changes what a student model learns, even when the student, the queries, and the training recipe stay fixed. The authors distilled 1.89 million shared queries with three teachers—AM-Thinking-v1, Qwen3-235B-A22B, and DeepSeek-R1—producing three parallel verified datasets. Students trained on the AM-Thinking-v1 data scored highest everywhere: 84.3 vs 79.4 vs 70.9 on AIME2024, 72.2 vs 62.2 vs 52.8 on AIME2025, 98.4 vs 93.9 vs 95.8 on MATH500, and 65.9 vs 59.6 vs 57.0 on LiveCodeBench. The paper argues this comes from the data itself: AM outputs had lower perplexity and a wider spread of response lengths, and the resulting model adapted its generation length to task difficulty. If true, the right distillation source is a practical lever for open-source reasoning models, and correct answers are not interchangeable training signal.

What carries the argument

The load-bearing object is a set of three parallel, verified distillation datasets built from a shared corpus of 1.89 million queries, each query answered by all three teachers and accepted only when an automatic verifier gave a score of at least 0.9. Per-category verifiers (math, code, science, instruction following, multi-turn chat, and general reasoning) enforce correctness, while perplexity filtering, n-gram deduplication, and logical checks shape quality. The comparison works because student model, data volume, training hyperparameters, and evaluation protocol are held constant; the only intended difference is which teacher produced the traces. The paper's explanatory mechanism is distributional: AM-Thinking-v1 data has lower perplexity and a wider token-length spread, which the authors link to the student's observed adaptive output lengths.

What would settle it

Run the same three-way comparison again with identical perplexity thresholds, a fixed and reported maximum number of regeneration attempts per query, and matched final dataset sizes; if the AM-Thinking-v1 student no longer beats the others by a wide margin, the teacher-source explanation fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that teacher identity, not just answer correctness, determines how much reasoning skill a student model absorbs. Holding the student base model (Qwen2.5-32B), the 1.89M query set, and the training recipe fixed, swapping only the teacher changes final benchmark performance by up to roughly 19 points on AIME2025. The AM-Thinking-v1-distilled student is best on all four benchmarks; its data is characterized by lower mean perplexity (2.5 vs 2.9 and 3.0) and a token-length distribution with both many short traces and many very long traces. The paper interprets these distributional properties as the mechanism: they teach the student to generate longer solutions when the problem demands it and shorter ones when it does not. This frames distillation quality as a property of the reasoning trace as a whole, not merely of the final answer's correctness.

Load-bearing premise

The comparison assumes the three datasets differ only in which teacher produced them; however, each teacher's outputs were cleaned with a different quality cutoff and regenerated until they passed a score threshold, so part of the measured gap could come from how aggressively each dataset was filtered rather than from the teacher itself.

Editorial extensions

If this is right

  • Training on AM-Thinking-v1-distilled data gives a Qwen2.5-32B student 84.3 on AIME2024, 72.2 on AIME2025, 98.4 on MATH500, and 65.9 on LiveCodeBench—higher than the Qwen3- and DeepSeek-R1-distilled students on every benchmark.
  • A student's response length becomes adaptive when its training data contains both very short and very long verified traces: it generates roughly 18,000 tokens on AIME2025 but only about 3,500 tokens on MATH500.
  • Lower perplexity of the training signal tracks lower training loss throughout optimization, so data quality is visible in training dynamics, not only in final scores.
  • The released AM-Thinking-v1 and Qwen3-235B-A22B distilled datasets let others reproduce the comparison and test whether the pattern holds for other student sizes and training budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A controlled follow-up that equalizes filtering thresholds and regeneration budgets across teachers would separate the teacher effect from the curation effect; the paper's current numbers cannot distinguish the two.
  • If the active ingredient is token-length diversity rather than the identity of the teacher, resampling the Qwen3 or DeepSeek-R1 outputs to match AM-Thinking-v1's length distribution while keeping content fixed would transfer much of the measured benefit.
  • The same three parallel verified datasets, trained on smaller and larger student bases, would show whether the teacher ordering is stable across model capacity and whether the adaptive-length benefit grows with scale.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a large-scale parallel distillation study: for a shared corpus of 1.89 million queries, it collects verified reasoning traces from three teacher models (AM-Thinking-v1, Qwen3-235B-A22B, and DeepSeek-R1), constructs three parallel datasets, and trains Qwen2.5-32B students on each. The authors analyze token-length and perplexity distributions, then evaluate the students on AIME2024, AIME2025, MATH500, and LiveCodeBench. The AM-Thinking-v1-distilled model consistently obtains the highest scores (e.g., 84.3 on AIME2024, 72.2 on AIME2025, 98.4 on MATH500, 65.9 on LiveCodeBench), leading to the conclusion that the distillation source materially affects student reasoning quality. The paper also analyzes generation-length behavior and releases two of the three distilled datasets.

Significance. If the comparison were clean, this would be a valuable empirical contribution: it provides large-scale, verified, parallel distillation data from multiple teachers and demonstrates that teacher identity can shape student performance on reasoning benchmarks. The release of the AM-Thinking-v1 and Qwen3-235B-A22B distilled datasets is a useful community resource, and the decontamination and verification pipeline is described with unusual care. However, the central causal claim—that teacher identity rather than data-construction choices drives the performance gaps—is weakened by protocol differences across teachers in filtering and verification effort, and by the absence of statistical uncertainty estimates on the headline benchmark numbers. The data-quality analysis (perplexity) is also partly self-referential because the scorer is a same-team model. The resource and descriptive findings are significant; the causal attribution requires additional controls.

major comments (4)
  1. [2.2 (Data Distilling), Perplexity-based Filtering] The paper states that perplexity filtering was applied with 'each model employing a different threshold.' This directly undermines the comparability of the three datasets: if AM-Thinking-v1 data were filtered more aggressively, the lower mean PPL in Figure 5 and the higher benchmark scores in Table 1 could result from data selection rather than from teacher output quality. Please report the exact threshold used for each teacher's data, the final dataset sizes after filtering, and ideally a control experiment in which all three datasets are filtered at the same threshold or matched in size and PPL distribution.
  2. [2.2 (Data Distilling), Distillation Framework] The verification loop repeats generation 'until the generated response satisfied the verification criterion (score ≥ 0.9)' with no reported cap on attempts, no per-teacher pass rate, and no final dataset sizes. If one teacher required substantially more generation attempts or failed verification more often, the resulting training data would reflect selection effort and query-level difficulty in a way that is not held constant across teachers. Please report the sampling budget, verification pass rate, and final instance counts for each teacher, or modify the protocol to use a fixed number of attempts per query.
  3. [3.2 and Table 1 (Results)] The benchmark differences that support the headline claim are small on the AIME tasks, which contain only 30 questions each: AIME2024 84.3 vs. 79.4 is a difference of 1.5 questions, and AIME2025 72.2 vs. 62.2 is a difference of 3 questions. No confidence intervals, significance tests, or per-question pass rates are reported, so it is unclear whether these gaps are robust to evaluation noise. Please provide confidence intervals (e.g., bootstrap over the 30 questions) or multiple evaluation seeds for all benchmarks, especially AIME2024 and AIME2025.
  4. [2.3 (Data Analysis), Figure 5] The claim that AM-Thinking-v1 data has 'better overall quality' because of its lower mean PPL is not supported as independent evidence. The PPL scorer is a 32B model from the same team (reference [30]), and the filtering threshold was allowed to differ per teacher. Lower PPL on the filtered dataset could be a direct consequence of a more aggressive PPL cutoff rather than of genuine output quality. Please report the PPL distribution before filtering, use an external or at least independently trained scorer for at least a subset, and avoid presenting PPL as a quality measure without calibration against downstream task performance.
minor comments (5)
  1. [Figure 2 caption] The caption contains a typo: 'AM-Thinkin-v1' should be 'AM-Thinking-v1'.
  2. [Section 3.2 (Benchmarks and Evaluation Setup), MATH500] The text says 'Each prompt was answered once, and we sampled 4 times to compute pass@1,' which is internally inconsistent: sampling 4 times and computing pass@1 is not standard; please clarify whether this is pass@1 from a single sample or a top-k or majority-based metric, and use a consistent nomenclature.
  3. [Section 4, Table 2] The text emphasizes that AM-Thinking-v1 produces longer outputs on AIME and LiveCodeBench and shorter on MATH500, but Table 2 shows that DeepSeek-R1 produces the longest outputs on LiveCodeBench (30731.0 tokens). Please clarify that the adaptivity claim is about the AM model across tasks, not about being the longest per task, and discuss whether length itself or length diversity is the more likely driver.
  4. [Section 4 (Results and Analysis), loss curves] Lower training loss on different target distributions is not by itself evidence of 'more learnable, coherent, and high-quality' data; it may simply reflect that the AM-Thinking-v1 responses are easier to model by the Qwen2.5-32B base. Please temper this interpretation or provide a control such as loss on a held-out set of same-distribution data.
  5. [Section 5 (Conclusion)] The manuscript releases only the AM-Thinking-v1 and Qwen3-235B-A22B distilled datasets, not the DeepSeek-R1 data; this limits independent replication of the three-way comparison. Consider releasing the DeepSeek-R1 data as well, or clearly state where users can obtain an equivalent dataset.

Circularity Check

1 steps flagged · score 2.0 of 10

Central benchmark comparison is externally grounded; only the PPL-based quality analysis is partly self-referential.

  1. fitted input called prediction [Section 2.2 (Quality Assurance Measures) and Section 2.3 (Figure 5, PPL distributions)]
    "Perplexity-based Filtering: We computed perplexity scores using a strong 32B language model [30], with each model employing a different threshold. Notably, responses distilled from AM-Thinking-v1 demonstrated the lowest perplexity among the three models. ... Among the three datasets, AM-Thinking-v1 achieves the lowest mean PPL (2.5), suggesting that its distilled outputs are generally of higher quality."

    The paper presents AM's lower perplexity as evidence of higher data quality, but the data were selected using that same perplexity metric, with thresholds allowed to differ per teacher. Post-filter PPL is a direct function of the chosen threshold: a stricter cutoff for AM-Thinking-v1 would mechanically lower its mean PPL, so the reported ranking is not an independent property of the teacher's outputs. Since the thresholds are not reported, the 'quality advantage' is at least partly an artifact of the filtering rule rather than a measured teacher characteristic, making the data-quality comparison self-referential rather than a prediction from teacher identity.

full rationale

The paper's headline claim is that distillation source matters: students trained on AM-Thinking-v1-distilled data outperform students trained on Qwen3-235B-A22B or DeepSeek-R1 distilled data under a shared training recipe. This claim is supported by evaluations on external benchmarks (AIME2024, AIME2025, MATH500, LiveCodeBench), which are independent of the paper's data construction and teacher selection. The same training recipe, learning rate, sequence length, and system prompt are applied to all three student arms, so the benchmark comparison is not circular. The use of same-team references for the training recipe and for the AM-Thinking-v1 teacher does not by itself make the central result circular, because the comparison is empirically evaluated against outside benchmarks. The one genuinely circular-adjacent element is the perplexity-based data-quality analysis: the same perplexity metric was used to filter the data (with per-teacher thresholds) and then cited as evidence that AM data is higher quality. That finding is not load-bearing for the external-benchmark conclusion, but it is a self-referential step in the paper's data-quality narrative, so the overall circularity score is low rather than zero.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or theoretical entities. Its central empirical contribution rests on data-processing choices and evaluative assumptions, which are listed above.

free parameters (5)
  • verify_score threshold = 0.9
    Single inclusion cutoff across all categories; controls which responses enter the training set.
  • per-model PPL filtering threshold = undisclosed, differs per teacher
    Different thresholds per teacher confound the data-quality comparison and may change dataset composition.
  • max sequence length = 32k tokens
    Samples longer than 32k tokens are excluded; affects the token-length distribution and training data.
  • evaluation sampling budgets = 64/16/4 for AIME/LCB/MATH500
    pass@1 estimates depend on these counts; no confidence intervals are reported.
  • training recipe = lr 8e-5, batch 64, 2 epochs, 32k seq len
    Chosen following prior work; all three models share it, so it does not differentiate teachers, but results depend on it.
assumptions (3)
  • domain assumption Automatic verification with score >= 0.9 is a valid proxy for correct reasoning traces.
    Used to certify all distilled data; if the verifier is biased toward certain teachers, the comparison inherits that bias.
  • domain assumption Perplexity from a 32B model measures data quality.
    The PPL comparisons in Section 2.3 and the filtering step both rely on this assumption; perplexity is a fluency measure, not a correctness measure.
  • domain assumption Decontamination removed overlap with evaluation benchmarks.
    Exact-match and semantic deduplication are described against AIME2024; contamination of AIME2025 or LiveCodeBench could inflate scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Not All Correct Answers Are Equal: Why Your Distillation Source Matters." pith.science (2026). https://pith.science/paper/XCMOLLDW

@misc{pith2026250514464,
  author       = {Pith},
  title        = {Pith review of: Not All Correct Answers Are Equal: Why Your Distillation Source Matters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XCMOLLDW}},
  note         = {Machine review of arXiv:2505.14464}
}
read the original abstract

Distillation has emerged as a practical and effective approach to enhance the reasoning capabilities of open-source language models. In this work, we conduct a large-scale empirical study on reasoning data distillation by collecting verified outputs from three state-of-the-art teacher models-AM-Thinking-v1, Qwen3-235B-A22B, and DeepSeek-R1-on a shared corpus of 1.89 million queries. We construct three parallel datasets and analyze their distributions, revealing that AM-Thinking-v1-distilled data exhibits greater token length diversity and lower perplexity. Student models trained on each dataset are evaluated on reasoning benchmarks including AIME2024, AIME2025, MATH500, and LiveCodeBench. The model distilled from AM-Thinking-v1 consistently achieves the best performance (e.g., 84.3 on AIME2024, 72.2 on AIME2025, 98.4 on MATH500, and 65.9 on LiveCodeBench) and demonstrates adaptive output behavior-producing longer responses for harder tasks and shorter ones for simpler tasks. These findings highlight the value of high-quality, verified reasoning traces. We release the AM-Thinking-v1 and Qwen3-235B-A22B distilled datasets to support future research on open and high-performing reasoning-oriented language models. The datasets are publicly available on Hugging Face\footnote{Datasets are available on Hugging Face: \href{https://huggingface.co/datasets/a-m-team/AM-Thinking-v1-Distilled}{AM-Thinking-v1-Distilled}, \href{https://huggingface.co/datasets/a-m-team/AM-Qwen3-Distilled}{AM-Qwen3-Distilled}.}.

Figures

Figures reproduced from arXiv: 2505.14464 by the authors.

Figure 1
Figure 1. Open-source model benchmarks on AIME2024/LiveCodeBench [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Instance-level and token-level output distributions are analyzed for AM-Thinkin-v1, Qwen3- [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Token span distribution of instances for AM-Thinking-v1, Qwen3-235B-A22B, and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Token count distributions for AM-Thinking-v1, Qwen3-235B-A22B, and DeepSeek-R1 [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Perplexity (PPL) distributions for AM-Thinking-v1, Qwen3-235B-A22B, and DeepSeek-R1 [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Loss curves of AM-Thinking-v1-Distilled, DeepSeek-R1-Distilled and Qwen3-235B-A22B [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Gold-Guided Programmatic Distillation for Financial Reasoning over Hybrid Tables and Text

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Execution-verified programmatic distillation with gold derivations and iterative recovery trains a 7B student that outperforms its 72B teacher on TAT-QA.

  2. When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models

    cs.CL 2026-03 conditional novelty 5.0 of 10

    Perplexity-based scoring systematically understates teacher–student gaps in distilled hybrid LLMs and can reverse design-choice rankings; generation-based evaluation is the more reliable protocol.

  3. A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models

    cs.CL 2025-09 conditional novelty 4.0 of 10

    A hybrid-granularity FP8 training recipe achieves near-lossless accuracy versus BF16 on reasoning benchmarks with up to 22% faster training and 14% lower memory.

Reference graph

Works this paper leans on

30 extracted references · 15 canonical work pages · cited by 3 Pith papers

  1. [30]

    1.4 million open-source distilled reasoning dataset to empower large language model training

    Han Zhao, Haotian Wang, Yiping Peng, Sitong Zhao, Xiaoyu Tian, Shuaiting Chen, Yunjie Ji, and Xiangang Li. 1.4 million open-source distilled reasoning dataset to empower large language model training. arXiv preprint arXiv:2503.19633, 2025. 11

    arXiv 2025

  2. [1]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    work page 2025

  3. [2]

    Deepdistill: Enhancing llm reasoning capabilities via large-scale difficulty- graded data training, 2025

    Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yiping Peng, Yunjie Ji, Han Zhao, and Xiangang Li. Deepdistill: Enhancing llm reasoning capabilities via large-scale difficulty- graded data training, 2025

    2025

  4. [3]

    Qwen3, April 2025

    Qwen Team. Qwen3, April 2025

    2025

  5. [4]

    Am-thinking-v1: Advancing the frontier of reasoning at 32b scale, 2025

    Yunjie Ji, Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yiping Peng, Han Zhao, and Xiangang Li. Am-thinking-v1: Advancing the frontier of reasoning at 32b scale, 2025

    work page 2025

  6. [5]

    American invitational mathematics examination - aime

    MAA. American invitational mathematics examination - aime. https://maa.org/ math-competitions/american-invitational-mathematics-examination-aime , feb 2024. Accessed in February 2024, from American Invitational Mathematics Examination - AIME 2024

    work page 2024

  7. [6]

    Aime-preview: A rigorous and imme- diate evaluation framework for advanced mathematical reasoning

    Yixin Ye, Yang Xiao, Tiantian Mi, and Pengfei Liu. Aime-preview: A rigorous and imme- diate evaluation framework for advanced mathematical reasoning. https://github.com/ GAIR-NLP/AIME-Preview, 2025. GitHub repository

    work page 2025

  8. [7]

    Let’s verify step by step, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023

    2023

Show all 30 references
  1. [8]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  2. [9]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025

    2025

  3. [10]

    Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models, 2025

    Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models, 2025

    2025

  4. [11]

    Numinamath

    Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. https://huggingface.co/ AI-MO/NuminaMath...

    2024

  5. [12]

    Free process rewards without process labels

    Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arXiv:2412.01981, 2024

    2024 arXiv

  6. [13]

    Deepcoder: A fully open-source 14b coder at o3-mini level

    Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder at o3-mini level. https://pretty-radio-b75. notion.s...

    2025

  7. [14]

    Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding

    Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding. 2025

    2025

  8. [15]

    Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks, 2022

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, et al. Super-naturalinstructions: Generalization via...

    2022

  9. [16]

    Chemistry-qa

    Microsoft. Chemistry-qa. https://github.com/microsoft/chemistry-qa, 2021. [GitHub repository]

    2021

  10. [17]

    Logic-701: A benchmark dataset for logical reasoning in english and russian

    hivaze. Logic-701: A benchmark dataset for logical reasoning in english and russian. https: //huggingface.co/datasets/hivaze/LOGIC-701, 2023. Hugging Face Dataset. 10

    2023

  11. [18]

    Llama-nemotron-post-training-dataset

    NVIDIA. Llama-nemotron-post-training-dataset. https://huggingface.co/datasets/ nvidia/Llama-Nemotron-Post-Training-Dataset , 2025. Version 1.1, released on April 8, 2025

    2025

  12. [19]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, et al

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, et al. Tülu 3: Pushing frontiers in open language model post-training. 2024

    2024

  13. [20]

    Infinity instruct

    Beijing Academy of Artificial Intelligence (BAAI). Infinity instruct. arXiv preprint arXiv:2406.XXXX, 2024

    2024

  14. [21]

    Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023

    Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023

    2023

  15. [22]

    Enhancing chat language models by scaling high-quality instructional conversations, 2023

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023

    2023

  16. [23]

    Wizardlm evol-instruct 70k dataset

    WizardLM Team. Wizardlm evol-instruct 70k dataset. https://huggingface.co/ datasets/WizardLMTeam/WizardLM_evol_instruct_70k, 2023. Accessed: 2025-04-23

    2023

  17. [24]

    Openorca: An open dataset of gpt augmented flan reasoning traces

    Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet V ong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https:// huggingface.co/datasets/Open-Orca/OpenOrca, 2023

    2023

  18. [25]

    Fine flan: Seqio to parquet so you don’t have to

    Bleys Goodson. Fine flan: Seqio to parquet so you don’t have to. https://huggingface. co/datasets/Open-Orca/FLAN, 2023

    2023

  19. [26]

    Bge m3- embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3- embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024

    2024

  20. [27]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  21. [28]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024

    2024

  22. [29]

    Decision-tree-reward-llama-3.1-8b.https://huggingface.co/ RLHFlow/Decision-Tree-Reward-Llama-3.1-8B , 2025

    Min Li and RLHFlow Team. Decision-tree-reward-llama-3.1-8b.https://huggingface.co/ RLHFlow/Decision-Tree-Reward-Llama-3.1-8B , 2025. Interpreting Language Model Preferences Through the Lens of Decision Trees

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.