Pith. sign in

REVIEW 3 major objections 4 minor 58 references

AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AttentionInfluence shows that masking a small model's retrieval heads yields a supervision-free score for selecting reasoning-intensive pretraining data.

desk verdict A novel, cheap data-selection recipe that is not yet proven: the headline gains are confounded with repeated exposure of the selected 73B subset, so the missing random-upsampling control is the decisive test. read the letter →

arxiv 2505.07293 v1 pith:F62P7YFE submitted 2025-05-12 cs.CL

Kai Hua , Steven Wu , Ge Zhang , Ke Shen This is my paper · ORCID
classification cs.CL
keywords pretrainingdataselectionattentionheadsretrievalweak-to-stronggeneralizationinfluencemeasurereasoning-intensivetraining-freecuration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AttentionInfluence proposes that the loss increase a small pretrained language model suffers when its retrieval attention heads are masked reveals how reasoning-intensive a pretraining document is. The paper uses this signal, computed on a 1.3B model, to pick the top 20% of the SmolLM corpus (about 73B tokens), then mixes those tokens back into the full corpus to pretrain a 7B model for 1T tokens. Against a baseline trained only on SmolLM, the selected-data model gains 1.4 to 3.5 percentage points on MMLU, MMLU-Pro, AGIEval-en, GSM8K, and HumanEval. The claim is that attention-head influence works as a weak-to-strong data selector: a small model, with no labels and no classifier training, can improve a larger model.

What carries the argument

The key machinery is the retrieval-head masking probe. First, the selector model's attention heads are scored on a synthetic key-passage retrieval task, and the top 5% become 'retrieval heads.' Then each candidate document receives an AttentionInfluence score: the relative increase in token-level cross-entropy loss when those heads are masked to uniform attention, compared with the intact model. This score is compared only within data domains, and a higher score is taken to mean higher reasoning intensity. The masking probe carries the entire selection: it is the only signal that ranks the 241B-token corpus.

What would settle it

Train a 7B model on a random 20% sample of the corpus (matched token count and length distribution) in addition to the full corpus, and compare with the AttentionInfluence-selected subset; if the random subset produces the same gains, the selection signal adds nothing. Alternatively, check whether the highest-scoring documents are simply the highest-perplexity documents for the intact 1.3B model; if so, the score reduces to a surprisal filter rather than a reasoning-specific signal.

Watch

Extended reading notes

Core claim

The paper's central claim is that the relative loss increase when the top 5% retrieval heads of a small pretrained model are masked—$(L_{\mathrm{ref}} - L_{\mathrm{base}})/L_{\mathrm{base}}$—measures a document's reasoning intensity, and that ranking a corpus by this score and upsampling the top fraction yields better downstream reasoning and knowledge performance than training on the original corpus alone. The authors validate this on a 1.3B selector and a 7B target, reporting consistent gains across knowledge and reasoning benchmarks, and observe a mirror effect: the tasks where masking the selector's retrieval heads hurts most are the tasks where the 7B model trained on the selected data improves most. They interpret this as evidence that the masking probe isolates the reasoning-relevant content in the data.

Load-bearing premise

The method assumes that the loss jump a 1.3B model shows when its retrieval heads are masked measures a document's reasoning intensity, and that this signal transfers to improve a 7B model trained on the selected data.

Editorial extensions

If this is right

  • A small pretrained model can act as a supervision-free data selector for a larger model, eliminating the need for human-labeled or LLM-labeled training data and classifier training.
  • The selected data is more balanced across content categories and longer than classifier-selected data, suggesting the signal captures procedural and reasoning content rather than just educational style.
  • The mirror effect offers a pre-training predictor: benchmarks that degrade under retrieval-head masking in the selector are the ones most likely to improve in the target model.
  • The approach scales with selector size: a 7B selector selects higher-scoring reasoning data than a 1.3B selector, improving downstream performance further.
  • Combining AttentionInfluence with an educational-value classifier yields comprehensive improvements on tasks requiring factual knowledge, reasoning, or both.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not include a random-upsampling control, so part of the observed gains could come from simply training on more tokens or on a longer, more diverse subset rather than from reasoning-specific selection; a matched random-20% baseline would isolate the mechanism.
  • Because the score is only compared within domains, a cross-domain normalization would be needed to use AttentionInfluence for global corpus reweighting; the current design avoids comparing math to general text.
  • The masking probe is a natural template for other capability-specific heads: replacing the retrieval task with, say, an induction-head probe could yield selectors for in-context learning or long-range coherence.
  • The mirror effect could be turned into a cheap diagnostic before large-scale runs, but its reliability across model families, tokenizers, and corpora has not been demonstrated.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes AttentionInfluence, a training-free data-selection score that uses a small pretrained 1.3B model to rank documents by the relative increase in loss when top-5% retrieval heads are masked (Eq. 5). The method is applied to the SmolLM corpus, selecting the top 20% of samples (~73.1B tokens). A 7B model is pretrained on the full SmolLM corpus plus this selected subset for 1T tokens (the treatment), and compared with a baseline 7B model trained on the SmolLM corpus alone for 1T tokens. The authors report consistent gains of 1.4--3.5pp on MMLU, MMLU-Pro, AGIEval-en, GSM8K, and HumanEval, along with a 'mirror effect' between masking-induced degradation on the 1.3B selector and downstream gains in the 7B model. They also provide analyses of data diversity, word-frequency overlap with FineWeb-Edu, clustering distributions, and GPT-4o-based education/reasoning scores.

Significance. If the reported gains were attributable to the AttentionInfluence selection signal rather than to the experimental design, the paper would make a useful contribution: it proposes a supervision-free, training-free scoring mechanism that is cheap to compute and connects mechanistic interpretability findings to data curation. The manuscript is generally careful in reporting hyperparameters, evaluation settings, and ablations, and it includes extensive appendices with case studies and implementation details. However, the central experimental comparison is confounded, and the paper does not currently provide the control needed to support its main claim that AttentionInfluence identifies high-quality reasoning-intensive data.

major comments (3)
  1. [§5.1, Table 1] The main comparison is confounded by token-level upsampling. The treatment model is trained on the full SmolLM corpus (241B tokens) plus the selected 73.1B subset, while the baseline is trained on the corpus alone; both consume 1T tokens. This means the treated model gives roughly twice the exposure to the selected documents and fewer exposures to the unselected documents, relative to the baseline. Any subset, including a random 73.1B slice, could produce gains under this protocol simply because of repeated exposure and reduced coverage of the rest of the corpus. The paper never runs a random-upsampling control at the same budget. Without such a control, the reported +1.4 to +3.5pp gains in Table 1 do not establish that the AttentionInfluence score, rather than the duplication protocol, is responsible for the improvement. Please add a training run that upsamples a randomly chosen 73.1B subset of the corpus in exactly the same way, and report the same benchmark table for that control.
  2. [§4.2, Eq. (5)] The claim that a larger relative loss increase when retrieval heads are masked measures 'reasoning intensity' is not tested against simpler alternative mechanisms. A high (L_ref - L_base)/L_base value could also flag out-of-distribution, high-surprisal, or repetitive text, since such texts typically have high loss and may be particularly sensitive to masking any important component. The paper provides no comparison with selection by base loss, perplexity, or random ranking, so the mechanism underlying the score is unidentified. Please add controls that select the same 73.1B budget by high base loss (or high perplexity) and by random ranking; if the high-loss control matches the reported gains, the reasoning-specific interpretation in Section 5.2 is not supported.
  3. [§5.2, Appendix D] The 'mirror effect' is presented as post-hoc evidence, but it is a selected correlation rather than a predictive test. The paper identifies tasks where masking retrieval heads degrades the 1.3B selector and then observes gains on those same tasks in the 7B model, but no a-priori list of tasks is defined and no rank correlation across all evaluated benchmarks is reported. Moreover, because the mirror effect is computed on the same benchmarks used to select the discussion points, it cannot rule out the possibility that the gains come from upsampling generally useful documents. To make this evidence load-bearing, specify the full benchmark list in advance, report the correlation between masking-induced degradation and downstream improvement across all benchmarks, and show that a random-up sampling control does not exhibit a similar pattern.
minor comments (4)
  1. [§5.2, Figure 1] The caption and text refer to the 7B model trained on selected data as 'AttentionInfluence-1.3B', which is confusing because the selector is 1.3B but the trained model is 7B; please use distinct names such as 'AI-1.3B-selected' for the trained model.
  2. [Appendix C] The masking operation is described only as setting attention weights to equal weights 1/L; please clarify whether the mask is applied to the attention logits before softmax or to the normalized attention probabilities, since this affects the resulting reference model and the score in Eq. (5).
  3. [Appendix E] The pretraining configuration says 'the amount of training tokens is 1 TB', which conflates bytes and tokens; this should read 1T tokens, and the same correction applies to Table 7 entries such as '1.5TB' and '9TB' for the pretrained 1.3B and 7B models.
  4. [§5.2, Figure 9] There is a typo in the figure caption: 'TirvialQA' should be 'TriviaQA', and the heading 'Increasing Parameter Size of AttentionInfluce' in Section 5.2 should be 'AttentionInfluence'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: AttentionInfluence is computed from a frozen model and a synthetic retrieval task; benchmark gains are out-of-sample empirical outcomes.

full rationale

The derivation chain is self-contained relative to the downstream benchmarks it claims to improve. Retrieval heads are identified on a synthetic key-value retrieval task built from random hash keys and web-document sentences (Appendix A), not from MMLU, GSM8K, HumanEval, or any other evaluation benchmark; the top-5% head threshold is taken from external prior work (Wu et al., 2024) with no author overlap. The AttentionInfluence score in Eq. (5) is a relative cross-entropy difference computed on a frozen 1.3B pretrained model, so no target-benchmark value is fitted into the scoring formula. The selected 73.1B subset is then mixed with the full SmolLM corpus to train a 7B model, and the reported 1.4-3.5pp gains are measured on held-out benchmarks. The mirror-effect discussion is post hoc rather than circular: it reports a correlation between masking-induced drops on the 1.3B selector and training-induced gains on the 7B model, but it does not feed benchmark results back into the data-selection mechanism. The absence of a random-upsampling control is a substantive validity/interpretation concern about whether gains come from reasoning-specific selection or token repetition, but it is not a circularity: the selection score would still be computed independently of the target benchmarks even if repetition, rather than reasoning intensity, drove the improvement. No load-bearing self-citation chain is present, and the prior mechanistic-interpretability citations are external evidence rather than restatements of this paper's conclusions.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method introduces no fitted constants from target benchmarks; the main degrees of freedom are the head threshold, selection budget, synthetic probe design, and masking scheme. The core hypothesis that loss gap equals reasoning intensity is an axiom, not derived. No new physical or ontological entities are postulated.

free parameters (4)
  • Selection budget (top 20%) = 20% of ranked SmolLM corpus (~73.1B tokens)
    Determines which documents enter the training mix; no sweep is reported, and the resulting 73B sample is the treatment whose effect is measured.
  • Retrieval head threshold = top 5% of heads by retrieval score
    Borrowed from Wu et al. [47]; controls which heads are masked to form the weak reference model.
  • Synthetic proxy dataset size and length = 800 samples; up to 4,096 tokens each
    Chosen to fit context limits and evaluation cost; no sensitivity analysis is reported, and the head ranking depends on this probe.
  • Masking scheme = uniform attention weights (1/L)
    Implementation choice for degrading the model; alternative masking schemes such as zeroing are not compared.
assumptions (5)
  • standard math Attention scores and loss computations follow standard transformer definitions, and the masking operation sets attention weights to uniform 1/L.
    Foundational definitions used in Eqs. 1-5; uncontroversial but required for the method.
  • domain assumption Retrieval heads are causally important for reasoning and retrieval, so masking them degrades reasoning ability.
    The paper relies on Wu et al. [47] and its own Table 6; this motivates why the loss gap is meaningful.
  • ad hoc to paper A larger relative loss increase when retrieval heads are masked identifies documents with higher reasoning intensity.
    Core hypothesis stated in Section 4.2; not derived or separately validated, and could confound out-of-distribution text with reasoning content.
  • domain assumption Attention-head sensitivity measured in a 1.3B model transfers to improving a 7B model's pretraining.
    The weak-to-strong claim rests on this transfer, asserted in Sections 1 and 5; no theoretical or multi-size evidence beyond one pair.
  • ad hoc to paper Loss values are comparable only within the same domain, so AttentionInfluence scores are ranked within domain.
    Introduced to avoid cross-domain loss distribution differences; this restriction changes what the top 20% means globally.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection." pith.science (2026). https://pith.science/paper/F62P7YFE

@misc{pith2026250507293,
  author       = {Pith},
  title        = {Pith review of: AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F62P7YFE}},
  note         = {Machine review of arXiv:2505.07293}
}
read the original abstract

Recently, there has been growing interest in collecting reasoning-intensive pretraining data to improve LLMs' complex reasoning ability. Prior approaches typically rely on supervised classifiers to identify such data, which requires labeling by humans or LLMs, often introducing domain-specific biases. Due to the attention heads being crucial to in-context reasoning, we propose AttentionInfluence, a simple yet effective, training-free method without supervision signal. Our approach enables a small pretrained language model to act as a strong data selector through a simple attention head masking operation. Specifically, we identify retrieval heads and compute the loss difference when masking these heads. We apply AttentionInfluence to a 1.3B-parameter dense model to conduct data selection on the SmolLM corpus of 241B tokens, and mix the SmolLM corpus with the selected subset comprising 73B tokens to pretrain a 7B-parameter dense model using 1T training tokens and WSD learning rate scheduling. Our experimental results demonstrate substantial improvements, ranging from 1.4pp to 3.5pp, across several knowledge-intensive and reasoning-heavy benchmarks (i.e., MMLU, MMLU-Pro, AGIEval-en, GSM8K, and HumanEval). This demonstrates an effective weak-to-strong scaling property, with small models improving the final performance of larger models-offering a promising and scalable path for reasoning-centric data selection.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 8 canonical work pages

  1. [1]

    Smollm2: When smol goes big–data-centric training of a small language model.arXiv preprint arXiv:2502.02737, 2025

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. Smollm2: When smol goes big–data-centric training of a small language model.arXiv preprint arXiv:2502.02737, 2025

    arXiv 2025

  2. [2]

    Per- plexed by perplexity: Perplexity-based data pruning with small reference models.arXiv preprintarXiv:2405.20541, 2024

    Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L Leavitt, and Mansheej Paul. Per- plexed by perplexity: Perplexity-based data pruning with small reference models.arXiv preprintarXiv:2405.20541, 2024

    arXiv 2024

  3. [3]

    Smollm-corpus,

    Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Smollm-corpus,

  4. [4]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  5. [5]

    Towards monoseman- ticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, and et al. Towards monoseman- ticity: Decomposing language models with dictionary learning. https://transformer-circuits.pub/2023/ monosemantic-features/index.html, 2023. Accessed: 2023-10-04

    work page 2023

  6. [6]

    Efficient intent detection with dual sentence encoders.arXiv preprint arXiv:2003.04807, 2020

    Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders.arXiv preprint arXiv:2003.04807, 2020

    arXiv 2003

  7. [7]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  8. [8]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

Show all 58 references
  1. [9]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  2. [10]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs.arXiv preprint arXiv:1903.00161, 2019

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs.arXiv preprint arXiv:1903.00161, 2019

    1903 arXiv

  3. [11]

    Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning.arXiv preprint arXiv:2410.19258, 2024

    Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning.arXiv preprint arXiv:2410.19258, 2024

    2024

  4. [12]

    Transformer feed-forward layers are key-value memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020

    2012 arXiv

  5. [13]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  6. [14]

    Optimizing pretraining data mixtures with llm-estimated utility.arXiv preprint arXiv:2501.11747, 2025

    William Held, Bhargavi Paranjape, Punit Singh Koura, Mike Lewis, Frank Zhang, and Todor Mihaylov. Optimizing pretraining data mixtures with llm-estimated utility.arXiv preprint arXiv:2501.11747, 2025

    2025 arXiv

  7. [15]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  8. [16]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  9. [17]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015. 12

    2015 arXiv

  10. [18]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024

    2024 arXiv

  11. [19]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36:6...

    2023

  12. [20]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.arXiv preprint arXiv:1705.03551, 2017

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.arXiv preprint arXiv:1705.03551, 2017

    2017 arXiv

  13. [21]

    Fasttext

    Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext. zip: Compressing text classification models.arXiv preprint arXiv:1612.03651, 2016

    2016 arXiv

  14. [22]

    The mirrored influence hypothesis: Efficient data influence estimation by harnessing forward passes

    Myeongseob Ko, Feiyang Kang, Weiyan Shi, Ming Jin, Zhou Yu, and Ruoxi Jia. The mirrored influence hypothesis: Efficient data influence estimation by harnessing forward passes. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26286–26295, 2024

    2024

  15. [23]

    Race: Large-scale reading comprehension dataset from examinations, 2017

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations, 2017. URLhttps://arxiv.org/abs/1704.04683

    2017 arXiv

  16. [24]

    Datacomp-lm: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processin...

    2024

  17. [25]

    Scalingfilter: Assessing data quality through inverse utilization of scaling laws.arXiv preprint arXiv:2408.08310, 2024

    Ruihang Li, Yixuan Wei, Miaosen Zhang, Nenghai Yu, Han Hu, and Houwen Peng. Scalingfilter: Assessing data quality through inverse utilization of scaling laws.arXiv preprint arXiv:2408.08310, 2024

    2024 arXiv

  18. [26]

    Rho-1: Not all tokens are what you need.arXiv preprint arXiv:2404.07965, 2024

    Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. Rho-1: Not all tokens are what you need.arXiv preprint arXiv:2404.07965, 2024

    2024 arXiv

  19. [27]

    Regmix: Data mixture as regression for language model pre-training.arXiv preprint arXiv:2407.01492, 2024

    Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training.arXiv preprint arXiv:2407.01492, 2024

    2024 arXiv

  20. [28]

    Interpreting key mechanisms of factual recall in transformer-based language models.arXiv preprint arXiv:2403.19521, 2024

    Ang Lv, Yuhan Chen, Kaiyi Zhang, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, and Rui Yan. Interpreting key mechanisms of factual recall in transformer-based language models.arXiv preprint arXiv:2403.19521, 2024

    2024 arXiv

  21. [29]

    Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  22. [30]

    2 olmo 2 furious.arXiv preprint arXiv:2501.00656, 2024

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious.arXiv preprint arXiv:2501.00656, 2024

    2024 arXiv

  23. [31]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  24. [32]

    The fineweb datasets: Decanting the web for the finest text data at scale.Advancesin Neural Information Processing Systems, 37:30811–30849, 2024

    Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale.Advancesin Neural Information Processing Systems, 37:30811–30849, 2024

    2024

  25. [33]

    Dataman: Data manager for pre-training large language models.arXiv preprint arXiv:2502.19363, 2025

    Ru Peng, Kexin Yang, Yawen Zeng, Junyang Lin, Dayiheng Liu, and Junbo Zhao. Dataman: Data manager for pre-training large language models.arXiv preprint arXiv:2502.19363, 2025

    2025 arXiv

  26. [34]

    Clongeval: A chinese benchmark for evaluating long-context large language models.arXiv preprint arXiv:2403.03514, 2024

    Zexuan Qiu, Jingjing Li, Shijue Huang, Xiaoqi Jiao, Wanjun Zhong, and Irwin King. Clongeval: A chinese benchmark for evaluating long-context large language models.arXiv preprint arXiv:2403.03514, 2024

    2024 arXiv

  27. [35]

    Scaling language models: Methods, analysis & insights from training gopher.arXiv preprint arXiv:2112.11446, 2021

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher.arXiv preprint arXiv:2112.11446, 2021

    2021 arXiv

  28. [36]

    Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  29. [37]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URLhttps: //arxiv.org/abs/2311.12022. 13

    2023 arXiv

  30. [38]

    Procedural knowledge in pretraining drives reasoning in large language models.arXiv preprint arXiv:2411.12580, 2024

    Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, and Max Bartolo. Procedural knowledge in pretraining drives reasoning in large language models.arXiv preprint arXiv:2411.12580, 2024

    2024 arXiv

  31. [39]

    Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    2021

  32. [40]

    Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset

    Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset. arXiv preprint arXiv:2412.02595, 2024

    2024 arXiv

  33. [41]

    Challenging big-bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261, 2022

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261, 2022

    arXiv

  34. [42]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge.arXiv preprint arXiv:1811.00937, 2018

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge.arXiv preprint arXiv:1811.00937, 2018

    2018 arXiv

  35. [43]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  36. [44]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. InThe Thirty-eightConference on Neural Information Proc...

    2024

  37. [45]

    Qurating: Selecting high-quality data for training language models.arXiv preprint arXiv:2402.09739, 2024

    Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models.arXiv preprint arXiv:2402.09739, 2024

    2024 arXiv

  38. [46]

    Organize the web: Constructing domains enhances pre-training data curation.arXiv preprint arXiv:2502.10341, 2025

    Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation.arXiv preprint arXiv:2502.10341, 2025

    2025 arXiv

  39. [47]

    Retrieval head mechanistically explains long-context factuality.arXiv preprint arXiv:2404.15574, 2024

    Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality.arXiv preprint arXiv:2404.15574, 2024

    2024 arXiv

  40. [48]

    Doremi: Optimizing data mixtures speeds up language model pretraining.Advances in Neural Information Processing Systems, 36:69798–69818, 2023

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining.Advances in Neural Information Processing Systems, 36:69798–69818, 2023

    2023

  41. [49]

    Data mixing laws: Optimizing data mixtures by predicting language modeling performance.arXiv preprint arXiv:2403.16952, 2024

    Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance.arXiv preprint arXiv:2403.16952, 2024

    2024 arXiv

  42. [50]

    Mates: Model-aware data selection for efficient pretraining with data influence models.Advances in Neural Information Processing Systems, 37:108735–108759, 2024

    Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models.Advances in Neural Information Processing Systems, 37:108735–108759, 2024

    2024

  43. [51]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  44. [52]

    Decoratelm: Data engineering through corpus rating, tagging, and editing with language models

    Ranchi Zhao, Zhen Leng Thai, Yifan Zhang, Shengding Hu, Yunqi Ba, Jie Zhou, Jie Cai, Zhiyuan Liu, and Maosong Sun. Decoratelm: Data engineering through corpus rating, tagging, and editing with language models. arXiv preprint arXiv:2410.05639, 2024

    arXiv

  45. [53]

    Attention heads of large language models: A survey.arXiv preprint arXiv:2409.03752, 2024

    Zifan Zheng, Yezhaohui Wang, Yuxin Huang, Shichao Song, Mingchuan Yang, Bo Tang, Feiyu Xiong, and Zhiyu Li. Attention heads of large language models: A survey.arXiv preprint arXiv:2409.03752, 2024

    2024 arXiv

  46. [54]

    Agieval: A human-centric benchmark for evaluating foundation models

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023

    2023 arXiv

  47. [55]

    Masked, Retrieval Heads

    Youxiang Zhu, Ruochen Li, Danqing Wang, Daniel Haehn, and Xiaohui Liang. Focus directions make your language models pay more attention to relevant contexts.arXiv preprint arXiv:2503.23306, 2025. 14 Appendix A Synthetic Test Sample model input: Please extract the value correspo...

    2025 arXiv

  48. [57]

    "This is a test string

    Respond in the following format: \#\#Educational Value Score <educational value score> \#\#Reasoning Intensive Score <reasoning intensive score> Although GPT-4o can also be used for scoring pretraining data, different domains require specially de- signed prompts. Moreover, the...

    2017

  49. [58]

    26 Figure 18 The cloud maps of the data selected by AttentionInfluence and FineWeb-Edu Classifier, respectively

    + 0 = 0\n\n\nExample 2:\n\nInput: nums1 = [0], nums2 = [0], nums3 = [0], nums4 = [0]\nOutput: 1\n\n\nConstraints:\n\nn == nums1.length\nn == nums2.length\nn == nums3.length\nn == nums4.length\n1 <= n <= 200\n- 228 <= nums1[i], nums2[i], nums3[i], nums4[i] <= 228\n"""\ndef fill...

  50. [2024]

    URL https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.