Pith. sign in

REVIEW 4 major objections 4 minor 31 references

From Data to Knowledge: Evaluating How Efficiently Language Models Learn Facts

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a language model's factual sample efficiency can be measured by fitting a power-law curve to its accuracy as a function of how often a fact appears in pre-training data, with the fitted slope $\alpha_m$ serving as a…

desk verdict Useful framework, honest limitations, but the headline ranking rests on an unvalidated frequency heuristic and no uncertainty quantification. read the letter →

arxiv 2506.16912 v1 pith:SQLEP2X6 submitted 2025-06-20 cs.CL cs.LG

Daniel Christoph , Max Ploner , Patrick Haller , Alan Akbik This is my paper · ORCID
classification cs.CLcs.LG
keywords sampleefficiencylanguagemodelsfactualknowledgefactfrequencypower-lawscalingLLaMAprobinglong-taildistribution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to define and measure sample efficiency for language models: how much exposure to a fact in the training corpus a model needs before it can answer questions about that fact. The authors count how often each relational fact appears in a Wikipedia pre-training corpus, train eight models of three architectures at two sizes on the same data, and probe them with multiple-choice factual questions. They fit a power-law curve to each model's accuracy as a function of fact frequency, and the fitted slope $\alpha_m$ becomes a single number summarizing that model's sample efficiency. They report that larger models, and LLaMA models in particular, have higher $\alpha_m$ values, meaning they gain more accuracy per additional exposure, especially on rare facts. The contribution is a concrete, reproducible way to compare architectures on learning efficiency rather than final accuracy alone.

What carries the argument

The central object is the power scaling function $F(x) = 1 - \frac{L_0 + x_0}{(1+x)^{\alpha_m}}$, fitted by maximum likelihood over all BEAR probe items and models, with $\alpha_m$ as the model-specific slope that the paper treats as the sample-efficiency score. It is paired with a weighted accuracy score over frequency buckets, $W_{ASB}$, which gives a more direct accuracy view, and both metrics rest on frequency counts produced by a co-occurrence heuristic that counts a fact whenever its subject and object appear in the same sentence. The function does the load-bearing work: it replaces a binary 'learned or not' threshold with a continuous curve, so sample efficiency becomes how quickly accuracy grows with exposure and can be compared across models with a single number.

What would settle it

Take a random sample of BEAR facts, recompute their frequencies with an entity-linking and relation-extraction pipeline (or manual annotation), and re-fit $\alpha_m$; if the relative frequencies change enough to reorder the $\alpha_m$ values across the eight models, the claim that $\alpha_m$ measures sample efficiency is not supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that a language model's factual sample efficiency can be quantified by fitting the function $F(x) = 1 - \frac{L_0 + x_0}{(1+x)^{\alpha_m}}$, where $x$ is the number of times a subject–relation–object fact co-occurs in a sentence in the pre-training corpus and $\alpha_m$ is a per-model parameter. A higher $\alpha_m$ means the probability of answering a factual question correctly rises faster per additional observed sample, so $\alpha_m$ ranks models by sample efficiency. Trained on identical five-billion-token Wikipedia data, the LLaMA models achieved the highest $\alpha$ values (0.103 for the small model, 0.120 for the medium), followed by Mamba 2, xLSTM, and GPT2, and in every architecture the medium model beat the small one. The paper also finds that models differ most on low-frequency facts (fewer than about 1024 occurrences), while high-frequency facts are learned to similar accuracy by all models, and that $\alpha$ values are robust across test splits with different frequency profiles.

Load-bearing premise

The whole analysis rests on the assumption that counting sentences where the subject and object merely appear together gives accurate relative frequencies of how often a fact is actually expressed in the training text.

Editorial extensions

If this is right

  • If $\alpha_m$ is a valid sample-efficiency measure, model comparisons no longer need matched final accuracies; one can rank architectures by how fast they convert exposures into knowledge.
  • The finding that size helps mainly on low-frequency facts implies that training larger models is a direct route to better long-tail knowledge, while smaller models can match them on common facts.
  • The reported robustness of $\alpha$ across test splits with different frequency profiles means a non-representative probe set could still yield a reliable efficiency ranking.
  • Because $\alpha$ values stabilize early in training, the metric measured at an intermediate checkpoint could predict final sample efficiency without completing full training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The co-occurrence heuristic may inflate frequencies when two entities appear in a sentence without actually expressing the target relation; recomputing counts with entity linking or manual annotation could reorder the $\alpha_m$ rankings across the eight models.
  • The fitted $\alpha_m$ could be connected to neural scaling laws: if it scales predictably with model size or compute, it could become a planning tool for estimating how much data a given architecture needs to reach a target fact-recall level.
  • The method could be applied to other corpora or languages to test whether the sample-efficiency rankings persist across data distributions, since the paper only uses one English Wikipedia dump.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces a framework for measuring how efficiently language models learn relational facts as a function of fact frequency in the pre-training corpus. The authors estimate fact frequencies by a co-occurrence heuristic applied to a Wikipedia dump, pre-train eight small-to-medium models (GPT-2, LLaMA, xLSTM, Mamba-2) on the same five-billion-token corpus, and propose two sample-efficiency metrics: a weighted accuracy over frequency buckets (W_ASB) and a fitted power-law curve F(x) = 1 - (L0 + x0)/(1+x)^alpha_m whose per-model slope alpha_m is interpreted as sample efficiency. The empirical results indicate that larger models are more sample-efficient, that LLaMA models achieve the highest scores, and that most models differ mainly on low-frequency facts. The paper also tracks metrics across training checkpoints and reports robustness experiments on frequency-split test sets.

Significance. If the proposed framework is valid, it would provide a useful, controlled methodology for comparing sample efficiency across architectures and sizes, and the released fact-frequency counts would be a reusable resource. The study has real strengths: all models are trained on the same corpus with the same tokenizer and comparable parameter counts, intermediate checkpoints are probed, and code and frequency statistics are released. The central qualitative trend, that accuracy increases with fact frequency and that model differences concentrate on low-frequency facts, is visible in the raw bucket accuracies and is not dependent on the parametric fit alone. However, the paper's headline ranking of architectures rests on the validity of the unvalidated co-occurrence frequency estimates and on the fitted alpha_m values, which are reported without uncertainty quantification. The significance of the contribution is therefore conditional on addressing these load-bearing methodological gaps.

major comments (4)
  1. [Section 3.1] The fact-frequency estimates that form the x-axis of the entire analysis are produced by a co-occurrence heuristic that counts a fact whenever the subject and object strings appear in the same sentence, with no verification that the relation is expressed and no entity disambiguation. The authors state that this heuristic is 'sufficiently accurate to predict the relative frequencies' but provide no precision/recall evaluation, no comparison against entity-linked counts, and no manual validation sample. The Limitations section itself concedes 'possible mismappings of entities due to likely ambiguity or relation misidentification.' Because every alpha_m and W_ASB value is computed from these frequency counts, systematic false positives (for example, alias co-occurrence in unrelated sentences) or false negatives can reorder or compress the true frequency axis and change the shape of the fitted curve. This is not a peripheral concern: the differences between non-LLaMA models in Table 1 are tiny (alpha_m = 0.084, 0.086, 0.087), so even modest x-axis noise could alter the reported ranking. I request a validation of the heuristic against a manually checked or entity-linked sample, and a sensitivity analysis showing that the main conclusions are stable under plausible frequency-count perturbations.
  2. [Section 3.3.2] The maximum-likelihood procedure fits L0, x0, and the model-specific alpha_m jointly on the same data that are then used to rank the models, and the paper reports no confidence intervals, bootstrap estimates, or repeated-seed variance for alpha_m. The conclusion that LLaMA is most sample-efficient and that larger models are more sample-efficient is essentially an ordering of fitted alpha_m values, yet the reported differences between several models are extremely small (for example, small GPT-2 at 0.084 versus small xLSTM at 0.086 versus small Mamba-2 at 0.087 in Table 1). Without uncertainty quantification or at least multiple training runs, it is impossible to determine whether these differences are statistically meaningful. I ask the authors to report profile-likelihood or bootstrap confidence intervals for alpha_m, and to state whether the ranking is stable across random seeds or replicated training runs.
  3. [Section 3.3.1] The weighting function in the W_ASB metric depends on an arbitrarily chosen decay parameter lambda = 0.05, and the paper does not report how the metric or the resulting model ranking changes as lambda varies. Since W_ASB is presented as one of the two proposed sample-efficiency metrics and is used to support the architecture-level conclusions, the choice of lambda should be justified either by a sensitivity analysis over a range of lambda values or by an explicit criterion for setting it. The current text states the value, but does not show that the ranking of models is robust to this choice.
  4. [Section 4.3] The metric-robustness experiments vary the frequency composition of the test split, but they do not test robustness to errors in the frequency estimates themselves, which is the more load-bearing uncertainty given the unvalidated heuristic in Section 3.1. In addition, the claim that alpha is robust across splits assumes that the sampled facts are 'on average equally hard,' an assumption that is stated but not tested. I recommend either directly validating the frequency estimates or adding a perturbation analysis that adds noise to the fact counts and measures the resulting change in alpha_m and W_ASB rankings.
minor comments (4)
  1. [Section 3.1] There is a typo in the first sentence: 'wether' should be 'whether.' Also, 'repostory' in the Contributions section should be 'repository.'
  2. [Figure 6] Figures 6(a) and 6(b) appear to display the same correlation matrix in the manuscript text; please check whether the two panels are intended to be different, and if so, label them clearly.
  3. [Throughout] The model name 'LL AMA' is written with a space throughout the paper; this is likely a formatting artifact and should be corrected to 'LLaMA.'
  4. [Section 4.1.2] The sentence reporting L0 = 0.00 and x0 = 0.88 as optimized values would benefit from a brief description of the uncertainty in these estimates, especially since the interpretation of x0 as reflecting the heuristic's underestimation is central to the discussion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper fits α_m to probing data and interprets the fitted ordering as sample efficiency, which is an operationalization rather than a prediction derived from the conclusion; the unvalidated frequency heuristic is a validity concern, not a circular step.

full rationale

The paper's central derivation is self-contained. The sample-efficiency metric α_m is a maximum-likelihood parameter fit to the BEAR probing outputs as a function of co-occurrence-based frequency buckets (Section 3.3.2), and the conclusion that LLaMA models are most sample-efficient is an empirical ordering of the fitted α_m values together with raw accuracy and W_ASB bucket scores (Table 1, Figure 16). No held-out quantity is renamed as a prediction: α_m is explicitly reported as a fitted parameter, and the robustness analysis re-fits α_m on low- and high-frequency splits rather than claiming an independent prediction. The statement that higher α_m means higher sample efficiency is a stated operationalization, not a derivation whose conclusion was presupposed. The BEAR probe is prior work by overlapping authors, but it is used as a fixed external evaluation instrument and is not fitted, redefined, or justified solely by self-citation in this paper. The paper's acknowledged limitation that the co-occurrence heuristic may produce noisy frequency counts is a measurement-validity risk that could affect reliability of the x-axis, but it does not make any equation or fitted parameter equal to the conclusion by construction. Therefore no circular step is exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central analysis depends on two unvalidated modeling choices: the co-occurrence frequency heuristic and the assumed power-law form of F(x). The free parameters lambda, L0, x0, and alpha are all fitted or hand-set, and no independent benchmark confirms that alpha measures true sample efficiency.

free parameters (4)
  • lambda (W_ASB decay) = 0.05
    Hand-chosen exponential decay used to weight rare facts more heavily; changes to lambda alter the weighted accuracy rankings.
  • L0 (irreducible error floor) = 0.00 for BEAR, 0.0 for BEAR-big
    Fitted globally via maximum likelihood in Section 3.3.2; interpreted as probe error rate but not independently measured.
  • x0 (frequency offset) = 0.88 for BEAR, 0.92 for BEAR-big
    Fitted globally via maximum likelihood; absorbs errors in the frequency heuristic and educated guessing, but not separately validated.
  • alpha_m (per-model slope) = GPT2 0.084/0.098; LLaMA 0.103/0.120; xLSTM 0.086/0.100; Mamba2 0.087/0.106
    Model-specific slope fitted by maximum likelihood; the paper's sample-efficiency ranking is essentially an ordering of these fitted values.
assumptions (3)
  • domain assumption Co-occurrence of two entities in a sentence implies their relational fact is present in that sentence.
    Section 3.1 states 'If so, we assume the relational fact is represented within the sentence.' This is the foundation of all frequency counts.
  • ad hoc to paper The probability of correct recall follows F(x) = 1 - (L0 + x0)/(1+x)^alpha.
    Section 3.3.2 proposes this power-law form without derivation from learning theory; alpha is only a sample-efficiency measure if this functional form is adequate.
  • domain assumption BEAR multiple-choice statements measure factual recall equivalently across causal and masked LMs.
    Taken from Wiland et al. 2024; the paper inherits this as a background assumption for all accuracy measurements.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Data to Knowledge: Evaluating How Efficiently Language Models Learn Facts." pith.science (2026). https://pith.science/paper/SQLEP2X6

@misc{pith2026250616912,
  author       = {Pith},
  title        = {Pith review of: From Data to Knowledge: Evaluating How Efficiently Language Models Learn Facts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SQLEP2X6}},
  note         = {Machine review of arXiv:2506.16912}
}
read the original abstract

Sample efficiency is a crucial property of language models with practical implications for training efficiency. In real-world text, information follows a long-tailed distribution. Yet, we expect models to learn and recall frequent and infrequent facts. Sample-efficient models are better equipped to handle this challenge of learning and retaining rare information without requiring excessive exposure. This study analyzes multiple models of varying architectures and sizes, all trained on the same pre-training data. By annotating relational facts with their frequencies in the training corpus, we examine how model performance varies with fact frequency. Our findings show that most models perform similarly on high-frequency facts but differ notably on low-frequency facts. This analysis provides new insights into the relationship between model architecture, size, and factual learning efficiency.

Figures

Figures reproduced from arXiv: 2506.16912 by the authors.

Figure 1
Figure 1. Sample efficiency evaluation of LMs. 2023). Models that rely predominantly on fre￾quent facts while struggling with rarer ones—an issue caused by the long-tailed distribution of in￾formation in natural text (Zhang et al., 2024)—are considered sample-inefficient. Conversely, sample￾efficient models should achieve higher accuracy on rare facts while maintaining strong performance on more common ones. To assess a model… view at source ↗
Figure 2
Figure 2. In BEAR, one statement per answer option [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Example fact frequency table constructed from a text corpus. A fact is counted if the subject and the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Number of matches for BEAR facts in the English Wikipedia dump (20231101.en; Wikimedia Foundation, 2023). the correct answer at a specific moment in train￾ing but may later give the incorrect answer after it has processed more data, leading to a different outcome. Ther…
Figure 5
Figure 5. Figure 5: Accuracy on frequency buckets during train [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Correlation matrix of the final scores and over [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Accuracy on frequency buckets after training [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Number of matches for BEAR-big facts in the [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 10
Figure 10. Figure 10: Development of the weighted accuracy (WASB) throughout the pre-training. [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Development of αm over the course of the pre-training. 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 Slices 0.15 0.20 0.25 0.30 0.35 Accuracy Score 0.2808 0.3049 0.2865 0.3213 0.2812 0.3076 0.3108 0…
Figure 12
Figure 12. Figure 12: Development of the accuracy throughout the pre-training. [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Development of the weighted accuracy (WASB) throughout the pre-training as measured on BEAR-big. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Development of αm throughout the pre-training as measured on BEAR-big [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Development of the accuracy throughout the pre-training as measured on BEAR-big. [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: Frequency Bucket Accuracy of the model’s final state as measured on BEAR. [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]
Figure 17
Figure 17. Figure 17: Accuracy, WASB and α scores on the low and high frequency splits and entire data set for comparison on BEAR [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 6 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. 2024. https://arxiv.org/abs/2405.04517 xlstm: Extended long short-term memory . Preprint, arXiv:2405.04517

    arXiv 2024

  4. [4]

    Wang, Zeb Kurth-Nelson , Charles Blundell, and Demis Hassabis

    Matthew Botvinick, Sam Ritter, Jane X. Wang, Zeb Kurth-Nelson , Charles Blundell, and Demis Hassabis. 2019. https://doi.org/10.1016/j.tics.2019.02.006 Reinforcement Learning , Fast and Slow . Trends in Cognitive Sciences, 23(5):408--422

    doi:10.1016/j.tics.2019.02.006 2019

  5. [5]

    Leshem Choshen, Yang Zhang, and Jacob Andreas. 2024. https://arxiv.org/abs/2410.11840 A Hitchhiker 's Guide to Scaling Law Estimation . Preprint, arXiv:2410.11840

    arXiv 2024

  6. [6]

    Tri Dao and Albert Gu. 2024. Transformers are SSM s: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML)

    2024

  7. [7]

    Kehua Feng, Keyan Ding, Kede Ma, Zhihua Wang, Qiang Zhang, and Huajun Chen. 2024. https://arxiv.org/abs/2404.08008 Sample-efficient human evaluation of large language models via maximum discrepancy competition . Preprint, arXiv:2404.08008

    arXiv 2024

  8. [8]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2024. https://doi.org/10.5281/zenodo.12608602 A framework...

    doi:10.5281/zenodo.12608602 2024

Show all 31 references
  1. [9]

    Nathan Godey, \'E ric de la Clergerie , and Beno \^i t Sagot. 2024. https://arxiv.org/abs/2402.19406 On the Scaling Laws of Geographical Representation in Language Models . Preprint, arXiv:2402.19406

    2024 arXiv

  2. [10]

    Patrick Haller, Jonas Golde, and Alan Akbik. 2024. https://aclanthology.org/2024.conll-babylm.7/ BabyHGRN : Exploring RNNs for Sample-Efficient Language Modeling . In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning , pages 82--94, Mia...

    2024

  3. [11]

    Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021. https://arxiv.org/abs/2102.01293 Scaling Laws for Transfer . Preprint, arXiv:2102.01293

    2021 arXiv

  4. [12]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 doi arXiv

  5. [13]

    Matthew Honnibal and Ines Montani. 2017. spaCy 2 : Natural language understanding with B loom embeddings, convolutional neural networks and incremental parsing. To appear

    2017

  6. [14]

    Jan-Christoph Kalo and Leandra Fichtel. 2022. https://www.akbc.ws/2022/assets/pdfs/15_kamel_knowledge_analysis_with_.pdf KAMEL : Knowledge Analysis with Multitoken Entities in Language Models . In Automated Knowledge Base Construction

    2022

  7. [15]

    Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. https://doi.org/10.48550/arXiv.2211.08411 Large Language Models Struggle to Learn Long-Tail Knowledge . Preprint, arXiv:2211.08411

    2023 doi arXiv

  8. [16]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://doi.org/10.48550/arXiv.2001.08361 Scaling Laws for Neural Language Models . Preprint, arXiv:2001.08361

    2020 doi arXiv

  9. [17]

    Jianghao Lin, Xinyi Dai, Rong Shan, Bo Chen, Ruiming Tang, Yong Yu, and Weinan Zhang. 2024. https://arxiv.org/abs/2406.02368 Large language models make sample-efficient recommender systems . Preprint, arXiv:2406.02368

    2024 arXiv

  10. [18]

    Liu, Ananya Kumar, Percy Liang, and Robin Jia

    Nelson F. Liu, Ananya Kumar, Percy Liang, and Robin Jia. 2023. https://arxiv.org/abs/2210.06456 Are sample-efficient nlp models more robust? Preprint, arXiv:2210.06456

    2023 arXiv

  11. [19]

    Xingyu Lu, Xiaonan Li, Qinyuan Cheng, Kai Ding, Xuanjing Huang, and Xipeng Qiu. 2024. https://arxiv.org/abs/2406.15720 Scaling Laws for Fact Memorization of Large Language Models . Preprint, arXiv:2406.15720

    2024 arXiv

  12. [20]

    Vincent Micheli, Eloi Alonso, and François Fleuret. 2023. https://arxiv.org/abs/2209.00588 Transformers are sample-efficient world models . Preprint, arXiv:2209.00588

    2023 arXiv

  13. [21]

    Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. https://aclanthology.org/P09-1113 Distant supervision for relation extraction without labeled data . In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Con...

    2009

  14. [22]

    Fabio Petroni, Tim Rockt \"a schel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. https://doi.org/10.18653/v1/D19-1250 Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language P...

    2019 doi

  15. [23]

    Max Ploner, Jacek Wiland, Sebastian Pohl, and Alan Akbik. 2024. https://doi.org/10.48550/arXiv.2408.15729 LM-PUB-QUIZ : A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models . Preprint, arXiv:2408.15729

    2024 doi arXiv

  16. [24]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

    2019

  17. [25]

    Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.437 How Much Knowledge Can You Pack Into the Parameters of a Language Model ? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , p...

    2020 doi

  18. [26]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama:...

    2023 arXiv

  19. [27]

    Wikimedia Foundation . 2023. https://dumps.wikimedia.org Dump of English Wikipedia of November 1st, 2023

    2023

  20. [28]

    Jacek Wiland, Max Ploner, and Alan Akbik. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.155 BEAR : A unified framework for evaluating relational knowledge in causal and masked language models . In Findings of the Association for Computational Linguistics: NAACL 2024, p...

    2024 doi

  21. [29]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen , Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and...

    2020 doi

  22. [30]

    Yang Yu. 2018. https://doi.org/10.24963/ijcai.2018/820 Towards sample efficient reinforcement learning . In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18 , pages 5739--5743. International Joint Conferences on Artificial I...

    2018 doi

  23. [31]

    Chongsheng Zhang, George Almpanidis, Gaojuan Fan, Binquan Deng, Yanbo Zhang, Ji Liu, Aouaidjia Kamel, Paolo Soda, and João Gama. 2024. https://arxiv.org/abs/2408.00483 A systematic review on long-tailed learning . Preprint, arXiv:2408.00483

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.