Pith. sign in

REVIEW 4 major objections 4 minor 33 references

Studying the Role of Input-Neighbor Overlap in Retrieval-Augmented Language Models Training Efficiency

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Deliberately increasing input–neighbor token overlap accelerates RETRO model training by about 40%.

desk verdict The threshold sweep is a solid, useful measurement, but the headline 40% speedup compares steps to different perplexity minima and the abstract's 'without compromising performance' is contradicted by the paper's own numbers. read the letter →

arxiv 2505.14309 v1 pith:NVBIZRJ3 submitted 2025-05-20 cs.CL

Ehsan Doostmohammadi , Marco Kuhlmann This is my paper · ORCID
classification cs.CL
keywords retrieval-augmentedlanguagemodelsRETROinput-neighboroverlaptrainingefficiencydataparaphrasingperplexityquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that the amount of token overlap between an input chunk and its retrieved neighbors is a controllable lever on how fast a retrieval-augmented language model learns to use retrieval. Training separate RETRO-fitted models under capped overlap levels, the authors find a critical threshold: below roughly 32 shared tokens the retrieved context is effectively ignored, and above it higher overlap speeds convergence and improves test-time perplexity. They then show that deliberately replacing one neighbor with a paraphrase of the input chunk raises overlap and cuts the data needed for activation by about 40%, with convergence around step 3,000 instead of step 5,000, while leaving question-answering accuracy essentially unchanged. If the claim holds, retrieval corpora and training schedules can be designed to manufacture overlap and make retrieval-augmented pretraining far more data-efficient, especially in low-resource settings.

What carries the argument

The central object is the input-neighbor overlap, defined as the number of tokens shared between a 64-token input chunk and its retrieved neighbors (including continuations), and the intervention is a paraphrased neighbor, a synthetic chunk generated from the input itself to guarantee high overlap. The RETRO architecture supplies the mechanism: at each new chunk, two retrieved neighbors and their continuations are fed through an encoder and cross-attention, and the paper shows that this machinery only starts contributing when overlap crosses roughly 32 tokens. The thresholded models and the paraphrase insertion are the instruments that expose this activation behavior and turn it into a training-efficiency result.

What would settle it

Train two RETRO-fitted models with the same average added overlap: one with LLaMA-3 paraphrases of the input chunk and one with a non-paraphrastic augmentation (back-translation or synonym substitution) matched to the same overlap. If the paraphrase model converges near step 3,000 while the other does not, the acceleration is not due to overlap per se but to synthetic-text properties; if both converge at the same step, overlap is confirmed as the causal driver.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that the driver of retrieval-augmented language model performance is surface-level overlap between the query chunk and its retrieved neighbors, and that this overlap acts as a switch rather than a smooth dial. Below a critical threshold—about 32 shared tokens for 64-token chunks—the RETRO model behaves like an unactivated model, ignoring the retrieved context; above that threshold, perplexity drops and further overlap monotonically accelerates learning. The practical consequence demonstrated is that a synthetic neighbor obtained by paraphrasing the input chunk with LLaMA 3 adds substantial overlap and makes previously unactivated models activate, moving convergence from roughly step 5,000 to step 3,000, a saving of about 40% of training data. The small cost is higher converged perplexity (6.1 vs 5.6), and the authors report that downstream exact match on Natural Questions is not degraded, validating perplexity as a predictor of task performance.

Load-bearing premise

The load-bearing premise is that the training-time speedup is caused by the increase in overlap itself, but the paraphrase intervention also changes the training distribution in other ways, so the paper does not isolate overlap from properties of synthetic text.

Editorial extensions

If this is right

  • Retrieval databases can be deliberately constructed or augmented so training chunks have high-overlap neighbors, enabling faster activation with less data.
  • A critical overlap threshold separates useless from useful retrieval; below it, added retrieval parameters do not engage, above it they do.
  • Synthetic paraphrase neighbors can substitute for naturally occurring high-overlap neighbors, making the speedup available when natural neighbors are scarce.
  • Faster convergence is paid for with a small perplexity penalty at convergence, so data efficiency and final quality trade off against each other.
  • Perplexity trends predict downstream question-answering trends, so overlap tuning can be evaluated on perplexity before committing to expensive QA runs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the same trick could be applied at test time by paraphrasing the query before retrieval, manufacturing high-overlap neighbors for chunks where natural neighbors are weak.
  • Editorial extension: if overlap is truly the causal lever, other cheap augmentations such as back-translation or synonym substitution should reproduce the speedup; if they do not, the effect is driven by synthetic-text properties rather than overlap.
  • Editorial extension: an annealing schedule that starts with synthetic high-overlap neighbors and later switches to natural neighbors could recover the lost final perplexity while keeping the faster activation.
  • Editorial extension: the findings suggest a new objective for retrieval corpus design, selecting or generating chunks to maximize query-neighbor overlap during pretraining rather than only retrieval relevance at inference.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies how token-level overlap between input chunks and retrieved neighbors affects RETRO-style retrieval-augmented language models during continued pretraining. It trains 345M-parameter GPT models with overlap-thresholded retrieval on the Pile, showing that test perplexity improves once average overlap exceeds roughly 26-28 tokens and that higher thresholds speed convergence. It then replaces one retrieved chunk with a LLaMA-3-8B paraphrase of the input to increase overlap, and reports roughly 40% faster convergence (step 3,000 vs 5,000) at a slightly higher final perplexity (6.1 vs 5.6), with downstream Natural Questions exact match scores that are on par. The paper concludes that deliberately increasing overlap through synthetic paraphrases improves data efficiency without compromising downstream QA performance.

Significance. If the threshold and speedup findings hold, they would offer a practical, low-cost lever for retrieval-augmented pretraining: synthetic paraphrase neighbors could reduce data requirements substantially, and the threshold result would inform retrieval-corpus design. The paper also provides a useful experimental framework (RETRO-fitting with overlap-filtered retrieval) and validates perplexity trends on a downstream QA task. However, the headline quantitative claim is currently not supported as stated because the speedup is measured to different final perplexity levels, and the causal role of overlap is confounded by properties of synthetic paraphrases; single-run curves without variance estimates further weaken the quantitative conclusions. The qualitative threshold pattern is plausible, and the downstream validation is a genuine strength.

major comments (4)
  1. [Section 7.2, Figure 4] The claim that 'this improvement corresponds to roughly 40% less data to reach optimal performance' compares the step at which each run approaches its own minimum: step 3,000 with paraphrases (minimum perplexity 6.1) and step 5,000 without (minimum perplexity 5.6). A data-efficiency comparison must hold the target performance fixed. At the no-paraphrase minimum of 5.6, the paraphrase models never arrive; at target 6.1, the no-paraphrase models appear to reach that level well before step 5,000 in Figure 2, and the paper does not establish that they require 5,000 steps. The 40% figure is therefore not supported as stated. Please report the step at which each run first reaches a common perplexity value, or rephrase the result as a perplexity/step trade-off without a percentage speedup claim.
  2. [Abstract and Section 7.2] The abstract's 'without compromising performance' conflicts with the paper's own report that the lowest perplexity achieved with paraphrased neighbors is 6.1 versus 5.6 without. The downstream QA results in Table 1 do show on-par exact match, so the performance claim should be explicitly limited to downstream question answering, and the perplexity degradation should be disclosed in the abstract and conclusions.
  3. [Section 7.1 and Section 8] The speedup is attributed to increased overlap, but replacing one retrieved neighbor with a LLaMA-3-8B paraphrase of the input changes multiple properties at once: paraphrase length, lexical distribution, repetitiveness or synthetic-text character, and the fact that the paraphrase is generated from the input itself rather than sampled from the retrieval corpus. Section 8 itself admits that 'factors beyond simple overlap' and 'the stronger signal provided by synthetic data compared to natural language' may contribute. The design therefore does not isolate overlap per se. A control that matches the overlap distribution while using natural neighbors, or that manipulates overlap independently of the paraphrase source, is needed before the paper can claim that overlap causes the faster activation.
  4. [Section 5.2, Figures 1-4, Table 1] All quantitative claims—the activation threshold around <32, the convergence steps, and the 40% figure—rest on a single training run per condition. Without multiple seeds or confidence intervals, the differences at stake (step 3,000 vs 5,000, perplexity 6.1 vs 5.6) cannot be distinguished from run-to-run variation. Please provide at least a small number of seeds for the central comparisons, or clearly present the convergence-step comparisons as illustrative rather than as measured speedups.
minor comments (4)
  1. [Section 5.1] The definition of overlap counts 'tokens shared between the input chunk and one of its neighbors (including continuations)', but it is not stated whether this is a bag-of-tokens count, a positional match count, or a longest-common-subsequence measure; please specify the exact matching criterion.
  2. [Section 4.3] The retrieval index uses approximate search (OPQ32_64, IVF65536_HNSW8, PQ32); please state whether the retrieved neighbor sets are deterministic across runs and whether the approximate index affects the overlap statistics reported in the figures.
  3. [References] The reference list contains two entries for Yogatama et al. (2021a and 2021b) with the same title 'Adaptive semiparametric language models'; please verify that these are distinct works and that citations are mapped correctly.
  4. [Section 7.1] The paper does not state whether the LLaMA-3 paraphrase generation is deterministic or sampled; if sampled, the reported overlap increases (e.g., +22 tokens) may vary across draws, which is relevant to the reproducibility of the speedup results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's empirical claims are supported by independently run experiments, and its self-citations are motivational rather than load-bearing.

full rationale

The paper does not derive any reported quantity from fitted constants or from definitions that presuppose the conclusions. The central claims—that higher input-neighbor overlap improves perplexity and accelerates RETRO activation, and that synthetic paraphrases of the input chunk similarly accelerate training—are supported by new experiments on the Pile and Natural Questions. The overlap thresholds are imposed as experimental conditions, not fitted. The paraphrase intervention increases overlap by construction, but the paper does not claim to isolate overlap as the sole cause; Section 8 explicitly notes that 'factors beyond simple overlap' contribute, and the Limitations section repeats that overlap 'likely does not capture the full complexity of retrieval utility.' The authors' prior work on overlap (Doostmohammadi et al. 2023; Norlund et al. 2023) is cited as motivation and background, not as the evidence for the new results, so the self-citations are not load-bearing. The reader-identified risk is a validity or interpretation concern, not circularity: the approximate 40% training-time reduction compares steps at which models approach their own perplexity minima (6.1 vs. 5.6), so the speedup is not measured against a fixed performance target. That is a questionable comparison, but it is not a case of the paper's claims reducing to their inputs by definition, and the paper is transparent about the perplexity trade-off. No circular step meeting the evidence bar is present.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper does not derive constants from first principles; it is empirical. The main free parameter is the experimental binning of overlap thresholds, and the main axioms are that the RETRO-fitting setup, the retrieval index, and the paraphrase intervention faithfully represent the phenomena under study.

free parameters (1)
  • Overlap threshold bins = Intervals: <12, <19, <25, <32, <38, <44, <51, <57, <=64
    The bin edges are chosen by hand. The claim that a critical threshold appears around <32 depends on this particular binning; a coarser or finer binning could shift the apparent activation point.
assumptions (5)
  • domain assumption RETRO-fitting a pretrained GPT model behaves like full RETRO pretraining for the overlap effects studied.
    Section 4.1 uses continued pretraining to save compute, citing Borgeaud et al. (2022) for comparability. The paper's own Limitations section notes that results may not extend to training from random initialization.
  • domain assumption The Pile training set, embedded with MiniLM-L6 and searched with FAISS, provides a retrieval distribution representative of larger RETRO setups.
    Section 4.3. Retrieval quality depends on the embedding model and approximate index; no retrieval-quality metrics are reported.
  • domain assumption Perplexity is a sufficient proxy for downstream QA performance when studying overlap.
    Section 6 validates this on Natural Questions exact match, so it is partially checked, but only for one task and without variance.
  • ad hoc to paper The LLaMA 3 8B paraphrase prompt yields paraphrases with controlled overlap and no harmful artifacts.
    Section 7.1 and Appendix A. The paraphrase model and prompt are specific to this intervention; no analysis of paraphrase quality or failure cases is provided.
  • domain assumption Single random seed results are representative of the training dynamics.
    No seeds are reported; every condition is run once, so all threshold and convergence conclusions could be influenced by run-to-run variance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Studying the Role of Input-Neighbor Overlap in Retrieval-Augmented Language Models Training Efficiency." pith.science (2026). https://pith.science/paper/NVBIZRJ3

@misc{pith2026250514309,
  author       = {Pith},
  title        = {Pith review of: Studying the Role of Input-Neighbor Overlap in Retrieval-Augmented Language Models Training Efficiency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NVBIZRJ3}},
  note         = {Machine review of arXiv:2505.14309}
}
read the original abstract

Retrieval-augmented language models have demonstrated performance comparable to much larger models while requiring fewer computational resources. The effectiveness of these models crucially depends on the overlap between query and retrieved context, but the optimal degree of this overlap remains unexplored. In this paper, we systematically investigate how varying levels of query--context overlap affect model performance during both training and inference. Our experiments reveal that increased overlap initially has minimal effect, but substantially improves test-time perplexity and accelerates model learning above a critical threshold. Building on these findings, we demonstrate that deliberately increasing overlap through synthetic context can enhance data efficiency and reduce training time by approximately 40\% without compromising performance. We specifically generate synthetic context through paraphrasing queries. We validate our perplexity-based findings on question-answering tasks, confirming that the benefits of retrieval-augmented language modeling extend to practical applications. Our results provide empirical evidence of significant optimization potential for retrieval mechanisms in language model pretraining.

Figures

Figures reproduced from arXiv: 2505.14309 by the authors.

Figure 1
Figure 1. Test perplexity (line) at training step 4,000 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Test perplexity trends for models with different overlap thresholds over training steps. The colors, as well [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Exact match percentages on the Natural Questions dataset over various training steps and overlap [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Test perplexity trends of models with varying overlap thresholds over training steps. The legend indicates [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 12 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  4. [4]

    Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020. https://openreview.net/forum?id=SJgVHkrYDH Learning to retrieve reasoning paths over wikipedia graph for question answering . In International Conference on Learning Representations

    work page 2020

  5. [5]

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206--2240. PMLR

    2022

  6. [6]

    Conover, M

    M. Conover, M. Hayes, A. Mathur, J. Xie, J. Wan, S. Shah, A. Ghodsi, P. Wendell, M. Zaharia, and R. Xin. 2023. https://databricks.com/ Free dolly: Introducing the world’s first truly open instruction-tuned llm . Technical report, Databricks

    work page 2023

  7. [7]

    Ehsan Doostmohammadi, Tobias Norlund, Marco Kuhlmann, and Richard Johansson. 2023. https://doi.org/10.18653/v1/2023.acl-short.45 Surface-based retrieval reduces perplexity of retrieval-augmented language models . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 521--529, Toronto, Ca...

    work page doi:10.18653/v1/2023.acl-short.45 2023

  8. [8]

    Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. https://doi.org/10.18653/v1/D18-1045 Understanding back-translation at scale . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489--500, Brussels, Belgium. Association for Computational Linguistics

    doi:10.18653/v1/d18-1045 2018

Show all 33 references
  1. [9]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027

    2020 arXiv

  2. [10]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: retrieval-augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org

    2020

  3. [11]

    Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. https://doi.org/10.18653/v1/2023.acl-long.806 Unnatural instructions: Tuning language models with (almost) no human labor . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics...

    2023 doi

  4. [12]

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1--43

    2023

  5. [13]

    Jeff Johnson, Matthijs Douze, and Herv \'e J \'e gou. 2019. Billion-scale similarity search with GPUs . IEEE Transactions on Big Data, 7(3):535--547

    2019

  6. [14]

    Micha Jungiewicz and Aleksander Smywi \'n ski-Pohl. 2019. Towards textual data augmentation for neural networks: synonyms and maximum loss. Computer Science, 20

    2019

  7. [15]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.550 Dense passage retrieval for open-domain question answering . In Proceedings of the 2020 Conference on Empiric...

    2020 doi

  8. [16]

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. https://openreview.net/forum?id=HklBjCEKvH Generalization through memorization: Nearest neighbor language models . In International Conference on Learning Representations

    2020

  9. [17]

    Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  10. [18]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  11. [19]

    Megatron . 2023. Nvidia Megatron-LM: tools/retro (commit 47e3bd3) . https://github.com/NVIDIA/Megatron-LM/tree/47e3bd3047aafbae361e1699d1d8785d678732ca/tools/retro. Accessed: 2025-05-05

    2023

  12. [20]

    Tobias Norlund, Ehsan Doostmohammadi, Richard Johansson, and Marco Kuhlmann. 2023. https://doi.org/10.18653/v1/2023.findings-eacl.109 On the generalization ability of retrieval-enhanced transformers . In Findings of the Association for Computational Linguistics: EACL 2023, pag...

    2023 doi

  13. [21]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners

    2019

  14. [22]

    Nils Reimers and Iryna Gurevych. 2019. http://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    2019 arXiv

  15. [23]

    David Samuel, Lucas Charpentier, and Sondre Wold. 2024. https://doi.org/10.18653/v1/2024.naacl-short.26 More room for language: Investigating the effect of retrieval on language models . In Proceedings of the 2024 Conference of the North American Chapter of the Association for...

    2024 doi

  16. [24]

    Smith, Luke Zettlemoyer, Wen tau Yih, and Mike Lewis

    Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Wen tau Yih, and Mike Lewis. 2024 a . https://openreview.net/forum?id=LXVswInHOo In-context pretraining: Language modeling beyond document boundaries . In The Twe...

    2024

  17. [25]

    Yucheng Shi, Qiaoyu Tan, Xuansheng Wu, Shaochen Zhong, Kaixiong Zhou, and Ninghao Liu. 2024 b . https://doi.org/10.1145/3627673.3679722 Retrieval-enhanced knowledge editing in language models for multi-hop question answering . In Proceedings of the 33rd ACM International Confe...

    2024

  18. [26]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053

    2019 arXiv

  19. [27]

    Amane Sugiyama and Naoki Yoshinaga. 2019. https://doi.org/10.18653/v1/D19-6504 Data augmentation using back-translation for context-aware neural machine translation . In Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019), pages 35--44, Hong K...

    2019 doi

  20. [28]

    Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Instructretro: instruction tuning post retrieval-augmented pretraining. In Proceedings of the 41st International Conference on Machine Learning, pages 51255--51272

    2024

  21. [29]

    Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, Anima Anandkumar, and Bryan Catanzaro. 2023 a . Shall we pretrain autoregressive language models with retrieval? a comprehensive study

    2023

  22. [30]

    Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. 2023 b . Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. arXiv preprint arXiv:2310.07521

    2023 arXiv

  23. [31]

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. http://arxiv.org/abs/2002.10957 Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    2020 arXiv

  24. [32]

    Xu, Uri Alon, and Graham Neubig

    Frank F. Xu, Uri Alon, and Graham Neubig. 2023. Why do nearest neighbor language models work? In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org

    2023

  25. [34]

    Dani Yogatama, Cyprien de Masson d’Autume, and Lingpeng Kong. 2021 b . https://doi.org/10.1162/tacl_a_00371 Adaptive semiparametric language models . Transactions of the Association for Computational Linguistics, 9:362--373

    2021 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.