Pith. sign in

REVIEW 22 cited by

Generalization through Memorization: Nearest Neighbor Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1911.00172 v2 pith:FFT6PXEU submitted 2019-11-01 cs.CL

Urvashi Khandelwal , Omer Levy , Dan Jurafsky , Luke Zettlemoyer , Mike Lewis This is my paper
classification cs.CL
keywords nearesttraininglanguagemodelneighborneighborsapproachdrawn
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

We introduce $k$NN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a $k$-nearest neighbors ($k$NN) model. The nearest neighbors are computed according to distance in the pre-trained LM embedding space, and can be drawn from any text collection, including the original LM training data. Applying this augmentation to a strong Wikitext-103 LM, with neighbors drawn from the original training set, our $k$NN-LM achieves a new state-of-the-art perplexity of 15.79 - a 2.9 point improvement with no additional training. We also show that this approach has implications for efficiently scaling up to larger training sets and allows for effective domain adaptation, by simply varying the nearest neighbor datastore, again without further training. Qualitatively, the model is particularly helpful in predicting rare patterns, such as factual knowledge. Together, these results strongly suggest that learning similarity between sequences of text is easier than predicting the next word, and that nearest neighbor search is an effective approach for language modeling in the long tail.

Discussion (0). Sign in to comment.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training-Free Token-Level Steering for LLM Personalized Co-Writing

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A token-level, training-free steering framework that improves LLM personalized co-writing by mixing the base model's posterior with a kernel-density estimate from a small user dataset.

  2. MemSFT: Mitigating Alignment Tax with an External Parametric Memory

    cs.LG 2026-07 conditional novelty 6.0 of 10

    MemSFT attaches a retriever-imitating 8B memory plus a word-level router to frozen Qwen3 backbones, boosting domain scores by ~36 points while holding general-benchmark averages essentially flat, where full SFT loses ...

  3. Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A DP-means allocate-on-novelty cache matches full-attention associative recall while storing only distinct items, and a minimal novelty gate recovers the rule end-to-end.

  4. Exploiting Structural Properties for Efficient Constraint-Aware HNSW Hyperparameter Tuning

    cs.DB 2026-07 conditional novelty 6.0 of 10

    CHAT uses HNSW-specific monotonic and unimodal structure plus resource surrogates to tune M, efc, and efs under constraints, beating black-box tuners by up to 45% throughput or 11% recall and up to 44× faster convergence.

  5. Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Aligning images to multi-view caption cores while suppressing orthogonal residual text and disagreement-aware temperature improves robust zero-shot recognition and LVLM transfer.

  6. Memory-Managed Long-Context Attention: Bounded Editable Memory with a Hard Lifecycle and Calibrated Sparse Fallback

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Editable bounded memory slots with lifecycle control plus sparse fallback cover overwrite and no-signal long-context cases that pure fixed-state or pure sparse methods fail under controlled conditions.

  7. Retrieval Feedback Memory Enhancement Large Model Retrieval Generation Method

    cs.IR 2025-08 reject novelty 6.0 of 10

    RFM-RAG improves RAG-based QA by iteratively building an evidence pool and stopping via a learned sufficiency classifier.

  8. RAGPPI: RAG Benchmark for Protein-Protein Interactions in Drug Discovery

    cs.CL 2025-05 conditional novelty 6.0 of 10

    RAGPPI introduces a QA benchmark for PPI biological impacts in drug target identification, with 500 expert-validated and 3,720 auto-labeled pairs (sum 4,220, though the abstract says 4,420).

  9. MaskSearch: A Universal Pre-Training Framework to Enhance Agentic Search Capability

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A pre-training task called RAMP, where models practice searching to fill masked text spans, improves downstream agentic open-domain QA performance across Qwen and LLaMA models.

  10. RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry

    cs.SE 2025-05 conditional novelty 6.0 of 10

    On a 160,000-file industrial C++ codebase, BM25-based retrieval-augmented generation outperformed fine-tuning for line-level code completion, and combining RAG with fine-tuning further improved accuracy.

  11. Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory

    cs.CL 2026-07 conditional novelty 5.0 of 10

    A pretrained parametric memory module, taught to copy nearest-neighbor retrieval for next-token prediction, lets small frozen LMs match or beat much larger LMs at the same total parameter budget.

  12. CogniConsole: Externalizing Inference-Time Control as a Formal Abstraction for Reliable LLM Interactions

    cs.AI 2026-04 conditional novelty 5.0 of 10

    Under a fixed model, increasing structural scaffolding of inference-time control systematically reduces LLM output variance and failure rates on controllability probes.

  13. GRACE: Graph-Guided Repository-Aware Code Completion through Hierarchical Code Fusion

    cs.SE 2025-09 conditional novelty 5.0 of 10

    GRACE combines a multi-level code graph, hybrid text-structure retrieval, and graph fusion to improve repository-level code completion over vanilla and graph-based RAG baselines.

  14. ExploreGS: Explorable 3D Scene Reconstruction with Virtual Camera Samplings and Diffusion Priors

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    Adding information-gain-selected virtual views refined by video diffusion priors to 3D Gaussian Splatting improves arbitrary-view rendering quality.

  15. A Rose by Any Other Name Would Smell as Sweet: Categorical Homotopy Theory for Large Language Models

    cs.CL 2025-08 reject novelty 5.0 of 10

    The paper argues that LLM next-token distributions form Markov categories whose paraphrase equivalences can be studied by homotopy theory, but its main theorem is unsupported.

  16. Beyond Independent Passages: Adaptive Passage Combination Retrieval for Retrieval Augmented Open-Domain Question Answering

    cs.CL 2025-07 conditional novelty 5.0 of 10

    AdaPCR jointly retrieves and reranks passage pairs for open-domain QA, showing small EM/F1 gains over an in-context retrieval baseline, mostly on multi-hop HotpotQA.

  17. Low-Perplexity LLM-Generated Sequences and Where To Find Them

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Only about 40% of low-perplexity 6-token spans generated by Pythia-6.9B can be exactly matched to The Pile, and the authors categorize matched and unmatched spans into four classes.

  18. TokenShapley: Token Level Context Attribution with Shapley Value

    cs.CL 2025-06 conditional novelty 5.0 of 10

    TokenShapley computes token-level Shapley attributions from context to response by treating context tokens as (prefix, token) data points in a KNN datastore.

  19. AdaDocVQA: Adaptive Framework for Long Document Visual Question Answering in Low-Resource Settings

    cs.CL 2025-08 conditional novelty 4.0 of 10

    A hybrid retrieval, data augmentation, and ensemble inference framework fine-tunes Qwen2.5-VL-72B to 59% on the Japanese LAVA document VQA benchmark.

  20. MobileRAG: A Fast, Memory-Efficient, and Energy-Efficient Method for On-Device RAG

    cs.DB 2025-07 conditional novelty 4.0 of 10

    A fully on-device RAG pipeline using a partitioned, partially disk-loaded graph index and selective sentence-window reduction claims 1.72-8.89x faster vector search and up to 40.2% lower power than baselines, with com...

  21. Rethinking Hybrid Retrieval: When Small Embeddings and LLM Re-ranking Beat Bigger Models

    cs.IR 2025-05 reject novelty 4.0 of 10

    In tri-modal hybrid retrieval with GPT-4o reranking, MiniLM-v6 matches or beats BGE-Large on SciFact, FIQA, and NFCorpus despite being far smaller.

  22. Generalizing vision-language models to novel domains: A comprehensive survey

    cs.CV 2025-06 conditional novelty 3.0 of 10

    A survey of VLM generalization literature organized by transferred module, with benchmark tables and a review of multimodal LLMs.

Pith tools