REVIEW 5 cited by
Test-Time Training on Nearest Neighbors for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Many recent efforts augment language models with retrieval, by adding retrieved data to the input context. For this approach to succeed, the retrieved data must be added at both training and test time. Moreover, as input length grows linearly with the size of retrieved data, cost in computation and memory grows quadratically for modern Transformers. To avoid these complications, we simply fine-tune the model on retrieved data at test time, using its standard training setup. We build a large-scale distributed index based on text embeddings of the Pile dataset. For each test input, our system retrieves its neighbors and fine-tunes the model on their text. Surprisingly, retrieving and training on as few as 20 neighbors, each for only one gradient iteration, drastically improves performance across more than 20 language modeling tasks in the Pile. For example, test-time training with nearest neighbors significantly narrows the performance gap between a small GPT-2 and a GPT-Neo model more than 10 times larger. Sufficient index quality and size, however, are necessary. Our work establishes a first baseline of test-time training for language modeling.
Forward citations
Cited by 5 Pith papers
-
Forget, Anticipate and Adapt: Test Time Training for Long Videos
FFN performs TTT on multi-hour videos by restricting updates to three frames and using a surprise metric for adaptive window sizing, plus a new EpicTours dataset.
-
Test3R: Learning to Reconstruct 3D at Test Time
Test3R improves 3D reconstruction by optimizing visual prompts at test time so that pointmaps from different image pairs are geometrically consistent.
-
How Benchmark Prediction from Fewer Data Misses the Mark
Benchmark prediction methods mostly work by interpolation among similar models and fail on better, unfamiliar models, where random sampling with an AIPW-style correction is the only consistent improvement.
-
Device-Cloud Collaborative Correction for On-Device Recommendation
CoCorrRec applies test-time training to on-device sequential recommendation, with a cloud-side correction network, and reports accuracy gains over RNN and Transformer baselines on three datasets.
-
Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selection
A test-time training method that fine-tunes LoRA adapters on verifier-selected high-confidence pseudo-labels, reporting large gains on math benchmarks, but evaluated on the same queries it adapts on.
Discussion (0). Continue with ORCID to comment.