Pith. sign in

REVIEW 12 cited by

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.07279 v2 pith:NDZ7CBXJ submitted 2024-11-11 cs.AI cs.CLcs.LG

Ekin Akyürek , Mehul Damani , Adam Zweiger , Linlu Qiu , Han Guo , Jyothish Pari , Yoon Kim , Jacob Andreas This is my paper
classification cs.AIcs.CLcs.LG
keywords in-contexttrainingexamplesfew-shotlearningtaskstest-timeeffectiveness
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Language models (LMs) have shown impressive performance on tasks within their training distribution, but often struggle with structurally novel tasks even when given a small number of in-context task examples. We investigate the effectiveness of test-time training (TTT) -- temporarily updating model parameters during inference using a loss derived from input data -- as a mechanism for improving LMs' reasoning and few-shot learning capabilities. On the Abstraction and Reasoning Corpus (ARC), performing TTT with in-context examples yields up to $6\times$ higher accuracy compared to fine-tuned baselines -- reaching $53.0\%$ on the public validation set with an 8B-parameter LM and $61.9\%$ when ensembled with program-synthesis methods, matching average human performance. On BIG-Bench Hard (BBH), TTT on in-context examples surpasses standard few-shot prompting in the $10$-shot setting by $7.3$ percentage points ($50.5\%$ to $57.8\%$). Our findings highlight the limitations of in-context learning for novel tasks and demonstrate the potential of test-time training to enhance language model adaptability.

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Consensus as Privileged Context for Label-Free Self-Distillation

    cs.LG 2026-07 conditional novelty 7.0 of 10

    CANON turns a model's majority-vote consensus into dense per-token supervision by distilling a frozen teacher conditioned on a consensus solution, improving label-free LLM reasoning by up to about 12 points and transf...

  2. Self-Guided Test-Time Training for Long-Context LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Model-selected evidence spans for test-time training beat random and full-context TTT on long-context reasoning benchmarks, with up to ~15% relative accuracy gains.

  3. No Time Like the Present: Agentic Test-Time Training for LLM Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Continuous test-time training that downweights tokens in repeated n-grams improves multi-turn LLM agent success by up to about 5 points by limiting self-reinforcing drift.

  4. LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems

    cs.AI 2025-07 conditional novelty 6.0 of 10

    LeMix co-locates LLM serving and retraining on shared GPUs with profiler-driven scheduling, reporting up to 3.53x throughput gains over separated deployments.

  5. Context Tuning for In-Context Optimization

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Context Tuning initializes a trainable prompt or KV-cache prefix from the few-shot demonstrations and refines it by gradient descent, matching Test-Time Training accuracy on MMLU, BBH, and ARC at up to 2-3x less train...

  6. EasyARC: Evaluating Vision Language Models on True Visual Reasoning

    cs.CV 2025-06 conditional novelty 6.0 of 10

    EasyARC is a new procedurally generated visual reasoning benchmark where state-of-the-art vision-language models score below 20%, despite tasks designed to be easy.

  7. From Generation to Generalization: Emergent Few-Shot Learning in Video Diffusion Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Frozen CogVideoX1.5, adapted with LoRA on 3 to 30 input-output videos, performs segmentation, pose estimation, and abstract reasoning (ARC-AGI 16.75%) with modest but real generalization.

  8. Memoir: Should a Model Write to Its Memory While It Thinks?

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Writing to fast memory during pondering slows associative-recall learning at a fixed budget, but does not reduce final performance once training is long enough.

  9. RTTC: Reward-Guided Collaborative Test-Time Compute

    cs.CL 2025-08 reject novelty 5.0 of 10

    RTTC uses a reward model to route each query to direct inference, retrieval-augmented generation, or test-time training, plus caching to cut redundant work.

  10. Channel-Wise MLPs Improve the Generalization of Recurrent Convolutional Networks

    cs.LG 2025-08 conditional novelty 5.0 of 10

    Adding a gated channel-wise MLP to a recurrent convolutional network raises median exact-match accuracy on 185 Re-ARC tasks from 78.75% to 92.19% in-distribution and from 2.34% to 14.58% on harder out-of-distribution tasks.

  11. Test-Time Learning for Large Language Models

    cs.CL 2025-05 reject novelty 5.0 of 10

    TLM adapts LLMs at test time by minimizing input perplexity on unlabeled data, using high-perplexity sample selection and LoRA updates.

  12. Few-Shot Test-Time Optimization Without Retraining for Semiconductor Recipe Generation and Beyond

    cs.LG 2025-05 reject novelty 4.0 of 10

    A two-loop reverse-model method for test-time input optimization is claimed to find semiconductor etching recipes in five iterations, but the supporting evidence is simulation-only and not reproducible.

Pith tools