Pith. sign in

REVIEW 4 major objections 5 minor 56 references

Context Tuning for In-Context Optimization

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Refining the key-value prefix built from a few demonstrations — instead of the model's weights — matches test-time training accuracy on NLP-LR, MMLU, BBH, and ARC while using at most half the training time.

desk verdict CT-KV is a clean, practical method with internally consistent results, but the comparison needs code and a stated hyperparameter-selection protocol before the headline efficiency claims are cited. read the letter →

arxiv 2507.04221 v3 pith:ROFBHCCJ submitted 2025-07-06 cs.CL cs.AIcs.LG

Jack Lu , Ryan Teehan , Zhenbang Yang , Mengye Ren This is my paper
classification cs.CLcs.AIcs.LG
keywords contexttuningin-contextlearningtest-timetrainingpromptprefixKVcacheadaptationfew-shotinference-timeoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a large language model can adapt to a new task more cheaply by refining the memory it forms from demonstrations than by updating its weights. Its method, Context Tuning, initializes a trainable soft prompt (CT-Prompt) or a per-layer trainable key-value prefix (CT-KV) from the layer-wise activations of the demonstration pairs, then optimizes that context with a leave-one-out masked objective and token dropout. The headline result is that CT-KV matches Test-Time Training's accuracy across NLP-LR, MMLU, BIG-Bench Hard, and ARC while requiring at most half the training time, and that running CT-KV after Test-Time Training improves on both. If true, this gives practitioners a weight-free alternative for few-shot adaptation that is cheaper than existing inference-time optimization and complementary to it.

What carries the argument

The load-bearing object is the trainable key-value prefix $\Theta_{\mathrm{CT}} = \{K_j, V_j\}_{j=1}^{L}$, one key and value matrix per transformer layer, initialized from the model's own activations on the demonstration context and refined by the leave-one-out, token-dropout objective of Equation 9. Its efficiency comes from an asymmetry in attention: the prefix tokens are inserted only as past keys and values, never as queries, so each training step costs $O(k\ell^2)$ per attention head instead of the $O((k\ell)^2)$ paid by Test-Time Training and CT-Prompt, which both attend over full concatenated contexts. The mechanism is that gradient refinement converts a single-pass demonstration memory into a task-tuned one, and the Leave-One-Out mask forces that refinement to encode the task rather than the answers.

What would settle it

Train CT-KV twice on the same tasks — once with Leave-One-Out Masking as specified, once with full attention to the prefix during training. Under the paper's account the unmasked version should overfit by retrieving stored answers and generalize worse; if unmasked training instead matches or beats the masked version, the reported gains come from the train/test gap rather than from transferable refinement and would not survive changes in demonstration composition.

Watch

Extended reading notes

Core claim

The central claim is that the context representation itself is a viable optimization surface for few-shot adaptation, not merely a static memory. CT-KV begins by caching the model's layer-wise keys and values on the concatenated demonstration context — exactly the representation a single in-context-learning forward pass would form — and then refines that key-value prefix by gradient descent so that the model solves each demonstration pair while its own pair's prefix entries are masked out (Leave-One-Out Masking), with random Token Dropout as regularization. At inference the query attends to all tuned entries. The paper reports that this matches Test-Time Training on four benchmarks (NLP-LR 44.2 vs 44.1, MMLU 43.7 vs 43.6, BBH 57.9 vs 57.8, ARC 23.8 vs 23.8) with at most half the training time, and that TTT+CT-KV reaches 47.6 on NLP-LR, exceeding either method alone. The two design choices are essential: without Leave-One-Out Masking the model retrieves answers from the prefix instead of learning the task, and without Token Dropout it overfits.

Load-bearing premise

During training each demonstration pair is solved without seeing its own key-value entries, but at test time the query sees all of them, and the paper assumes that refining under the masked objective transfers to the unmasked inference setting.

Editorial extensions

If this is right

  • CT-KV delivers accuracy statistically tied with Test-Time Training on NLP-LR, MMLU, BBH, and ARC at roughly half the training time per task, so the same few-shot gains are available without weight updates.
  • Context adaptation and weight adaptation are complementary: TTT+CT-KV beats both alone (47.6 vs 44.1 and 44.2 on NLP-LR), so the two routes can be stacked.
  • Initializing trainable prompts or prefixes from demonstration activations instead of random tokens improves accuracy and lowers seed-to-seed variance.
  • CT-KV's advantage over ICL grows with demonstration count — the gap widens to 16.1 points at 100 shots on BBH — and survives up to 75% label corruption, so it scales where plain ICL plateaus.
  • Because it tunes a linear-cost prefix on frozen weights, CT-KV offers a path to many-shot adaptation without the quadratic cost of Test-Time Training or full weight fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's Fisher analysis — value parameters carry far more task information than key parameters — suggests that what in-context learning stores is largely value-side associations; a testable extension is whether random keys combined with tuned values retain most of CT-KV's gains across tasks.
  • The unmeasured train/test gap is the natural probe: training hides each pair's own key-value entries while inference reveals all of them, so the method bets that refinement transfers across the mask; directly training CT-KV with unmasked attention on the same tasks would show whether the gains are genuine refinement or retrieval of trained answers.
  • Nothing in the method is specific to text, so CT-KV should transfer to frozen vision or multimodal backbones for few-shot adaptation, and the ARC grid-format results already exercise non-linguistic inputs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Context Tuning, a family of inference-time adaptation methods that initialize a trainable prompt (CT-Prompt) or per-layer key-value prefix (CT-KV) from the activations of in-context demonstration pairs and refine it with gradient descent under leave-one-out masking and token dropout. The authors frame Test-Time Training (TTT) as an instance of a broader 'In-Context Optimization' framework and propose TTT+CT-KV as a combination. Experiments on NLP-LR, MMLU, BBH, and ARC report that CT-KV matches TTT accuracy with at most half the training time, outperforms ICL and prompt/prefix tuning, and that TTT+CT-KV improves further. An appendix gives an asymptotic complexity argument for CT-KV's linear scaling in the number of demonstrations.

Significance. If the reported comparisons are sound, CT-KV is a practically valuable method: it is simple, training-efficient, and complementary to weight-based test-time training. The paper's strengths include evaluation across four benchmarks with multiple backbones, ablations of the two design choices, an asymptotic complexity analysis, and detailed per-method hyperparameter tables. However, the main quantitative claims depend on an auditable hyperparameter-selection procedure and on a justification of the train/inference masking mismatch, both of which need additional support before the claims can be fully credited.

major comments (4)
  1. [Section 5.3 and Appendix C] The hyperparameter selection procedure is not auditable. The paper states that for all methods it searches over learning rates {3e-4, 1e-3, 3e-3}, token dropout rates {0, 0.05, 0.1}, and benchmark-specific iteration counts, but it never specifies the selection criterion or a held-out validation split. Because Table 1 reports test accuracy, any selection based on the test set would inflate the reported gaps (e.g., CT-KV vs Prefix Tuning (m=32): 44.2 vs 42.0 on NLP-LR, 43.7 vs 39.9 on MMLU). Please report the validation protocol, the per-seed selected hyperparameters, and ideally a fixed-hyperparameter comparison across methods.
  2. [Section 4.3, Equations (7), (9), and (11)] Training uses leave-one-out masking while inference does not: during optimization, pair i is predicted without attending to its own KV entries, but at inference the query attends to all k tuned entries. The paper does not analyze this train/inference mismatch. Add an experiment that evaluates the trained context under the same mask used for gradients, or a diagnostic that measures to what extent the optimized KV entries encode the answer of the corresponding demonstration, to support the transfer assumption.
  3. [Table 3 vs Table 1, ARC row] The ARC row of Table 3 reports 'Both' = 22.5, but Table 1 reports CT-KV = 23.8 and Section 5.3 states that ARC does not use Leave-One-Out Masking. The configuration reported in Table 1 corresponds to the 'No Leave-One-Out Masking' row (23.8), not to 'Both.' Please relabel or annotate the table so the ablation is consistent with the main configuration.
  4. [Table 2] Table 2 reports only point estimates for five large models, with no standard deviations and no statement of how many demonstration sets were used. The claim that CT-KV 'significantly outperforms' ICL and Prefix Tuning on these models is therefore not statistically supported. Report variability and the hyperparameter settings used for each large-model run.
minor comments (5)
  1. [Appendix A] The symbol p is overloaded: it is first defined as the number of additional context/prompt/prefix tokens per layer, but later set to (k-1)ℓ for all methods because leave-one-out masking removes one pair. Clarify that for CT-KV, p is the number of prefix tokens per layer after masking.
  2. [Related Work] The paper does not compare against In-Context Vectors (Liu et al., 2024b), which is closely related in spirit. A sentence explaining the difference between optimizing a KV cache and adding a latent vector would help position the contribution.
  3. [Table 1] ARC entries have no standard deviation; please state explicitly that these are single fixed-seed numbers because ARC has fixed demonstration pairs, as done in the table caption.
  4. [Figure 4] The error bars in Figure 4 are not described; please specify whether they are standard deviations over the five demonstration-set seeds and how they are computed for the k variations.
  5. [Section 5.7] The text says 'CT-KV without Token Dropout performs marginally worse than CT-KV with both components,' but Table 3 shows that on ARC, No Token Dropout (21.0) is worse than Both (22.5) while No Leave-One-Out Masking (23.8) is best. Consider reporting the ARC discussion in a way that directly maps to the configuration used in Table 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central empirical claims rest on external benchmark comparisons, and the complexity argument is self-contained.

full rationale

I find no circular step in the derivation chain. The central quantitative claims (Section 5.4, Table 1) are relative accuracies on external benchmarks (NLP-LR, MMLU, BBH, ARC) for held-out queries; CT-KV's trainable prefixes are optimized by Eq. 9 on demonstration pairs, and the reported test accuracies are not optimands, so no fitted value is renamed as a prediction. The initialization from demonstrations and the Leave-One-Out Masking and Token Dropout regularizers are method design choices, not circular inputs: the ablation in Table 3 is an empirical check, and the train/inference mismatch between Eq. 7/9 and Eq. 11 is a transfer assumption (a robustness risk) rather than an identity. The ICO taxonomy is a definitional framing; the sentence excluding Prompt and Prefix Tuning 'by definition' is a naming choice and does not carry the accuracy claims, which stand on external comparisons and on the external TTT baseline (Akyurek et al., 2025). The Appendix A complexity bound is a self-contained asymptotic analysis following from the architecture's Q/K shapes, not a result imported from a self-citation. The only author self-citation (Lu et al., 2024, ProCreate) appears in related work and is not load-bearing. Remaining concerns--per-method hyperparameter sweeps in Section 5.3/Appendix C without a disclosed held-out validation split, and the unverified LOO-mask-to-inference transfer--are correctness and auditability risks, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central empirical claims rest on standard fine-tuning machinery plus per-benchmark hyperparameters (LR, iterations, dropout, LOO on/off), four of which are listed above. No new physical or mathematical entities are postulated; the only invented abstraction is the ICO umbrella term. The ARC setup additionally assumes that fine-tuning on the ARC training split is a permitted and transferable preparation step.

free parameters (4)
  • Token Dropout rate = 0.05 (NLP-LR), 0.1 (MMLU, BBH, ARC)
    Regularization probability applied to the trainable context tokens (Section 4.3, Table 6). Selected per benchmark; the headline CT-KV numbers depend on it.
  • Training iterations = 200/20/16/200 for NLP-LR/MMLU/BBH/ARC (CT-KV, Table 6)
    Optimization steps per task selected by per-method sweep; results are reported at the selected point.
  • Learning rate = 1e-3/3e-3/1e-3/3e-3 for NLP-LR/MMLU/BBH/ARC (CT-KV)
    Selected per benchmark and method (Table 6); the method's results are sensitive to these choices.
  • Leave-One-Out Masking on/off = off for ARC, on elsewhere
    Section 5.3 discloses that ARC uses no LOO masking because it improves performance; the headline ARC number comes from the modified configuration.
assumptions (4)
  • domain assumption The KV activations of the demonstrations, computed in one forward pass, are a differentiable and sufficient initialization for gradient refinement with frozen weights.
    Section 4.3 and Equation 9: CT-KV optimizes the prefix KV tensors directly; no analysis is given of when this initialization is degenerate.
  • standard math Attention cost dominates training time, so the Appendix A complexity comparison governs wall-clock efficiency.
    Appendix A and Table 4; the one-time initialization forward pass over all demonstrations, O((k*l)^2), is excluded from the per-step comparison.
  • domain assumption For ARC, fine-tuning Llama 3.2-1B on the 400-task ARC training split transfers to the evaluation tasks.
    Section 5.2, following Akyurek et al. (2025) and Franzen et al. (2025); all ARC results, including the ICL baseline of 13.3%, depend on this training step.
  • domain assumption Per-benchmark hyperparameter selection on the evaluation set is a fair comparison protocol.
    Appendix C Table 6 selects LR, iterations, and dropout for every method on each benchmark without a stated validation split.
invented entities (1)
  • In-Context Optimization (ICO) framework
    purpose: Defines a taxonomy (Section 4.1) under which TTT optimizes model weights and Context Tuning optimizes the context representation.
    A definitional umbrella introduced by this paper; no falsifiable predictions beyond the methods it names. Its stem 'Context Optimization' is also the name of the CoOp prompt-learning line (not cited), creating a terminology collision.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Context Tuning for In-Context Optimization." pith.science (2026). https://pith.science/paper/ROFBHCCJ

@misc{pith2026250704221,
  author       = {Pith},
  title        = {Pith review of: Context Tuning for In-Context Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ROFBHCCJ}},
  note         = {Machine review of arXiv:2507.04221}
}
read the original abstract

We introduce Context Tuning, a simple and effective method to significantly enhance few-shot adaptation of large language models (LLMs) without weight updates. In-Context Learning (ICL) forms a memory representation of the demonstrations in a single forward pass but cannot refine it when insufficient. Prompt-based methods offer lightweight adaptation by optimizing a trainable prompt or prefix but initialize it independently of the demonstrations. In contrast, Context Tuning leverages the model's inherent ICL ability to initialize a trainable memory representation from demonstrations, then refines it through gradient-based optimization. Extensive evaluations on benchmarks such as CrossFit, UnifiedQA, MMLU, BIG-Bench Hard, and ARC demonstrate that Context Tuning outperforms both ICL and traditional prompt-based adaptation methods while achieving competitive accuracy with Test-Time Training at significantly higher training efficiency.

Figures

Figures reproduced from arXiv: 2507.04221 by the authors.

Figure 1
Figure 1. Comparison of training-free, prompt-based adaptation, and In-Context Optimization methods on solving 26 NLP-LR tasks from [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. CT-KV, the variant of Context Tuning that optimizes the key-value prefixes derived from in-context demonstration pairs. CT-KV (left) first initializes a prefix {𝐾𝑖 , 𝑉𝑖 } 𝑘 𝑖=1 from demonstration pairs {(𝑥𝑖 , 𝑦𝑖 )}𝑘 𝑖=1, then trains it to solve each pair. To prevent the model from simply retrieving the demonstration pair from the prefix, Leave-One-Out Masking prevents the model from attending to 𝐾𝑖 , 𝑉𝑖 when solving… view at source ↗
Figure 3
Figure 3. One test pair from BBH, NLP-LR, and MMLU each, and three demonstration pairs followed by a test pair from ARC. BBH contains instructions that we prepend to model inputs. NLP-LR and MMLU contain multiple-choice options for the model to select. To avoid clutter, we show demonstration pairs from BBH, NLP-LR, and MMLU in Appendix K. 5.3 Experiment Setup On top of zero-shot inference and ICL, we compare a variety of few-… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Performance of ICL, Prefix Tuning, and CT-KV as (a) the number of demonstration pairs and (b) the demonstration-label corruption probability vary on NLP-LR and MMLU. CT-KV is more efficient than CT-Prompt. CT-KV exhibits significantly lower training time per task compa…
Figure 5
Figure 5. Figure 5: ARC example showing 4 demonstration query-answer pairs, the test query, and LLM predictions at CT-KV training iterations 0, 50, 100, 150, and 200. Iteration 0 is equivalent to ICL, with correct prediction labels in green and incorrect prediction labels in red. Page 12 …
Figure 6
Figure 6. Figure 6: Additional ARC example showing 4 demonstration query-answer pairs, the test query, and LLM predictions at CT-KV training iterations 0, 50, 100, 150, and 200. Iteration 0 is equivalent to ICL, with correct prediction labels in green and incorrect prediction labels in re…
Figure 7
Figure 7. Figure 7: We display LLM predictions at CT-KV training iterations 0, 12, 16, and 20 for two queries from the task “word sorting” in BBH. We omit the 16 demonstration pairs provided with each query for brevity. We color-code correct prediction labels in green and incorrect predic…
Figure 8
Figure 8. Figure 8: Left is an ARC task that CT-KV successfully solves, but ICL does not. Right is an ARC task that ICL solves, but CT-KV does not. Page 25 of 27 [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: Three demonstration pairs for the BBH task from [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]
Figure 10
Figure 10. Figure 10: Three demonstration pairs for the NLP-LR task from [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: Three demonstration pairs for the MMLU task from [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 41 canonical work pages

  1. [1]

    Akyürek, E., Damani, M., Zweiger, A., Qiu, L., Guo, H., Pari, J., Kim, Y., and Andreas, J. (2024). The surprising effectiveness of test-time training for few-shot learning. arXiv preprint arXiv:2411.07279

    arXiv 2024

  2. [2]

    and Macfarlane, M

    Bonnet, C. and Macfarlane, M. V. (2024). Searching latent program spaces. arXiv preprint arXiv:2411.08706

    arXiv 2024

  3. [3]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

    work page 2020

  4. [4]

    Chen, Y., Zhong, R., Zha, S., Karypis, G., and He, H. (2022). Meta-learning via language model in-context tuning. In ACL

    work page 2022

  5. [5]

    Chollet, F. (2019a). Abstraction and reasoning corpus for artificial general intelligence (arc-agi)

    work page 2019

  6. [6]

    Chollet, F. (2019b). On the measure of intelligence. arXiv preprint arXiv:1911.01547

    arXiv 2019

  7. [7]

    Chollet, F., Knoop, M., Kamradt, G., and Landers, B. (2025). Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604

    arXiv 2025

  8. [8]

    Dai, D., Sun, Y., Dong, L., Hao, Y., Ma, S., Sui, Z., and Wei, F. (2023). Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. In ICLR

    work page 2023

Show all 56 references
  1. [9]

    C., Kautz, J., Guestrin, C., Hashimoto, T., Koyejo, S., Choi, Y., Sun, Y., and Wang, X

    Dalal, K., Koceja, D., Hussein, G., Xu, J., Zhao, Y., Song, Y., Han, S., Cheung, K. C., Kautz, J., Guestrin, C., Hashimoto, T., Koyejo, S., Choi, Y., Sun, Y., and Wang, X. (2025). One-minute video generation with test-time training. In CVPR

    2025

  2. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    DeepSeek-AI (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  3. [11]

    Deutch, G., Magar, N., Natan, T., and Dar, G. (2024). In-context learning and gradient descent revisited. In NAACL

    2024

  4. [12]

    Devoto, A., Zhao, Y., Scardapane, S., and Minervini, P. (2024). A simple and effective l\_2 norm-based strategy for kv cache compression. In EMNLP

    2024

  5. [13]

    and Nichol, A

    Dhariwal, P. and Nichol, A. Q. (2021). Diffusion models beat gans on image synthesis. In NeurIPS

    2021

  6. [14]

    Franzen, D., Disselhoff, J., and Hartmann, D. (2024). The llm architect: Solving the arc challenge is a matter of perspective. arXiv preprint arXiv:2505.07859

    2024 arXiv

  7. [15]

    Gandelsman, Y., Sun, Y., Chen, X., and Efros, A. A. (2022). Test-time training with masked autoencoders. In NeurIPS

    2022

  8. [16]

    Garg, S., Tsipras, D., Liang, P., and Valiant, G. (2022). What can transformers learn in-context? a case study of simple function classes. arXiv preprint arXiv:2208.01066

    2022 arXiv

  9. [17]

    Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., and Gao, J. (2024). Model tells you what to discard: Adaptive kv cache compression for llms. In ICML

    2024

  10. [18]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., and et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  11. [19]

    and Sun, Y

    Hardt, M. and Sun, Y. (2024). Test-time training on nearest neighbors for large language models. In ICLR

    2024

  12. [20]

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding. In ICLR

    2021

  13. [21]

    Ho, J. (2022). Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598

    2022 arXiv

  14. [22]

    Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. arXiv preprint arxiv:2006.11239

    2020 arXiv

  15. [23]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022). Lo RA : Low-rank adaptation of large language models. In ICLR

    2022

  16. [24]

    Jang, J., Jang, S., Kweon, W., Jeon, M., and Yu, H. (2024). Rectifying demonstration shortcut in in-context learning. In ACL

    2024

  17. [25]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. (2023). Mistral 7b. arXiv ...

    2023 arXiv

  18. [26]

    Kalajdzievski, D. (2023). A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732

    2023 arXiv

  19. [27]

    Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., and Hajishirzi, H. (2020). UNIFIEDQA : Crossing format boundaries with a single QA system. In EMNLP (Findings)

    2020

  20. [28]

    A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R

    Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. (2017). Overcoming catastrophic forgetting in neural networks. In PNAS

    2017

  21. [29]

    Lester, B., Al-Rfou, R., and Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. In EMNLP

    2021

  22. [30]

    and Qiu, X

    Li, X. and Qiu, X. (2023). Finding supporting examples for in-context learning. In EMNLP (Findings)

    2023

  23. [31]

    Li, X. L. and Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. In ACL

    2021

  24. [32]

    Liu, A., Liu, J., Pan, Z., He, Y., Haffari, G., and Zhuang, B. (2024a). Minicache: KV cache compression in depth dimension for large language models. In NeurIPS

    2024

  25. [33]

    Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. (2021). What makes good in-context examples for gpt- 3 ? In ACL

    2021

  26. [34]

    Liu, S., Ye, H., Xing, L., and Zou, J. (2024b). In-context vectors: making in context learning more effective and controllable through latent space steering. In ICML

    2024

  27. [35]

    F., Cheng, K.-T., and Chen, M.-H

    Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. (2024c). Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353

    2024 arXiv

  28. [36]

    Liu, X., Ji, K., Fu, Y., Du, Z., Yang, Z., and Tang, J. (2022a). P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. In ACL

    2022

  29. [37]

    Liu, X., Ji, K., Fu, Y., Tam, W., Du, Z., Yang, Z., and Tang, J. (2022b). P -tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In ACL

    2022

  30. [38]

    Lu, J., Teehan, R., and Ren, M. (2024). Procreate, don’t reproduce! propulsive energy diffusion for creative generation. In ECCV

    2024

  31. [39]

    Min, S., Lewis, M., Zettlemoyer, L., and Hajishirzi, H. (2022a). Meta ICL : Learning to learn in context. In NAACL

    2022

  32. [40]

    Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. (2022b). Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP

    2022

  33. [41]

    Mistral nemo

    Mistral-AI-team (2024). Mistral nemo

    2024

  34. [42]

    Muhtar, D., Shen, Y., Yang, Y., Liu, X., Lu, Y., Liu, J., Zhan, Y., Sun, H., Deng, W., Sun, F., Zhang, X., Gao, J., Chen, W., and Zhang, Q. (2024). Streamadapter: Efficient test time adaptation from contextual streams. arXiv preprint arXiv:2411.09289

    2024 arXiv

  35. [43]

    Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M

    Nichol, A. Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. (2022). GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In ICML

    2022

  36. [44]

    Qwen3 technical report

    Qwen-Team (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388

    2025 arXiv

  37. [45]

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI

    2019

  38. [46]

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In CVPR

    2022

  39. [47]

    Shin, T., Razeghi, Y., IV, R. L. L., Wallace, E., and Singh, S. (2020). AutoPrompt : Eliciting knowledge from language models with automatically generated prompts. In EMNLP

    2020

  40. [48]

    S., Wei, J., Chung, H

    Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., Payne, P., Seneviratne, M., Gamble, P., Kelly, C., Scharli, N., Chowdhery, A., Mansfield, P., y Arcas, B. A., Webster, D., Corrado, G. S., Matias, Y., Cho...

    2023

  41. [49]

    Srivastava, A., Rastogi, A., Rao, A., Md-Shoeb, A.-A., Abid, A., Fisch, A., Brown, A., Santoro, A., Gupta, A., and et al. (2023). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. In TMLR

    2023

  42. [50]

    Sun, Y., Wang, X., Zhuang, L., Miller, J., Hardt, M., and Efros, A. A. (2020). Test-time training with self-supervision for generalization under distribution shifts. In ICML

    2020

  43. [51]

    W., Chowdhery, A., Le, Q

    Suzgun, M., Scales, N., Scharli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., and Wei, J. (2022). Challenging big-bench tasks and whether chain-of-thought can solve them. In ACL

    2022

  44. [52]

    Wallace, B., Gokul, A., Ermon, S., and Naik, N. (2023). End-to-end diffusion latent optimization improves classifier guidance. In ICCV

    2023

  45. [53]

    V., Chi, E

    Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models. In ICML

    2023

  46. [54]

    H., Le, Q

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS

    2022

  47. [55]

    Y., and Ren, X

    Ye, Q., Lin, B. Y., and Ren, X. (2021). C ross F it: A few-shot learning challenge for cross-task generalization in NLP . In EMNLP

    2021

  48. [56]

    Zhao, J. (2023). In-context exemplars as clues to retrieving from large associative memory. In ICML Neural Conversational AI

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.