Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models

Chungpa Lee; Jy-yong Sohn; Kangwook Lee

REVIEW 4 major objections 4 minor 1 cited by

Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models

T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning all attention parameters to minimize zero-shot loss collapses the query–key mechanism that powers in-context learning; fine-tuning only the value matrix avoids this collapse.

desk verdict Solid theory paper: the full-FT kills ICL result is robust to the free-parameter w, but Fig 4(b) fits w=0.52 without derivation — still deserves peer review. read the letter →

arxiv 2602.23197 v2 pith:RPRGYRJ7 submitted 2026-02-26 cs.CL cs.LGstat.ML

Chungpa Lee , Jy-yong Sohn , Kangwook Lee This is my paper

classification cs.CLcs.LGstat.ML

keywords in-contextlearningfine-tuninglinearattentionregressionzero-shotvsfew-shotquery-keymatrixvalue-matrixwithoutforgetting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that, in a tractable linear-attention model of linear regression, the way you fine-tune decides whether a pretrained model keeps its in-context learning ability. Its central result is that fine-tuning all attention parameters to minimize zero-shot loss destroys the query–key block that makes few-shot prompting work, so the resulting model's few-shot predictions are strictly worse than its own zero-shot predictions, even on the exact task it was fine-tuned on. The paper also proves a safer route: freeze the query–key block and update only the value matrix; zero-shot error approaches the irreducible noise level as the input dimension grows, while few-shot error stays close to the pretrained level. Finally, adding an auxiliary few-shot loss during value-matrix fine-tuning improves few-shot accuracy on the fine-tuning task but degrades it on dissimilar tasks, with the excess error growing with task dissimilarity. If correct, these results give a parameter-level explanation for the observed fine-tuning/ICL trade-off and a concrete prescription: preserve the query–key channel to keep few-shot ability.

What carries the argument

The merged query–key matrix Q11 and the value row v21 inside the linear attention layer. The pretrained in-context learning mechanism is the condition Q11=Σ^{-1}: it is what makes the n examples in a prompt act like a step of preconditioned gradient descent on the task. Full fine-tuning's global minimizers (Theorem 4.3) set Q11=0 and v21=wθ0, so the predictor becomes a fixed zero-shot estimator plus a negligible prompt term—this is the 'erase' step. Value-matrix fine-tuning freezes Q11 at Σ^{-1} and only sets v21=θ0/(d+4), keeping the in-context channel open. The remaining scalar w parameterizes all zero-shot-optimal value matrices; the paper resolves it by minimal Frobenius update from the

What would settle it

Train the paper's linear attention model (d=5, Σ=I₅, σ²=0.1, m=n=20) and fully fine-tune on a fixed θ0 but with L2 regularization or early stopping before the zero-shot loss reaches its global minimum. If the resulting model keeps n-shot error below zero-shot error on θ0 for large n, or keeps Q11 noticeably nonzero, the claim that every zero-shot-optimal full fine-tune destroys in-context learning is refuted. A second check on a real language model: fine-tune with adapters restricted to value projections and compare 7-shot accuracy on a held-out category; the paper predicts near-zero degradati

Watch

Extended reading notes

Core claim

Central claim: in-context learning lives in one parameter block, and fine-tuning either preserves or destroys it. In the linear attention model, pretraining gives Q11≈Σ^{-1}, the query–key block that makes demonstrations work like a gradient step. Full fine-tuning's global minimizers set Q11=0 and the value row to wθ0, so prompts stop mattering: few-shot error exceeds zero-shot error for every n and tends to σ²+θ0^TΣθ0. Value-matrix fine-tuning keeps Q11=Σ^{-1} and sets the value row to θ0/(d+4), giving zero-shot error σ²+2/(d+4)θ0^TΣθ0 and asymptotic few-shot error σ²+1/(d+4)^2 θ0^TΣθ0. An auxiliary few-shot loss tunes a residual parameter w toward the target task, improving its few-shot er

Load-bearing premise

The theory assumes fine-tuning reaches a global minimizer of the expected zero-shot loss, with no regularization, early stopping, or optimization noise; if real optimization stops short of that global optimum, the predicted collapse of in-context learning need not occur.

Editorial extensions

If this is right

Full fine-tuning on a target task can make the fine-tuned model's few-shot performance worse than its zero-shot performance, even on that same task (Corollary 4.4).
Freezing the query–key side and fine-tuning only the value matrix keeps few-shot error near the pretrained level while zero-shot error drops to σ²+2/(d+4)θ0^TΣθ0, which approaches the noise floor as d grows (Theorem 4.6, Corollary 4.8).
Adding an auxiliary few-shot loss to value-matrix fine-tuning further reduces few-shot error on the fine-tuning task but increases few-shot error on other tasks, by an amount proportional to (1−ρ)² where ρ is the Σ-inner-product cosine similarity between tasks (Proposition 4.10).
The free parameter w can be chosen at test time to trade off zero-shot and few-shot behavior; the minimal-update choice w=m/(m+1+d) approximates the task-averaged few-shot optimum (Proposition 4.7, Corollary 4.8).
The degradation from full fine-tuning is structural, not a small-sample artifact: even with infinitely many shots, few-shot error converges to σ²+θ0^TΣθ0, strictly above the zero-shot error σ² (Corollary 4.4).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: if the same collapse happens with softmax attention, parameter-efficient methods that leave query/key projections untouched (value-only adapters) should be a safer default than methods that also alter query/key projections; this is directly testable with controlled comparisons.
Inference: the global-minimizer analysis implies that regularization, early stopping, or optimization noise that keeps Q11 away from zero may retain some in-context learning; Q11's distance from Σ^{-1} after fine-tuning could serve as a diagnostic for few-shot degradation.
Inference: the paper's annealing observation suggests a practical rule—schedule any auxiliary few-shot loss to zero during fine-tuning—to avoid the zero-shot penalty a fixed combination would incur.
Inference: in high-dimensional settings the 2/(d+4) zero-shot penalty of value-matrix fine-tuning becomes negligible, so the cost of preserving in-context learning shrinks for wide models; value-only fine-tuning is most attractive there.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a genuinely useful theory paper, and the central result is not derailed by the free-parameter issue the stress-test note flags. The qualitative prediction — full fine-tuning can kill in-context learning in linear attention, value-matrix fine-tuning preserves it — holds across the whole family of optima. Send it to review.

What's new: Ahn et al. (2023) gave the pretrained solution; this paper goes further and characterizes what task-specific fine-tuning does. The clean observation is that minimizing the zero-shot loss on a fixed task admits global minimizers with Q11=0, which by construction dismantles the query-key pathway that enables ICL. Value-matrix-only fine-tuning keeps Q11 intact and pays only a O(1/d) zero-shot penalty, and the auxiliary few-shot loss trades off target-task few-shot gains against OOD degradation with explicit excess-error formulas. The proofs are detailed, use standard Wishart/Isserlis machinery, and look checkable.

Soft spots: Figure 4(b) matches the full-fine-tune theory using w=0.52, but the theory says w is free. Nothing derives that value from the initialization or the optimizer, so the quantitative agreement is partly a fit. The qualitative conclusion is robust — few-shot error degrades for every w — but the empirical validation would be much stronger with a principled tie-breaking rule or a sensitivity plot. The same issue appears in value-matrix fine-tuning (V11, v12, and w free at the optimum); Proposition 4.7's minimal-update choice is reasonable but heuristic, and the preservation claim holds across the family asymptotically, so the mechanism is not threatened. The MMLU section supports 'degrades much less' better than 'preserves': value-matrix LoRA still drops 7-shot accuracy by a couple of points, just far less than full FT. No code or configs are released, which matters for a paper claiming empirical validation. The global-minimizer assumption is a limitation but a standard one in this literature, and the qualitative story is insensitive to it.

Bottom line: the paper deserves a serious referee. Clarify how w is chosen, report sensitivity, release code, and soften 'preserves' — then it's a clear accept.

Referee Report

4 major / 4 minor

Summary. The paper studies a linear attention model for linear regression, with pretraining on a task distribution followed by fine-tuning on a target task. It derives closed-form expressions for global minimizers of the zero-shot fine-tuning loss under three regimes: full fine-tuning, value-matrix fine-tuning, and value-matrix fine-tuning with an auxiliary few-shot loss. The central qualitative claim is that full fine-tuning can destroy the query-key mechanism (Q11=0 at the zero-shot optimum), causing few-shot error to exceed zero-shot error even on the target task, while value-matrix fine-tuning preserves few-shot performance. The theoretical results are compared with linear-regression experiments and with LoRA fine-tuning of Qwen2.5-3B on MMLU.

Significance. The paper gives a clean, detailed mathematical analysis of a practically important phenomenon: task-specific fine-tuning can degrade in-context learning. The moment computations (Isserlis/Wishart identities) are careful, and the core mechanism—that the zero-shot optimum forces Q11=0, removing the preconditioned-gradient-descent mechanism for ICL—is simple and convincing. The value-matrix fine-tuning result is a useful, non-obvious qualitative prediction. If the gaps identified below are addressed, the paper would be a solid theoretical contribution. The MMLU experiments, while small-scale and using LoRA rather than full fine-tuning, provide suggestive support.

major comments (4)

[§4.2, Theorem 4.3] The theorem shows only that the displayed family (9) attains the Bayes error σ²; it does not characterize all global minimizers of the zero-shot loss. The section's wording ('characterize the optimal parameters') and the conclusion of §4.2 imply that full fine-tuning generically degrades ICL. But the zero-shot objective is invariant to several blocks (V11, v12, v22, q12, q21, q22) and to the rescaling q·v21=θ0, so other global minimizers outside family (9) exist and may have different few-shot behavior. The proof in Appendix A.8 does not analyze these directions or which member gradient flow from the pretrained initialization would select. Please either characterize the full set of global minimizers and prove the few-shot property for all of them, or explicitly restrict the claim to the displayed family and to 'can' rather than 'does'.
[§5.1, Figure 4(b)] The theoretical curve for full fine-tuning uses w=0.52, a free parameter in the optimum family of Theorem 4.3. No procedure is given for deriving this value from the pretrained initialization or the optimizer, so the empirical agreement is not a parameter-free validation of the theory. The same point affects the quantitative prediction of the ICL loss after full fine-tuning. The authors should either derive w from the initialization (e.g., by analyzing gradient flow in the flat directions of the zero-shot loss) or report w as a fitted parameter and discuss the sensitivity of the comparison.
[§4.3, Proposition 4.7] The claim that value-matrix fine-tuning preserves ICL depends on selecting w by a minimal-Frobenius-distance rule that is not part of the zero-shot fine-tuning objective. Theorem 4.6 leaves w free; Proposition 4.7 is a heuristic. Corollary 4.8 shows that the minimal-update w approximates the task-averaged optimal w* when m=n and n+d is large, but this is an external justification. A more natural and direct argument is available: because the zero-shot loss is independent of v22, gradient flow from the pretrained v22 keeps it at m/(m+1+d), so the minimal-update choice is exactly what an unregularized gradient method would produce. Please make this explicit or otherwise justify the selection; without it, the 'preserves ICL' conclusion is conditional on a heuristic.
[General assumptions, §4 and §5] All main theorems concern global minimizers of the expected population loss, with no regularization, early stopping, or optimization noise. Real fine-tuning—including the paper's own MMLU experiments, which use LoRA and select the best zero-shot checkpoint—will generally stop at a point that is not a global minimizer of the unregularized objective. The Limitations section acknowledges this in general terms, but the main-text conclusions (abstract, Table 1, §6) are easy to misread as statements about actual fine-tuning dynamics. Please state the exact idealization (population loss, global optimum, no regularization) in the statements of the main results and soften the causal wording where appropriate.

minor comments (4)

[Figure 4 caption] Please explain in the main text how w=0.52 is chosen in panel (b); the caption merely states the value.
[§4.2, family (9)] Consider adding a sentence that (9) is a subfamily of the zero-shot optima, not necessarily the full set, to avoid overclaiming.
[References] The MMLU experiments cite Achiam et al. (2023) for gpt-4.1-mini; this reference is the GPT-4 technical report and likely does not describe gpt-4.1-mini. Please use the appropriate model documentation.
[Table 2 / §5.2] The notation 'ZS fine-tuning on Q/K/V' is a bit cryptic when first used; a one-line definition would improve readability.

Circularity Check

1 steps flagged · score 4.0 of 10

Core derivations are self-contained and do not rest on self-citation; the main circularity concern is the quantitative full-fine-tuning curve in Fig. 4(b), whose free parameter w=0.52 is not predicted by the theory.

fitted input called prediction [Section 5.1 / Figure 4 caption, with Theorem 4.3 and Corollary 4.4]
"Curves show theoretical predictions from Section 4, while points correspond to models trained empirically in Section 5.1. Empirical results match the theoretical values. The parameters used for the theoretical predictions are given by: ... (b) (9) from Theorem 4.3 with w= 0.52."

Theorem 4.3 states that its displayed parameters globally minimize the zero-shot loss 'for any w > 0'; the zero-shot objective does not determine w. The paper provides no derivation of w=0.52 from the pretrained initialization or the optimizer. Therefore the Fig. 4(b) curve is not a parameter-free prediction: w is a free label of a one-parameter family, and choosing w=0.52 after seeing the trained model makes the quantitative agreement a consistency check rather than a test. The qualitative ordering (zero-shot error sigma^2 < n-shot error, with limit sigma^2 + theta0^T Sigma theta0) is w-independent, so the circularity is confined to this quantitative empirical curve, not to the main theorem.

full rationale

The central mathematical claims are derived in a self-contained way from the explicit loss expressions in Lemma A.5 and Appendix A. The pretraining solution is imported from Ahn et al. (2023), which is an external, non-self citation and is used with stated assumptions; it is not a load-bearing self-citation. Theorem 4.3 correctly shows that the displayed family attains the Bayes lower bound sigma^2, so those parameters are global minimizers of the zero-shot loss; there is no circularity in that argument, though the paper does not characterize all global minimizers. Corollary 4.4's qualitative prediction that full fine-tuning degrades few-shot performance holds uniformly over the free parameter w, so the main finding does not reduce to a fitted value. The value-matrix fine-tuning results fix w by the minimal-update rule (Proposition 4.7) or by explicit optimization (Corollary 4.8, Theorem 4.9), so those predictions are parameter-free given the stated conventions. The only concrete circular element is the quantitative full-fine-tuning curve in Figure 4(b), where w=0.52 is used without derivation from the theory; this makes that particular empirical match partially constructed. Overall, the derivation chain is largely independent, with one underdetermined validation curve, so the circularity score is moderate rather than severe.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

All theoretical results depend on global minimization of expected losses in the linear-attention model; the empirical full-FT comparison additionally relies on an unspecified choice of the degenerate parameter w. No new physical or architectural entities are introduced beyond the existing linear-attention parameterization.

free parameters (2)

w (full fine-tuning) = 0.52 in Fig. 4(b); otherwise arbitrary w>0
The zero-shot loss in Theorem 4.3 admits a one-parameter family of global minimizers. The paper does not derive w from the fine-tuning objective, so the exact few-shot curve used to match empirical data in Fig. 4(b) is not predicted.
w (value-matrix fine-tuning) = m/(m+1+d) ≈ 0.77 in synthetic experiments
The value-matrix zero-shot optimum in Theorem 4.6 also has a free w. It is resolved by a minimal-update heuristic (Proposition 4.7) rather than by the objective; this is a modeling choice, not a prediction.

assumptions (4)

domain assumption One-layer linear self-attention with a merged query-key matrix is an adequate proxy for Transformer in-context learning and fine-tuning behavior.
The entire paper works with the model (1); the LLM experiments are described as illustrative, so the transfer from linear attention to softmax attention is assumed, not proven.
domain assumption Pretraining reaches the global minimizer of the m-shot loss, so Q11 → Σ⁻¹ as m→∞.
Corollary 4.1 imports the global-minimizer result from Ahn et al. (2023); the paper then fixes Q at this optimum for value-matrix fine-tuning. Real pretraining may not reach this exact solution.
domain assumption Fine-tuning minimizes the exact expected loss and reaches a global minimizer, with no regularization or early stopping.
Theorem 4.3 and Theorem 4.6 characterize global optima of the population zero-shot loss; the claim that full fine-tuning destroys ICL relies on Q11 actually being driven to zero.
standard math Wishart moment identities (Hagedorn/Kollo-von Rosen), Isserlis' theorem, and Magnus-Neudecker formulas hold as stated.
The proofs in Appendix A use these results without derivation; they are standard, but the paper provides no machine-checked verification.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models." pith.science (2026). https://pith.science/paper/RPRGYRJ7

@misc{pith2026260223197,
  author       = {Pith},
  title        = {Pith review of: Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RPRGYRJ7}},
  note         = {Machine review of arXiv:2602.23197}
}

read the original abstract

Transformer-based large language models exhibit in-context learning, enabling adaptation to downstream tasks via few-shot prompting with demonstrations. In practice, such models are often fine-tuned to improve zero-shot performance on downstream tasks, allowing them to solve tasks without examples and thereby reducing inference costs. However, fine-tuning can degrade in-context learning, limiting the performance of fine-tuned models on tasks not seen during fine-tuning. Using linear attention models, we provide a theoretical analysis that characterizes how fine-tuning objectives modify attention parameters and identifies conditions under which this leads to degraded few-shot performance. We show that fine-tuning all attention parameters can harm in-context learning, whereas restricting updates to the value matrix improves zero-shot performance while preserving in-context learning. We further show that incorporating an auxiliary few-shot loss enhances in-context learning primarily on the target task, at the expense of degraded in-context learning ability on tasks not seen during fine-tuning. We provide empirical evidence from synthetic and real-world datasets consistent with the qualitative predictions of our theory.

Figures

Figures reproduced from arXiv: 2602.23197 by the authors.

**Figure 1.** n-shot test error on the target task θ0 of the pretrained model using the theory-derived parameters in Theorem 4.1. Each curve corresponds to a model pretrained with context length m = 1000 and a different input dimension d. We set σ 2 = 0, under which the condition in (8) predicts that the n-shot performance is worse than zero-shot performance for n ranging from 1 to d − 2 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

**Figure 2.** n-shot test error on the target task θ0 of the fully finetuned model using the theory-derived parameters in Theorem 4.3. Each curve corresponds to a model with a different choice of the parameter w, evaluated under σ 2 = 0.1 and θ ⊤ 0 Σθ0 = 1. For all values of w, the zero-shot error is 0.1, whereas the few-shot error is larger than the zero-shot error and converges to 1 as n → ∞ [PITH_FULL_IMAGE:figures/full_fig_… view at source ↗

**Figure 3.** (Top) n-shot test error on the target task θ0 for the value-matrix fine-tuned model with the theory-derived parameters [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: n-shot test errors on the target task θ0 for linear attention models under different training regimes. Curves show theoretical predictions from Section 4, while points correspond to models trained empirically in Section 5.1. Empirical results match the theoretical valu…

**Figure 5.** Figure 5: n-shot test errors evaluated on the in-distribution task θ0 (Left) and the out-of-distribution task −θ0 (Right) . Empty circles denote models fine-tuned with the zero-shot (ZS) loss only, whereas filled circles denote models fine-tuned with both the zeroshot and few-s…

**Figure 7.** Figure 7: Test errors on (Left) the in-distribution task θ0 and (Right) the out-of-distribution task −θ0, using a 1-head attention model. 0 10 20 10 1 10 0 10 1 10 2 Test error evaluate on the task 0 0 10 20 evaluate on the task 0 Number of shots (n) ZS (all) ZS+FS (all) ZS (Q) …

**Figure 8.** Figure 8: Test errors on (Left) the in-distribution task θ0 and (Right) the out-of-distribution task −θ0, using a 2-head attention model. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates
cs.LG 2026-05 unverdicted novelty 5.0 of 10

FINCH is a loss-adaptive learning-rate schedule that reduces forgetting by 93% on average during LLM fine-tuning while matching standard task performance across several benchmarks.

Reference graph

Works this paper leans on

17 extracted references · 8 linked inside Pith · cited by 1 Pith paper

[1]

L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

arXiv
[2]

Proofs for Optimal Parameters Corollary A.7(Optimal Parameters of Pretrained Models; Corollary of Theorem 1 in Ahn et al

20 Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models A.3. Proofs for Optimal Parameters Corollary A.7(Optimal Parameters of Pretrained Models; Corollary of Theorem 1 in Ahn et al. (2023)).Suppose that V and Q in (1) satisfy q≥0 and that Q11 is positive definite. Consider the loss L(V,Q) in (4) with conte...

2023
[4]

30 Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models B

4η∥θ0∥2 2 2 ∥θ0∥2 2 = 16(d+ 2)(d+ 4)η 2∥θ0∥6 2 >0, and by continuity the same holds in a neighborhood of(1,1,0). 30 Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models B. Experimental Details We provide additional details on the experimental setup and supplementary results for the linear regression tasks i...

2023
[7]

Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971,

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971,

arXiv
[8]

On the loss of context-awareness in general instruction fine-tuning

Wang, Y ., Bai, A., Peng, N., and Hsieh, C.-J. On the loss of context-awareness in general instruction fine-tuning. arXiv preprint arXiv:2411.02688, 2024a. Wang, Y ., Si, S., Li, D., Lukasik, M., Yu, F., Hsieh, C.- J., Dhillon, I. S., and Kumar, S. Two-stage LLM fine- tuning with less specialization and more generalization. InThe Twelfth International Con...

arXiv
[9]

Qwen2.5 technical report

Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Xia, T., Ren, X., Ren, X., Fan, Y ., Su, Y ., Zhang, Y ., Wan, Y ., Liu, Y ....

arXiv
[10]

Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y ., Min, Y ., Zhang, B., Zhang, J., Dong, Z., Du, Y ., Yang, C., Chen, Y ., Chen, Z., Jiang, J., Ren, R., Li, Y ., Tang, X., Liu, Z., Liu, P., Nie, J.-Y ., and Wen, J.-R. A survey of large language models.arXiv preprint arXiv:2303.18223,

arXiv
[11]

Seq2sql: Generating structured queries from natural language using reinforce- ment learning.arXiv preprint arXiv:1709.00103,

Zhong, V ., Xiong, C., and Socher, R. Seq2sql: Generating structured queries from natural language using reinforce- ment learning.arXiv preprint arXiv:1709.00103,

arXiv

Show all 17 references

[13]

Therefore, it suffices to determine their second moments, as they coincide with the (co)variances. 14 Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models From the definition ofˆyNet-FS(Z[n],x n+1;V,Q)in (16), we have Var ˆyNet-...

1918
[15]

(2023), because ˆyFS(Zm,x;V,Q)≈ˆy Net-FS(Zm,x;V,Q) = v21 v22 ⊤ 1 m X i∈[m] xi yi xi yi ⊤ Q11 q⊤ 21 xn+1

Whenq= 0andmis sufficiently large, the prediction reduces to that of Theorem 1 in Ahn et al. (2023), because ˆyFS(Zm,x;V,Q)≈ˆy Net-FS(Zm,x;V,Q) = v21 v22 ⊤ 1 m X i∈[m] xi yi xi yi ⊤ Q11 q⊤ 21 xn+1. By Theorem 1 in Ahn et al. (2023), the following parameters globally minimize t...

2023
[17]

For each fine-tuning instance, we use gpt-4.1-mini (Achiam et al.,

on theHumanitiessubset of MMLU (Hendrycks et al., 2021), which contains 1,000 training examples. For each fine-tuning instance, we use gpt-4.1-mini (Achiam et al.,

2021
[2017]

Proofs A.1

11 Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models A. Proofs A.1. Proofs of Preliminary Lemma Lemma A.1(Theorem 1 in Hagedorn (2022); specialized version of Theorem 2.2.9 (ii) in Kollo & V on Rosen (2005)).Let Q∼ W(Σ, n)be ...

2022
[2020]

Why can GPT learn in-context? language models implicitly perform gradient descent as meta-optimizers

Dai, D., Sun, Y ., Dong, L., Hao, Y ., Ma, S., Sui, Z., and Wei, F. Why can GPT learn in-context? language models implicitly perform gradient descent as meta-optimizers. InICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models,

2023
[2022]

Gemini: a family of highly capable multi- modal models.arXiv preprint arXiv:2312.11805,

Gemini Team. Gemini: a family of highly capable multi- modal models.arXiv preprint arXiv:2312.11805,

arXiv
[2023]

BERT: Pre-training of deep bidirectional transformers for lan- guage understanding

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. InProceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- ...

2019
[2024]

Rethinking the role of demonstrations: What makes in-context learning work? InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, December

Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, December

2022
[2025]

Expected value of matrix quadratic forms with wishart distributed random matrices.arXiv preprint arXiv:2212.01412,

Hagedorn, M. Expected value of matrix quadratic forms with wishart distributed random matrices.arXiv preprint arXiv:2212.01412,

arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

arXiv

[2] [2]

Proofs for Optimal Parameters Corollary A.7(Optimal Parameters of Pretrained Models; Corollary of Theorem 1 in Ahn et al

20 Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models A.3. Proofs for Optimal Parameters Corollary A.7(Optimal Parameters of Pretrained Models; Corollary of Theorem 1 in Ahn et al. (2023)).Suppose that V and Q in (1) satisfy q≥0 and that Q11 is positive definite. Consider the loss L(V,Q) in (4) with conte...

2023

[3] [4]

30 Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models B

4η∥θ0∥2 2 2 ∥θ0∥2 2 = 16(d+ 2)(d+ 4)η 2∥θ0∥6 2 >0, and by continuity the same holds in a neighborhood of(1,1,0). 30 Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models B. Experimental Details We provide additional details on the experimental setup and supplementary results for the linear regression tasks i...

2023

[4] [7]

Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971,

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971,

arXiv

[5] [8]

On the loss of context-awareness in general instruction fine-tuning

Wang, Y ., Bai, A., Peng, N., and Hsieh, C.-J. On the loss of context-awareness in general instruction fine-tuning. arXiv preprint arXiv:2411.02688, 2024a. Wang, Y ., Si, S., Li, D., Lukasik, M., Yu, F., Hsieh, C.- J., Dhillon, I. S., and Kumar, S. Two-stage LLM fine- tuning with less specialization and more generalization. InThe Twelfth International Con...

arXiv

[6] [9]

Qwen2.5 technical report

Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Xia, T., Ren, X., Ren, X., Fan, Y ., Su, Y ., Zhang, Y ., Wan, Y ., Liu, Y ....

arXiv

[7] [10]

Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y ., Min, Y ., Zhang, B., Zhang, J., Dong, Z., Du, Y ., Yang, C., Chen, Y ., Chen, Z., Jiang, J., Ren, R., Li, Y ., Tang, X., Liu, Z., Liu, P., Nie, J.-Y ., and Wen, J.-R. A survey of large language models.arXiv preprint arXiv:2303.18223,

arXiv

[8] [11]

Seq2sql: Generating structured queries from natural language using reinforce- ment learning.arXiv preprint arXiv:1709.00103,

Zhong, V ., Xiong, C., and Socher, R. Seq2sql: Generating structured queries from natural language using reinforce- ment learning.arXiv preprint arXiv:1709.00103,

arXiv