Pith. sign in

REVIEW 4 major objections 5 minor 44 references

Can Gradient Descent Simulate Prompting?

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Meta-training makes a single gradient step behave like prompting.

desk verdict A fresh inverse of ICL-as-GD with real within-task evidence, but the 'surprisingly expressive' claim needs a held-out test; send to a serious referee. read the letter →

arxiv 2506.20989 v1 pith:TANRDYDQ submitted 2025-06-26 cs.CL cs.LG

Eric Zhang , Leshem Choshen , Jacob Andreas This is my paper
classification cs.CLcs.LG
keywords meta-learningpromptingfine-tuninggradientdescentlanguagemodelsreversalcursein-contextlearningMAML
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether fine-tuning can be made to work like prompting: a model that, after one gradient step on a new piece of text, answers questions about that text as well as if the text had been put in the prompt. The authors propose a meta-learning procedure that trains the model's initialization so that a single gradient descent step on a context reproduces the predictions the model would make if that context were in the prompt. The targets for this meta-training come from the model's own prompted predictions, so no ground-truth labels are needed. Across reversal-curse style tasks, passage QA, and language modeling, the procedure recovers some, and occasionally all, of the prompted model's performance. If the claim holds, gradient-based learning is far more expressive than standard fine-tuning experiments suggest, and parameter updates could serve as a low-cost alternative to long prompts.

What carries the argument

The load-bearing object is the meta-learning objective in Equation 8: a teacher LM with frozen parameters $\theta_B$ conditions on the context to produce a target distribution, and a student initialization $\theta^*$ is trained so that after one inner-loop gradient step $\theta^* - \eta \nabla_\theta \mathcal{L}(\text{context}, \theta^*)$ (Equation 5) it matches that distribution on the query. The teacher's own greedy-decoded continuations serve as labels, replacing the ground-truth answers that MAML ordinarily requires (Equation 9 is an oracle upper bound using gold labels). The outer loop optimizes the sum of KL divergences between the conditioned teacher and the post-update student, plus a language-modeling regularizer that preserves general capabilities. This construction converts 'prompting behaves like an update' from a metaphor into a training signal.

What would settle it

Train the meta-learning procedure on a synthetic task where the correct answer to a query is computable from the context only through a multi-step logical rule (e.g., transitive closure: 'a<b', 'b<c' implies a<c), and measure whether one gradient step on the context recovers the prompted answer. If, with ample meta-training data, the post-update accuracy stays at chance, the claim that gradient descent can simulate prompting is false for that class of reasoning.

Watch

Extended reading notes

Core claim

The central claim is that after meta-training with the objective in Equation 8, a single gradient step of the next-token prediction loss on a raw context (the inner loop) can emulate conditioning on that context. Concretely, the meta-trained Llama 3.2 1B model improves from 47.3% to 58.6% fine-tune accuracy on SQuAD after one update (versus 87.7% with prompting), recovers roughly half the gap on WikiText, and achieves high accuracy on both directions of the reversal-curse task. The procedure also finds that a rank-1 LoRA update in the inner loop is sufficient for much of the gain, and that models meta-trained on one dataset do not transfer the capability to another. The paper's own framing is that with appropriate initialization, gradient descent can be surprisingly expressive, and that this points toward new approaches to long-context modeling.

Load-bearing premise

The method depends on a single gradient step of ordinary next-token prediction loss being able to carry the facts or instructions contained in a piece of text after meta-training; if that update mechanism cannot express the needed information, the emulation fails.

Editorial extensions

If this is right

  • If the effect holds, one documented fact or instruction could be injected into a model by a single fine-tuning step, eliminating the inference-time cost and context-window limit of prompting.
  • Meta-trained models could make fine-tuning competitive with prompting for single-example knowledge updates, at least on tasks within the training distribution.
  • The success of rank-1 updates suggests that low-rank adapters can carry the expressive information needed for prompt-like generalization, making the approach practical for large models.
  • The failure of cross-dataset transfer implies that the ability to 'learn by prompting' is itself task-specific and must be meta-trained per domain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit: the same objective could meta-train for multi-step or composed updates, where several gradient steps must collectively emulate a longer prompt; the paper's multi-context results suggest this is far from solved.
  • If single-step gradient descent can encode the content of a prompt, then the effective 'context window' of gradient-based methods is bounded only by what one update can express — a hypothesis that could be tested with synthetic key-value stores of increasing size.
  • The method's label-free distillation could be applied beyond question-answering, e.g., to instruction following or preference alignment, wherever prompted behavior is better than fine-tuned behavior.
  • The lack of transfer across datasets raises the question of whether a scale-up in meta-training diversity (as the authors hypothesize) would actually yield a general 'gradient-descent learns like prompting' capability, or whether the effect is intrinsically task-local.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a meta-learning method to make a single gradient descent step on a new context behave like conditioning on that context. The authors use a teacher language model's own prompted predictions as targets (Eq. 8), avoiding ground-truth labels, and compare against a gold-label oracle meta-learner (Eq. 9). Experiments on Character Description, Reversal Curse, SQuAD, and WikiText show that the label-free objective (LC) closely tracks the gold-label objective (LG) and improves fine-tuning accuracy relative to the base model, with control experiments indicating the improvement is often context-dependent. However, results are partial on harder tasks, cross-task transfer is negligible, and several implementation details are underspecified.

Significance. If the central claim holds, the method provides a new way to distill context-based behavior into parameters and adds evidence that gradient updates can be expressive beyond simple memorization. The paper's strengths include a simple self-supervised objective, a clean comparison with an oracle upper bound, control experiments with irrelevant contexts and with direct answering, and the surprising finding that a rank-1 update suffices for the studied tasks. The limitations are equally clear: the method is demonstrated only in-distribution, with one model and one epoch, and the abstract's general phrasing ('gradient descent can be surprisingly expressive') goes beyond what the experiments currently support.

major comments (4)
  1. [§5.5, Table 3] The cross-dataset results show negligible transfer from WikiText meta-training to SQuAD (47.8 and 48.0 vs. 47.3 base FT, compared to 58.6 in-domain). Since the central claim is that gradient descent can emulate the semantic effect of conditioning on new contexts, the absence of any held-out template or domain test is a load-bearing gap. The current evidence supports in-distribution adaptation only, and the abstract's 'surprisingly expressive' claim needs to be tempered or accompanied by a held-out generalization experiment.
  2. [§4.3, §4.2] The 'fine-tuning preparation' step is underspecified: the paper does not state which subset of each dataset is used, how many examples are selected, or what exactly 'Prompt and NC configurations' means. In addition, Figure 2 reports standard errors below 2% but does not state the number of random seeds or runs; only one epoch and one model (Llama 3.2 1B) are described. These omissions make the main results difficult to reproduce and to assess for variance.
  3. [§4.1, Figure 2] The sentence 'the original fine-tuning accuracy of the base model serves as an upper bound for the effectiveness of our method' is the opposite of what the experiments show: the base model's fine-tuning accuracy is the baseline that LC and LG attempt to improve, while the prompted accuracy is the upper bound. This wording obscures the intended comparison and should be corrected.
  4. [§5.1, §5.5] The SQuAD result recovers only about a quarter of the prompted gap, and the authors state the meta-training loss was still decreasing when training data ran out. Combined with the lack of cross-task transfer, this suggests the learned update is task-specific and possibly underfit rather than a general capability to encode context semantics. The paper should either provide additional evidence (e.g., scaling the meta-training data or testing a held-out task template) or explicitly restrict the claims to the in-distribution setting.
minor comments (5)
  1. [Table 4] The column header 'Base Base Step ML ML Step' is ambiguous; clarify which entries correspond to the base model with and without the gradient step and which correspond to the meta-trained model.
  2. [Table 5] The WikiText row under 'Irrelevant Context' shows the value '.010', which appears to be a typo or leftover placeholder; report the actual accuracy.
  3. [§5.3] When describing the irrelevant-context control, specify whether the randomly sampled context is a different passage from the same distribution but paired with the original question, or a completely unrelated context; this affects the interpretation of the performance drop.
  4. [§3.2, Eq. (8)] The use of greedy decoding from the teacher as the only targets for the KL objective is an important design choice; state explicitly that this is an approximation and discuss why greedy outputs are sufficient rather than sampling multiple continuations.
  5. [§4.1] The generated datasets for Character Description and Reversal Curse are described but not released; the paper should state whether code and data will be made available.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the meta-training objective is self-referential, but the reported results are evaluated against held-out ground truth.

full rationale

I walked the derivation chain from the inner-loop update (Eq. 5) through the meta-training objective (Eq. 8) to the reported evaluations. Eq. 8 does define the goal that a gradient update on a context emulates conditioning, using the same model's prompted predictions as targets; on training contexts, this match is literally the optimization objective. However, the paper's central claims are not that this objective is satisfied, but that the meta-trained initialization produces correct answers on held-out test contexts after a single gradient step. Those claims are supported by ground-truth accuracy in Figure 2 and Tables 1-5, which are not forced by Eq. 8. Table 3's lack of cross-task transfer and the partial SQuAD recovery further show that the effect is not trivially guaranteed by construction. The self-citations in Related Work are not load-bearing, and the format warm-start in Section 4.3 is a preprocessing step, not a circular prediction. The self-referential training signal is a mild design choice, but the evaluation against external ground truth keeps the derivation non-circular.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or ontological entities are introduced; the meta-learned initialization is a parameter configuration, not an invented entity in the ledger sense. The method relies on the standard LM next-token prediction loss and standard MAML machinery, plus the domain assumptions listed above.

free parameters (3)
  • eta (inner learning rate) = 0.001
    Fixed across tasks; chosen because it works with and without meta-training (Section 4.2). This controls the size of the simulated prompting update and is central to the method.
  • outer loop learning rate = not reported (tuned per task)
    Tuned on held-out data for each task (Section 4.2). The final values are not listed, affecting reproducibility.
  • lambda (LM loss weight) = not reported
    Weight of the language modeling loss in Equations 7 and 8; no value given in the paper.
assumptions (4)
  • domain assumption The teacher model's prompted distribution P_thetaB(·|c_i ⊕ q_i) is a reliable target for the behavior to be emulated.
    The method's training signal is the teacher's own prompted outputs (Eq. 8); if prompting is inaccurate, the student inherits those errors.
  • domain assumption A single gradient step on the context, using the standard next-token prediction loss, is sufficiently expressive to encode the information needed to answer queries, given a suitable initialization.
    This is the core mechanism tested in all experiments (Eq. 5); the SQuAD results show it only partially holds.
  • domain assumption Meta-training on a finite set of in-distribution contexts generalizes to new contexts from the same distribution.
    The method is evaluated on held-out contexts from the same dataset; cross-dataset transfer fails (Section 5.5).
  • ad hoc to paper Greedy decoding from the teacher provides a sufficient set of prefix targets for the KL objective in Eq. 8.
    The authors use greedy decoding to avoid sampling multi-token continuations, a design choice specific to this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Gradient Descent Simulate Prompting?." pith.science (2026). https://pith.science/paper/TANRDYDQ

@misc{pith2026250620989,
  author       = {Pith},
  title        = {Pith review of: Can Gradient Descent Simulate Prompting?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TANRDYDQ}},
  note         = {Machine review of arXiv:2506.20989}
}
read the original abstract

There are two primary ways of incorporating new information into a language model (LM): changing its prompt or changing its parameters, e.g. via fine-tuning. Parameter updates incur no long-term storage cost for model changes. However, for many model updates, prompting is significantly more effective: prompted models can generalize robustly from single examples and draw logical inferences that do not occur under standard fine-tuning. Can models be modified so that fine-tuning does emulate prompting? This paper describes a method for meta-training LMs such that gradient updates emulate the effects of conditioning on new information. Our approach uses tools from gradient-based meta-learning but uses an LM's own prompted predictions as targets, eliminating the need for ground-truth labels. Subsequent gradient descent training recovers some (and occasionally all) of prompted model performance -- showing improvement on the ``reversal curse'' tasks, and answering questions about text passages after a single gradient update. These results suggest that, with appropriate initialization, gradient descent can be surprisingly expressive. Our results suggest new avenues for long-context modeling and offer insight into the generalization capabilities of gradient-based learning.

Figures

Figures reproduced from arXiv: 2506.20989 by the authors.

Figure 1
Figure 1. Meta-training a model to emulate conditioning with fine-tuning. In the low-sample or 1-sample regime, fine-tuning can fail due to a variety of reasons, including the inability to generalize, the inability to handle reversed phrasing, and difficulty overriding existing priors. On the other hand, models are typically able to use information presented in prompts. We investigate a meta-training procedure to distill the … view at source ↗
Figure 2
Figure 2. Performance of models across different training strategies and tasks. We evaluate the base model (BM), the model after meta-learning from conditioning procedure (LC), and meta-learning from gold labels (LG). We evaluate each task in three different settings. In the No Context (NC) setting, the model is asked to perform the task without any context. This is a lower bound for the performance of our method. In the prom… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 15 canonical work pages

  1. [1]

    Transformers learn to implement preconditioned gradient descent for in-context learning

    Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. ArXiv, abs/2306.00297, 2023. URL https://api.semanticscholar.org/CorpusID:258999480

    arXiv 2023

  2. [2]

    u rek, Ekin Aky \

    Afra Feyza Aky \"u rek, Ekin Aky \"u rek, Leshem Choshen, Derry Tanti Wijaya, and Jacob Andreas. Deductive closure training of language models for coherence, accuracy, and updatability. In Annual Meeting of the Association for Computational Linguistics, 2024. URL https://api.semanticscholar.org/CorpusID:267028613

    work page 2024

  3. [3]

    What learning algorithm is in-context learning? investigations with linear models

    Ekin Aky \"u rek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. ArXiv, abs/2211.15661, 2022. URL https://api.semanticscholar.org/CorpusID:254043800

    arXiv 2022

  4. [4]

    Hoffman, David Pfau, Tom Schaul, and Nando de Freitas

    Marcin Andrychowicz, Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In Neural Information Processing Systems, 2016. URL https://api.semanticscholar.org/CorpusID:2928017

    work page 2016

  5. [5]

    Antreas Antoniou, Harrison Edwards, and Amos J. Storkey. How to train your maml. ArXiv, abs/1810.09502, 2018. URL https://api.semanticscholar.org/CorpusID:53036488

    arXiv 2018

  6. [6]

    Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova Dassarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, John Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a l...

    arXiv 2021

  7. [7]

    a is b" fail to learn

    Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on "a is b" fail to learn "b is a". ArXiv, abs/2309.12288, 2023. URL https://api.semanticscholar.org/CorpusID:262083829

    arXiv 2023

  8. [8]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz Litwin, ...

    arXiv 2005

Show all 44 references
  1. [9]

    Editing factual knowledge in language models

    Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Conference on Empirical Methods in Natural Language Processing, 2021. URL https://api.semanticscholar.org/CorpusID:233289412

    2021

  2. [10]

    Meta-learning via language model in-context tuning

    Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via language model in-context tuning. ArXiv, abs/2110.07814, 2021. URL https://api.semanticscholar.org/CorpusID:239009828

    2021 arXiv

  3. [11]

    Prompt injection: Parameterization of fixed inputs

    Eunbi Choi, Yongrae Jo, Joel Jang, and Minjoon Seo. Prompt injection: Parameterization of fixed inputs. ArXiv, abs/2206.11349, 2022. URL https://api.semanticscholar.org/CorpusID:249953762

    2022 arXiv

  4. [12]

    Fusing finetuned models for better pretraining

    Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining. arXiv preprint arXiv:2204.03044, 2022

    2022 arXiv

  5. [13]

    Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and P. Abbeel. Model-based reinforcement learning via meta-policy optimization. In Conference on Robot Learning, 2018. URL https://api.semanticscholar.org/CorpusID:52282277

    2018

  6. [14]

    Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers. ArXiv, abs/2212.10559, 2023. URL https://api.semanticscholar.org/CorpusID:254877715

    2023 arXiv

  7. [15]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston ...

    2024 arXiv

  8. [16]

    Abbeel, and Sergey Levine

    Chelsea Finn, P. Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017. URL https://api.semanticscholar.org/CorpusID:6719686

    2017

  9. [17]

    Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685, 2021. URL https://api.semanticscholar.org/CorpusID:235458009

    2021 arXiv

  10. [18]

    Beneath the surface of consistency: Exploring cross-lingual knowledge representation sharing in llms

    Maxim Ifergan, Leshem Choshen, Roee Aharoni, Idan Szpektor, and Omri Abend. Beneath the surface of consistency: Exploring cross-lingual knowledge representation sharing in llms. arXiv preprint arXiv:2408.10646, 2024

    2024 arXiv

  11. [19]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. URL https://api.semanticscholar.org/CorpusID:6628106

    2014 arXiv

  12. [20]

    Meta-sgd: Learning to learn quickly for few shot learning

    Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. ArXiv, abs/1707.09835, 2017. URL https://api.semanticscholar.org/CorpusID:25316837

    2017 arXiv

  13. [21]

    Scaling down to scale up: A guide to parameter-efficient fine-tuning

    Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: A guide to parameter-efficient fine-tuning. ArXiv, abs/2303.15647, 2023. URL https://api.semanticscholar.org/CorpusID:257771591

    2023 arXiv

  14. [22]

    Taming maml: Efficient unbiased meta-reinforcement learning

    Hao Liu, Richard Socher, and Caiming Xiong. Taming maml: Efficient unbiased meta-reinforcement learning. In International Conference on Machine Learning, 2019. URL https://api.semanticscholar.org/CorpusID:174800385

    2019

  15. [23]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. ArXiv, abs/2205.05638, 2022. URL https://api.semanticscholar.org/CorpusID:248693283

    2022 arXiv

  16. [24]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In Neural Information Processing Systems, 2022 a . URL https://api.semanticscholar.org/CorpusID:255825985

    2022

  17. [25]

    Mass-editing memory in a transformer

    Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. ArXiv, abs/2210.07229, 2022 b . URL https://api.semanticscholar.org/CorpusID:252873467

    2022 arXiv

  18. [26]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. ArXiv, abs/1609.07843, 2016. URL https://api.semanticscholar.org/CorpusID:16299141

    2016 arXiv

  19. [27]

    Metaicl: Learning to learn in context

    Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. ArXiv, abs/2110.15943, 2021. URL https://api.semanticscholar.org/CorpusID:240288835

    2021 arXiv

  20. [28]

    Rethinking the role of demonstrations: What makes in-context learning work? ArXiv, abs/2202.12837, 2022

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? ArXiv, abs/2202.12837, 2022. URL https://api.semanticscholar.org/CorpusID:247155069

    2022 arXiv

  21. [29]

    From insights to actions: The impact of interpretability and analysis research on nlp

    Marius Mosbach, Vagrant Gautam, Tom \'a s Vergara-Browne, Dietrich Klakow, and Mor Geva. From insights to actions: The impact of interpretability and analysis research on nlp. arXiv preprint arXiv:2406.12618, 2024

    2024 arXiv

  22. [30]

    D isent QA : Disentangling parametric and contextual knowledge with counterfactual question answering

    Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. D isent QA : Disentangling parametric and contextual knowledge with counterfactual question answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61s...

    2023 doi

  23. [32]

    On first-order meta-learning algorithms

    Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. ArXiv, abs/1803.02999, 2018. URL https://api.semanticscholar.org/CorpusID:4587331

    2018 arXiv

  24. [33]

    Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova Dassarma, Tom Henighan, Benjamin Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, John Kernion, Liane Lovitt, K...

    2022 arXiv

  25. [34]

    Zhang, Eunsol Choi, and Greg Durrett

    Yasumasa Onoe, Michael J.Q. Zhang, Eunsol Choi, and Greg Durrett. Entity cloze by date: What lms know about unseen entities. In NAACL-HLT, 2022. URL https://api.semanticscholar.org/CorpusID:248525074

    2022

  26. [35]

    Kakade, and Sergey Levine

    Aravind Rajeswaran, Chelsea Finn, Sham M. Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Neural Information Processing Systems, 2019. URL https://api.semanticscholar.org/CorpusID:202542766

    2019

  27. [36]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing, 2016. URL https://api.semanticscholar.org/CorpusID:11816014

    2016

  28. [37]

    Larochelle

    Sachin Ravi and H. Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations, 2016. URL https://api.semanticscholar.org/CorpusID:67413369

    2016

  29. [38]

    forgetting

    A. Sha, Bernardo Pereira Nunes, and Armin Haller. "forgetting" in machine learning and beyond: A survey. ArXiv, abs/2405.20620, 2024. URL https://api.semanticscholar.org/CorpusId:270199793

    2024 arXiv

  30. [39]

    Maml-en-llm: Model agnostic meta-training of llms for improved in-context learning

    Sanchit Sinha, Yuguang Yue, Victor Soto, Mayank Kulkarni, Jianhua Lu, and Aidong Zhang. Maml-en-llm: Model agnostic meta-training of llms for improved in-context learning. ArXiv, abs/2405.11446, 2024. URL https://api.semanticscholar.org/CorpusID:269921787

    2024 arXiv

  31. [40]

    Pyrkin, Sergei Popov, and Artem Babenko

    Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy V. Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. ArXiv, abs/2004.00345, 2020. URL https://api.semanticscholar.org/CorpusID:213938729

    2004 arXiv

  32. [41]

    Randazzo, Jo \ a o Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov

    Johannes von Oswald, Eyvind Niklasson, E. Randazzo, Jo \ a o Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, 2022. URL https://api.semanticscholar.org/C...

    2022

  33. [42]

    Towards zero-label language learning

    Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. Towards zero-label language learning. ArXiv, abs/2109.09193, 2021. URL https://api.semanticscholar.org/CorpusID:237572306

    2021 arXiv

  34. [43]

    An explanation of in-context learning as implicit bayesian inference

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. ArXiv, abs/2111.02080, 2021. URL https://api.semanticscholar.org/CorpusID:241035330

    2021 arXiv

  35. [44]

    Unforgettable generalization in language models, 2024

    Eric Zhang, Leshem Chosen, and Jacob Andreas. Unforgettable generalization in language models, 2024. URL https://arxiv.org/abs/2409.02228

    2024 arXiv

  36. [45]

    Yu, and Sanjiv Kumar

    Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix X. Yu, and Sanjiv Kumar. Modifying memories in transformer models. ArXiv, abs/2012.00363, 2020. URL https://api.semanticscholar.org/CorpusID:227238659

    2012 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.