Pith. sign in

REVIEW 4 major objections 6 minor 42 references

Efficient Knowledge Feeding to Language Models: A Novel Integrated Encoder-Decoder Architecture

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A compact vector added to attention keys can replace demonstration examples in retrieval-augmented generation.

desk verdict The in-context vector idea is a reasonable recombination of existing activation-editing work, but the paper's own numbers contradict its headline claim and the method is too underspecified to reproduce. read the letter →

arxiv 2502.05233 v1 pith:FWH7CIXQ submitted 2025-02-07 cs.CL cs.IR

S Santosh Kumar , Rishi Gottimukkala , Supriya Devidutta , Karthikeyan S This is my paper · ORCID
classification cs.CLcs.IR
keywords in-contextlearningretrieval-augmentedgenerationvectorscross-attentionencoder-decoderarchitecturequestionansweringlatentembeddingsknowledgeintegration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single compact vector, called an in-context vector (ICV), can carry the task information that in-context learning normally gets from demonstration examples, and that injecting this vector into cross-attention lets a retrieval-augmented generator answer questions without paying the token cost of a long prompt. The paper proposes an integrated encoder-decoder architecture in which the ICV is pooled from latent embeddings of example demonstrations and added to the key matrix in cross-attention. If the claim holds, retrieval-augmented systems could feed large amounts of knowledge into a model with a fixed, short context and a fraction of the parameters of large LLMs, while staying competitive on question-answering benchmarks. The experimental evidence is most direct on retrieval accuracy and on the multi-hop HotpotQA dataset, where the ICV model reports the highest Exact Match.

What carries the argument

The load-bearing object is the in-context vector (ICV), a single vector obtained by running demonstration examples through the model and pooling their latent embeddings. Its operation site is the cross-attention formula $A = \operatorname{softmax}(Q(K + v_{\mathrm{ICV}})^\top / \sqrt{d_k})$, where the vector is added to every key before the softmax. The paper's argument is that this additive shift aligns the query context with relevant document vectors and thereby feeds retrieved knowledge into the decoder without concatenating tokens to the prompt.

What would settle it

A controlled ablation that removes $v_{\mathrm{ICV}}$ from the cross-attention formula, keeping the same retrieved documents, decoder, and training, would settle the core claim: if Exact Match scores do not drop, the vector is not doing the work assigned to it. A second test compares the ICV against a prompt containing as many tokens as the demonstrations used to build the vector; if the uncompressed prompt matches or beats the vector, the claim of lossless compression fails.

Watch

Extended reading notes

Core claim

The central discovery is that in-context learning can be recast as an additive shift in latent space. The paper's ICV is computed by a forward pass over demonstrations, $H = f(D)$, followed by a pooling operation $v_{\mathrm{ICV}} = g(H)$, and it is then added to the key vectors in cross-attention: $A = \operatorname{softmax}(Q(K + v_{\mathrm{ICV}})^\top/\sqrt{d_k})$. The result, as reported, is that a 140-million-parameter encoder-decoder with this shift achieves Exact Match scores of 61 on Natural Questions, 67.5 on TriviaQA, and 72 on HotpotQA, and retrieval top-1 accuracy of 65.2%, top-3 of 77.4%, and top-5 of 85.6%, beating the BGE embedding-plus-reranker baseline. The paper claims this shows a single vector can substitute for demonstrations, reduce prompt length, and make knowledge integration computationally cheaper than fine-tuning.

Load-bearing premise

The mechanism assumes that one pooled vector added uniformly to all attention keys can preserve the information that full demonstration examples would convey, and the paper gives no proof, analysis, or ablation to show that this equivalence holds.

Editorial extensions

If this is right

  • If the ICV mechanism is correct, retrieval-augmented generation can operate with prompts that stay short no matter how many documents are retrieved, easing token-limit constraints.
  • The reported retrieval gains imply that the same architecture can improve ranking quality without a separate reranker, simplifying the retrieval stack.
  • The claim that 140 million parameters can approach billion-parameter models on factual QA suggests the bottleneck for these tasks is knowledge access, not raw scale, which is a testable hypothesis for architecture design.
  • Because the vector is trained end-to-end with cosine and generation losses, the approach offers a middle path between prompt engineering and full fine-tuning, with a single learned artifact per task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This mechanism is a close cousin of activation steering: the paper's additive key shift could in principle be applied to decoder-only LLMs at every layer, which would let the idea generalize beyond the encoder-decoder tested here.
  • A possible hidden confound is that the retrieval gains alone, rather than the vector itself, may drive the HotpotQA improvement; the paper does not separate the ICV contribution from the improved retriever it trains.
  • A natural stress test is multi-hop or long-document reasoning, where a single pooled vector may discard the compositional structure that demonstrations carry; if ICV fails there, the vector is best seen as a short-context approximation rather than a general replacement for in-context learning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an integrated encoder-decoder architecture for retrieval-augmented generation that uses an 'In-Context Vector' (ICV) derived from latent embeddings of demonstration examples. The ICV is meant to replace explicit demonstrations by additively shifting latent states, and the method is evaluated on Natural Questions, TriviaQA, and HotpotQA, with reported Exact Match scores for generation and retrieval metrics for document selection. The abstract claims that ICV outperforms standard in-context learning and fine-tuning across question answering and other tasks, while also reducing prompt length and computational cost. The manuscript also reports a small-parameter model (about 140M parameters) achieving results comparable to larger models such as LLaMA-3, Gemma, and Phi-3.

Significance. The underlying research direction—compressing task or demonstration information into a single additive vector that steers a language model's latent states—is plausible and connects to existing work on task vectors and activation editing. If fully specified and empirically supported, such a mechanism could reduce prompt token usage and improve retrieval-augmented generation efficiency. The paper, however, does not deliver this: the ICV mechanism is not precisely defined, the two central cross-attention equations describe different mechanisms, and the reported experimental results contradict the abstract's central claim. No code, data, hyperparameters, ablations, error bars, or statistical tests are provided, so the claimed advantages cannot be verified or attributed to the proposed component. The contribution is therefore not established in its current form.

major comments (4)
  1. [Abstract and Section 6.1, Table 1] The central claim that ICV outperforms fine-tuning across QA is contradicted by the paper's own results. In Table 1, the ICV Model achieves EM 0.61 versus 0.62 for Fine-Tuned BART on Natural Questions, and 0.67 versus 0.70 on TriviaQA; only on HotpotQA does ICV exceed Fine-Tuned BART (0.72 versus 0.68). The abstract states that 'ICV outperforms standard in-context learning and fine-tuning across question-answering, information retrieval, and other tasks,' and Section 6.1 describes the ICV model as 'notably outperforming the baselines' while also conceding it 'did not achieve the highest EM scores on NQ or TriviaQA.' This internal contradiction undermines the main empirical claim, and no error bars, standard deviations, or statistical tests are reported anywhere in the experimental section.
  2. [Sections 3.3 and 4.6] The proposed ICV mechanism is not consistently specified. Section 3.3 defines A = softmax(Q(K+vICV)^T / sqrt(dk)) with vICV = g(H) pooled from demonstration hidden states, and says that ICVs are 'integrated into the cross-attention mechanism.' Section 4.6, which presents the cross-attention used in the proposed architecture, defines Across = softmax(cquery K_TopN^T / sqrt(dk)) with no vICV term at all. These two equations describe different attention mechanisms, and the paper never clarifies which one was implemented and evaluated, making it impossible to know which mechanism produced the reported results.
  3. [Sections 3.3, 4.2-4.6] The method is not reproducible from the text. The pooling function g is only exemplified ('mean, max, or attention-based pooling') and never defined; no layer index is given for the hidden states h_i in H = f(D); and there is no description of how vICV is trained, optimized, or selected. The encoder, DB encoder, and decoder are described only as generic transformer layers with unspecified dimensions, numbers of layers, and training details. No code, data splits, hyperparameters, or implementation details are provided, so the central mechanism cannot be implemented or independently checked from the manuscript.
  4. [Section 6.2 and Table 2] The reported retrieval improvements are not attributable to the ICV component. Table 2 compares an 'ICV Retrieval Approach' with BGE embedding and reranker baselines, but Section 4.5 only describes cosine similarity between the transformed query vector and precomputed database vectors; no retrieval algorithm, reranking strategy, or additional ICV-based retrieval procedure is specified for the ICV row. Section 5.2 mentions that 'ablation studies were conducted,' but no ablation results appear anywhere in the paper. Consequently, the retrieval gains in Table 2 could be due to any part of the architecture or training objective, not specifically to the ICV mechanism.
minor comments (6)
  1. [Section 7.1 vs Section 6.1] The conclusion reports a TriviaQA EM of '67.5' while Table 1 reports 0.67; please standardize the scaling and fix the inconsistency.
  2. [Section 4.8.3] The dynamic weighting schedule sets alpha(t) to 'decay' when Lcos <= 1, but 'decay' is never defined; please specify the decay schedule and its exact values.
  3. [Section 5.2] The text states that ablation studies and additional baselines were evaluated, but no ablation table or standard transformer baseline results are presented, so the stated procedure is not verifiable.
  4. [Table 2] The caption for Table 2 does not state which dataset or datasets the retrieval metrics are computed on; please clarify the experimental setting.
  5. [References] Reference [30] has a corrupted author list ('Jason Wei, Jeffrey Wei, Yi Tay, Dai Tran, ...') and several other references contain formatting inconsistencies; please correct the bibliography.
  6. [Section 2.2 vs Sections 3.3 and 4.6] Section 2.2 says ICV 'enhances latent states across all layers,' but the formal equations only apply vICV as an additive term in a single cross-attention operation; the relationship between these descriptions should be explained.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: the ICV equations are explicit construction choices, and the reported gains are empirical claims tested on external benchmarks rather than consequences of the definitions.

full rationale

The central mechanism is defined, not derived: Section 3.3 gives v_ICV = g(H) and H_adjusted_q = H_q + v_ICV, and the cross-attention form A = softmax(Q(K+v_ICV)^T / sqrt(d_k)) is presented as a design choice. These equations do not by themselves force the Exact Match or retrieval numbers in Tables 1 and 2; those numbers are measured on NQ, TriviaQA, and HotpotQA and could have come out differently, so the empirical claim has independent content. No load-bearing self-citation, uniqueness theorem, or ansatz-smuggling citation is present; reference [8] is explicitly cited as a similar concurrent method, and the paper claims a difference. The acknowledged relationship to task vectors and activation manipulation is a novelty/attribution concern, not a circular reduction. The paper does contain serious correctness and reproducibility problems: Section 4.6's cross-attention uses A = softmax(c_query K_TopN^T / sqrt(d_k)) with no v_ICV term, contradicting Section 3.3; Table 1 shows the ICV model below fine-tuned BART on NQ (0.61 vs 0.62) and TriviaQA (0.67 vs 0.70) while the abstract claims ICV outperforms fine-tuning; and pooling g is unspecified ('typically through a pooling operation g (e.g., mean, max, or attention-based pooling)'). These are missing-support and internal-inconsistency issues, not circularity in the derivation sense, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claims rely on an unproven equivalence between additive latent shifts and attention over demonstrations, an unnamed precomputed encoder for database vectors, and vague training hyperparameters. The paper introduces one invented entity, the in-context vector, with no external falsifiable handle.

free parameters (4)
  • alpha threshold = 1 (threshold for cosine loss)
    Section 4.8.3: the dynamic weighting alpha switches from 1 to a decay when Lcos <= 1; the threshold is hand-chosen and the decay rate is unspecified.
  • pooling function g for ICV = unspecified (mean, max, or attention)
    Section 3.3 says pooling is 'typically' mean, max, or attention, but the actual choice is never given, and this choice determines vICV.
  • top-N retrieved documents = unspecified
    Section 4.5.1 selects top N documents for cross-attention, but N is not reported, and retrieval results are reported only at top-1/3/5 ranks.
  • precomputed encoder = open-source encoder, unspecified
    Section 4.4 relies on an unnamed open-source encoder to generate database vectors; the choice affects the whole retrieval space.
assumptions (4)
  • ad hoc to paper Adding a pooled latent vector to keys in cross-attention transfers task information equivalent to demonstrations.
    Section 3.3, equations for vICV and cross-attention; no derivation or proof is given, and the claim is the central premise.
  • domain assumption The decoder can generate answers from a single attention vector catt as initial hidden state.
    Section 4.7 sets H_dec(0) = catt, which is not a standard decoder initialization; no justification is given.
  • domain assumption The precomputed database vectors and the DB encoder output lie in the same embedding space.
    Sections 4.3-4.4 assume cDB aligns with VDB; this is trained with cosine loss but the space compatibility is assumed.
  • standard math The attention equation Attn(x_query W_q, X W_k, X W_v) = alpha h(X_query) + (1-alpha) h(X_demos) is an accurate model of how demonstrations influence attention.
    Section 3.2 presents a simplified additive decomposition of attention; it is a known approximation but not rigorously established for all cases.
invented entities (1)
  • In-Context Vector (vICV)
    purpose: A pooled latent embedding intended to capture task information and shift latent states or keys.
    Defined in Section 3.3 as vICV = g(H), and used as an additive shift. No external falsifiable handle is provided; its existence and effect are only asserted in this paper's framework.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Knowledge Feeding to Language Models: A Novel Integrated Encoder-Decoder Architecture." pith.science (2026). https://pith.science/paper/FWH7CIXQ

@misc{pith2026250205233,
  author       = {Pith},
  title        = {Pith review of: Efficient Knowledge Feeding to Language Models: A Novel Integrated Encoder-Decoder Architecture},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FWH7CIXQ}},
  note         = {Machine review of arXiv:2502.05233}
}
read the original abstract

This paper introduces a novel approach to efficiently feeding knowledge to language models (LLMs) during prediction by integrating retrieval and generation processes within a unified framework. While the Retrieval-Augmented Generation (RAG) model addresses gaps in LLMs' training data and knowledge limits, it is hindered by token limit restrictions and dependency on the retrieval system's accuracy. Our proposed architecture incorporates in-context vectors (ICV) to overcome these challenges. ICV recasts in-context learning by using latent embeddings of LLMs to create a vector that captures essential task information. This vector is then used to shift the latent states of the LLM, enhancing the generation process without adding demonstration examples to the prompt. ICV directly integrates information into the model, enabling it to process this information more effectively. Our extensive experimental evaluation demonstrates that ICV outperforms standard in-context learning and fine-tuning across question-answering, information retrieval, and other tasks. This approach mitigates the limitations of current RAG models and offers a more robust solution for handling extensive and diverse datasets. Despite leveraging a fraction of the parameters, our ICV-enhanced model achieves competitive performance against models like LLaMA-3, Gemma, and Phi-3, significantly reducing computational costs and memory requirements. ICV reduces prompt length, is easy to control, surpasses token limitations, and is computationally efficient compared to fine-tuning.

Figures

Figures reproduced from arXiv: 2502.05233 by the authors.

Figure 1
Figure 1. Proposed methodology integrating encoder-decoder [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Model Size vs. Performance. Despite its smaller size, [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 15 canonical work pages

  1. [8]

    Ronen Hendel, Mor Geva, and Amir Globerson. 2023. In-Context Learning Creates Task Vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023. 9318–9333. https://doi.org/10.18653/v1/2023.emnlp-long.890

    work page doi:10.18653/v1/2023.emnlp-long.890 2023

  2. [26]

    Seongmin Shin, Sungmin Lee, Hyeonseo Ahn, Sangwoo Kim, Hyunsoo Kim, Byoungjun Kim, Kyunghyun Cho, Gyuwan Lee, Woosung Park, Jangwon Ha, et al. 2022. On the Effect of Pretraining Corpora on In-Context Learning by a Large-Scale Language Model. In 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T...

    doi:10.18653/v1/2022.naacl-main.379 2022

  3. [1]

    Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou

  4. [2]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long- Document Transformer. arXiv preprint arXiv:2004.05150 (2020). https://doi.org/ 10.48550/arXiv.2004.05150

    arXiv doi:10.48550/arxiv.2004.05150 2020

  5. [3]

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Katie Mil- lican, Susannah Young, Eliza Rutherford, Tom Hennigan, et al. 2022. Improv- ing Language Models by Retrieving from Trillions of Tokens. arXiv preprint arXiv:2201.11193 (2022). https://doi.org/10.48550/arXiv.2201.11193

    work page Pith review arXiv doi:10.48550/arxiv.2201.11193 2022

  6. [6]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv preprint arXiv:2312.10997 (2024). https: //doi.org/10.48550/arXiv.2312.10997

    arXiv doi:10.48550/arxiv.2312.10997 2024

  7. [7]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang

  8. [9]

    Gautier Izacard and Edouard Grave. 2021. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 874–880. https://doi.org/10.18653/v1/2021.eacl-main.74

    doi:10.18653/v1/2021.eacl-main.74 2021

Show all 42 references
  1. [10]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Re...

    2017 doi

  2. [11]

    Jean Kaddour, James Harris, Marzieh Mozes, Hayley Bradley, Roberta Raileanu, and Rosie McHardy. 2023. Challenges and Applications of Large Language Models. arXiv preprint arXiv:2301.11943 (2023). https://doi.org/10.48550/arXiv.2301.11943

    2023 doi arXiv

  3. [12]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense Passage Retrieval for Open- Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP...

    2020

  4. [13]

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for C...

    2019 doi

  5. [14]

    Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Urvashi Khandelwal, Mike Lewis, et al. 2020. Retrieval- Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems 33 (2020), 9459–947...

    2020

  6. [15]

    Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg

    Ke Li, Andrew K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2022. Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. arXiv preprint arXiv:2210.13382 (2022). https: //doi.org/10.48550/arXiv.2210.13382

    2022 doi arXiv

  7. [16]

    NF Liu, K Lin, J Hewitt, A Paranjape, M Bevilacqua, F Petroni, and P Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024), 132–148. https://doi.org/ 10.1162/tacl_a_00563

    2024 doi

  8. [17]

    Yian Lu, Massimo Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp

  9. [18]

    Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Noisy Channel Language Model Prompting for Few-Shot Text Classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 5316–5330. http...

    2022 doi

  10. [19]

    Umang Mini, Peter Grietzer, Mukund Sharma, Alex Meek, Malachy MacDiarmid, and Alec M. Turner. 2023. Understanding and Controlling a Maze-Solving Policy Network. arXiv preprint arXiv:2310.08043 (2023). https://doi.org/10.48550/arXiv. 2310.08043

    2023 doi arXiv

  11. [20]

    In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Fantastically Ordered Prompts and Where to Find Them: Overcoming Few- Shot Prompt Order Sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 8086–8098. https://doi.org/10.18653/v1/2022.acl-long.553

    doi

  12. [21]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. Ope- nAI blog 1, 8 (2019), 9. https://openai.com/research/language-models-are- unsupervised-multitask-learners

    2019

  13. [22]

    Or Hon Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to Retrieve Prompts for In-Context Learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2655–2671. https:/...

    2022 doi

  14. [23]

    OpenAI. 2023. GPT-4 Technical Report. https://doi.org/10.48550/arXiv.2303.08774

    2023 doi arXiv

  15. [24]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971 (2023). https://ar...

    2023 arXiv

  16. [25]

    Hugo Touvron, Louis Martin, Kevin Stone, Pierre-Emmanuel Albert, Amjad Almahairi, Yasmine Babaei, Siddharth Batra, Shubham Bhosale, et al. 2023. LLaMA 2: Open Foundation and Fine-Tuned Chat Models.arXiv preprint arXiv:2307.09288 (2023). https://doi.org/10.48550/arXiv.2307.09288

    2023 doi arXiv

  17. [27]

    Arik, and Thomas Pfister

    Xiaodong Wan, Ruiqi Sun, Hanjun Dai, Sercan O. Arik, and Thomas Pfister

  18. [28]

    Eisenschlos, and Thomas Pfister Sercan O

    Xiaodong Wan, Ruiqi Sun, Hootan Nakhost, Hanjun Dai, Jose M. Eisenschlos, and Thomas Pfister Sercan O. Arik. 2023. Universal Self-Adaptive Prompting. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). 7437–7462. https://doi.org/1...

    2023 doi

  19. [29]

    Alec Turner, Leon Thiergart, David Udell, Graeme Leech, Umang Mini, and Malachy MacDiarmid. 2023. Activation Addition: Steering Language Models Without Optimization. arXiv preprint arXiv:2308.10248 (2023). https://doi.org/10. 48550/arXiv.2308.10248

    2023 doi arXiv

  20. [30]

    Jason Wei, Jeffrey Wei, Yi Tay, Dai Tran, Adam Webson, Yian Lu, Xiaodong Chen, Hao Liu, Dianqiang Huang, Denny Zhou, et al . 2023. Larger Language Models Do In-Context Learning Differently. arXiv preprint arXiv:2303.03846 (2023). https://doi.org/10.48550/arXiv.2303.03846

    2023 doi arXiv

  21. [31]

    Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma

    Shengjia M. Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An Expla- nation of In-Context Learning as Implicit Bayesian Inference. InInternational Con- ference on Learning Representations . https://doi.org/10.48550/arXiv.2101.04655

    2021 doi arXiv

  22. [32]

    Baiqiang Xu, Qi Wang, Zifan Mao, Yixin Lyu, Qian She, and Yichen Zhang

  23. [33]

    Misra, Maarten Bosma, Denny Zhou, Maarten Ma, et al

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Albert Yu, Karan Goel, William W. Misra, Maarten Bosma, Denny Zhou, Maarten Ma, et al. 2022. Emergent Abilities of Large Language Models. arXiv preprint arXiv:2206.07682 (2022). https://doi.org/...

    2022 doi arXiv

  24. [34]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Rus- lan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-Hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Langua...

    2018 doi

  25. [35]

    Sungjin Ye, Donghwan Kim, Jiho Jang, Janghoon Shin, and Minjoon Seo. 2023. Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero- Shot Learners. In Proceedings of the Eleventh International Conference on Learning Representations. https://doi.org/10.48550/...

    2023 doi arXiv

  26. [36]

    Fangyuan Yin, Jesse Vig, Shafiq Joty Philippe Laban, Caiming Xiong, and Chien- Sheng Wu. 2023. Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning. In Proceedings of the 61st Annual Meeting of the Association for Computationa...

    2023 doi

  27. [37]

    In Proceedings of the Eleventh International Conference on Learning Representations

    KNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inference. In Proceedings of the Eleventh International Conference on Learning Representations. https://doi.org/10.48550/arXiv.2210.07896

    doi arXiv

  28. [38]

    Jing Yang, Binghui Hui, Mingqiang Yang, Bin Li, Fei Huang, and Yining Li. 2024. Iterative Forward Tuning Boosts In-Context Learning in Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 15460–154...

    2024 doi

  29. [39]

    Allen Zou, Long Phan, Sheng Chen, John Campbell, Ping Guo, Rui Ren, Albert Pan, Xinyi Yin, Matas Mazeika, Alexandra-Kate Dombrowski, et al. 2023. Rep- resentation Engineering: A Top-Down Approach to AI Transparency. arXiv preprint arXiv:2310.01405 (2023). https://doi.org/10.48...

    2023 doi arXiv

  30. [42]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shu Chen, Chris Dewan, Mona Diab, Xian Li, Xiang Lin, et al. 2022. OPT: Open Pre- trained Transformer Language Models. arXiv preprint arXiv:2205.01068 (2022). https://arxiv.org/abs/2205.01068

    2022 arXiv

  31. [43]

    Eric Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate Before Use: Improving Few-Shot Performance of Language Models. InProceedings of the 38th International Conference on Machine Learning (ICML) . 12697–12706. https://doi.org/10.48550/arXiv.2102.09690

    2021 doi arXiv

  32. [2020]

    In The Thirty-seventh International Conference on Machine Learning

    Retrieval Augmented Language Model Pre-Training. In The Thirty-seventh International Conference on Machine Learning

  33. [2022]

    In Proceedings of the Eleventh International Conference on Learning Representations

    What Learning Algorithm Is In-Context Learning? Investigations with Linear Models. In Proceedings of the Eleventh International Conference on Learning Representations. https://doi.org/10.48550/arXiv.2210.10282

    doi arXiv

  34. [2023]

    In Findings of the Association for Computational Linguistics: ACL 2023

    Better Zero-Shot Reasoning with Self-Adaptive Prompting. In Findings of the Association for Computational Linguistics: ACL 2023 . 3493–3514. https: //doi.org/10.18653/v1/2023.acl-main.197

    2023 doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.