Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

LoRA-Augmented Generation (LAG) for Knowledge-Intensive Language Tasks

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-stage, data-free router—cheap arrow-vector filtering followed by spectral reranking—lets a language model select from a library of 1,000 LoRA adapters on a per-token basis, capturing 92.1% of oracle performance on…

desk verdict A solid, well-scoped engineering combo of Arrow and SpectR whose main empirical claim against Arrow is believable, but the abstract overreaches by declaring victory over data-free methods without ever running SpectR. read the letter →

arxiv 2507.05346 v2 pith:V3NQZJHI submitted 2025-07-07 cs.CL cs.AIcs.LG

William Fleshman , Benjamin Van Durme This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords LoRAadapterroutingspectralretrieval-augmentedgenerationparameter-efficientfine-tuningknowledge-intensivetasksdata-freeinferenceselectionlow-rankadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LoRA-Augmented Generation (LAG) claims that a library of over a thousand LoRA adapters can be routed at inference time with no training and no access to the adapters' data, by combining two existing routing signals in a cascade. The paper argues that cheap arrow-vector scores can filter the library down to a shortlist, after which expensive spectral scores rerank only that shortlist and pick one adapter per token and per layer. On a 1,000-adapter KILT benchmark built from Wikipedia experts, this two-stage procedure outperforms Arrow routing on most datasets and reaches 92.1% of the oracle model's average normalized score, while making full spectral reranking computationally feasible through top-k filtering. If correct, LAG turns large, heterogeneous adapter libraries into a practical, swap-in knowledge source for language models, with a tunable knob k that trades compute for accuracy.

What carries the argument

The load-bearing object is the spectrally aligned adapter pair $(B^*, A^*)$, obtained by SVD of the LoRA update $BA$. One phrase: $A^*$ stores the adapter's orthogonal directions of maximum variation, scaled by singular values, so the norm $\|A^* x\|_2$ is a per-token activation score. The paper's two-stage routing uses the first row of $A^*$ (Arrow's arrow vector) for a top-$k$ filter over all $n$ adapters, then uses the full $A^*$ over the $k$ survivors for reranking; the FLOP identity $2nh + 2khr$ with $rk \ll n$ is what makes 1000-adapter libraries tractable where full SpectR is not.

What would settle it

Run LAG on the KILT evaluation and check, per token, whether the ground-truth oracle adapter is the one LAG selects and whether it survives the top-20 arrow filter. If the oracle adapter is frequently filtered out at $k=20$ on tokens where it is correct, yet LAG's downstream scores remain high, the magnitude-based routing signal in Eqs. (6)-(7) is not carrying the performance and the mechanism would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that Arrow's rank-1 prototype and spectral routing's (SpectR's) full spectral score are complementary, and that chaining them—cheap filter first, precise rerank second—yields near-oracle adapter selection at roughly Arrow's cost. Each LoRA update $BA$ is aligned offline by a rank-$r$ SVD, $BA = U S V^T$, and stored as $B^* = U$, $A^* = S V^T$, so that $A^*$ contains scaled eigenvectors of the adapter's covariance and $\|A^* x\|_2$ measures the activation of the adapter on token vector $x$. At inference, LAG computes $|a^*_i x|$ for every adapter ($a^*$ being the first row of $A^*$), keeps the $k$ largest, and selects the winner on that shortlist by $\|A^* x\|_2$, applying $h = W_l x + \hat{B}(\hat{A} x)$ with the low-rank product reused from the score. With $k=20$, this filters 98% of the 1000 adapters before the expensive step; the paper reports an average normalized score of 92.1 against an oracle that always applies ground-truth adapters, versus 84.8 for Arrow.

Load-bearing premise

The load-bearing premise is that the adapter with the largest projection magnitude $\|A^* x\|_2$ on a token is the adapter whose knowledge is most useful for generating that token; the paper never validates this head-on and relies on downstream task scores as indirect evidence.

Editorial extensions

If this is right

  • LAG can be applied to libraries far too large for full SpectR reranking, with $k$ tuned to the compute budget; the paper shows performance flattening around $k=5$, so aggressive filtering costs little accuracy.
  • Because the arrow vectors are already contained in the aligned $A^*$ matrices, LAG needs no extra disk storage beyond the aligned adapter library, matching SpectR's storage while keeping Arrow's per-token compute.
  • LAG is compatible with RAG and PRAG: using LAG for task adapters and RAG for knowledge reaches a 95.0 average normalized score, beating either alone, and can exceed the oracle on slot filling with 102.7.
  • Per-token and per-layer selection gives flexibility on noisy tasks like entity linking, where LAG's normalized score of 89.2 far exceeds Arrow's 62.6 and approaches the oracle's 67.6.
  • LAG interpolates between its two parents: at $k=1$ it reduces to Arrow routing, and at $k=n$ it is equivalent to SpectR, making it a single tunable mechanism for the whole efficiency-accuracy spectrum.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The routing signal is never directly validated; a fair test would compare LAG's per-token selection against the oracle adapter to see whether $\|A^* x\|_2$ actually ranks the ground-truth adapter first, or whether downstream gains come from task-level biases of the adapters.
  • The performance plateau around $k=5$ suggests most of the value comes from a small set of highly activated adapters, so $k$ could be set dynamically per token based on the margin between top arrow scores as a form of test-time compute scaling.
  • The strong RAG+LAG result indicates parametric and non-parametric knowledge are complementary; an obvious extension is to route between document retrieval and adapter selection based on query characteristics such as lexical recall versus semantic activation.
  • Since the knowledge adapters were trained on Wikipedia with synthetic continued pretraining, LAG's success may depend on these adapters being highly separable; testing on heterogeneous community-contributed LoRA libraries would show whether per-token routing generalizes beyond this curated setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes LoRA-Augmented Generation (LAG), a two-stage data- and training-free adapter routing method. Offline SVD aligns all LoRA adapters; at inference, Arrow routing selects the top-k adapters by |a^*·x|, then SpectR reranks the surviving adapters by ||A^* x||_2 and applies the winner per token and layer. The authors construct 1,000 knowledge adapters (one per Wikipedia article) plus task adapters on a filtered subset of KILT, compare LAG against Arrow and an instruction-tuned base model, report normalized gains including 92.1% of an Oracle adapter-selection upper bound, and explore combinations with RAG and PRAG. The central empirical claim is that LAG outperforms existing data-free methods while remaining computationally feasible at a scale where SpectR is intractable.

Significance. The strength of the paper is its concrete, parameter-free architecture: LAG has only k as a free parameter, inherits Arrow's disk efficiency and SpectR's reranking, and the experiments are conducted on a substantial 1,000-adapter library. The comparison against Arrow from a different group is credible, and the Oracle normalization provides a useful upper-bound reference. The efficiency analysis in Table 2 is careful. However, the evaluation does not include the key data-free competitor SpectR, reports no variance or significance tests, and uses a self-filtered benchmark with two unnamed drops; these gaps leave the strength of the main claim unproven. If the missing controls confirm the approximation, LAG would be a valuable practical contribution to adapter routing.

major comments (4)
  1. [Experiments, 'Data and Training Free'] SpectR, the method LAG is most directly built on, is never run. The authors justify this by saying SpectR is intractable at n=1000, but the abstract and conclusion claim 'superior performance over existing data-free methods.' Without a small-scale SpectR baseline (e.g., on a 50-100 adapter subset or with a reduced rank) and a per-token recall of Arrow's top-k set (Eq. 6), the normalized gains in Tables 3-4 cannot be attributed to LAG's reranking rather than Arrow's filtering. This is a missing control, not a demonstrated failure, but it is load-bearing for the paper's central claim.
  2. [Conclusions] The manuscript states in Knowledge Access that 'no statistically significant winner' was found among combined approaches, yet the Conclusion says LAG 'significantly outperformed' RAG/PRAG. Either report standard errors and significance tests for the dataset-level metrics in Tables 3-5, or temper the significance language. Without any error bars, the numerical differences among LAG, RAG-LAG, and P-LAG (e.g., 92.1 vs 92.2 vs 95.0 average normalized score) are not interpretable.
  3. [Data and Metrics] The evaluation is based on a self-built filtered subset of KILT with two unnamed dropped datasets. The reader cannot assess whether the drop is benign or systematically favorable to LAG; name the dropped datasets and report their sizes and the reason for dropping. Ideally also show that LAG's advantage is stable across the subsets.
  4. [Eq. (9)] The normalized score S_T normalizes by the Oracle model's score per dataset. Because some Oracle scores are low (e.g., WnCw and WnWi in Table 3), the ratio can be sensitive to small changes in the numerator; moreover, the average is weighted by sample counts, which makes the aggregate dominated by the largest datasets (WoW, FEV). Please report the per-dataset raw scores alongside the normalized aggregates and justify the Oracle normalization as a stable reference.
minor comments (4)
  1. [Introduction] The phrase 'methods for for effectively deploying them' contains a duplicated 'for'.
  2. [Background] There are several typographical errors, including 'Fleshman and Van Durme (2025)’sspectral alignment procedure' and 'adapter retreival'; these should be corrected.
  3. [Table 2 footnote] The footnote '1e.g. 1 million rank-6 adapters with k = 20 and h = 4096 would require 49B extra parameters w/ SpectR versus 4B w/ LAG' is not fully clear about whether the comparison is GPU memory or disk; please clarify the setting.
  4. [Figure 3] Figure 3 shows the average normalized score without error bars or a per-task breakdown; adding them would let the reader judge the plateau behavior more concretely.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LAG composes two existing routing methods and its measured gains are against Arrow, an external baseline from a different group.

full rationale

The derivation chain is self-contained and does not reduce to its inputs. Equations (1)-(4) are an exact SVD reparameterization of the LoRA product BA, preserving function exactly (USV^T = BA). Equation (5) is Arrow's prototype definition, attributed to Ostapenko et al. (2024); Eq. (6) is Arrow filtering; Eq. (7) is SpectR reranking from Fleshman and Van Durme (2025). LAG's novelty is the two-stage composition of these existing, externally introduced routing signals, not a derivation of those signals. No parameter is fitted to the evaluation data: LAG has no training step, and k=20 is a compute-budget choice, not an optimized hyperparameter. The Oracle normalization (Eq. 9) is a reporting metric used to average across datasets and is not fed back into routing. The boundary identities the paper states — k=1 makes LAG equivalent to Arrow, and k=n makes it equivalent to SpectR — are explicit properties of the composition, not hidden equivalences. The main concern is that SpectR is never run on the 1000-adapter library, so the abstract's phrase 'superior performance over existing data-free methods' is broader than what is directly demonstrated; that is a missing baseline or over-claim about scope, not a circularity. The self-citation to SpectR is load-bearing as a component choice, but SpectR is an arXiv paper with its own independent experiments, and using its routing score as a building block is legitimate composition rather than an unverified uniqueness claim. Therefore no circular step can be exhibited with a specific reduction.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The stated method has no fitted constants; routing is a deterministic function of adapter weights and the token vector. The hand-set value k trades compute against accuracy, and the adapter training hyperparameters determine library quality but not the routing logic. The domain assumptions listed above are the load-bearing premises inherited from prior work.

free parameters (1)
  • k (top-k filter size) = 20
    User-specified compute budget for Arrow filtering; used in all main experiments. The paper shows performance plateaus around k=5, but k is not tuned on a held-out set.
assumptions (5)
  • domain assumption The magnitude of the LoRA adapter projection onto the token representation, ||A* x||_2, indicates how useful that adapter is for the current token.
    Central routing assumption; enters at Eqs. (5)-(7) where the adapter maximizing this projection is selected. No direct correlation analysis is provided.
  • standard math The rank-r SVD exactly reconstructs the LoRA product, U S V^T = BA (Eq. 2), so the aligned adapter is functionally equivalent to the original.
    Standard linear algebra used in the offline spectral alignment step.
  • domain assumption LoRA adapters trained on the pretrained Llama-3.2-3B can be applied directly to the instruction-tuned version without retraining.
    Used in the Knowledge Library section; follows Fleshman and Van Durme (2024), not independently verified here.
  • domain assumption Synthetic continued pretraining (Yang et al. 2025) produces knowledge adapters that encode article-specific knowledge required by KILT tasks.
    Used to build the 1,000-adapter knowledge library; if this assumption fails, the library is low quality and the LAG results are not meaningful.
  • domain assumption The filtered KILT subset, restricted to samples grounded in the top-1000 Wikipedia provenance articles, is a representative evaluation of knowledge-intensive tasks.
    Evaluation protocol; two datasets are dropped without names and only grounded samples are kept, which may bias results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LoRA-Augmented Generation (LAG) for Knowledge-Intensive Language Tasks." pith.science (2026). https://pith.science/paper/V3NQZJHI

@misc{pith2026250705346,
  author       = {Pith},
  title        = {Pith review of: LoRA-Augmented Generation (LAG) for Knowledge-Intensive Language Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V3NQZJHI}},
  note         = {Machine review of arXiv:2507.05346}
}
read the original abstract

The proliferation of fine-tuned language model experts for specific tasks and domains signals the need for efficient selection and combination methods. We propose LoRA-Augmented Generation (LAG) for leveraging large libraries of knowledge and task-specific LoRA adapters. LAG requires no additional training or access to data, and efficiently filters, retrieves, and applies experts on a per-token and layer basis. We evaluate LAG on various knowledge-intensive tasks, achieving superior performance over existing data-free methods. We explore scenarios where additional data is available, demonstrating LAG's compatibility with alternative solutions such as retrieval-augmented generation (RAG).

Figures

Figures reproduced from arXiv: 2507.05346 by the authors.

Figure 1
Figure 1. Overview of LAG. LoRA adapters are converted offline via SVD to align representations and extract [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. RAG and PRAG both identify the document most [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The average normalized performance of LAG goes [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RING: Retrieval-Internalized Generation for Continual Large-Scale Knowledge Injection

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A mixture-of-experts LLM trained with reinforcement learning to perform retrieval from its own parametric memory can replace external retrieval in some settings, at lower latency.

  2. Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memory

    cs.LG 2026-07 conditional novelty 6.0 of 10

    PMDRouter selects LoRAs zero-shot by decoding scale-normalized linear response energy from one adapter-free backbone prefill, and leads most internal-signal baselines on a new multi-granularity EPM bench.

Reference graph

Works this paper leans on

49 extracted references · 38 canonical work pages · cited by 2 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alani, H.; Guo, Z.; and Barbosa, D. 2018. Robust named entity disambiguation with random walks. Semant. Web, 9(4): 459–479

    work page 2018

  4. [4]

    Allen-Zhu, Z.; and Li, Y. 2024. Physics of language models: part 3.1, knowledge storage and extraction. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org

    work page 2024

  5. [5]

    Bapna, A.; and Firat, O. 2019. Simple, Scalable Adaptation for Neural Machine Translation. In Inui, K.; Jiang, J.; Ng, V.; and Wan, X., eds., Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 1538--1548. Hong Kong, China: Associat...

    work page 2019

  6. [6]

    Barnett, S.; Kurniawan, S.; Thudumu, S.; Brannelly, Z.; and Abdelrazek, M. 2024. Seven Failure Points When Engineering a Retrieval Augmented Generation System. In Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI, CAIN '24, 194–199. New York, NY, USA: Association for Computing Machinery. ISBN 9798400705915

    work page 2024

  7. [7]

    Caccia, L.; Ansell, A.; Ponti, E.; Vulić, I.; and Sordoni, A. 2025. Training Plug-n-Play Knowledge Modules with Deep Context Distillation. arXiv:2503.08727

    arXiv 2025

  8. [8]

    L.; and Sordoni, A

    Caccia, L.; Ponti, E.; Su, Z.; Pereira, M.; Roux, N. L.; and Sordoni, A. 2023. Multi-Head Adapter Routing for Cross-Task Generalization. In Thirty-seventh Conference on Neural Information Processing Systems

    work page 2023

Show all 49 references
  1. [9]

    Chronopoulou, A.; Peters, M.; Fraser, A.; and Dodge, J. 2023. A dapter S oup: Weight Averaging to Improve Generalization of Pretrained Language Models. In Vlachos, A.; and Augenstein, I., eds., Findings of the Association for Computational Linguistics: EACL 2023, 2054--2063. D...

    2023

  2. [10]

    Dinan, E.; Roller, S.; Shuster, K.; Fan, A.; Auli, M.; and Weston, J. 2019. Wizard of Wikipedia: Knowledge-Powered Conversational Agents. In International Conference on Learning Representations

    2019

  3. [11]

    Elsahar, H.; Vougiouklis, P.; Remaci, A.; Gravier, C.; Hare, J.; Laforest, F.; and Simperl, E. 2018. T - RE x: A Large Scale Alignment of Natural Language with Knowledge Base Triples. In Calzolari, N.; Choukri, K.; Cieri, C.; Declerck, T.; Goggi, S.; Hasida, K.; Isahara, H.; M...

    2018

  4. [12]

    Fleshman, W.; Khan, A.; Marone, M.; and Durme, B. V. 2024. AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guarantees. In Proceedings of Conference on Applied Machine Learning in Information Security (CAMLIS) 2024

    2024

  5. [13]

    Fleshman, W.; and Van Durme , B. 2024. RE-Adapt: Reverse Engineered Adaptation of Large Language Models. arXiv:2405.15007

    2024 arXiv

  6. [14]

    Fleshman, W.; and Van Durme , B. 2025. SpectR: Dynamically Composing LM Experts with Spectral Routing. arXiv:2504.03454

    2025 arXiv

  7. [15]

    Gao, Y.; Xiong, Y.; Gao, X.; Jia, K.; Pan, J.; Bi, Y.; Dai, Y.; Sun, J.; Guo, Q.; Wang, M.; and Wang, H. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997

    2024 arXiv

  8. [16]

    Geva, M.; Schuster, R.; Berant, J.; and Levy, O. 2021. Transformer Feed-Forward Layers Are Key-Value Memories. In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S. W.-t., eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 5484--5495. O...

    2021

  9. [17]

    Glass, M.; Rossiello, G.; Chowdhury, M. F. M.; Naik, A.; Cai, P.; and Gliozzo, A. 2022. R e2 G : Retrieve, Rerank, Generate. In Carpuat, M.; de Marneffe, M.-C.; and Meza Ruiz, I. V., eds., Proceedings of the 2022 Conference of the North American Chapter of the Association for ...

    2022

  10. [18]

    Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783

    2024 arXiv

  11. [19]

    A.; Bordino, I.; F \"u rstenau, H.; Pinkal, M.; Spaniol, M.; Taneva, B.; Thater, S.; and Weikum, G

    Hoffart, J.; Yosef, M. A.; Bordino, I.; F \"u rstenau, H.; Pinkal, M.; Spaniol, M.; Taneva, B.; Thater, S.; and Weikum, G. 2011. Robust Disambiguation of Named Entities in Text. In Barzilay, R.; and Johnson, M., eds., Proceedings of the 2011 Conference on Empirical Methods in ...

    2011

  12. [20]

    Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019. Parameter-Efficient Transfer Learning for NLP . In Proceedings of the 36th International Conference on Machine Learning

    2019

  13. [21]

    J.; yelong shen; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; yelong shen; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. Lo RA : Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations

    2022

  14. [22]

    Y.; Pang, T.; Du, C.; and Lin, M

    Huang, C.; Liu, Q.; Lin, B. Y.; Pang, T.; Du, C.; and Lin, M. 2024. LoraHub: Efficient Cross-Task Generalization via Dynamic Lo RA Composition. In First Conference on Language Modeling

    2024

  15. [23]

    T.; Wortsman, M.; Schmidt, L.; Hajishirzi, H.; and Farhadi, A

    Ilharco, G.; Ribeiro, M. T.; Wortsman, M.; Schmidt, L.; Hajishirzi, H.; and Farhadi, A. 2023. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations

    2023

  16. [24]

    Joshi, M.; Choi, E.; Weld, D.; and Zettlemoyer, L. 2017. T rivia QA : A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Barzilay, R.; and Kan, M.-Y., eds., Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ...

    2017

  17. [25]

    Karpukhin, V.; Oguz, B.; Min, S.; Lewis, P.; Wu, L.; Edunov, S.; Chen, D.; and Yih, W.-t. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Webber, B.; Cohn, T.; He, Y.; and Liu, Y., eds., Proceedings of the 2020 Conference on Empirical Methods in Natural La...

    2020

  18. [26]

    M.; Uszkoreit, J.; Le, Q.; and Petrov, S

    Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K.; Toutanova, K.; Jones, L.; Kelcey, M.; Chang, M.-W.; Dai, A. M.; Uszkoreit, J.; Le, Q.; and Petrov, S. 2019. Natural Questions: A Benchmark for Q...

    2019

  19. [27]

    Lester, B.; Al-Rfou, R.; and Constant, N. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S. W.-t., eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 3045--3059. Online...

    2021

  20. [28]

    Levy, O.; Seo, M.; Choi, E.; and Zettlemoyer, L. 2017. Zero-Shot Relation Extraction via Reading Comprehension. In Levy, R.; and Specia, L., eds., Proceedings of the 21st Conference on Computational Natural Language Learning ( C o NLL 2017) , 333--342. Vancouver, Canada: Assoc...

    2017

  21. [29]

    u ttler, H.; Lewis, M.; Yih, W.-t.; Rockt\

    Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; K\" u ttler, H.; Lewis, M.; Yih, W.-t.; Rockt\" a schel, T.; Riedel, S.; and Kiela, D. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Conf...

    2020

  22. [30]

    L.; and Liang, P

    Li, X. L.; and Liang, P. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Zong, C.; Xia, F.; Li, W.; and Navigli, R., eds., Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference ...

    2021

  23. [31]

    F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; and Liang, P

    Liu, N. F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; and Liang, P. 2023. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12: 157--173

    2023

  24. [32]

    Lu, Q.; Dou, D.; and Nguyen, T. H. 2021. Parameter-Efficient Domain Knowledge Integration from Multiple Sources for Biomedical Pre-trained Language Models. In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S. W.-t., eds., Findings of the Association for Computational Linguistic...

    2021

  25. [33]

    Mangrulkar, S.; Gugger, S.; Debut, L.; Belkada, Y.; Paul, S.; and Bossan, B. 2022. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. https://github.com/huggingface/peft

    2022

  26. [34]

    M.; Charlin, L.; Le Roux, N.; Caccia, L.; and Sordoni, A

    Ostapenko, O.; Su, Z.; Ponti, E. M.; Charlin, L.; Le Roux, N.; Caccia, L.; and Sordoni, A. 2024. Towards modular LLMs by building and reusing a library of LoRAs. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org

    2024

  27. [35]

    Petroni, F.; Piktus, A.; Fan, A.; Lewis, P.; Yazdani, M.; De Cao, N.; Thorne, J.; Jernite, Y.; Karpukhin, V.; Maillard, J.; Plachouras, V.; Rockt \"a schel, T.; and Riedel, S. 2021. KILT : a Benchmark for Knowledge Intensive Language Tasks. In Toutanova, K.; Rumshisky, A.; Zet...

    2021

  28. [36]

    Pfeiffer, J.; Kamath, A.; R \"u ckl \'e , A.; Cho, K.; and Gurevych, I. 2021. A dapter F usion: Non-Destructive Task Composition for Transfer Learning. In Merlo, P.; Tiedemann, J.; and Tsarfaty, R., eds., Proceedings of the 16th Conference of the European Chapter of the Associ...

    2021

  29. [37]

    M.; Sordoni, A.; Bengio, Y.; and Reddy, S

    Ponti, E. M.; Sordoni, A.; Bengio, Y.; and Reddy, S. 2023. Combining Parameter-efficient Modules for Task-level Generalisation. In Vlachos, A.; and Augenstein, I., eds., Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics...

    2023

  30. [38]

    Robertson, S.; and Zaragoza, H. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Found. Trends Inf. Retr., 3(4): 333–389

    2009

  31. [39]

    V.; Lee, J.; Xu, K.; and Kumar, A

    Snell, C. V.; Lee, J.; Xu, K.; and Kumar, A. 2025. Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning. In The Thirteenth International Conference on Learning Representations

    2025

  32. [40]

    Su, W.; Tang, Y.; Ai, Q.; Yan, J.; Wang, C.; Wang, H.; Ye, Z.; Zhou, Y.; and Liu, Y. 2025. Parametric Retrieval Augmented Generation. arXiv:2501.15915

    2025 arXiv

  33. [41]

    Thorne, J.; Vlachos, A.; Christodoulopoulos, C.; and Mittal, A. 2011. FEVER : a Large-scale Dataset for Fact Extraction and VERification . In NAACL-HLT

    2011

  34. [42]

    Wang, X.; Mikaelyan, L.; Isazawa, T.; and Hensman, J. 2024. KBLaM: Knowledge Base augmented Language Model. ArXiv

    2024

  35. [43]

    H.; and Gao, J

    Wang, Y.; Agarwal, S.; Mukherjee, S.; Liu, X.; Gao, J.; Awadallah, A. H.; and Gao, J. 2022. A da M ix: Mixture-of-Adaptations for Parameter-efficient Model Tuning. In Goldberg, Y.; Kozareva, Z.; and Zhang, Y., eds., Proceedings of the 2022 Conference on Empirical Methods in Na...

    2022

  36. [44]

    W.; Lester, B.; Du, N.; Dai, A

    Wei, J.; Bosma, M.; Zhao, V.; Guu, K.; Yu, A. W.; Lester, B.; Du, N.; Dai, A. M.; and Le, Q. V. 2022. Finetuned Language Models Are Zero-Shot Learners. In Proceedings of the 10th International Conference on Learning Representations (ICLR 2022)

    2022

  37. [45]

    L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A

    Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Scao, T. L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. M. 2020. HuggingFac...

    2020 arXiv

  38. [46]

    Yang, Z.; Band, N.; Li, S.; Candès, E.; and Hashimoto, T. 2025. Synthetic continued pretraining. In International Conference on Learning Representations

    2025

  39. [47]

    Zadouri, T.; \"U st \"u n, A.; Ahmadian, A.; Ermis, B.; Locatelli, A.; and Hooker, S. 2024. Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning. In The Twelfth International Conference on Learning Representations

    2024

  40. [48]

    Zhang, Z.; Zeng, Z.; Lin, Y.; Wang, H.; Ye, D.; Xiao, C.; Han, X.; Liu, Z.; Li, P.; Sun, M.; and Zhou, J. 2023. Plug-and-Play Knowledge Injection for Pre-trained Language Models. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of the 61st Annual Meeting of t...

    2023

  41. [49]

    Zhao, Z.; Gan, L.; Wang, G.; Zhou, W.; Yang, H.; Kuang, K.; and Wu, F. 2024. L ora R etriever: Input-Aware L o RA Retrieval and Composition for Mixed Tasks in the Wild. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Findings of the Association for Computational Linguistics...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.