Pith. sign in

REVIEW 4 major objections 6 minor 56 references

Joint Localization and Activation Editing for Low-Resource Fine-Tuning

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Jointly learning which attention heads to edit, and whether to add or scale, consistently beats LoRA and prior activation-editing methods when only a few hundred examples are available.

desk verdict JoLA is a clean, well-ablationed method paper whose joint head-selection and intervention-type learning is genuinely new, but the 'consistent superiority' claim needs multi-seed runs and disclosed hyperparameters before it is established. read the letter →

arxiv 2502.01179 v4 pith:LRRX6LOI submitted 2025-02-03 cs.CL cs.AIcs.LG

Wen Lai , Alexander Fraser , Ivan Titov This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords parameter-efficientfine-tuningactivationeditinglow-resourceattentionheadselectionHardConcretegatesL0regularizationLLMadaptationcommonsensereasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes JoLA, a method for adapting large language models when only a few hundred training examples are available. Instead of fine-tuning weights with LoRA-style low-rank updates, JoLA learns, for each attention head, a small gate that decides whether the head should be edited and whether the edit is an additive offset, a multiplicative scaling, or both. The paper claims that this joint localization-and-editing approach consistently beats LoRA and all prior activation-editing baselines across 26 tasks in commonsense reasoning, natural language understanding, and natural language generation. If true, it shows that a sparse set of attention-head interventions is a stronger low-resource adaptation mechanism than updating weights, and that the right place to intervene can be learned rather than hand-selected.

What carries the argument

The load-bearing mechanism is the gated hybrid intervention on attention-head outputs. For head $(l,i)$ with output $z^{(l,i)}$, JoLA applies $z'^{(l,i)} = (1 + g_m^{(l,i)} m^{(l,i)}) \odot z^{(l,i)} + g_a^{(l,i)} a^{(l,i)}$, where $m$ and $a$ are learned vectors and $g_m, g_a \in [0,1]$ are scalar gates drawn from a HardConcrete distribution with a single learnable parameter each. Expected-$L_0$ regularization pushes most gates to exactly zero, so the model simultaneously learns where to intervene, whether each intervention is additive, multiplicative, or both, and the values of the edits; at inference each surviving gate is replaced by its expectation, making the intervention deterministic.

What would settle it

Re-run the Table 1 comparison with at least ten random 200-example samples per task and report confidence intervals; if the JoLA-versus-LoRA margin overlaps zero on most tasks, the consistent-superiority claim fails. A second check: fix the same number of active heads chosen at random or by LoFIT-style importance and train the same offset and scaling vectors; if random heads match JoLA, then the learned localization is not what carries the gain.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that attention-head outputs are the most impactful Transformer component to edit for task adaptation, and that the selection of which heads to edit and the choice of intervention type should be optimized jointly rather than fixed in advance. JoLA equips every head with two scalar HardConcrete gates, one for an additive offset vector and one for a multiplicative scaling vector, and trains them together with the vectors under expected-$L_0$ regularization. During training most gates close, so at inference only a small fraction of heads are altered; the paper reports an average reasoning accuracy of 70.55 for LLaMA-3 with JoLA versus 66.58 for LoRA, with similar margins on understanding and generation tasks and on Qwen-2.5. The paper also reports that combining attention-head edits with edits to other components hurts performance, which it reads as evidence that precision, not breadth, is what matters.

Load-bearing premise

The claim that attention heads are the best place to edit and that the learned gate selection generalizes rests on experiments run with a single 200-example sample per task using one random seed, so if the reported margins are mostly sampling noise the method's advantage is not established.

Editorial extensions

If this is right

  • A few hundred examples suffice to learn both the location and the form of effective edits, so low-resource fine-tuning no longer needs to rely on low-rank weight updates.
  • Attention heads alone carry the adaptation signal; editing additional components such as MLP outputs or bias terms adds parameters without adding accuracy.
  • The method's active parameter count is around 0.0002% of model parameters, roughly one-thousandth of the LoRA budget reported in the paper, which lowers the storage and serving cost of each adapted model.
  • The advantage persists from 100 training examples up to about 10,000 and across model sizes from 1B to 70B; only at very large data scales does LoRA catch up.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable implication the paper leaves implicit: the heads selected per task could serve as a map of the model's task-specific reasoning circuitry, so comparing gate patterns across the 26 tasks might reveal shared versus task-specific mechanisms.
  • Because the gates are trained end-to-end with the edits, the method could be combined with inference-time steering: the learned additive offsets may double as interpretable directions that can be inspected or transferred to other prompts.
  • The data-scaling curves suggest that the small active-parameter budget caps expressiveness in data-rich regimes; testing whether raising per-head capacity restores the advantage over LoRA at 20,000-100,000 examples is a natural extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes JoLA, a low-resource fine-tuning method that jointly learns which attention heads to edit, whether to apply additive offsets, multiplicative scalings, or both, and the intervention vectors themselves. Training uses HardConcrete gates with expected-L0 regularization, and inference prunes gates with low expected values. The method is evaluated on 26 tasks spanning commonsense reasoning, MMLU-Pro, and GEM generation benchmarks with 200 training examples per task, comparing against LoRA, BitFit, RED, RePE, ReFT, and LoFIT on LLaMA-3 and Qwen-2.5. The authors report average accuracy, BLEU, ROUGE-L, and BERTScore, and include ablations on gate mechanisms, gate count, head-selection strategies, data size, and model size, concluding that JoLA consistently outperforms all baselines.

Significance. If the empirical claim holds, the contribution is practically useful: JoLA unifies localization and intervention in a single end-to-end framework, removes manual head selection, and achieves strong low-resource adaptation with very few active parameters. The paper includes a broad evaluation across 26 tasks, useful ablations, and released code, which are concrete strengths. The main methodological novelty is moderate—it combines known HardConcrete pruning with LoFIT-style head interventions—but the joint formulation and per-head hybrid gating are sensible and clearly motivated by the Q1/Q2 analyses. However, the central claim of consistent superiority rests on a single training draw per task without variance estimation, and some stated parameter-efficiency claims are internally inconsistent. With additional statistical support and hyperparameter disclosure, the paper could be a solid contribution.

major comments (4)
  1. [Appendix D / Section 5] Appendix D states that all low-resource training sets were drawn once with seed=42, and no confidence intervals, repeated seeds, or variance estimates are reported anywhere. With only 200 training examples per task, the point-estimate margins in Table 1 (e.g., 70.55 vs 66.58 for LLaMA-3 reasoning) cannot be distinguished from sampling noise, and the central claim that JoLA 'consistently outperforms all baselines... by a significant margin' is not statistically supported. Please add multi-seed runs (or at least paired bootstrap confidence intervals) and report per-task variance, or explicitly soften the consistency claim to what the evidence supports.
  2. [Section 3.2 / Eq. (5) / Appendix F] The sparsity weight λ in Eq. (5) and the inference threshold ε in Section 3.2 are never reported in the main text or in Appendix F, even though they directly control the trained sparsity and the active-parameter count. Appendix F.3 describes a validation-based grid search for baseline hyperparameters; without equivalent reporting of λ and ε for JoLA, the comparison cannot be judged symmetric. Please report the values used for every task or run and include a sensitivity analysis for both hyperparameters.
  3. [Appendix B / Table 3] Appendix B's comparative analysis states that 'JOLA is more parameter-efficient than LoFIT', but Table 3 lists JOLA trainable parameters as 0.0065% versus LoFIT's 0.0035%, with identical active parameters (0.0002% for both). The claim is therefore contradicted by the paper's own table if 'parameter-efficient' refers to trainable parameters. Please clarify whether the intended notion is active or trainable parameters, and correct the statement or the table accordingly.
  4. [Abstract / Section 5 / Tables 10–11] The abstract and Section 5 say that JoLA 'consistently outperforms' all baselines, but the per-task tables contain clear counterexamples. Table 11 shows zero-shot accuracy of 75 on MMLU-Pro Biology versus JoLA's 70 for LLaMA-3, and Table 10 shows zero-shot accuracy of 87.38 on PIQA versus JoLA's 87.05 for Qwen-2.5. The claim should be quantified (e.g., average across tasks, or number of tasks where JoLA wins) rather than stated categorically as consistent superiority.
minor comments (6)
  1. [Throughout] The method name is spelled inconsistently as 'JoLA' in the title and abstract and 'JOLA' in most of the main text and figures; please standardize.
  2. [Section 4.3 / Appendix F.3] Section 4.3 says the authors 'select five hyperparameters and average the results', while Appendix F.3 says a validation set selects the best configuration per task and then results are averaged across tasks; please clarify which procedure produced Table 1.
  3. [Table 3 / Appendix A] Table 3 lists LoRA as 'Robust? No', but Appendix A describes LoRA as 'relatively robust' and notes that it still requires manual rank tuning; please align these characterizations.
  4. [Appendix F.3 / Table 7] The hyperparameter search description would benefit from stating how the validation set was constructed and how many tasks used each of the five configurations, since Table 7 lists a wider range of values than five for some baselines.
  5. [Figures 4 and 7] Figures 4 and 7 are hard to read in print: the legend markers are small, and the axes in Figure 7 lack explicit labels explaining whether the plotted probability is P(gate=0) or P(gate=1). Please enlarge and relabel.
  6. [Minor typos] There are several typographical issues, including 'V oita' in the references, 'L OFIT' in Table 4, 'seperately' in Table 4, and inconsistent spacing in 'T ransactions'; a copyedit pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: JoLA's gains are measured on held-out test sets, and the head-editing design is motivated by controlled experiments rather than by construction.

full rationale

JoLA is an empirical methods paper. Its claimed contribution—that jointly learning attention-head gates and additive/multiplicative intervention vectors outperforms PEFT and activation-editing baselines—is supported by training on 200-example samples and evaluating on held-out test sets (Section 4.3, Tables 9-14). No equation in the derivation sets the reported accuracies equal to a fitted parameter or to the gate objective by construction; Eq. 2 defines the intervention, Eq. 5 is the training loss, and the test metrics are measured on data not used to fit m, a, or the gate parameters. The self-citations (Voita et al. 2019; Schlichtkrull et al. 2021) are cited only as sources of the expected-L0/Hard-Concrete pruning mechanism, not as evidence for the target result. The decision to edit attention heads is motivated by the controlled comparison in Figure 2, which is an empirical observation rather than an imported uniqueness theorem or ansatz. Concerns about a single seed (seed=42, Appendix D) and unreported lambda and epsilon values are threats to the stability of the empirical ranking, not circularity, and they are checkable with the released code. I therefore find no circular step that reduces the central claim to its own inputs.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method rests on standard ML machinery: HardConcrete gates, expected-L0 regularization, and cross-entropy training. No new physical or theoretical entities are introduced. The main unstated premises are that head-output editing suffices and that single-seed low-resource evaluation is representative. Several hyperparameters that control sparsity and inference behavior are not reported.

free parameters (4)
  • Sparsity regularization weight lambda = not reported
    Equation (5) defines the trade-off between task loss and expected-L0 sparsity; no value or tuning procedure is given in the main text, yet it controls how many heads are edited and therefore the active parameter count.
  • Inference gate threshold epsilon = not reported
    Section 3.2 says interventions with expected gate value below epsilon are disregarded, but the value of epsilon is not stated, which affects the reported active parameter counts.
  • HardConcrete temperature = 0.33
    Fixed to 0.33 following prior sparsification work, but it controls the sharpness of the gate distribution and is not ablated.
  • Learning rate schedule constants = lr0=5e-4, lambda=0.1, decay=0.01
    Equation (9) defines the exponential decay schedule; these constants were chosen after comparing four schedules on a small set of tasks and then applied uniformly.
assumptions (3)
  • domain assumption HardConcrete gates with expected-L0 regularization induce useful sparse selections in low-data regimes.
    Section 3.2, Equations (4) and (5); this is a borrowed modeling assumption from pruning literature, not proven for activation editing here.
  • domain assumption Task adaptation can be achieved by editing attention-head outputs only, without weight updates.
    Section 3.1 Q1 concludes attention heads are the most impactful component based on a limited comparison over six tasks; the whole method depends on this.
  • domain assumption The 200-example samples with seed 42 are representative enough to support the claimed ranking of methods.
    Section 4.3 and Appendix D state a single seed is used for data sampling; with low-resource data, sampling variance could be substantial.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Joint Localization and Activation Editing for Low-Resource Fine-Tuning." pith.science (2026). https://pith.science/paper/LRRX6LOI

@misc{pith2026250201179,
  author       = {Pith},
  title        = {Pith review of: Joint Localization and Activation Editing for Low-Resource Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LRRX6LOI}},
  note         = {Machine review of arXiv:2502.01179}
}
read the original abstract

Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, are commonly used to adapt LLMs. However, the effectiveness of standard PEFT methods is limited in low-resource scenarios with only a few hundred examples. Recent advances in interpretability research have inspired the emergence of activation editing (or steering) techniques, which modify the activations of specific model components. Due to their extremely small parameter counts, these methods show promise for small datasets. However, their performance is highly dependent on identifying the correct modules to edit and often lacks stability across different datasets. In this paper, we propose Joint Localization and Activation Editing (JoLA), a method that jointly learns (1) which heads in the Transformer to edit (2) whether the intervention should be additive, multiplicative, or both and (3) the intervention parameters themselves - the vectors applied as additive offsets or multiplicative scalings to the head output. Through evaluations on three benchmarks spanning commonsense reasoning, natural language understanding, and natural language generation, we demonstrate that JoLA consistently outperforms existing methods. The code for the method is released at https://github.com/wenlai-lavine/jola.

Figures

Figures reproduced from arXiv: 2502.01179 by the authors.

Figure 1
Figure 1. Comparison of previous representative activation editing methods with proposed JOLA. (a) includes BitFIT (Ben Zaken et al., 2022), which fine-tunes only the bias term; RED (Wu et al., 2024a) introduces scaling and bias vectors in the MLP layer; ReFT (Wu et al., 2024b), which fine-tunes the hidden layer representations; and LoFIT (Yin et al., 2024) intervenes with attention heads in two steps. (b) JOLA introduces a g… view at source ↗
Figure 2
Figure 2. Performance comparison of activation editing across different Transformer modules: bias terms, MLP layers, hidden states, and attention heads. Q1: Component Selection. We evaluate activation edit￾ing across four Transformer components: bias terms, MLP layers, hidden states, and attention heads4 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the performance impact of scaling factors versus bias offsets in activation editing. tions for more effective adaptation. 3.2. Joint Localization and Editing Based on our insights from Section 3.1, JOLA focuses on adaptive attention head interventions to maximize activation editing effectiveness. Existing methods like LoFIT (Yin et al., 2024) require manual hyperparameter tuning to select the number of… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Performance comparison of JOLA and baseline methods across commonsense reasoning, natural language understanding, and natural language generation tasks for LLaMA-3 (Dubey et al., 2024) and Qwen-2.5 (Yang et al., 2024). Commonsense Reasoning. For commonsense reasoning, …
Figure 5
Figure 5. Figure 5: Ablation 2: Performance comparison of models with separate gating units for scaling and offset vectors versus a shared gating unit. 6.2. Ablation 2: Number of Gates In Equation (2), we employ separate gating units for the scaling vector and the bias vector. To investig…
Figure 6
Figure 6. Figure 6: Ablation 3: Comparison of different head selection strategies: SMP, DSP, PASS, and JOLA. 6.3. Ablation 3: Different Head Selection Strategies Head selection is a critical component of JOLA’s design. To evaluate whether alternative selection strategies could achieve sim…
Figure 7
Figure 7. Figure 7: Gate pruning probabilities for the additive gate (ga) and multiplicative gate (gm) during training on the OBQA dataset. A probability of 1 indicates a fully closed gate for the corresponding attention head. 100 200 300 400 500 600 800 1000 60 63 66 69 72 75 Zero-shot: …
Figure 9
Figure 9. Figure 9: Performance comparison of JOLA across different model sizes: Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, and Llama￾3.1-70B-Instruct. 8. Conclusions In this paper, we introduce JOLA, a novel approach to low-resource fine-tuning that jointly learns to dynamically local…
Figure 10
Figure 10. Figure 10: Performance comparison of interventions across different Transformer components and training sample sizes. Empirical Observations. JOLA demonstrates robust performance across 26 NLP tasks, particularly under low-resource settings. In addition to improved accuracy, JOL…
Figure 11
Figure 11. Figure 11: Performance comparison across six tasks under different experimental settings. The three subplots illustrate the sensitive of various configurations on task performance: (a) Different learning rates in RED (5e-5, 2e-4, 6e-2), (b) Different prefix and suffix positions …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 11 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    B it F it: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Ben Zaken, E., Goldberg, Y., and Ravfogel, S. B it F it: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp.\ 1--9, Dublin, Ireland, May 2022. Associ...

    doi:10.18653/v1/2022.acl-short.1 2022

  3. [3]

    Piqa: Reasoning about physical commonsense in natural language

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

    2020

  4. [4]

    Cheng, H., Zhang, M., and Shi, J. Q. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46 0 (12): 0 10558--10578, 2024. doi:10.1109/TPAMI.2024.3447085

    work page arXiv 2024

  5. [5]

    Examining modularity in multilingual LM s via language-specialized subnetworks

    Choenni, R., Shutova, E., and Garrette, D. Examining modularity in multilingual LM s via language-specialized subnetworks. In Duh, K., Gomez, H., and Bethard, S. (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp.\ 287--301, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi:10.18653/v1/2024.findi...

    work page doi:10.18653/v1/2024.findings-naacl.21 2024

  6. [6]

    B ool Q : Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. B ool Q : Exploring the surprising difficulty of natural yes/no questions. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol...

    doi:10.18653/v1/ 2019

  7. [7]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  8. [8]

    S., Desai, A., Poli, M., Grogan, J., Liu, A., Rao, A., Rudra, A., and R \'e , C

    Dao, T., Chen, B., Sohoni, N. S., Desai, A., Poli, M., Grogan, J., Liu, A., Rao, A., Rudra, A., and R \'e , C. Monarch: Expressive structured matrices for efficient and accurate training. In International Conference on Machine Learning, pp.\ 4690--4721. PMLR, 2022

    work page 2022

Show all 56 references
  1. [9]

    Qlora: Efficient finetuning of quantized llms

    Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

  2. [10]

    Ding, D., Jawahar, G., and Lakshmanan, L. V. S. PASS : Pruning attention heads with almost-sure sparsity targets. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=S4duStTKGL

    2024

  3. [11]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  4. [12]

    and Alistarh, D

    Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp.\ 10323--10337. PMLR, 2023

    2023

  5. [13]

    Creating training corpora for NLG micro-planners

    Gardent, C., Shimorina, A., Narayan, S., and Perez-Beltrachini, L. Creating training corpora for NLG micro-planners. In Barzilay, R. and Kan, M.-Y. (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 179...

    2017 doi

  6. [14]

    I., Strobelt, H., Hayashi, H., Novikova, J., Kanerva, J., Chim, J., Zhou, J., Clive, J., Maynez, J., Sedoc, J., Juraska, J., Dhole, K., Chandu, K

    Gehrmann, S., Bhattacharjee, A., Mahendiran, A., Wang, A., Papangelis, A., Madaan, A., Mcmillan-major, A., Shvets, A., Upadhyay, A., Bohnet, B., Yao, B., Wilie, B., Bhagavatula, C., You, C., Thomson, C., Garbacea, C., Wang, D., Deutsch, D., Xiong, D., Jin, D., Gkatzia, D., Rad...

    2022

  7. [15]

    Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S. Q. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608, 2024

    2024 arXiv

  8. [16]

    X., and Hasan, S

    He, J., Rungta, M., Koleczek, D., Sekhon, A., Wang, F. X., and Hasan, S. Does prompt formatting have any impact on llm performance? arXiv preprint arXiv:2411.10541, 2024

    2024 arXiv

  9. [17]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  10. [18]

    Parameter-efficient transfer learning for nlp

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp.\ 2790--2799. PMLR, 2019

    2019

  11. [19]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  12. [20]

    LLM -adapters: An adapter family for parameter-efficient fine-tuning of large language models

    Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.-P., Bing, L., Xu, X., Poria, S., and Lee, R. LLM -adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical...

    2023 doi

  13. [21]

    Categorical reparameterization with gumbel-softmax

    Jang, E., Gu, S., and Poole, B. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=rkE3y85ee

    2017

  14. [22]

    LLM s beyond E nglish: Scaling the multilingual capability of LLM s with cross-lingual feedback

    Lai, W., Mesgar, M., and Fraser, A. LLM s beyond E nglish: Scaling the multilingual capability of LLM s with cross-lingual feedback. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 8186--8213, Bangko...

    2024 doi

  15. [23]

    Differentiable subset pruning of transformer heads

    Li, J., Cotterell, R., and Sachan, M. Differentiable subset pruning of transformer heads. Transactions of the Association for Computational Linguistics, 9: 0 1442--1459, 2021. doi:10.1162/tacl_a_00436. URL https://aclanthology.org/2021.tacl-1.86/

    2021 doi

  16. [24]

    and Arora, S

    Li, Z. and Arora, S. An exponential learning rate schedule for deep learning. arXiv preprint arXiv:1910.07454, 2019

    1910 arXiv

  17. [25]

    Y., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y., and Ren, X

    Lin, B. Y., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y., and Ren, X. C ommon G en: A constrained text generation challenge for generative commonsense reasoning. In Cohn, T., He, Y., and Liu, Y. (eds.), Findings of the Association for Computational Linguistics: EMNL...

    2020 doi

  18. [26]

    ROUGE : A package for automatic evaluation of summaries

    Lin, C.-Y. ROUGE : A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp.\ 74--81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013/

    2004

  19. [27]

    Decoupled weight decay regularization

    Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  20. [28]

    Louizos, C., Welling, M., and Kingma, D. P. Learning sparse neural networks through l\_0 regularization. In International Conference on Learning Representations, 2018

    2018

  21. [29]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural L...

    2018 doi

  22. [30]

    A., Veness, J., Bellemare, M

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518 0 (7540): 0 529--533, 2015

    2015

  23. [31]

    B., and Lapata, M

    Narayan, S., Cohen, S. B., and Lapata, M. Don`t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Method...

    2018 doi

  24. [32]

    and Sennrich, R

    Nooralahzadeh, F. and Sennrich, R. Improving the cross-lingual generalisation in visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 13419--13427, 2023

    2023

  25. [33]

    The E 2 E dataset: New challenges for end-to-end generation

    Novikova, J., Du s ek, O., and Rieser, V. The E 2 E dataset: New challenges for end-to-end generation. In Jokinen, K., Stede, M., DeVault, D., and Louis, A. (eds.), Proceedings of the 18th Annual SIG dial Meeting on Discourse and Dialogue , pp.\ 201--206, Saarbr \"u cken, Germ...

    2017 doi

  26. [34]

    B leu: a method for automatic evaluation of machine translation

    Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. B leu: a method for automatic evaluation of machine translation. In Isabelle, P., Charniak, E., and Lin, D. (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp.\ 311--318, Philad...

    2002

  27. [35]

    Identifying semantic induction heads to understand in-context learning

    Ren, J., Guo, Q., Yan, H., Liu, D., Zhang, Q., Qiu, X., and Lin, D. Identifying semantic induction heads to understand in-context learning. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 6916--6932,...

    2024 doi

  28. [36]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  29. [37]

    Social IQ a: Commonsense reasoning about social interactions

    Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social IQ a: Commonsense reasoning about social interactions. In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Intern...

    2019 doi

  30. [38]

    S., De Cao, N., and Titov, I

    Schlichtkrull, M. S., De Cao, N., and Titov, I. Interpreting graph neural networks for nlp with differentiable edge masking. ICLR, 2021

    2021

  31. [39]

    Smith, L. N. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pp.\ 464--472. IEEE, 2017

    2017

  32. [40]

    Smith, L. N. A disciplined approach to neural network hyper-parameters: Part 1--learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09820, 2018

    2018 arXiv

  33. [41]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  34. [42]

    Spdf: Sparse pre-training and dense fine-tuning for large language models

    Thangarasa, V., Gupta, A., Marshall, W., Li, T., Leong, K., DeCoste, D., Lie, S., and Saxena, S. Spdf: Sparse pre-training and dense fine-tuning for large language models. In Uncertainty in Artificial Intelligence, pp.\ 2134--2146. PMLR, 2023

    2023

  35. [43]

    Causal mediation analysis for interpreting neural nlp: The case of gender bias

    Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Sakenis, S., Huang, J., Singer, Y., and Shieber, S. Causal mediation analysis for interpreting neural nlp: The case of gender bias. arXiv preprint arXiv:2004.12265, 2020

    2004 arXiv

  36. [44]

    Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned

    Voita, E., Talbot, D., Moiseev, F., Sennrich, R., and Titov, I. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Korhonen, A., Traum, D., and M \`a rquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Associatio...

    2019 doi

  37. [45]

    Parameter-efficient fine-tuning in large models: A survey of methodologies

    Wang, L., Chen, S., Jiang, L., Pan, S., Cai, R., Yang, S., and Yang, F. Parameter-efficient fine-tuning in large models: A survey of methodologies. arXiv preprint arXiv:2410.19878, 2024 a

    2024 arXiv

  38. [46]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024 b

    2024 arXiv

  39. [47]

    Advancing parameter efficiency in fine-tuning via representation editing

    Wu, M., Liu, W., Wang, X., Li, T., Lv, C., Ling, Z., JianHao, Z., Zhang, C., Zheng, X., and Huang, X. Advancing parameter efficiency in fine-tuning via representation editing. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Ass...

    2024 doi

  40. [48]

    D., and Potts, C

    Wu, Z., Arora, A., Wang, Z., Geiger, A., Jurafsky, D., Manning, C. D., and Potts, C. Reft: Representation finetuning for language models. arXiv preprint arXiv:2404.03592, 2024 b

    2024 arXiv

  41. [49]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  42. [50]

    Lofit: Localized fine-tuning on llm representations

    Yin, F., Ye, X., and Durrett, G. Lofit: Localized fine-tuning on llm representations. arXiv preprint arXiv:2406.01563, 2024

    2024 arXiv

  43. [51]

    H ella S wag: Can a machine really finish your sentence? In Korhonen, A., Traum, D., and M \`a rquez, L

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. H ella S wag: Can a machine really finish your sentence? In Korhonen, A., Traum, D., and M \`a rquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4...

    2019 doi

  44. [52]

    and Nanda, N

    Zhang, F. and Nanda, N. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arXiv:2309.16042, 2023

    2023 arXiv

  45. [53]

    Q., and Artzi, Y

    Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  46. [54]

    Know what you don't need: Single-shot meta-pruning for attention heads

    Zhang, Z., Qi, F., Liu, Z., Liu, Q., and Sun, M. Know what you don't need: Single-shot meta-pruning for attention heads. AI Open, 2: 0 36--42, 2021

    2021

  47. [55]

    X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023

    2023 arXiv

  48. [56]

    Representation engineering: A top-down approach to ai transparency

    Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.