Pith. sign in

REVIEW 5 major objections 7 minor 84 references

PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment

T0 review · 5 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PARA inserts a lightweight prompt-conditional vector generator into each Transformer layer and claims it outperforms LoRA and AdaLoRA on five benchmarks while reducing per-token serving latency.

desk verdict A plausible PEFT variant with a real multi-tenant latency story, but the accuracy claims are not yet supported and one table contains an impossible COPA number. read the letter →

arxiv 2502.01033 v1 pith:RKFTC7WN submitted 2025-02-03 cs.CL

Zequan Liu , Yi Zhao , Ming Tan , Wei Zhu , Aaron Xuxiang Tian This is my paper · ORCID
classification cs.CL
keywords parameter-efficientfine-tuningprompt-awareadaptationvectorgeneratorrepresentationadjustmentLoRAKVcachemulti-tenantservinginstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PARA is a parameter-efficient fine-tuning method that places a small vector generator before each Transformer layer. The generator reads the pooled hidden state of the input prompt and emits three vectors that element-wise rescale the Query and Value projections in attention and the Up projection in the feed-forward network, so the adaptation is prompt-specific rather than fixed after training. On five tasks spanning reading comprehension, binary question answering, math reasoning, and SQL generation, the authors report median-of-five-seed scores above LoRA, AdaLoRA, adapters, and (IA)^3 with roughly the same or fewer tunable parameters (8.9M on a 7B backbone). Because the vectors are computed once when the prompt is first processed and reused with the KV cache, PARA keeps per-token inference cost close to (IA)^3 and below LoRA, and the same accuracy pattern appears on Llama-2 13B and Gemma 2B.

What carries the argument

The carrying object is the vector generator (VG), a bottleneck module added before each Transformer layer. It consists of a pooling step over the prompt's hidden states, a down-projection to dimension $r=12$, a GELU activation, and an up-projection with bias that outputs a vector of size $2d_{\text{model}}+d_{\text{ffn}}$, split into $l_q$, $l_v$, $l_u$. These vectors are applied as Hadamard products to the Query, Value, and Up hidden states, and the mechanism's key property is that the generator runs only once per prompt, before generation, so its cost is amortized over all generated tokens and it composes with the KV cache.

What would settle it

Re-run the five benchmark comparisons with a full hyperparameter sweep over LoRA and AdaLoRA budgets (rank, alpha, dropout, learning rate), report per-seed scores with standard deviations and confidence intervals, and check whether PARA separates from the best-tuned baseline. A complementary check is to replace the final-token pooler with mean pooling and see whether the accuracy advantage survives.

Watch

Extended reading notes

Core claim

The central claim is that a prompt-aware, per-layer rescaling of hidden representations can match or beat reparameterization-based PEFT. In each Transformer layer, a vector generator takes the hidden states of the input prompt, pools them at the final token, projects through a bottleneck of dimension $r=12$, and outputs vectors $l_q$, $l_v$, $l_u$ that modify attention and feed-forward computations as $Q'=l_q\odot Q$, $V'=l_v\odot V$, and $U'=l_u\odot U$. This makes the adjustment depend on the prompt, unlike (IA)^3, and it avoids per-token low-rank matrix multiplications, unlike LoRA. The reported experiments use Llama-2 7B as the main backbone and show PARA above all baselines on HSM10K, Q2SQL, SQuAD, BoolQ, and COPA at 8.9M tunable parameters, with the same pattern on Llama-2 13B and Gemma 2B.

Load-bearing premise

The superiority claim depends on the baselines being tuned as aggressively as PARA; the paper reports median scores over five seeds but no variance, significance tests, or baseline hyperparameters, so a mis-tuned LoRA or AdaLoRA could account for the reported gaps.

Editorial extensions

If this is right

  • If the reported results hold, PEFT no longer has to choose between (IA)^3-style speed and LoRA-style expressiveness: prompt-conditioned rescaling gives both.
  • In a single-backbone multi-tenant deployment, PARA removes the per-token low-rank matrix multiplications that LoRA adds, so tenants can share the base model without a per-tenant latency penalty.
  • The one-time-per-prompt calculation of the adjusting vectors slots directly into the KV-cache generation path, so the method's inference cost does not grow with generation length.
  • The same design transferred to Llama-2 13B and Gemma 2B, suggesting the benefit is not specific to one backbone size.
  • Instruction tuning with PARA also improved MT-Bench, MMLU, and the internal LLM-Eval1 benchmark over AdaLoRA, pointing to general-purpose use beyond task-specific fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's tasks, the prompt-conditioned vector generator suggests a testable continuum: interpolating between fixed vectors as in (IA)^3 and fully per-token adaptation by regenerating vectors at intervals during long generations.
  • The use of the final token's pooled hidden state as the prompt summary is a choice the paper does not ablate; mean or attention-pooled summaries could behave differently for long or multi-turn prompts, and that is a cheap experiment.
  • If the accuracy gains are real, they imply that a low-dimensional readout of the prompt (here $r=12$) carries enough information to route model behavior, connecting PEFT to conditional computation and mixture-of-expert style routing.
  • The method's multi-tenant benefit depends on prompts being short enough to amortize the one-time generator cost; very long prompts or streaming inputs would erode the latency advantage, an edge case the paper does not address.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces PARA, a parameter-efficient fine-tuning method that inserts a lightweight "vector generator" into each Transformer layer. The generator pools the hidden state of the input prompt, projects it through a bottleneck, and outputs three scaling vectors that are applied elementwise to the Q and V projections in self-attention and to the U (up) projection in the FFN. The authors report results on five supervised benchmarks (SQuAD, BoolQ, COPA, HSM10K, Q2SQL) and on MT-Bench, MMLU, and a proprietary LLM-Eval1 after instruction tuning, and they measure inference speed and memory in a multi-tenant setting. They claim that PARA outperforms existing PEFT methods such as LoRA, AdaLoRA, (IA)3, adapters, and SSP with comparable or fewer tunable parameters, while being substantially faster than LoRA during decoding because the scaling vectors are computed once per prompt and reused via the KV cache.

Significance. If the performance claims were adequately supported, PARA would be a simple and useful PEFT method with a genuine practical advantage: it is architecturally straightforward, compatible with the KV cache, and shows a latency benefit over LoRA in single-backbone multi-tenant inference. The efficiency/memory comparison in Table 4 is credible and could support a focused contribution on deployment-oriented PEFT. However, the paper's central accuracy-superiority claim currently rests on small, statistically unsupported margins over baselines whose tuning is not documented, and on several unverifiable proprietary datasets. The method itself is plausible, but the empirical demonstration is not yet convincing enough to justify the abstract's claim that PARA "surpasses current PEFT benchmarks."

major comments (5)
  1. [§4.4 (Tables 2 and 3)] The central claim that PARA outperforms all standard methods on the five benchmarks is supported only by median-of-five-seed point estimates. No standard deviations, confidence intervals, or significance tests are provided, and the margins over the second-best baseline are 0.6–1.0 points in Table 2 and 0.08–0.9 points in Table 3, which are of the same order as typical seed variance for these tasks. The authors should report per-seed results with dispersion, perform a significance test (e.g., bootstrap or paired test), and show that the margins are not within run-to-run variation.
  2. [Tables 1 and 2] The COPA test set is listed as 0.05k (50 examples) in Table 1, but Table 2 reports COPA accuracies of 91.9 and 92.0 with a "median over five seeds" protocol. With only 50 test examples, each correct classification changes accuracy by 2 percentage points, so the reported resolutions of 91.9 and 92.0 are not representable as medians of integer counts, and the 0.8-point PARA-vs-AdaLoRA gap is below the resolution of a single test example. This indicates either a different evaluation protocol (e.g., averaging or a larger test set) or an arithmetic error; please clarify and correct.
  3. [§4.3 vs Tables 2–3] Section 4.3 states that all main experiments use "LlaMA-2 7B released by Meta," but Tables 2 and 3 label the backbone as "LLM-Assist 7B," which is never defined. If LLM-Assist is a different model or a fine-tuned variant of LlaMA-2, the parameter counts and comparisons need to be revisited; if it is just a rename, the text should say so. As written, the reader cannot tell which model was actually evaluated, which undermines reproducibility.
  4. [§4.3 (baseline hyperparameters)] Hyperparameters are specified for PARA (bottleneck r=12, learning rate 1e-4, batch sizes, etc.), but no hyperparameter search or final settings are reported for LoRA, AdaLoRA, (IA)3, Houlsby-Adapter, Learned-Adapter, or SSP. The baselines are said to be implemented with Transformers/PEFT or the original code, but no ranks, alphas, dropout rates, or tuning budgets are given. Without evidence that the baselines were tuned to a comparable level of effort, the reported improvements in Tables 2–3 could reflect under-tuned baselines rather than a genuine advantage of PARA.
  5. [§4.2 and Appendix A] Three of the datasets used for the main comparison (HSM10K, Q2SQL, LLM-Eval1) are proprietary and cannot be inspected or accessed by readers. This is especially concerning for HSM10K and Q2SQL in Table 2, where the claimed wins are among the largest (0.7 and 0.6 points). Unless these datasets are released or a reproducible public surrogate is provided, an external reader cannot verify the data, the annotation protocol, or the evaluation script, which weakens the verifiability of the central performance claim.
minor comments (7)
  1. [Equation (1)] The softmax expression appears malformed; it should be softmax(QK^T / sqrt(d_model))V rather than "softmax( QK√dmodel ) V." Please correct the typesetting.
  2. [§4.3] "Computing infrastures" is a typo for "Computing infrastructure."
  3. [Table 2] "Housbly-Adapter" should be "Houlsby-Adapter."
  4. [§4.4] "M MLU" should be "MMLU."
  5. [Appendix A.1 (UltraChat)] The sentence "split it into a 99:1 train/validate set for the FanLoRA workflow" appears to be a leftover from another manuscript; FanLoRA is not otherwise discussed, and the split purpose should be stated in terms of the current paper or removed.
  6. [§3.2 (vector generator)] The pooling mechanism is underspecified: it is unclear whether the "input prompt" is the full input sequence or only the instruction prefix, how the final prompt token is identified when the input includes the target during training, and whether the generated vectors are applied to prompt tokens themselves as well as to generated tokens. Please clarify the training-time procedure and how the pooled token is chosen.
  7. [Limitations section] The Limitations section acknowledges the lack of experiments on 70B-scale models and on other task types, but it does not mention the statistical reliability issues (absence of error bars or significance tests) or the fact that key benchmarks are proprietary. The limitations discussion should also cover these empirical-support constraints.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claims are empirical evaluation results, and no derivation step reduces to its inputs or to a load-bearing self-citation.

full rationale

The paper's central claims are that PARA outperforms PEFT baselines on five benchmarks at comparable or lower parameter counts and that PARA has lower multi-tenant inference latency than LoRA. Both are empirical evaluation claims rather than results derived from assumptions. The method is defined in Section 3.2 as a vector generator VG(h) that produces lq, lv, lu and rescales Q, V, and U; the generator is a concrete bottleneck architecture with stated initialization and no hidden reference to the reported outcomes. No equation or theorem in the paper derives the reported superiority; the numbers in Tables 2-5 are the evaluation itself. There is no fitted parameter later renamed as a prediction: the tuning protocol in Section 4.3 is the standard train-on-train, select-on-dev, report-on-test workflow. Self-citations are numerous in the introduction, related work, and references, but none is load-bearing for the central claims: the latency argument rests on the KV-cache compatibility observation in Sections 3.2 and 4.5, and the comparison procedure is stated in Section 4.3. The citations to prior PEFT work by the same authors (AloRA, FanLoRA, MiLoRA, IAPT) are contextual background, not evidence for PARA's superiority. The limitations section is an honest scope caveat about unexamined larger models and other tasks, and it does not assert any missing derivation. The legitimate concerns raised by the manuscript are statistical and reproducibility issues, such as the absence of variance or significance testing, the inconsistent backbone naming between 'LLM-Assist 7B' and LlaMA-2 7B, the COPA median granularity relative to the stated 50-example test set, and the proprietary nature of HSM10K, Q2SQL, and LLM-Eval1. These concerns affect whether the empirical superiority claim is convincing, but they are not circularity. Under the circularity standard requiring a quoted reduction of a claim to its own inputs or a load-bearing self-citation, no such step is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The method's empirical success rests on several unablated design choices (bottleneck size, target modules, initialization, pooling) and on the domain assumption that a single prompt-token summary suffices for all conditioning. No additional hidden entities are needed beyond the vector generator itself.

free parameters (4)
  • Bottleneck dimension r = 12
    Chosen by hand for the vector generator; no ablation study over r is reported, so its suitability is untested.
  • Adjustment target modules = Q, V, U (not K or G)
    Selected after unreported preliminary experiments (footnote 3), making it a data-dependent design choice rather than a fixed design.
  • Vector generator initialization = W_down ~ N(0, 0.02), W_up = 0, b_up = 1
    Chosen to make the initial scaling vectors equal to one; no ablation of this initialization is presented.
  • Pooling method = Final token hidden state of prompt
    Adopted from GPT/BART conventions (Radford 2018); no comparison with mean pooling or other pooling is shown.
assumptions (4)
  • domain assumption The hidden state of the final prompt token contains sufficient information to generate effective per-layer scaling vectors for all positions and tokens.
    Core design premise of PARA; motivated by in-context learning analogies but not proven or ablated.
  • domain assumption Recomputing the scaling vectors only once during prompt processing and never during token generation does not degrade generation quality.
    Required for the efficiency claim; no experiment tests an alternative that recomputes vectors for each new token.
  • standard math The pretrained backbone weights remain frozen and only the vector generators are updated, following standard PEFT practice.
    Standard PEFT assumption; keeps the parameter count comparable to other PEFT methods.
  • domain assumption Evaluation with beam search (beam size 3) and the LM head is appropriate and comparable across all methods and tasks.
    The evaluation protocol presumes these decoding settings are non-biasing, but no sensitivity analysis is provided.
invented entities (1)
  • Vector Generator (VG) independent evidence
    purpose: Lightweight bottleneck network that maps the pooled prompt hidden state to three scaling vectors (lq, lv, lu) applied to Q, V, and U in each Transformer layer.
    This is the newly introduced module; it is fully specified and its performance is tested across tasks and backbones, so it is not an unfalsifiable postulated entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment." pith.science (2026). https://pith.science/paper/RKFTC7WN

@misc{pith2026250201033,
  author       = {Pith},
  title        = {Pith review of: PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RKFTC7WN}},
  note         = {Machine review of arXiv:2502.01033}
}
read the original abstract

In the realm of parameter-efficient fine-tuning (PEFT) methods, while options like LoRA are available, there is a persistent demand in the industry for a PEFT approach that excels in both efficiency and performance within the context of single-backbone multi-tenant applications. This paper introduces a new and straightforward PEFT technique, termed \underline{P}rompt \underline{A}ware \underline{R}epresentation \underline{A}djustment (PARA). The core of our proposal is to integrate a lightweight vector generator within each Transformer layer. This generator produces vectors that are responsive to input prompts, thereby adjusting the hidden representations accordingly. Our extensive experimentation across diverse tasks has yielded promising results. Firstly, the PARA method has been shown to surpass current PEFT benchmarks in terms of performance, despite having a similar number of adjustable parameters. Secondly, it has proven to be more efficient than LoRA in the single-backbone multi-tenant scenario, highlighting its significant potential for industrial adoption.

Figures

Figures reproduced from arXiv: 2502.01033 by the authors.

Figure 1
Figure 1. A schematic representation of our PARA approach is depicted below. On the left, the vector generator [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

84 extracted references · 44 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021. https://doi.org/10.18653/v1/2021.acl-long.568 Intrinsic dimensionality explains the effectiveness of language model fine-tuning . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V...

    doi:10.18653/v1/2021.acl-long.568 2021

  4. [4]

    Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. ArXiv, abs/2106.10199

    arXiv 2021

  5. [5]

    Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, Arvind Krishnamurthy University of Washington, and Duke University. 2023. https://api.semanticscholar.org/CorpusID:264590197 Punica: Multi-tenant lora serving . ArXiv, abs/2310.18547

    arXiv 2023

  6. [6]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. https://api.semanticscholar.org/CorpusID:263605623 Ultrafeedback: Boosting language models with high-quality feedback . ArXiv, abs/2310.01377

    arXiv 2023

  7. [7]

    Tim Dettmers , Artidoro Pagnoni , Ari Holtzman , and Luke Zettlemoyer . 2023. https://doi.org/10.48550/arXiv.2305.14314 QLoRA: Efficient Finetuning of Quantized LLMs . arXiv e-prints, page arXiv:2305.14314

    arXiv doi:10.48550/arxiv.2305.14314 2023

  8. [8]

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233

    arXiv 2023

Show all 84 references
  1. [9]

    Ning Ding, Yujia Qin, Guang Yang, Fu Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Haitao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juan Li, and Maosong Sun. 2022. Delta tuning: A comprehensiv...

    2022 arXiv

  2. [10]

    Wensheng Gan, Shicheng Wan, and S Yu Philip. 2023. Model-as-a-service (maas): A survey. In 2023 IEEE International Conference on Big Data (BigData), pages 4636--4645. IEEE

    2023

  3. [11]

    Xiangxiang Gao, Wei Zhu, Jiasheng Gao, and Congrui Yin. 2023. F-pabee: Flexible-patience-based early exiting for single-label and multi-label text classification tasks. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages...

    2023

  4. [12]

    Demi Guo, Alexander Rush, and Yoon Kim. 2021 a . https://doi.org/10.18653/v1/2021.acl-long.378 Parameter-efficient transfer learning with diff pruning . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint ...

    2021 doi

  5. [13]

    Zhao Guo, Yuan Ni, Keqiang Wang, Wei Zhu, and Guotong Xie. 2021 b . https://doi.org/10.18653/v1/2021.findings-acl.122 Global attention decoder for C hinese spelling error correction . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1419--14...

    2021 doi

  6. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  7. [15]

    Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv: Learning

    2016

  8. [16]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790--2799. PMLR

    2019

  9. [17]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  10. [18]

    Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang, Yasheng Wang, Zhiyuan Liu, and Maosong Sun. 2022. Sparse structure search for parameter-efficient tuning. ArXiv, abs/2206.07382

    2022 arXiv

  11. [19]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691

    2021 arXiv

  12. [20]

    Mike Lewis. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461

    2019 arXiv

  13. [21]

    Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long ...

    2021 arXiv

  14. [22]

    Xiaonan Li, Kai Lv, Hang Yan, Tianya Lin, Wei Zhu, Yuan Ni, Guo Tong Xie, Xiaoling Wang, and Xipeng Qiu. 2023. https://api.semanticscholar.org/CorpusID:258557751 Unified demonstration retriever for in-context learning . ArXiv, abs/2305.04320

    2023 arXiv

  15. [23]

    Xiepeng Li, Zhexi Zhang, Wei Zhu, Zheng Li, Yuan Ni, Peng Gao, Junchi Yan, and Guotong Xie. 2019. https://doi.org/10.18653/v1/D19-6011 Pingan smart health and SJTU at COIN - shared task: utilizing pre-trained language models and common-sense knowledge in machine reading tasks ...

    2019 doi

  16. [24]

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022 a . https://api.semanticscholar.org/CorpusID:248693283 Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning . ArXiv, abs/2205.05638

    2022 arXiv

  17. [25]

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022 b . P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Annual Meeting of the Association for Computational Linguistics

    2022

  18. [26]

    Zequan Liu, Jiawen Lyn, Wei Zhu, and Xing Tian. 2024. Alora: Allocating low-rank adaptation for fine-tuning large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ...

    2024

  19. [27]

    Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. In NeurIPS

    2021

  20. [28]

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft

    2022

  21. [29]

    OpenAI . 2023. https://doi.org/10.48550/arXiv.2303.08774 GPT-4 Technical Report . arXiv e-prints, page arXiv:2303.08774

    2023 doi arXiv

  22. [30]

    Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476

    2023 arXiv

  23. [31]

    Alec Radford. 2018. Improving language understanding by generative pre-training. OpenAI Blog

    2018

  24. [32]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-...

    2016 doi

  25. [33]

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633

    2021 arXiv

  26. [34]

    Andreas R \"u ckl \'e , Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2020. Adapterdrop: On the efficiency of adapters in transformers. In Conference on Empirical Methods in Natural Language Processing

    2020

  27. [35]

    Haixia Sun, Jin Xiao, Wei Zhu, Yilong He, Sheng Zhang, Xiaowei Xu, Li Hou, Jiao Li, Yuan Ni, and Guotong Xie. 2020. https://doi.org/10.2196/17653 Medical knowledge graph to enhance fraud, waste, and abuse detection on claim data: Model development and performance evaluation . ...

    2020 doi

  28. [36]

    Tianxiang Sun, Xiangyang Liu, Wei Zhu, Zhichao Geng, Lingling Wu, Yilong He, Yuan Ni, Guotong Xie, Xuanjing Huang, and Xipeng Qiu. 2022. https://doi.org/10.18653/v1/2022.findings-acl.189 A simple hash-based early exiting approach for language understanding and generation . In ...

    2022 doi

  29. [37]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  30. [38]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  31. [39]

    Aaron Tian, Yi Zhao, Congrui Yin, Wei Zhu, Xing Tian, and Yi Ge. 2024. Fanlora: Fantastic loras and where to find them in large language model fine-tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 515--528

    2024

  32. [40]

    Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M

    Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cant \'o n Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes,...

    2023 arXiv

  33. [41]

    Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

    Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. ArXiv, abs/1706.03762

    2017 arXiv

  34. [42]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. ArXiv, abs/1905.00537

    2019 arXiv

  35. [43]

    Li Wang, Wei Zhu, Sihang Jiang, Sheng Zhang, Keqiang Wang, Yuan Ni, Guo Tong Xie, and Yanghua Xiao. 2020. https://api.semanticscholar.org/CorpusID:224281022 Mining infrequent high-quality phrases from domain-specific corpora . Proceedings of the 29th ACM International Conferen...

    2020

  36. [44]

    Xuwu Wang, Lihan Chen, Wei Zhu, Yuan Ni, Guo Tong Xie, Deqing Yang, and Yanghua Xiao. 2023. https://api.semanticscholar.org/CorpusID:258975891 Multi-task entity linking with supervision from a taxonomy . Knowledge and Information Systems, 65:4335 -- 4358

    2023

  37. [45]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2020 a . Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical ...

    2020

  38. [46]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  39. [47]

    Wenjing Yue, Xiaoling Wang, Wei Zhu, Ming Guan, Huanran Zheng, Pengfei Wang, Changzhi Sun, and Xin Ma. 2024. Tcmbench: A comprehensive benchmark for evaluating large language models in traditional chinese medicine. arXiv preprint arXiv:2406.01126

    2024 arXiv

  40. [48]

    Jingfan Zhang, Yi Zhao, Dan Chen, Xing Tian, Huanran Zheng, and Wei Zhu. 2024. Milora: Efficient mixture of low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2410.18035

    2024 arXiv

  41. [49]

    Jingfang Zhang, Ming Tan, Pengyu Dai, and Wei-Guo Zhu. 2023 a . https://api.semanticscholar.org/CorpusID:259370796 Leco: Improving early exiting via learned exits and comparison-based exiting mechanism . In Annual Meeting of the Association for Computational Linguistics

    2023

  42. [50]

    Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao

    Qingru Zhang, Minshuo Chen, Alexander W. Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023 b . https://api.semanticscholar.org/CorpusID:257631760 Adaptive budget allocation for parameter-efficient fine-tuning . ArXiv, abs/2303.10512

    2023 arXiv

  43. [51]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi. 2020. Revisiting few-sample bert fine-tuning. ArXiv, abs/2006.05987

    2020 arXiv

  44. [52]

    Xinpeng Zhang, Ming Tan, Jingfan Zhang, and Wei Zhu. 2023 c . https://api.semanticscholar.org/CorpusID:259370837 Nag-ner: a unified non-autoregressive generation framework for various ner tasks . In Annual Meeting of the Association for Computational Linguistics

    2023

  45. [53]

    Yuming Zhang, Xiangxiang Gao, Wei Zhu, and Xiaoling Wang. 2023 d . https://api.semanticscholar.org/CorpusID:265214231 Fastner: Speeding up inferences for named entity recognition tasks . In International Conference on Advanced Data Mining and Applications

    2023

  46. [54]

    Yuming Zhang, Peng Wang, Ming Tan, and Wei-Guo Zhu. 2023 e . https://api.semanticscholar.org/CorpusID:259858833 Learned adapters are better than manually designed adapters . In Annual Meeting of the Association for Computational Linguistics

    2023

  47. [55]

    Zhen Zhang, Wei Zhu, Jinfan Zhang, Peng Wang, Rize Jin, and Tae-Sun Chung. 2022. https://doi.org/10.18653/v1/2022.findings-naacl.25 PCEE - BERT : Accelerating BERT inference via patient and confident early exiting . In Findings of the Association for Computational Linguistics:...

    2022 doi

  48. [56]

    Zhexi Zhang, Wei Zhu, Junchi Yan, Peng Gao, and Guowang Xie. 2021. Automatic student network search for knowledge distillation. 2020 25th International Conference on Pattern Recognition (ICPR), pages 2446--2453

    2021

  49. [57]

    Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Sch \"u tze. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.174 Masking as an efficient alternative to finetuning for pretrained language models . In Proceedings of the 2020 Conference on Empirical Methods in Natural ...

    2020 doi

  50. [58]

    Wayne Xin Zhao , Kun Zhou , Junyi Li , Tianyi Tang , Xiaolei Wang , Yupeng Hou , Yingqian Min , Beichen Zhang , Junjie Zhang , Zican Dong , Yifan Du , Chen Yang , Yushuo Chen , Zhipeng Chen , Jinhao Jiang , Ruiyang Ren , Yifan Li , Xinyu Tang , Zikang Liu , Peiyu Liu , Jian-Yu...

    2023 doi arXiv

  51. [59]

    Huanran Zheng, Wei Zhu, Pengfei Wang, and Xiaoling Wang. 2023. https://api.semanticscholar.org/CorpusID:256358677 Candidate soups: Fusing candidate results improves translation quality for non-autoregressive translation . ArXiv, abs/2301.11503

    2023 arXiv

  52. [60]

    Huanran Zheng, Wei Zhu, and Xiaoling Wang. 2024 a . Chimera model of candidate soups for non-autoregressive translation. In International Conference on Database Systems for Advanced Applications, pages 416--425. Springer

    2024

  53. [61]

    Huanran Zheng, Wei Zhu, and Xiaoling Wang. 2024 b . Nat4at: Using non-autoregressive translation makes autoregressive translation faster and better. In Proceedings of the ACM on Web Conference 2024, pages 4181--4192

    2024

  54. [62]

    Huanran Zheng, Wei Zhu, and Xiaoling Wang. 2024 c . Sca: Selective compression attention for efficiently extending the context window of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6166--6178

    2024

  55. [63]

    P Xing , Hao Zhang , Joseph E

    Lianmin Zheng , Wei-Lin Chiang , Ying Sheng , Siyuan Zhuang , Zhanghao Wu , Yonghao Zhuang , Zi Lin , Zhuohan Li , Dacheng Li , Eric. P Xing , Hao Zhang , Joseph E. Gonzalez , and Ion Stoica . 2023. https://doi.org/10.48550/arXiv.2306.05685 Judging LLM-as-a-Judge with MT-Bench...

    2023 doi arXiv

  56. [64]

    Xiaofeng Zhou, Yuan Ni, Guotong Xie, Wei Zhu, Cai Chen, Tianhao Wang, and Zhigang Pan. 2019. Analysis of the health information needs of diabetics in china. In MEDINFO 2019: Health and Wellbeing e-Networks for All, pages 487--491. IOS Press

    2019

  57. [65]

    Wei Zhu. 2021 a . Leebert: Learned early exit for bert with cross-level optimization. In ACL

    2021

  58. [66]

    Wei Zhu. 2021 b . https://api.semanticscholar.org/CorpusID:237331564 Mvp-bert: Multi-vocab pre-training for chinese bert . In Annual Meeting of the Association for Computational Linguistics

    2021

  59. [67]

    Wei Zhu, Yilong He, Ling Chai, Yuanchun Fan, Yuan Ni, Guo Tong Xie, and Xiaoling Wang. 2021 a . https://api.semanticscholar.org/CorpusID:235097590 paht\_nlp @ mediqa 2021: Multi-grained query focused multi-answer summarization . In Workshop on Biomedical Natural Language Processing

    2021

  60. [68]

    Wei Zhu, Wenfeng Li, Xiaoling Wang, Wendi Ji, Yuanbin Wu, Jin Chen, Liang Chen, and Buzhou Tang. 2023 a . Extracting decision trees from medical texts: An overview of the text2dt track in chip2022. In Health Information Processing. Evaluation Track Papers, pages 89--102, Singa...

    2023

  61. [69]

    Wei Zhu, Wenfeng Li, Xiaoling Wang, Wendi Ji, Yuanbin Wu, Jin Chen, Liang Chen, and Buzhou Tang. 2023 b . Extracting decision trees from medical texts: An overview of the text2dt track in chip2022. In Health Information Processing. Evaluation Track Papers, pages 89--102, Singa...

    2023

  62. [70]

    Wei Zhu, Yuan Ni, Xiaoling Wang, and Guotong Xie. 2021 b . https://doi.org/10.18653/v1/2021.naacl-industry.29 Discovering better model architectures for medical query understanding . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com...

    2021 doi

  63. [71]

    Wei Zhu, Yuan Ni, Guo Tong Xie, Xiaofeng Zhou, and Cai Chen. 2019 a . https://api.semanticscholar.org/CorpusID:208207213 The dr-kgqa system for automatically answering medication related questions in chinese . 2019 IEEE International Conference on Healthcare Informatics (ICHI)...

    2019

  64. [72]

    Wei Zhu, Yuan Ni, Guotong Xie, Xiaofeng Zhou, and Cai Chen. 2019 b . The dr-kgqa system for automatically answering medication related questions in chinese. In 2019 IEEE International Conference on Healthcare Informatics (ICHI), pages 1--6. IEEE

    2019

  65. [73]

    Wei Zhu and Ming Tan. 2023. https://aclanthology.org/2023.emnlp-main.727 SPT : Learning to selectively insert prompts for better prompt tuning . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11862--11878, Singapore. Associatio...

    2023

  66. [74]

    Wei Zhu, Aaron Xuxiang Tian, Congrui Yin, Yuan Ni, Xiaoling Wang, and Guotong Xie. 2024. Iapt: Instruction-aware prompt tuning for large language models. arXiv preprint arXiv:2405.18203

    2024 arXiv

  67. [75]

    Wei Zhu, Peifeng Wang, Yuan Ni, Guo Tong Xie, and Xiaoling Wang. 2023 c . https://api.semanticscholar.org/CorpusID:259370582 Badge: Speeding up bert inference after deployment via block-wise bypasses and divergence-based early exiting . In Annual Meeting of the Association for...

    2023

  68. [76]

    Wei Zhu, Peng Wang, Xiaoling Wang, Yuan Ni, and Guotong Xie. 2023 d . Acf: aligned contrastive finetuning for language and vision tasks. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5. IEEE

    2023

  69. [77]

    Wei Zhu, Xiaoling Wang, Mosha Chen, and Buzhou Tang. 2023 e . https://api.semanticscholar.org/CorpusID:266690968 Overview of the promptcblue shared task in chip2023 . ArXiv, abs/2312.17522

    2023 arXiv

  70. [78]

    Wei Zhu, Xiaoling Wang, Yuan Ni, and Guotong Xie. 2021 c . https://aclanthology.org/2021.emnlp-main.242 GAML - BERT : Improving BERT early exiting by gradient aligned mutual learning . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p...

    2021

  71. [79]

    Wei Zhu , Xiaoling Wang , Huanran Zheng , Mosha Chen , and Buzhou Tang . 2023. https://doi.org/10.48550/arXiv.2310.14151 PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain . arXiv e-prints, page arXiv:2310.14151

    2023 doi arXiv

  72. [80]

    Wei Zhu, Xiaofeng Zhou, Keqiang Wang, Xun Luo, Xiepeng Li, Yuan Ni, and Guotong Xie. 2019 c . Panlp at mediqa 2019: Pre-trained language models, transfer learning and knowledge distillation. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 380--388

    2019

  73. [81]

    Yuhui Zuo, Wei Zhu, and Guoyong GUET Cai. 2022. https://aclanthology.org/2022.coling-1.268 Continually detection, rapidly react: Unseen rumors detection based on continual prompt-tuning . In Proceedings of the 29th International Conference on Computational Linguistics, pages 3...

    2022

  74. [82]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  75. [83]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  76. [84]

    ժ u/14V*8 F1(

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.