Pith. sign in

REVIEW 3 major objections 6 minor 63 references

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper aims to establish that zero-initialized attention is a mixture-of-experts model in which both the prompt parameters and the scalar gating factor are recoverable at the optimal parametric rate, giving polynomial rather than…

desk verdict The MoE-style analysis is careful, but the central rates require an injectivity condition on the projection matrices that the actual zero-initialized attention architecture does not satisfy. read the letter →

arxiv 2502.03029 v3 pith:QRZVECMP submitted 2025-02-05 cs.LG

Nghiem T. Diep , Huy Nguyen , Chau Nguyen , Minh Le , Duy M. H. Nguyen , Daniel Sonntag , Mathias Niepert , Nhat Ho This is my paper · ORCID
classification cs.LG
keywords zero-initializedattentionLLaMA-Adaptermixtureofexpertspromptestimationgatingfactorsamplecomplexityparameter-efficientfine-tuningnon-linearprompts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to prove that zero-initialized attention, the gating mechanism used by LLaMA-Adapter, is not a heuristic but a statistically principled design. It recasts the mechanism as a mixture-of-experts model and proves, under a least-squares regression model, that the prompt parameters and the scalar gating factor $\alpha$ can be estimated at the parametric rate $O_P(\sqrt{\log n/n})$ when the number of prompts is overspecified. This rate means approximating the true prompts to accuracy $\epsilon$ takes polynomially many samples, $O(\epsilon^{-2})$ or $O(\epsilon^{-4})$, whereas the cited rates for random-initialized attention require exponentially many samples. The same rates are claimed to hold for non-linear (MLP-reparameterized) prompts under injectivity and smoothness assumptions on the activation. On Open LLM benchmarks, the paper reports that zero-initialized prompts outperform random-initialized ones and that non-linear prompts outperform linear ones, especially at low data fractions.

What carries the argument

The load-bearing machinery is the MoE reformulation of zero-initialized attention: the output is a sum of pretrained experts weighted by a softmax gate plus $\tanh(\alpha)$ times a sum of prompt experts weighted by their own softmax gate. To turn the regression rate of Proposition 4.1 into prompt-level rates, the paper introduces the Voronoi loss $D(G, G^*)$ (Eq. 15), which accounts separately for mass errors, single-atom location errors, and multi-atom squared location errors. The proof shows a coercivity lower bound, $\|f_{G,\alpha} - f_{G^*,\alpha^*}\|_{L^2(\mu)} \gtrsim D(G,G^*) + |\alpha-\alpha^*|$, by Taylor expansion around the true prompts, linear independence of the resulting exponential and polynomial terms, and Fatou's lemma.

What would settle it

On synthetic data drawn from the MoE model (12) with known $G^*$, $\alpha^*$, and a non-injective activation such as ReLU, run the least-squares estimator (18) for sample sizes $n$ from $10^2$ to $10^6$ and measure $D(\hat{G}_n, G^*)$ and $|\hat{\alpha}_n - \alpha^*|$; if the errors do not decrease at the rate $\sqrt{\log n/n}$, the assumptions behind Theorem 4.4 are violated in the exact regime the experiments use.

Watch

Extended reading notes

Core claim

The central claim is that zero-initialized attention can be written as a mixture of frozen pretrained experts and trainable prompt experts, with a single learnable gating factor $\alpha$ wrapping the prompt experts (Eq. 10). Under the MoE regression model (12), the least-squares estimator (13) satisfies $D(\hat{G}_n, G^*) = O_P(\sqrt{\log n/n})$ and $|\hat{\alpha}_n - \alpha^*| = O_P(\sqrt{\log n/n})$ (Theorem 4.2), where $D$ is the Voronoi loss. Theorem 4.4 extends these rates to non-linear prompts of the form $\sigma(p)$ with $\sigma$ injective and uniformly Lipschitz. The authors conclude that zero-initialized attention is more sample-efficient than random-initialized attention, needing polynomial rather than exponential data, and that non-linear prompts retain this efficiency while adding flexibility.

Load-bearing premise

The non-linear results require the prompt activation $\sigma$ to be injective, uniformly Lipschitz, and sufficiently differentiable in the sense of Assumptions A.1-A.2; the paper's experiments instead use a ReLU-based MLP, which is neither injective nor differentiable everywhere, so the proven non-linear rates do not formally cover the reported non-linear experiments.

Editorial extensions

If this is right

  • Zero-initialized attention with linear prompts requires $O(\epsilon^{-2})$ or $O(\epsilon^{-4})$ samples to estimate prompts to error $\epsilon$, versus exponentially many samples for random-initialized attention, so it is the sample-efficient choice for low-data fine-tuning.
  • Non-linear prompts can be estimated at the same rates as linear prompts, so reparameterizing prompts through a smooth injective MLP adds flexibility without sacrificing statistical efficiency.
  • The gating factor $\alpha$ is identifiable and consistently estimated at the parametric rate, so the learned gate can be used as a reliable indicator of how strongly prompts influence the frozen model.
  • The paper's empirical findings — zero-initialized attention beating random-initialized attention at every data fraction, with non-linear prompts matching or beating linear prompts — are direct consequences of the claimed rates and the benchmark setup.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the theorem covers only a scalar $\alpha$, extending the gate to be per-layer or per-head is a natural testable step; the current proof's Taylor argument for $\tanh(\alpha)$ would need to be reworked for vector-valued gates.
  • The reported 1-2 point advantage of non-linear over linear prompts is larger than the theory predicts, since both share the same rates; this suggests the advantage comes from optimization dynamics or initialization geometry rather than estimation accuracy.
  • The experimental activation is a ReLU-based MLP, which is not injective, so the paper's own non-linear experiments operate outside the assumptions of Theorem 4.4; a direct synthetic check with an injective activation (e.g., tanh) would cleanly separate the theory from the heuristic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies zero-initialized attention in LLaMA-Adapter from a statistical perspective. It rewrites the zero-initialized attention output as a mixture-of-experts model (Eq. 10), introduces a regression model (11)-(12) with L unknown linear prompt atoms and a gating factor α*, and proves via a least-squares estimator (13) that the Voronoi loss D(Ĝ_n, G*) and |α̂_n - α*| are O_P(√(log n/n)), for both linear and non-linear prompts under stated assumptions (Theorems 4.2 and 4.4). The paper then claims these polynomial rates are optimal, concludes that zero-initialized attention is more sample-efficient than random-initialized attention, and reports LLaMA-7B/13B experiments where non-linear prompts outperform linear ones.

Significance. The paper's strength is the systematic transfer of Voronoi-loss and Hellinger-rate machinery from the mixture-of-experts literature to a prompt-tuning model, together with detailed proofs and large-scale experiments with public code. If the identifiability issues are resolved, the polynomial sample-complexity result would be a meaningful contribution toward a statistical theory of zero-initialized attention. Currently, however, the main theorems are unproven as stated, and the experimental non-linear prompts violate the theory's assumptions. The significance of the contribution is therefore conditional on a substantial revision.

major comments (3)
  1. [Theorem 4.2 and Appendix A.1 (Step 2 and 'Proof for the identifiability property')] The theorem requires an unstated injectivity condition on the projection matrices ĀB and ĀC. If v ∈ ker(ĀB) ∩ ker(ĀC), then p*_j and p*_j + v produce identical softmax scores and expert values in the model (12), so distinct mixing measures generate the same regression function, and D(Ĝ_n, G*) cannot converge to zero for the true G*. The proof's Step 2 asserts linear independence of {exp((ĀB p*_j)^⊤X)} and the identifiability proof concludes {p_j} = {p*_j} from {( ĀB p_j, ĀC p_j)} = {(ĀB p*_j, ĀC p*_j)}; both steps fail without injectivity of ĀB. In zero-initialized attention, ĀB = W^Q W^{K^⊤} and ĀC = W^V have rank at most d_k and d_v, typically far below the token dimension d, so the kernel intersection is generically non-trivial, meaning the claimed prompt-identification rates do not apply to the intended attention setting.
  2. [Theorem 4.4 and Assumptions A.1-A.2 (Appendix A.2)] Assumption A.2 only states that σ is injective, which is insufficient: the objects that must be injective are the combined maps ĀB∘σ and ĀC∘σ. The identifiability proof in Appendix A.2 again equates {p_j} with {p*_j} from equality of {(ĀB σ(p_j))^⊤, σ(p_j)} without requiring ĀB to be injective on the range of σ. Moreover, the experimental σ in Eq. (21), implemented as a two-layer MLP with ReLU, is neither injective nor differentiable everywhere, so Theorem 4.4's assumptions do not cover the reported non-linear prompt experiments.
  3. [Section 4.1 (after Theorem 4.2) and Section 4.2] The paper labels the rates as 'optimal' and concludes that zero-initialized attention is more sample-efficient than random-initialized attention. No minimax lower bound is proved or cited for the model (12), so 'optimal' is an overclaim. In addition, the comparison to random-initialized attention relies on external rates O(1/log^τ n) from Akbarian et al. (2024) and Le et al. (2024) without defining the corresponding estimator or model in this paper; as it stands, the sample-complexity comparison is not self-contained.
minor comments (6)
  1. [Proposition 4.1 (Appendix A.3) and Eq. (11)] The proof of Proposition 4.1 assumes Gaussian noise ε_i|X_i ∼ N(0_d, σ² I_{d'}) and invokes maximum likelihood estimation, but the model in Eq. (11) only states E[ε_i|X_i]=0 and Var(ε_i|X_i)=σ² I_{d'}. Please either add the Gaussian assumption explicitly to the model or provide a proof that does not require normality.
  2. [Section 5.2, item II and Table 2] The statement that non-linear prompts 'consistently' match or outperform linear prompts is too strong, since for LLaMA-7B the non-linear prompt is slightly worse than the linear prompt on HellaSwag (76.67 vs 76.79) and TruthfulQA (45.04 vs 45.53).
  3. [Table 1] The task name 'TruthfullQA' is misspelled and should be 'TruthfulQA'.
  4. [Appendix A.1, Step 2] The same limit for ĀL_{2,n}(p*_j)(u)/(D_n + |α_n - α*|) appears multiple times consecutively; this appears to be a copy-and-paste error that should be cleaned up.
  5. [Appendix A.1 and A.2] The notation ĀM_{n,j,0_d} is used before it is defined; the index 0_d is easy to confuse with the zero vector, so the definition should be stated at first use.
  6. [Eq. (15) and Section 4.1 text] The paper states that the Voronoi-loss rates range from O_P((log n/n)^{1/2}) to O_P((log n/n)^{1/4}) without a formal derivation from the definition of D; a short derivation or pointer to the relevant inequalities would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the MoE equivalence in Eq. (10) is explicit algebraic rewriting, and the convergence theorems are proved in the appendix from standard empirical-process and Voronoi-loss arguments rather than being assumed or fitted.

full rationale

The paper's claimed connection between zero-initialized attention and mixture-of-experts is an explicit identity, not a hidden reduction: Eqs. (7)-(9) define the experts and gates so that Eq. (10) is exactly the attention output of Eq. (6). This is a modeling equivalence, and the subsequent least-squares analysis is a genuine statistical theorem rather than a restatement of the definition. Theorem 4.2 and Theorem 4.4 are proved in Appendix A through the standard route of a parametric regression-function rate (Proposition 4.1, via van de Geer 2000 and Pardo 2018) plus a Voronoi-loss lower bound that is reproduced in the appendix, not merely imported by citation. The heavy use of prior MoE theory, including Voronoi cells and softmax-gating rates from Manole & Ho (2022) and Ho et al. (2022), is consistent with normal mathematical dependency; those results are not this paper's own fitted values, and the proofs here do not collapse into the cited statements. The comparison to random-initialized attention does cite papers with overlapping authorship for the slow OP(1/log^tau(n)) rates, but that is an external comparative claim about a different initialization scheme, not a parameter fitted in this paper and renamed as a prediction. The most serious issue is not circularity: the identifiability proof at the end of Appendix A.1, and its non-linear analogue, silently requires injectivity of p -> (B p, C p) or p -> (B sigma(p), C sigma(p)), a condition that is neither stated nor satisfied by the low-rank projections in actual zero-initialized attention. That is a soundness gap about identifiability, not a reduction of the derivation to its own inputs, so it does not raise the circularity score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The theoretical results rest on a stylized MoE data-generating process, compact parameter spaces, identifiability, and regularity conditions on sigma; the practical implementation adds an MLP that the theory does not cover. No new physical or architectural entities are postulated, so the invented-entity ledger is empty.

free parameters (4)
  • True prompt atoms p*_j and biases b*_j = unknown (targets of estimation)
    Model (12) assumes a true mixing measure G* = sum exp(b*_j) delta_{p*_j}; all rates are relative to these inaccessible parameters.
  • Gating factor alpha* = unknown
    Equation (12) includes unknown tanh(alpha*); the theorem provides |alpha_hat - alpha*| rate but alpha* is not specified.
  • Number of true prompts L = unknown
    The analysis requires L' >= L, with L unknown; rates depend on the Voronoi partition induced by the L true atoms.
  • Experimental prompt length L and number of adapted layers K = L=10, K=30 (7B) or K=38 (13B)
    Section 5.1 specifies these hyperparameters by hand; performance depends on them, but they are standard choices from LLaMA-Adapter rather than fitted.
assumptions (5)
  • domain assumption The data follow the MoE regression model (12) with Gaussian noise.
    Section 4.1, Eq. (11)-(12) postulates Yi = f_{G*,alpha*}(Xi) + epsilon_i. This is the central modeling assumption; the theorems only describe this process, not the actual transformer training dynamics.
  • domain assumption Parameter spaces Theta and Omega are compact.
    Used in the global part of the proof of Theorem 4.2 (Appendix A.1) to extract convergent subsequences of the mixing measures.
  • domain assumption Activation sigma satisfies injectivity and uniform Lipschitz conditions A.1-A.2.
    Appendix A.2, Assumptions A.1-A.2. These are needed for Theorem 4.4; the practical ReLU MLP (Eq. 21) does not satisfy them.
  • domain assumption Identifiability of the MoE parameterization holds on the support of mu.
    The proof of Theorem 4.2 requires f_{G,alpha} = f_{G*,alpha*} almost surely to imply (G,alpha) = (G*,alpha*); a proof is sketched at the end of Appendix A.1, relying on linear independence of exponential functions.
  • standard math Hellinger rate for the least-squares estimator follows van de Geer (2000).
    Proposition 4.1's proof invokes the empirical process result from van de Geer (2000) for the Hellinger rate OP(sqrt(log n/n)); the paper does not re-derive it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation." pith.science (2026). https://pith.science/paper/QRZVECMP

@misc{pith2026250203029,
  author       = {Pith},
  title        = {Pith review of: On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QRZVECMP}},
  note         = {Machine review of arXiv:2502.03029}
}
read the original abstract

The LLaMA-Adapter has recently emerged as an efficient fine-tuning technique for LLaMA models, leveraging zero-initialized attention to stabilize training and enhance performance. However, despite its empirical success, the theoretical foundations of zero-initialized attention remain largely unexplored. In this paper, we provide a rigorous theoretical analysis, establishing a connection between zero-initialized attention and mixture-of-expert models. We prove that both linear and non-linear prompts, along with gating functions, can be optimally estimated, with non-linear prompts offering greater flexibility for future applications. Empirically, we validate our findings on the open LLM benchmarks, demonstrating that non-linear prompts outperform linear ones. Notably, even with limited training data, both prompt types consistently surpass vanilla attention, highlighting the robustness and adaptability of zero-initialized attention.

Figures

Figures reproduced from arXiv: 2502.03029 by the authors.

Figure 1
Figure 1. LLaMA-Adapter with non-linear prompt structures. Trainable prompts are integrated into the final layers of the LLaMA model, where a zero-gating mechanism modulates the added prompts. This approach enables progressive learning of instruc￾tional knowledge while keeping the remaining model parameters frozen. While the original zero-initialization approach considered only linear prompts (Zhang et al., 2024), most curren… view at source ↗
Figure 2
Figure 2. Accuracy of different prompt strategies (Linear, Random, and Non-Linear) with a LLaMa-7B model across varying fractions of the training set. The Non-Linear prompt consistently outper￾forms the other methods, especially in low-data regimes. 6. Conclusion and Limitations In this paper, we demonstrate that zero-initialization prompt￾tuning for adapting LLMs is not just an engineering trick but can be rigorously explain… view at source ↗
Figure 3
Figure 3. Accuracy of different prompt strategies (Linear, Random, and Non-Linear) using the LLaMa-13B model across varying fractions of the training set. The Non-Linear prompt consistently achieves the highest accuracy, particularly in low-resource settings. In contrast, the Random prompt performs significantly worse when only a small portion of the training data is available, but gradually improves as more data is introduce… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Prompt templates for ARC dataset MMLU Prompt Template Below is an instruction that describes a task. Write a response that appropriately completes the multiple-choice question about {task}. ### Instruction: Question: <question> Options: - Option 1 - Option 2 - Option 3…
Figure 5
Figure 5. Figure 5: Prompt template for MMLU dataset. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_5.png]
Figure 6
Figure 6. Figure 6: Prompt templates for TruthfulQA datasets. C.3. Visualize question-answering We provide in [PITH_FULL_IMAGE:figures/full_fig_p033_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 26 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Quadratic gating functions in mixture of experts: A statistical insight

    Akbarian, P., Nguyen, H., Han, X., and Ho, N. Quadratic gating functions in mixture of experts: A statistical insight. arXiv preprint arXiv:2410.11222, 2024

    arXiv 2024

  3. [3]

    M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al

    Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    arXiv 2023

  4. [4]

    Open llm leaderboard

    Beeching, E., Fourrier, C., Habib, N., Han, S., Lambert, N., Rajani, N., Sanseviero, O., Tunstall, L., and Wolf, T. Open llm leaderboard. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2024. Accessed:2024

    work page 2024

  5. [5]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  6. [6]

    Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning mllms

    Chen, S., Jie, Z., and Ma, L. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning mllms. arXiv preprint arXiv:2401.16160, 2024

    arXiv 2024

  7. [7]

    Adamv-moe: Adaptive multi-task vision mixture-of-experts

    Chen, T., Chen, X., Du, X., Rashwan, A., Yang, F., Chen, H., Wang, Z., and Li, Y. Adamv-moe: Adaptive multi-task vision mixture-of-experts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 17346--17357, 2023

    work page 2023

  8. [8]

    Towards understanding the mixture-of-experts layer in deep learning

    Chen, Z., Deng, Y., Wu, Y., Gu, Q., and Li, Y. Towards understanding the mixture-of-experts layer in deep learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 23049--23062. Curran Associates, Inc., 2022

    work page 2022

Show all 63 references
  1. [9]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  2. [10]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  3. [11]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022

    2022

  4. [12]

    Clip-adapter: Better vision-language models with feature adapters

    Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., and Qiao, Y. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132 0 (2): 0 581--595, 2024

    2024

  5. [13]

    Fusemoe: Mixture-of-experts transformers for fleximodal fusion

    Han, X., Nguyen, H., Harris, C., Ho, N., and Saria, S. Fusemoe: Mixture-of-experts transformers for fleximodal fusion. In Advances in Neural Information Processing Systems, 2024

    2024

  6. [14]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  7. [15]

    Ho, N., Yang, C.-Y., and Jordan, M. I. Convergence rates for G aussian mixtures of experts. Journal of Machine Learning Research, 23 0 (323): 0 1--81, 2022

    2022

  8. [16]

    Parameter-efficient transfer learning for nlp

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp.\ 2790--2799. PMLR, 2019

    2019

  9. [17]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  10. [18]

    A., Jordan, M

    Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural Computation, 3, 1991

    1991

  11. [19]

    Jordan, M. I. and Jacobs, R. A. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6 0 (2): 0 181--214, 1994

    1994

  12. [20]

    Compacter: Efficient low-rank hypercomplex adapter layers

    Karimi Mahabadi, R., Henderson, J., and Ruder, S. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34: 0 1022--1035, 2021

    2021

  13. [21]

    J., Blankevoort, T., and Asano, Y

    Kopiczko, D. J., Blankevoort, T., and Asano, Y. M. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454, 2023

    2023 arXiv

  14. [22]

    Mixture of experts meets prompt-based continual learning

    Le, M., Nguyen, A., Nguyen, H., Nguyen, T., Pham, T., Van Ngo, L., and Ho, N. Mixture of experts meets prompt-based continual learning. Advances in Neural Information Processing Systems, 38, 2024

    2024

  15. [23]

    Revisiting prefix-tuning: Statistical benefits of reparameterization among prompts

    Le, M., Nguyen, C., Nguyen, H., Tran, Q., Le, T., and Ho, N. Revisiting prefix-tuning: Statistical benefits of reparameterization among prompts. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  16. [24]

    The power of scale for parameter-efficient prompt tuning

    Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  17. [25]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day

    Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., and Gao, J. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36: 0 28541--28564, 2023

    2023

  18. [26]

    Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts

    Li, D., Ma, Y., Wang, N., Cheng, Z., Duan, L., Zuo, J., Yang, C., and Tang, M. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. arXiv preprint arXiv:2404.15159, 2024 a

    2024 arXiv

  19. [27]

    Li, H., Lin, S., Duan, L., Liang, Y., and Shroff, N. B. Theory on mixture-of-experts in continual learning. arXiv preprint arXiv:2406.16437, 2024 b

    2024 arXiv

  20. [28]

    Graphadapter: Tuning vision-language models with dual knowledge graph

    Li, X., Lian, D., Lu, Z., Bai, J., Chen, Z., and Wang, X. Graphadapter: Tuning vision-language models with dual knowledge graph. Advances in Neural Information Processing Systems, 36, 2024 c

    2024

  21. [29]

    Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  22. [30]

    Truthfulqa: Measuring how models mimic human falsehoods

    Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021

    2021 arXiv

  23. [31]

    Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. A. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35: 0 1950--1965, 2022

    1950

  24. [32]

    L., Du, Z., Yang, Z., and Tang, J

    Liu, X., Ji, K., Fu, Y., Tam, W. L., Du, Z., Yang, Z., and Tang, J. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021

    2021 arXiv

  25. [33]

    Peft: State-of-the-art parameter-efficient fine-tuning methods

    Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., and Bossan, B. Peft: State-of-the-art parameter-efficient fine-tuning methods. URL: https://github. com/huggingface/peft, 2022

    2022

  26. [34]

    and Ho, N

    Manole, T. and Ho, N. Refined convergence rates for maximum likelihood estimation under finite mixture models. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.\ 14979--15006. PMLR, 17--23 Jul 2022

    2022

  27. [35]

    Large language models: A survey

    Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., and Gao, J. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024

    2024 arXiv

  28. [36]

    M., Diep, N

    Nguyen, D. M., Diep, N. T., Nguyen, T. Q., Le, H.-B., Nguyen, T., Nguyen, T., Nguyen, T., Ho, N., Xie, P., Wattenhofer, R., et al. Logra-med: Long context multi-graph alignment for medical vision-language model. arXiv preprint arXiv:2410.02615, 2024 a

  29. [37]

    M., Le, A

    Nguyen, D. M., Le, A. T., Nguyen, T. Q., Diep, N. T., Nguyen, T., Duong-Tran, D., Peters, J., Shen, L., Niepert, M., and Sonntag, D. Dude: Dual distribution-aware context prompt learning for large vision-language model. Asian Conference on Machine Learning (ACML), 2024 b

    2024

  30. [38]

    Demystifying softmax gating function in gaussian mixture of experts

    Nguyen, H., Nguyen, T., and Ho, N. Demystifying softmax gating function in gaussian mixture of experts. Advances in Neural Information Processing Systems, 36: 0 4624--4652, 2023

    2023

  31. [39]

    Is temperature sample efficient for softmax G aussian mixture of experts? In Proceedings of the ICML, 2024 c

    Nguyen, H., Akbarian, P., and Ho, N. Is temperature sample efficient for softmax G aussian mixture of experts? In Proceedings of the ICML, 2024 c

    2024

  32. [40]

    A general theory for softmax gating multinomial logistic mixture of experts

    Nguyen, H., Akbarian, P., Nguyen, T., and Ho, N. A general theory for softmax gating multinomial logistic mixture of experts. In Proceedings of the ICML, 2024 d

    2024

  33. [41]

    W., Saria, S., and Ho, N

    Nguyen, H., Han, X., Harris, C. W., Saria, S., and Ho, N. On expert estimation in hierarchical mixture of experts: Beyond softmax gating functions. arxiv preprint arxiv 2410.02935, 2024 e

    2024 arXiv

  34. [42]

    Sigmoid gating is more sample efficient than softmax gating in mixture of experts

    Nguyen, H., Ho, N., and Rinaldo, A. Sigmoid gating is more sample efficient than softmax gating in mixture of experts. In Advances in Neural Information Processing Systems, 2024 f

    2024

  35. [43]

    Statistical advantages of perturbing cosine router in mixture of experts

    Nguyen, H., Akbarian, P., Pham, T., Nguyen, T., Zhang, S., and Ho, N. Statistical advantages of perturbing cosine router in mixture of experts. In International Conference on Learning Representations, 2025

    2025

  36. [44]

    OpenAI. Chatgpt. https://chat.openai.com, 2025. Accessed: 2025

    2025

  37. [45]

    A., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F

    OpenAI, J. A., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report, 2024. URL https://arxiv. org/abs/2303.08774, 2: 0 6, 2024

    2024 arXiv

  38. [46]

    Statistical Inference Based on Divergence Measures

    Pardo, L. Statistical Inference Based on Divergence Measures. Statistics: A Series of Textbooks and Monographs. CRC Press, 2018. ISBN 9781420034813. URL https://books.google.com/books?id=ziDGGIkhqlMC

    2018

  39. [47]

    Instruction tuning with gpt-4

    Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023

    2023 arXiv

  40. [48]

    T., Ramasamy, S., Li, X., Hoi, S., and Ho, N

    Pham, Q., Do, G., Nguyen, H., Nguyen, T., Liu, C., Sartipi, M., Nguyen, B. T., Ramasamy, S., Li, X., Hoi, S., and Ho, N. Competesmoe -- effective training of sparse mixture of experts via competition. arXiv preprint arXiv:2402.02526, 2024

    2024 arXiv

  41. [49]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  42. [50]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  43. [51]

    Adapterdrop: On the efficiency of adapters in transformers

    R \"u ckl \'e , A., Geigle, G., Glockner, M., Beck, T., Pfeiffer, J., Reimers, N., and Gurevych, I. Adapterdrop: On the efficiency of adapters in transformers. arXiv preprint arXiv:2010.11918, 2020

    2010 arXiv

  44. [52]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  45. [53]

    and Lipani, A

    Shi, Z. and Lipani, A. Dept: Decomposed prompt tuning for parameter-efficient fine-tuning. arXiv preprint arXiv:2309.05173, 2023

    2023 arXiv

  46. [54]

    Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model, 2023

    2023

  47. [56]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 b

    2023 arXiv

  48. [57]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 c

    2023 arXiv

  49. [58]

    Empirical processes in M-estimation

    van de Geer, S. Empirical processes in M-estimation. Cambridge University Press, 2000

    2000

  50. [59]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  51. [60]

    Understanding expert structures on minimax parameter estimation in contaminated mixture of experts

    Yan, F., Nguyen, H., Le, D., Akbarian, P., and Ho, N. Understanding expert structures on minimax parameter estimation in contaminated mixture of experts. In Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, 2025

    2025

  52. [61]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  53. [62]

    Adalora: Adaptive budget allocation for parameter-efficient fine-tuning

    Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., and Zhao, T. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023

    2023 arXiv

  54. [63]

    Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention

    Zhang, R., Han, J., Liu, C., Zhou, A., Lu, P., Qiao, Y., Li, H., and Gao, P. Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations, 2024

    2024

  55. [64]

    C., and Liu, Z

    Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. International Journal of Computer Vision, 130 0 (9): 0 2337--2348, 2022

    2022

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.