Pith. sign in

REVIEW 4 major objections 5 minor 37 references

TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Tensorising multi-head attention weights into a shared low-rank subspace improves LLM reasoning accuracy on four benchmarks while compressing attention parameters by up to roughly 250x.

desk verdict Per-head tensorisation with shared-factor Tucker is a genuine new compression variant, but the paper never isolates the shared subspace; gains could be generic low-rank denoising. read the letter →

arxiv 2501.15674 v2 pith:36ATZ4OL submitted 2025-01-26 cs.CL cs.LG

Yuxuan Gu , Wuyang Zhou , Giorgos Iacovides , Danilo Mandic This is my paper · ORCID
classification cs.CLcs.LG
keywords LargeLanguageModelsMulti-headAttentionTensorisationTuckerDecompositionReasoningCompressionWeightdenoisingPost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that the reasoning performance of large language models can be improved, and their attention-block parameters compressed, by treating the four weight matrices of every attention head as one small 3D tensor and then approximating all head tensors in a layer with a Tucker decomposition that shares factor matrices across heads. The shared factor matrices force all heads in a layer to live in a common low-dimensional subspace, which the authors argue removes training noise from the weights rather than useful signal. On four reasoning benchmarks and three models spanning encoder-only and decoder-only architectures, the method improves accuracy while compressing the multi-head attention weights by up to roughly 250 times, without any extra data, training, or fine-tuning. The method is also designed to combine with existing feed-forward-only denoising techniques for additional gains. A sympathetic reader would care because it suggests part of LLM reasoning ability is carried by a small shared structure inside attention weights, accessible by a purely post-hoc algebraic operation.

What carries the argument

The load-bearing object is the multi-head tensorisation plus a special variant of Tucker decomposition with shared factor matrices. Each head's four projection matrices $W^Q_i, W^K_i, W^V_i, (W^O_i)^T$ are stacked along a third mode into $\mathcal{W}_i \in \mathbb{R}^{d_{\text{model}} \times d_v \times 4}$, and the $h$ head tensors are collected into a 4D tensor. Tucker decomposition is then applied jointly to the heads with a common set of factor matrices $U^{(1)} \in \mathbb{R}^{d_{\text{model}} \times R_1}$, $U^{(2)} \in \mathbb{R}^{d_v \times R_2}$, $U^{(3)} \in \mathbb{R}^{4 \times R_3}$ and an individual core $\mathcal{G}_i$ per head; the shared factors define the common subspace, and the cores carry per-head specialization. The decomposition is computed with the Higher-Order Orthogonal Iterations algorithm, which makes the denoising a purely post-hoc approximation with no retraining.

What would settle it

Run the method one transformer layer at a time and compare each layer's accuracy change with the reconstruction error of the shared-subspace Tucker fit relative to a per-head low-rank fit. If layers with high relative reconstruction error still show accuracy gains, the shared-subspace premise is not the mechanism; if accuracy gains appear only where the shared fit is nearly lossless, the premise is confirmed. A simpler check: apply the method to a layer whose heads are known from attention-output analysis to be highly dissimilar; if accuracy drops sharply there, the shared-subspace assumption is doing real work.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that multi-head attention weights contain reasoning-related information in a subspace shared across heads, and that this structure can be exploited by a 'shared-factor Tucker' decomposition. For each head $i$, the query, key, value, and transposed output matrices are stacked into a 3D tensor $\mathcal{W}_i \in \mathbb{R}^{d_{\text{model}} \times d_v \times 4}$, and all heads are collected into $\mathcal{W}_{\text{all}}$; the decomposition then writes each head tensor as $\mathcal{W}_i = \mathcal{G}_i \times_1 U^{(1)} \times_2 U^{(2)} \times_3 U^{(3)}$ with common factor matrices $U^{(1)}, U^{(2)}, U^{(3)}$ and a per-head core $\mathcal{G}_i$. Approximating with small multilinear ranks ($R_1, R_2, R_3$) both denoises the weights and reduces parameter count, yielding accuracy gains on HotPotQA, FEVER, Bios Profession, and BigBench-WikidataQA and MHA compression up to about 250x. The authors interpret this as each head encoding different information within a common subspace, consistent with prior observations that heads in a layer capture similar-level patterns with different specialisations.

Load-bearing premise

The load-bearing premise is that all attention heads in a layer contain their useful information inside one common low-dimensional subspace, so squeezing them onto that shared subspace removes noise instead of throwing away real signal.

Editorial extensions

If this is right

  • Accuracy on four reasoning benchmarks improves for all three tested models after the MHA weights alone are tensorised and compressed, with no extra data or training.
  • Multi-head attention parameters in a transformer layer can be reduced by up to roughly 250x while improving, not merely preserving, accuracy.
  • The method works in both encoder-only and decoder-only architectures, suggesting the shared-subspace structure is not an artifact of one training recipe.
  • Combining this MHA denoising with existing FFN-only denoising yields higher accuracy than either alone in most tested settings.
  • Ablation results indicate that tensorising query, key, value, and output matrices together outperforms compressing each matrix type separately.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the shared-subspace claim is right, attention heads in a layer should be representable as points in one low-dimensional coordinate system; the Tucker cores could be used directly to measure per-head specialization and locate redundant heads.
  • The compression/accuracy trade-off might be improved by choosing multilinear ranks per layer based on how well the shared-subspace assumption holds there, rather than one global setting.
  • The same tensorisation-with-shared-factors recipe could be tested on other multi-part transformer structures, such as grouped-query or cross-attention blocks, where shared structure is also plausible.
  • Because the method is purely post-hoc, it could be applied on top of already-quantized or pruned models, possibly compounding gains; the paper does not test this combination.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes TensorLLM, a post-training, training-free method for compressing the multi-head attention (MHA) blocks of transformer LLMs while improving reasoning accuracy. The method tensorises the four MHA weight matrices of each attention head into a 3D tensor (Eq. 7), stacks these across heads into a 4D tensor (Eq. 8), and applies a Tucker decomposition with factor matrices shared across heads and per-head core tensors (Eqs. 9-11). The authors argue this enforces a shared higher-dimensional subspace across attention heads, thereby structurally denoising MHA weights. Experiments on RoBERTa, GPT-J, and LLaMA2 across HotPotQA, FEVER, Bios Profession, and BigBench-WikidataQA report accuracy improvements and MHA compression rates up to about 247x, with additional hybrid experiments combining the method with LASER on FFN blocks.

Significance. If the empirical claims are robust, the paper addresses a real gap: prior weight-denoising methods such as LASER and TRAWL have been effective mainly on FFN blocks, not MHA blocks. The proposed tensorisation and shared-factor Tucker decomposition are mathematically natural and clearly described, and the paper makes its code available. The central conjecture, that attention heads in a layer share a common subspace, is domain-grounded and testable. However, the current experimental support is not yet sufficient to establish the causal claim that the shared-subspace construction is responsible for the improvements; the missing independent-head control, lack of uncertainty estimates, and per-dataset hyperparameter selection all need to be addressed before the results can be fully credited.

major comments (4)
  1. [Section IV-D, Table IV] The ablation study compares decomposing the four MHA weight matrices jointly versus separately, but both conditions use the shared-factor Tucker construction. There is no control where each attention head is Tucker-decomposed independently with its own factor matrices at a matched compression budget. Without this control, the accuracy gains in Table II cannot be attributed to the shared-subspace conjecture stated in Section I; they could arise from generic low-rank denoising. Please add this control and report whether the shared-factor design outperforms it.
  2. [Section IV-C, Table II] Several reported accuracy improvements are very small (e.g., GPT-J HotPotQA 19.6 to 20.15; RoBERTa FEVER 50.0 to 50.45), no multiple seeds or confidence intervals are reported, and the test set is a single 20% tail of each dataset. Meanwhile, the loss metric degrades sharply in some rows (LLaMA2 HotPotQA 3.15 to 9.80). The claim of consistent reasoning improvement needs variance estimates and ideally significance testing; as reported, the results are difficult to distinguish from noise.
  3. [Section IV-A and Limitations paragraph (Section V)] The method has free hyperparameters: the multilinear ranks R1, R2, R3 and the choice of transformer layer(s) for compression. The Limitations paragraph admits that different datasets require different hyperparameter settings, but the paper does not specify how these were selected or whether test accuracy influenced the selection. If ranks and layers were chosen using test-set accuracy, the reported numbers are optimistic. Please describe the selection protocol (e.g., a validation split) and include a sensitivity analysis over ranks and layers.
  4. [Section IV-C, Table III] Table III compares hybrid configurations without reporting the compression rate for each case. Since the paper simultaneously claims improved reasoning and compression, the comparison is incomplete unless the parameter budgets of Cases 1, 2, and 3 are quantified and made commensurate; otherwise the accuracy differences could be an artefact of different compression levels.
minor comments (5)
  1. [Eq. (13)] The notation in Eq. (13) is ambiguous: the summation appears inside the norm, but the intended objective is likely a sum of per-head Frobenius norms. Please rewrite it as sum over i of ||W_i - G_i ×1 U(1) ×2 U(2) ×3 U(3)||_F^2.
  2. [References] Reference [37] is cited for BigBench-WikidataQA, but the cited paper is about a natural language inference corpus; the BIG-bench dataset should be cited to the actual BIG-bench paper.
  3. [Section IV-B] There are several typos: 'appearred' appears twice in Section IV-B, 'transfomer' appears in Section IV, and 'preforming' appears in the Conclusion. Please proofread.
  4. [Section IV-C and Tables II-III] The text says the method is applied in a layer-selective fashion, but the compression ratios are reported only for the MHA parameters in a single layer. Please clarify explicitly whether the reported CR is for one selected layer only and what the resulting whole-model compression rate would be.
  5. [Section IV-C] The loss metric is described as 'included for completeness,' but several entries show large loss increases that seem inconsistent with the accuracy improvements. Please discuss these discrepancies in the text rather than leaving them only in the tables.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the shared-subspace Tucker method fits only model weights, and the reasoning gains are judged against external benchmarks.

full rationale

The paper's derivation chain is not circular. The tensorisation in Eqs. (7)-(8) and the shared-factor Tucker approximation in Eqs. (9)-(11) are purely functional operations on the MHA weight matrices; Eqs. (13)-(14) minimize reconstruction error with respect to those same weights, and no benchmark label or task output is used in the fit. The 'denoising' claim is the low-rank projection itself, and the claim that this improves reasoning is an empirical claim tested on HotPotQA, FEVER, Bios Profession, and BigBench-WikidataQA, which are external to the method. No fitted constant is renamed as a prediction: the compression ratio is computed from the chosen ranks and the original parameter counts, and the accuracy numbers come from running the reconstructed model on held-out data. The paper does rely on a conjecture about shared subspaces (Section I), but that conjecture is an assumption of the design, not a conclusion derived from it; a failure to isolate the shared-subspace mechanism (the missing matched-budget per-head independent Tucker control) is a mechanistic attribution weakness, not circularity. Self-citations ([14], [21], [26]) are contextual references to tensor tools and rank-search literature and are not load-bearing; there is no imported uniqueness theorem that forbids alternatives. The Limitations section's admission that different datasets require different hyperparameter settings is a legitimate evaluation-protocol concern, since test-set-based selection could inflate reported gains, but nothing in the paper states or requires that ranks or layers were chosen by test accuracy, and this does not make the derivation equivalent to its inputs by construction. Overall the method is self-contained against external benchmarks.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. The shared factor matrices are a mathematical construction, not a new force, particle, or dimension. The central claim rests on domain assumptions about attention head structure and on standard tensor decomposition results.

free parameters (2)
  • Multilinear ranks R1, R2, R3 per model and dataset = Not reported in paper
    The Tucker ranks determine compression and denoising; the Limitations section states different hyperparameters work best per dataset, but the chosen values or selection procedure are not given.
  • Layer index chosen for compression = Not reported
    The method is applied to a single transformer layer in a layer-selective fashion similar to LASER, but the specific layer per model and dataset is not stated.
assumptions (3)
  • domain assumption Attention heads in the same layer share a common subspace that contains reasoning-related information.
    Introduced as a conjecture in Section I and cited to prior analyses [17]-[19]; operationalized by sharing Tucker factor matrices across heads in Eqs. (9)-(11).
  • domain assumption Low-rank approximation of weight matrices removes training noise while preserving useful signal.
    Adopted from LASER's argument in Section I ('removing the weight noise caused by the randomness introduced during training') and applied to the MHA weights.
  • standard math Higher Order Orthogonal Iterations (HOOI) yields a good local optimum for the Tucker decomposition.
    The implementation relies on TensorLy's HOOI algorithm [31]; the paper assumes this standard algorithm behaves as expected.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs." pith.science (2026). https://pith.science/paper/36ATZ4OL

@misc{pith2026250115674,
  author       = {Pith},
  title        = {Pith review of: TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/36ATZ4OL}},
  note         = {Machine review of arXiv:2501.15674}
}
abstract

The reasoning abilities of Large Language Models (LLMs) can be improved by structurally denoising their weights, yet existing techniques primarily focus on denoising the feed-forward network (FFN) of the transformer block, and can not efficiently utilise the Multi-head Attention (MHA) block, which is the core of transformer architectures. To address this issue, we propose a novel intuitive framework that, at its very core, performs MHA compression through a multi-head tensorisation process and the Tucker decomposition. This enables both higher-dimensional structured denoising and compression of the MHA weights, by enforcing a shared higher-dimensional subspace across the weights of the multiple attention heads. We demonstrate that this approach consistently enhances the reasoning capabilities of LLMs across multiple benchmark datasets, and for both encoder-only and decoder-only architectures, while achieving compression rates of up to $\sim 250$ times in the MHA weights, all without requiring any additional data, training, or fine-tuning. Furthermore, we show that the proposed method can be seamlessly combined with existing FFN-only-based denoising techniques to achieve further improvements in LLM reasoning performance.

Figures

Figures reproduced from arXiv: 2501.15674 by the authors.

Figure 1
Figure 1. Structures of our proposed framework for the compression of the MHA block and the existing methods (such as LASER) which apply to the FFN [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Tensor network [20] topologies of different decomposition methods applied to the MHA weights in a single transformer layer. Note that while [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 21 canonical work pages

  1. [1]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language mod- els are few-shot learners,” Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901, 2020

    work page 1901

  2. [2]

    GPT-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. , “GPT-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  3. [3]

    LLaMA: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, et al. , “LLaMA: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [4]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  5. [5]

    The Llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al., “The Llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    arXiv 2024

  6. [6]

    Learning Deep Transformer Models for Machine Translation,

    Q. Wang, B. Li, T. Xiao, J. Zhu, C. Li, D. F. Wong, and L. S. Chao, “Learning Deep Transformer Models for Machine Translation,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1810–1822, July 2019

    work page 2019

  7. [7]

    Language Modeling with Deep Transformers,

    K. Irie, A. Zeyer, R. Schl ¨uter, and H. Ney, “Language Modeling with Deep Transformers,” in Proceedings of Interspeech 2019 , pp. 3905– 3909, 2019

    work page 2019

  8. [8]

    Text Summarization with Pretrained Encoders,

    Y . Liu and M. Lapata, “Text Summarization with Pretrained Encoders,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3730–3740, 2019

    work page 2019

Show all 37 references
  1. [9]

    Distilling the Knowledge in a Neural Network,

    G. E. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” ArXiv, vol. abs/1503.02531, 2015

    2015 arXiv

  2. [10]

    On the effect of dropping layers of pre-trained transformer models,

    H. Sajjad, F. Dalvi, N. Durrani, and P. Nakov, “On the effect of dropping layers of pre-trained transformer models,” Computer Speech & Language, vol. 77, p. 101429, 2023

    2023

  3. [11]

    Reducing Transformer Depth on Demand with Structured Dropout,

    A. Fan, E. Grave, and A. Joulin, “Reducing Transformer Depth on Demand with Structured Dropout,” in Proceedings of the International Conference on Learning Representations , 2020

    2020

  4. [12]

    The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction,

    P. Sharma, J. T. Ash, and D. Misra, “The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction,” in Proceedings of The Twelfth International Conference on Learning Representations, 2024

    2024

  5. [13]

    Com- pressing Large Language Models using Low Rank and Low Precision Decomposition,

    R. Saha, N. Sagan, V . Srivastava, A. Goldsmith, and M. Pilanci, “Com- pressing Large Language Models using Low Rank and Low Precision Decomposition,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

    2024

  6. [14]

    Com- pression and Interpretability of Deep Neural Networks Via Tucker Ten- sor Layer: From First Principles to Tensor Valued Back-Propagation,

    G. G. Calvi, A. Moniri, M. Mahfouz, Q. Zhao, and D. P. Mandic, “Com- pression and Interpretability of Deep Neural Networks Via Tucker Ten- sor Layer: From First Principles to Tensor Valued Back-Propagation,” arXiv preprint arXiv:1903.06133 , 2019

    1903 arXiv

  7. [15]

    TensorGPT: Efficient compression of the embedding layer in LLMs based on the tensor-train decomposi- tion,

    M. Xu, Y . L. Xu, and D. P. Mandic, “TensorGPT: Efficient compression of the embedding layer in LLMs based on the tensor-train decomposi- tion,” arXiv preprint arXiv:2307.00526 , 2023

    2023 arXiv

  8. [16]

    TRAWL: Tensor Reduced and Approximated Weights for Large Language Models,

    Y . Luo, H. Patel, Y . Fu, D. Ahn, J. Chen, Y . Dong, and E. E. Papalexakis, “TRAWL: Tensor Reduced and Approximated Weights for Large Language Models,” arXiv preprint arXiv:2406.17261 , 2024

    2024 arXiv

  9. [17]

    What does BERT look at? an analysis of BERT’s attention,

    K. Clark, U. Khandelwal, O. Levy, and C. D. Manning, “What does BERT look at? an analysis of BERT’s attention,” in Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pp. 276–286, Aug. 2019

    2019

  10. [18]

    A Multiscale Visualization of Attention in the Transformer Model,

    J. Vig, “A Multiscale Visualization of Attention in the Transformer Model,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations , pp. 37–42, July 2019

    2019

  11. [19]

    Analyzing the Structure of Attention in a Transformer Language Model,

    J. Vig and Y . Belinkov, “Analyzing the Structure of Attention in a Transformer Language Model,” in Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 63–76, Aug. 2019

    2019

  12. [20]

    A practical introduction to tensor networks: Matrix product states and projected entangled pair states,

    R. Or ´us, “A practical introduction to tensor networks: Matrix product states and projected entangled pair states,” Annals of Physics , vol. 349, pp. 117–158, 2014

    2014

  13. [21]

    Tensor decompositions for signal processing applica- tions: From two-way to multiway component analysis,

    A. Cichocki, D. Mandic, L. De Lathauwer, G. Zhou, Q. Zhao, C. Caiafa, and H. A. Phan, “Tensor decompositions for signal processing applica- tions: From two-way to multiway component analysis,” IEEE Signal Processing Magazine, vol. 32, no. 2, pp. 145–163, 2015

    2015

  14. [22]

    Tensor decompositions and applications,

    T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,” SIAM Review, vol. 51, no. 3, pp. 455–500, 2009

    2009

  15. [23]

    Some mathematical notes on three-mode factor analysis,

    L. R. Tucker, “Some mathematical notes on three-mode factor analysis,” Psychometrika, vol. 31, no. 3, pp. 279–311, 1966

    1966

  16. [24]

    A multilinear singular value decomposition,

    L. De Lathauwer, B. De Moor, and J. Vandewalle, “A multilinear singular value decomposition,” SIAM Journal on Matrix Analysis and Applications, vol. 21, no. 4, pp. 1253–1278, 2000

    2000

  17. [25]

    Multilinear analysis of image ensembles: Tensorfaces,

    M. A. O. Vasilescu and D. Terzopoulos, “Multilinear analysis of image ensembles: Tensorfaces,” in Proceedings of Computer Vision—ECCV 2002: 7th European Conference on Computer Vision Copenhagen, Denmark, May 28–31, 2002 Proceedings, Part I 7 , pp. 447–460, 2002

    2002

  18. [26]

    Towards LLM-guided Efficient and Interpretable Multi-linear Tensor Network Rank Selection,

    G. Iacovides, W. Zhou, and D. Mandic, “Towards LLM-guided Efficient and Interpretable Multi-linear Tensor Network Rank Selection,” arXiv preprint arXiv:2410.10728, 2024

    2024 arXiv

  19. [27]

    SVDinsTN: A Tensor Network Paradigm for Efficient Structure Search from Regularized Modeling Perspective,

    Y .-B. Zheng, X.-L. Zhao, J. Zeng, C. Li, Q. Zhao, H. C. Li, and T.-Z. Huang, “SVDinsTN: A Tensor Network Paradigm for Efficient Structure Search from Regularized Modeling Perspective,” in Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR...

    2024

  20. [28]

    Attention is All you Need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is All you Need,” in Advances in Neural Information Processing Systems , vol. 30, 2017

    2017

  21. [29]

    Layer normalization,

    J. Lei Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” ArXiv e-prints, pp. arXiv–1607, 2016

    2016

  22. [30]

    TensorLy: Tensor Learning in Python,

    J. Kossaifi, Y . Panagakis, A. Anandkumar, and M. Pantic, “TensorLy: Tensor Learning in Python,” Journal of Machine Learning Research , vol. 20, no. 26, pp. 1–6, 2019

    2019

  23. [31]

    On the best rank-1 and rank-( r1, r2, ..., rn) approximation of higher-order tensors,

    L. De Lathauwer, B. De Moor, and J. Vandewalle, “On the best rank-1 and rank-( r1, r2, ..., rn) approximation of higher-order tensors,” SIAM Journal on Matrix Analysis and Applications , vol. 21, no. 4, pp. 1324– 1342, 2000

    2000

  24. [32]

    RoBERTa: A robustly optimized BERT pretraining approach,

    Y . Liu, “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv preprint arXiv:1907.11692 , vol. 364, 2019

    1907 arXiv

  25. [33]

    GPT-J-6B: A 6 Billion Param- eter Autoregressive Language Model

    B. Wang and A. Komatsuzaki, “GPT-J-6B: A 6 Billion Param- eter Autoregressive Language Model.” https://github.com/kingoflolz/ mesh-transformer-jax, May 2021

    2021

  26. [34]

    HotpotQA: A Dataset for Diverse, Explainable Multi- hop Question Answering,

    Z. Yang, P. Qi, S. Zhang, Y . Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning, “HotpotQA: A Dataset for Diverse, Explainable Multi- hop Question Answering,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pp. 2369–2380, Oct.-Nov. 2018

    2018

  27. [35]

    FEVER: a Large-scale Dataset for Fact Extraction and VERification,

    J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal, “FEVER: a Large-scale Dataset for Fact Extraction and VERification,” in Pro- ceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies...

    2018

  28. [36]

    Bias in bios: A case study of semantic representation bias in a high-stakes setting,

    M. De-Arteaga, A. Romanov, H. Wallach, J. Chayes, C. Borgs, A. Chouldechova, S. Geyik, K. Kenthapadi, and A. T. Kalai, “Bias in bios: A case study of semantic representation bias in a high-stakes setting,” in Proceedings of the Conference on Fairness, Accountability, and Trans...

    2019

  29. [37]

    A large an- notated corpus for learning natural language inference,

    S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, “A large an- notated corpus for learning natural language inference,” in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632–642, Sept. 2015

    2015

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.