Pith. sign in

REVIEW 3 major objections 7 minor 41 references

GTA: Grouped-head latenT Attention

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Grouped-head Latent Attention cuts attention FLOPs by up to 62.5% and KV cache by up to 70% versus GQA while matching or improving sub-1B benchmarks.

desk verdict GTA is a real efficiency contribution for small-model inference, with clean FLOP/cache arithmetic, but its quality claim at 1B is only single-seed and its gating formulation needs a clarifying rewrite. read the letter →

arxiv 2506.17286 v2 pith:RSZLNFYK submitted 2025-06-15 cs.CL cs.AI

Luoyang Sun , Cheng Deng , Jiwen Jiang , Xinjian Wu , Haifeng Zhang , Lei Chen , Lionel Ni , Jun Wang This is my paper · ORCID
classification cs.CLcs.AI
keywords Grouped-headLatentAttentionKVcachecompressionefficiencygrouped-queryvaluedecoderlanguagemodelinferencesharedmapsnonlineargating
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Grouped-head Latent Attention (GTA), an attention mechanism for language models that claims to cut attention computation FLOPs by up to 62.5% and KV cache size by up to 70% compared with Grouped-Query Attention, while matching or slightly improving benchmark quality on models from 160M to 1B parameters. The design rests on two redundancies the authors observe in standard attention: attention score maps are highly similar across heads, and the value cache can be compressed into a much smaller latent space. GTA shares one attention map across groups of heads and reconstructs head-specific values through a learned nonlinear decoder, so the cache stores only grouped keys and latent values. If the claims hold, sub-1B LLMs could run roughly twice as fast end-to-end on commodity hardware, with the prefill stage saving compute and the decode stage saving memory.

What carries the argument

The load-bearing mechanism is the nonlinear value decoder combined with shared attention maps. The decoder starts from a latent value matrix $C \in \mathbb{R}^{N \times n_c d_l}$ and a per-head projection $W_{P,i} \in \mathbb{R}^{d_l \times d_h}$, modulated element-wise by a sigmoid gate $\mathrm{Sigmoid}(x_t W_{G,i})$ that depends only on the current token. This lets the gate be pulled outside the softmax-weighted sum, so the cache stores only $(n_k d_h + n_c d_l)N$ values per layer instead of $2 n_k d_h N$ for GQA; the shared attention map reduces the quadratic attention cost from $2 n_h d_h N^2$ to $n_q(d_h + d_l)N^2$. The gating nonlinearity is what preserves head diversity despite the shared map and compressed latent.

What would settle it

Track the cache contents during an actual GTA decode run: if the implementation must store a gate value for each cached position rather than recomputing it from the current token, the per-layer cache will exceed $(n_k d_h + n_c d_l)N$ and the claimed memory reduction fails. Alternatively, profile prefill FLOPs at long sequence lengths; if attention cost grows as $2 n_h d_h N^2$ rather than $n_q(d_h + d_l)N^2$, the computational claim is falsified.

Watch

Extended reading notes

Core claim

GTA replaces per-head key and value caches with grouped keys $K \in \mathbb{R}^{N \times n_k d_h}$ and a compressed latent value representation $C \in \mathbb{R}^{N \times n_c d_l}$, then generates head-specific values on the fly as $V_i = C_{c(i)} W_{P,i} \odot \mathrm{Sigmoid}(x_t W_{G,i})$, where the sigmoid gate depends on the current token's representation $x_t$ only. Because the gate factors out of the softmax-weighted sum, the efficient form $O_i = (\mathrm{Softmax}(Q_i K_{k(i)}^\top/\sqrt{d_h}) C_{c(i)} W_{P,i}) \odot \mathrm{Sigmoid}(x_t W_{G,i}) W_{O,i}$ requires caching only $C$ and $K$, not per-position gate values. The authors train models at 160M, 500M, and 1B scale and report that GTA matches or improves over GQA and MHA on evaluation loss, Wikitext perplexity, and downstream accuracy, while using as little as 12.5% of MHA's per-layer cache in the 160M and 500M setups and 30% of GQA-1B's cache at 1B scale. Inference benchmarks on server, consumer, and mobile-class hardware show lower prefill and decode latency than GQA-1B, including with cache offload.

Load-bearing premise

The claimed cache savings assume the sigmoid gate is a function of the current query token only and therefore factors out of the softmax sum, so the cache never stores per-position gate values; if the intended per-token value gating actually requires a gate for every cached position, the 70% cache reduction and 2x decode speedup would not hold as written.

Editorial extensions

If this is right

  • GTA's per-layer cache of $(n_k d_h + n_c d_l)N$ means longer contexts and larger batches fit in the same memory, which directly improves decode throughput on memory-bound hardware.
  • At the 1B configuration, attention computation drops to $n_q(d_h + d_l)N^2$, about 37.5% of GQA's $2 n_h d_h N^2$, so prefill latency should keep shrinking relative to GQA as sequence length grows.
  • Because the controlled comparison fixes non-attention parameters, the reported quality match attributes the gains to the attention design rather than to a larger MLP.
  • Fine-tuning with SFT preserves and even improves the quality comparison, so the efficiency mechanism does not appear to break under instruction tuning.
  • The efficiency gains compound in I/O-bound regimes: with cache offload, GTA-1B shows larger decode improvements because the smaller cache reduces data transfers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The redundancy argument suggests a natural extension: sharing attention maps across layers could yield further cache reductions, though the paper does not test this.
  • The query-dependent gate makes GTA's stated cache math specific to autoregressive decoding; in bidirectional or encoder-style attention, gates would need per-position caching, so the same 70% claim should not be assumed there.
  • Because the sigmoid gate is what keeps value representations high-rank, trying sparser gates such as ReLU2 or SiLU would likely require rethinking the value decoder rather than swapping activations.
  • If GTA's efficiency holds at larger scale, its smaller cache should combine multiplicatively with other decode-time techniques such as speculative decoding or quantization, since those also reduce memory traffic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes Grouped-head latenT Attention (GTA), an attention mechanism that shares attention matrices across query groups and compresses values into a latent state with a nonlinear sigmoid gate. The central efficiency claims are that GTA reduces attention FLOPs from 2n_h d_h N^2 (GQA) to n_q(d_h+d_l)N^2 and KV cache from 2n_k d_h N to (n_k d_h+n_c d_l)N, yielding up to 62.5% attention-FLOP reduction and 70% cache reduction, while achieving quality comparable to or better than GQA on 160M to 1B parameter language models. The paper also reports up to 2x end-to-end inference speedup in prefill and decode measurements across several hardware platforms.

Significance. If the efficiency and quality claims hold, GTA is a plausible and useful contribution: the arithmetic of the cache and FLOP reductions is elementary and checks out for the stated configurations, and the paper makes a good-faith effort to control for non-attention parameters and to measure latency on multiple hardware platforms. The design could be of practical interest for deploying sub-1B models. However, the statistical basis for the quality parity claim is weak (all benchmark tables are single-seed), the factorization that underlies the cache-reduction claim is not explicitly derived, and the headline 2x speedup is never directly quantified. These issues are fixable but currently prevent full verification of the paper's central claims.

major comments (3)
  1. [Sec. 3.3, Eq. (8); Appendix B.1.1] The factorization O_i = (Softmax(QK^T) C W_P) ⊙ σ(x_t W_G) W_O in Eq. (8) is asserted without derivation, and the gate placement in the main text appears to differ from Appendix B.1.1, where the gate is applied to the attention output after the weighted sum (Eqs. 12-14). These formulations are algebraically equivalent only if the gate is constant over the cached positions, i.e., a function of the current query token only. The paper states this in words, but it never proves the equivalence, and the appendix notation does not clarify it. Since the cache-size claim of (n_k d_h + n_c d_l)N and the FLOP claim n_q(d_h+d_l)N^2 depend on this factorization, please provide an explicit step-by-step derivation and reconcile the notation, or release the implementation so the exact computation can be checked.
  2. [Tables 1-3, Sec. 4.1-4.3] All quality comparisons are single-seed with no error bars, confidence intervals, or significance tests. The conclusion that GTA-1B achieves comparable performance to GQA-1B in both base and fine-tuned settings is not well supported by Table 3: in the base models the average is 39.56 for GTA-1B versus 40.62 for GQA-1B, and in the 160M/500M tables several reported differences are small enough to be within run-to-run variance. Please provide at least three independent training runs, or confidence intervals, for the headline comparisons, or explicitly discuss the expected noise in the evaluation protocol.
  3. [Abstract; Sec. 5.2-5.3, Figs. 4-6] The abstract claims a '2x increase in end-to-end inference speed,' but no end-to-end speedup ratio is reported anywhere in the paper. Figures 4-6 present prefill and decode phase times, yet the speedup depends on the ratio of prefill to decode work, sequence length, batch size, and hardware, and the paper never ties these phase numbers into an end-to-end number. Please quantify the actual end-to-end speedup under clearly stated assumptions, or revise the claim to what the measurements actually support.
minor comments (7)
  1. [Title] The word 'latenT' in the title should be 'latent' for readability, unless the capitalization is a deliberate acronym styling that should be explained.
  2. [Table 4] The GTA row reads 'n q(dk+dl)N2' and should be 'n_q(d_h+d_l)N^2', with d_k replaced by d_h; also the MLA row mixes 'dnope' and 'd_nope'.
  3. [Eq. (13), Appendix B.1.1] Equation (13) says g_{t,i} ∈ R^{d_h×H}, but g_{t,i} is a vector of length d_h; W_{G,i} should be R^{d_h×H} and the dimension of g_{t,i} should be corrected.
  4. [Appendix B.1.1] The notation for the latent dimension and the number of value groups is inconsistent: the text uses d_c, d_l, n_v, and n_c interchangeably; please unify the symbols.
  5. [Sec. 2, 'Comparison with [16]'] This paragraph mentions 'Grouped Tied Attention' and 'Grouped Latent Attention' without references or definitions, making it difficult to understand the claimed distinction; please clarify with proper citations.
  6. [Tables 5 and 6] There are typos: 'Tie Embeddinng' should be 'Tie Embedding' and 'consine' should be 'cosine'.
  7. [Abstract] The abstract ends with 'Project website' but no URL is provided; either add a link or remove the placeholder.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: GTA's headline FLOP/cache reductions follow directly from the stated projection counts and are benchmarked with external tools; the only author-overlapping citation is a methodology reference and is not load-bearing.

full rationale

The claimed efficiency results are arithmetic consequences of the model definition, not fitted quantities renamed as predictions. Table 4 gives attention cost n_q(d_h+d_l)N^2 and cache (n_k d_h + n_c d_l)N for GTA, versus 2n_h d_h N^2 and 2n_k d_h N for GQA; plugging the GTA-1B configuration (n_q=5, n_k=1, n_c=1, d_h=64, d_l=128) gives 960N^2 vs 2560N^2 and 192N vs 640N, i.e. the advertised 62.5% and 70% reductions. These are counts of the defined architecture, so no circularity arises. The 2x latency results come from LLM-Viewer and transformers-library measurements, which are external benchmarks. The step from Eq. (6) to Eq. (8) is valid given the paper's explicit definition that the sigmoid gate Sigmoid(x_t W_G,i) is a function of the current token and is 'broadcasting across the sequence'; Appendix B.1.1 applies the same gate after the weighted sum, which is equivalent for a query-dependent gate. Whether this is the intended or implemented form is a reproducibility/correctness question, not a circularity: the paper does not secretly fit the headline numbers or import them via definition. The only author-overlapping reference, PLM [13], is cited for the deployment benchmark setup ('Refer to PLM [13]'), not as evidence for GTA's efficiency or performance, so it is a minor non-load-bearing self-citation at most.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central efficiency claims depend only on hand-chosen group counts and the latent dimension, not on fitted parameters. The quality claims rest on standard pretraining assumptions and on the assertion of attention redundancy, which is never measured. No new physical entities are introduced.

free parameters (3)
  • query-group count n_q = 3, 6 (160M); 5, 10 (500M); 5 (1B)
    Chosen per model size; directly controls the attention-FLOP reduction. Not fitted to data, but the headline 62.5% reduction depends on this design choice.
  • latent value dimension d_l = 128
    Set to 128 for all GTA variants, larger than d_h=64 to preserve expressiveness. This choice determines cache size and compute.
  • compressed value group count n_c = 1
    All reported GTA variants use one value group; this keeps the cache small and is central to the 70% cache-reduction claim.
assumptions (3)
  • domain assumption Attention mechanisms exhibit substantial redundancy: attention maps across heads are highly similar and KV caches can be significantly compressed.
    Stated in the abstract and Section 1 as an observation, but the paper provides no measurement of map similarity or compressibility. The whole architecture is motivated by this premise.
  • ad hoc to paper The sigmoid gate factorizes out of the attention sum, so values in the cache can be stored as the latent C without per-position gate storage.
    Needed to make Eq. 8 match the cache footprint formula in Section 3.3. Eq. 6, if read literally, requires gate values for every stored position.
  • standard math Standard transformer background (softmax attention, RoPE, AdamW, C4 and smollm corpora) is reliable.
    The paper relies on these standard tools for all training and evaluation; no new mathematical framework is introduced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GTA: Grouped-head latenT Attention." pith.science (2026). https://pith.science/paper/RSZLNFYK

@misc{pith2026250617286,
  author       = {Pith},
  title        = {Pith review of: GTA: Grouped-head latenT Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RSZLNFYK}},
  note         = {Machine review of arXiv:2506.17286}
}
read the original abstract

Attention mechanisms underpin the success of large language models (LLMs), yet their substantial computational and memory overhead poses challenges for optimizing efficiency and performance. A critical bottleneck arises as KV cache and attention computations scale rapidly with text length, challenging deployment on hardware with limited computational and memory resources. We observe that attention mechanisms exhibit substantial redundancy, since the KV cache can be significantly compressed and attention maps across heads display high similarity, revealing that much of the computation and storage is unnecessary. Leveraging these insights, we propose \textbf{G}rouped-Head Laten\textbf{T} \textbf{A}ttention (GTA), a novel attention mechanism that reduces memory usage and computational complexity while maintaining performance. GTA comprises two components: (1) a shared attention map mechanism that reuses attention scores across multiple heads, decreasing the key cache size; and (2) a nonlinear value decoder with learned projections that compresses the value cache into a latent space, further cutting memory needs. GTA cuts attention computation FLOPs by up to \emph{62.5\%} versus Grouped-Query Attention and shrink the KV cache by up to \emph{70\%}, all while avoiding the extra overhead of Multi-Head Latent Attention to improve LLM deployment efficiency. Consequently, GTA models achieve a \emph{2x} increase in end-to-end inference speed, with prefill benefiting from reduced computational cost and decoding benefiting from the smaller cache footprint.

Figures

Figures reproduced from arXiv: 2506.17286 by the authors.

Figure 1
Figure 1. Attention Architecture: Comparing MHA with GVA and GHA, highlighting key, query, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. GTA architecture: GTA employs a compressed latent value representation C with dimension dl , combined with a nonlinear decoder that uses element-wise multiplication with a sigmoid gate. This design preserves expressive power while reducing the key-value cache size and computational costs compared to traditional attention mechanisms. where WQ ∈ R H×nqdh , WK ∈ R H×nkdh , and WC ∈ R H×ncdl are projection matrices. Her… view at source ↗
Figure 3
Figure 3. Loss and gradient norm curves over 50,000 training steps for GTA-1B and GQA-1B, [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Prefill and decode times for GTA-1B and GQA-1B across configurations on an NVIDIA [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison of prefill (top row) and decode (bottom row) times for GTA-1B and GQA [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison of GTA-1B and GQA-1B with cache offload enabled, showing [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Loss Curve for 160M with 2048 sequence length [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Loss Curve for 160M with 4096 sequence length [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Loss Curve for 500M with 2048 sequence length [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Loss Curve for 500M with 4096 sequence length [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Loss curve for SFT regularization effect, supporting the hypothesis that traditional attention mechanisms may be over￾parameterized [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Prefill and decode times for GTA-1B and GQA-1B across configurations on an NVIDIA [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Prefill and decode times for GTA-1B and GQA-1B across configurations on an NVIDIA [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Prefill and decode times for GTA-1B and GQA-1B across configurations on an NVIDIA [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Prefill and decode times for GTA-1B and GQA-1B across configurations on an NVIDIA [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: Half-precision prefill and decode times for GTA-1B and GQA-1B across configurations [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: Half-precision prefill and decode performance of GTA-1B and GQA-1B models with [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 13 canonical work pages

  1. [16]

    Hardware-efficient attention for fast decoding, 2025

    Ted Zadouri, Hubert Strauss, and Tri Dao. Hardware-efficient attention for fast decoding, 2025

    work page 2025

  2. [1]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  3. [2]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [3]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  5. [4]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344–16359, 2022

    2022

  6. [5]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023

    2023

  7. [6]

    Fast transformer decoding: One write-head is all you need

    Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019

    arXiv 1911

  8. [7]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023

    arXiv 2023

Show all 41 references
  1. [8]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  2. [9]

    Differential transformer, 2025

    Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer, 2025

    2025

  3. [10]

    Multi-token attention, 2025

    Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. Multi-token attention, 2025

    2025

  4. [11]

    Glu variants improve transformer, 2020

    Noam Shazeer. Glu variants improve transformer, 2020

    2020

  5. [12]

    You only cache once: Decoder-decoder architectures for language models

    Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. Advances in Neural Information Processing Systems, 37:7339–7361, 2024

    2024

  6. [13]

    Ni, Haifeng Zhang, and Jun Wang

    Cheng Deng, Luoyang Sun, Jiwen Jiang, Yongcheng Zeng, Xinjian Wu, Wenxin Zhao, Qingfa Xiao, Jiachuan Wang, Haoyang Li, Lei Chen, Lionel M. Ni, Haifeng Zhang, and Jun Wang. Plm: Efficient peripheral language models hardware-co-designed for ubiquitous computing, 2025

    2025

  7. [14]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  8. [15]

    Gated linear attention transformers with hardware-efficient training, 2024

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training, 2024

    2024

  9. [17]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023

    2023

  10. [18]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  11. [19]

    Tinyllama: An open-source small language model

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024. 11

    2024 arXiv

  12. [20]

    Sigmoid-weighted linear units for neural network function approximation in reinforcement learning, 2017

    Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning, 2017

    2017

  13. [21]

    Relu 2 wins: Discovering efficient activation functions for sparse llms, 2024

    Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, and Maosong Sun. Relu 2 wins: Discovering efficient activation functions for sparse llms, 2024

    2024

  14. [22]

    Smollm-corpus, 2024

    Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Smollm-corpus, 2024

    2024

  15. [23]

    The llama 3 herd of models, 2024

    Meta llama team. The llama 3 herd of models, 2024

    2024

  16. [24]

    Mobilellm: Optimizing sub-billion parameter language models for on-device use cases, 2024

    Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases, 2024

    2024

  17. [25]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  18. [26]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

    2018 arXiv

  19. [27]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  20. [28]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    1905 arXiv

  21. [29]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020

    2020

  22. [30]

    MathQA: Towards interpretable math word problem solving with operation-based formalisms

    Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Associatio...

    2019

  23. [31]

    Truthfulqa: Measuring how models mimic human falsehoods, 2022

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. URL https://arxiv. org/abs/2109.07958, 2021

    2022 arXiv

  24. [32]

    Socialiqa: Com- monsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019

    1904 arXiv

  25. [33]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    2021 arXiv

  26. [34]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023

    2023 arXiv

  27. [35]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020. 12

    2007 arXiv

  28. [36]

    Challenging big- bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    arXiv

  29. [37]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Lu...

    2024

  30. [38]

    Llm inference unveiled: Survey and roofline model insights, 2024

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. Llm inference unveiled: Survey and roofline model insights, 2024

    2024

  31. [39]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System ...

    2024

  32. [40]

    Validation

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. 13 A Training Detail A.1 Pretrain Detail This section provides a comprehensive overview of the pretrainin...

    2024

  33. [128]

    Doubling the head dimension improves performance in both GQA and GTA models, with GTA consistently outperforming GQA. Notably, GTA with doubled head dimensions achieves our best performance (2.492), suggesting that allocating more capacity to each head while sharing attention ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.