REVIEW 3 major objections 5 minor 1 cited by
R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Training-free activation sparsity matches full-model accuracy at 50% sparsity
desk verdict A genuine training-free 40% speedup with small average accuracy loss, but the 'no performance loss' claim oversells and the low-rank residual mechanism is empirically motivated, not proven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The score matrix $S_{i,j}=\sigma_i X_j V_{[j,i]}$ is the paper's diagnostic object: it decomposes each output contribution by input channel and by singular component of the weight, and it reveals a sparse lower-right concentration. The construction that carries the argument is the two-path decomposition of each linear layer, a magnitude-thresholded sparse path through the raw weight plus a rank-$r$ SVD-residual path $A_rB_r$ (with $A_r=U_r\Sigma_r^{1/2}$, $B_r=\Sigma_r^{1/2}V_r^T$) applied to the masked input; an offline evolutionary search sets the per-layer balance $\rho$ between the sparse budget and the rank budget. Since the low-rank factors are computed once from the pretrained weights and the input threshold is read off the current token's activation magnitudes, nothing in the method requires training or predicting which channels will be active.
What would settle it
Measure the residual approximation error $\|Y - \sigma_{t(s)}(X)W^T - (X-\sigma_{t(s)}(X))(A_rB_r)^T\|_F$ on out-of-distribution tokens for a fixed searched recipe; if the relative error under 50% sparsity on, say, code or multilingual text is substantially larger than on the C4 validation set, the transferability of the low-rank residual assumption is refuted.
Extended reading notes
Core claim
The paper's central discovery is an approximation identity for a linear layer $Y = XW^T$: writing $W = U\Sigma V^T$ and letting $\sigma_{t(s)}(X)$ keep only entries of $X$ above a magnitude threshold, the full computation is approximated by $Y_s + Y_r = \sigma_{t(s)}(X)W^T + (X-\sigma_{t(s)}(X))(A_r B_r)^T$, where $A_rB_r$ is a low-rank factor of the weight. The paper supports this with two measurements on Llama-2-7B using 16 C4 samples: the non-sparse (small-magnitude) input components behave like a few data-dependent biases whose span across thousands of tokens has stable rank about 400, and the contribution matrix $S_{i,j} = \sigma_i X_j V_{[j,i]}$, which scores how much input channel $j$ and singular component $i$ matter to the output, concentrates its large entries in a small lower-right corner. Sparsifying the input directly removes the top-left of that score matrix, and the low-rank residual term recovers most of what was removed; no prediction of future active channels is needed because the input is already known when the layer runs. An evolutionary search over the sparse-to-low-rank ratio per layer tunes the recipe, and the paper reports that at 50% model-level sparsity the accuracy drop across Llama-2, Llama-3, and Mistral is small.
Load-bearing premise
The load-bearing premise is that the small-magnitude input channels that get masked out leave behind a contribution that a fixed low-rank subspace of the original weights can approximate; if the relative importance of channels and singular values shifts on inputs unlike the 16 C4 samples used to measure it, the residual path will miss the lost computation.
Editorial extensions
If this is right
- At 50% model-level sparsity, R-Sparse keeps average accuracy within a few points of the dense model on eight common-sense tasks for Llama-2-7B, Llama-3-8B, and Mistral-7B, and on easy tasks like SciQ the match is exact even at 70% sparsity.
- Because sparsification targets the input side, it works in both attention and MLP linear layers, raising the achievable model-level sparsity beyond the one-third ceiling of MLP-only methods.
- With a customized kernel, skipping half the linear-layer I/O yields up to 43% faster token generation at generation lengths from 128 to 2048 tokens.
- The method composes with 4-bit weight quantization, so sparsity and quantization can be stacked for further memory savings.
Reading between the lines
- If the channel-and-singular-value importance pattern is as stable across domains as the paper's five-dataset comparison suggests, the searched sparsity recipe should transfer to new tasks and possibly to other models in the same family without rerunning the search; this is a direct, testable prediction the paper does not make.
- The same two-path idea, magnitude sparsity on one factor of a bilinear map corrected by a low-rank approximation of the other factor, may apply outside transformers, for example to attention key-value computations or state-space model projections.
- A per-token or per-head adaptive threshold, rather than a global percentile threshold, might push beyond 50% sparsity while controlling the residual error; the paper's fixed-threshold design leaves that margin unexplored.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes R-Sparse, a training-free activation-sparsity method for LLMs with SiLU/GELU activations. For each linear layer, small-magnitude input channels are zeroed by a threshold, and the contribution of the masked channels is approximated by a low-rank SVD factor pair (A_r, B_r) of the weight matrix, so the layer output is computed as Y_s + Y_r. The per-layer trade-off between the sparse and low-rank budgets is found by an evolutionary search that minimizes perplexity on 16 C4 samples. The method is evaluated at 40% and 50% model-level sparsity on Llama-2-7B, Llama-3-8B, and Mistral-7B across eight common-sense reasoning tasks plus WikiText-2 and XSUM, with an end-to-end kernel speedup and compatibility with 4-bit GPTQ quantization.
Significance. R-Sparse addresses a real bottleneck in on-device LLM inference: it avoids the expensive ReLUfication pre-training used by prior activation-sparsity methods, does not need an active-channel predictor, and extends sparsity to attention layers. The empirical coverage is solid, with three model families, ten tasks, comparisons against CATS and GRIFFIN, ablations of the sparse and low-rank components, and a released codebase. The central methodological claim, however, rests on a low-rank residual approximation that is only qualitatively illustrated, and the paper overstates its accuracy results by claiming 'no performance loss' where Table 1 shows several-point drops. The contribution would be significant if the approximation error is directly measured and the claims are qualified; at present it is a promising empirical technique whose mechanism is under-verified.
major comments (3)
- [§3.4, Eq. for Y_r] The decomposition Y ≈ σ_t(s)(X) W^T + (X − σ_t(s)(X)) (A_r B_r)^T is the load-bearing approximation of the paper, but its accuracy is never directly measured. Figure 3 shows only sorted heatmaps of S_{i,j} for 16 C4 samples; since rows and columns are sorted independently, the visual concentration in the lower-right corner is partly a consequence of the sorting. The paper should report the relative approximation error ||Y − Y_s − Y_r||_F / ||Y||_F for the actually deployed per-layer recipes on calibration data not used for the search, and should compare the chosen A_r B_r against the top-r subspace of the masked-channel rows of W. Without such a measurement, the retained accuracy at 50% sparsity could be due to downstream task robustness rather than to the proposed rank-aware mechanism.
- [§5 Conclusion and Abstract] The claim that R-Sparse achieves 50% sparsity 'without any performance loss' is contradicted by Table 1. On Llama-2-7B at 50% sparsity, the average accuracy is 64.06 versus 65.88 for the dense model; BoolQ drops from 77.71 to 72.84 (4.87 points), HellaSwag from 57.13 to 54.26, and ARC-Challenge from 43.43 to 40.78. Similar multi-point drops appear for Llama-3-8B and Mistral-7B. The 'comparable performance' wording used in the abstract is supported, but the stronger 'without any performance loss' wording should be removed or replaced with a statistically grounded statement, for example error bars over multiple evaluation runs or seeds.
- [§3.5 and Table 4] The per-layer sparsity/rank recipe is obtained by minimizing perplexity on 16 C4 samples, and the rank choices are also derived from those samples, so the method has a calibration set built in. This is a legitimate design, but the paper does not quantify how sensitive the end-task results are to the calibration choice: no standard deviation is reported for Tables 1, 2, or 4, and no ablation varies the number of calibration samples or the calibration corpus for the final benchmark results. Such a sensitivity analysis, even on a subset of tasks, is needed to distinguish a robust approximation from a recipe that overfits to C4.
minor comments (5)
- [§4.3 and Abstract] The abstract promises 'up to 43% end-to-end efficiency improvements,' while Section 4.3 reports 42% for Llama-2-7B and 40% for Llama-3-8B; please reconcile the numbers.
- [§3.4] The formula for memory I/O overhead reads 'rm+n/mn + s' and is ambiguous; it should be r(n+m)/(mn) + s.
- [§3.3] There is a duplicated word: 'aligns with with recent studies.'
- [§3.4 and Figure 3] Because the heatmaps are sorted along both axes, please state this limitation in the main text and provide at least one unsorted heatmap in the appendix to support the claimed concentrated structure.
- [§3.4] The terms 'rank' and 'sparse ratio' are sometimes used interchangeably; define clearly in Section 3.4.
Circularity Check
No significant circularity: R-Sparse's claims are validated on held-out benchmarks; C4-based calibration and supportive self-citations do not make the derivation self-referential.
full rationale
The derivation chain is not circular. R-Sparse's accuracy claims are evaluated on held-out benchmarks (eight common-sense tasks, WikiText-2, and XSUM) that are not used to fit the method; the only calibration uses 16 C4 samples, both to estimate the importance maps in Figure 3 and to run the evolutionary search for per-layer rho. This is standard model selection, not a prediction forced by construction: the reported task accuracies do not reduce to the C4 perplexity objective. The sparsity budget s and threshold t(s) are user-defined controls, so reaching 50% model-level sparsity is definitional, but the question of whether performance is maintained is an external measurement. The low-rank residual approximation Y_r = (X - sigma_t(s)(X))(A_r B_r)^T is an ansatz validated empirically; the lack of a formal approximation bound is a correctness risk, not circularity. Self-citations (e.g., Jaiswal et al. 2024, Yin et al. 2023) are used only as contextual support for layer-wise compressibility observations and are not load-bearing derivation steps.
Assumptions & free parameters
free parameters (2)
- Per-layer sparse-rank ratio rho_i =
not reported (per-layer vector found by search)
- Per-layer singular component subset (rank r selection) =
not reported (chosen by ranking scores from 16 C4 samples)
assumptions (4)
- domain assumption Output contributions can be separated by S_{i,j} = sigma_i X_j V[j,i], and the dominant contributions lie in the combination of small input magnitudes with large singular values.
- domain assumption The static low-rank weight approximation A_r B_r^T faithfully approximates the residual contribution (X - sigma(X)) W^T for arbitrary decode-time inputs.
- domain assumption Importance patterns and sparsification recipes estimated from 16 C4 samples transfer to the ten benchmark tasks and to other data domains.
- standard math The weight matrix admits an SVD and a truncated rank-r decomposition.
Cite this review
Pith. "Pith review of R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference." pith.science (2026). https://pith.science/paper/AXUJM2YQ
@misc{pith2026250419449,
author = {Pith},
title = {Pith review of: R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference},
year = {2026},
howpublished = {\url{https://pith.science/paper/AXUJM2YQ}},
note = {Machine review of arXiv:2504.19449}
}
read the original abstract
Large Language Models (LLMs), while demonstrating remarkable capabilities across various applications, present significant challenges during inference due to their substantial model size, especially when deployed on edge devices. Activation sparsity offers a promising solution to reduce computation and memory movement, enabling more efficient inference, particularly for small-batch on-device applications. However, current approaches face limitations with non-ReLU activation function, which are foundational to most advanced LLMs, or require heavy continual training. Additionally, the difficulty in predicting active channels and limited achievable sparsity ratios constrain the effectiveness of activation sparsity-based methods. In this paper, we introduce R-Sparse, a training-free activation sparsity approach capable of achieving high sparsity levels in advanced LLMs. We conducted two preliminary investigations into how different components contribute to the output within a single linear layer and found two key observations: (i) the non-sparse components of the input function can be regarded as a few bias terms, and (ii) The full computation can be effectively approximated by an appropriate combination of input channels and weight singular values. Building on this, we replace the linear layers in LLMs with a rank-aware sparse inference method that leverages the sparsity of input channels and singular value components, eliminating the need for active channel prediction like the output sparsity based approaches. Experiments on Llama-2/3 and Mistral models across ten diverse tasks demonstrate that R-Sparse achieves comparable performance at 50% model-level sparsity, resulting in a significant 43% end-to-end efficient improvements with customized kernels.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 1 Pith paper
-
Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization
SNIPER uses a 0/1 knapsack dynamic program for coarse depth pruning plus a fine-grained MLP width stage, achieving near-exact compression budgets and better average retention than six LLM pruners.
Reference graph
Works this paper leans on
-
[1]
Llm in a flash: Efficient large language model inference with limited memory
Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. Llm in a flash: Efficient large language model inference with limited memory. arXiv preprint arXiv:2312.11514,
-
[6]
Documenting large webtext corpora: A case study on the colossal clean crawled corpus
Jesse Dodge, Maarten Sap, Ana Marasovi ´c, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758,
-
[8]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,
-
[9]
Extreme compression of large language models via additive quantization
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06118,
-
[10]
Gptq: Accurate post-training quantization for generative pre-trained transformers
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323,
-
[11]
Transformer feed-forward layers are key-value memories
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913,
arXiv 2012
-
[13]
Gaussian error linear units (gelus)
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415,
-
[16]
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,
Show all 38 references
-
[17]
Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490,
-
[18]
Squeezellm: Dense-and-sparse quantization
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629,
-
[19]
Cats: Contextually- aware thresholding for sparsity in large language models
Je-Yong Lee, Donghyun Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. Cats: Contextually- aware thresholding for sparsity in large language models. arXiv preprint arXiv:2404.08763,
-
[20]
The lazy neuron phenomenon: On emergence of activation sparsity in transformers
11 Published as a conference paper at ICLR 2025 Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, et al. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. arXiv p...
2025 arXiv
-
[21]
Training- free activation sparsity in large language models
James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. Training- free activation sparsity in large language models. arXiv preprint arXiv:2408.14690, 2024a. Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava,...
-
[22]
Kivi: A tuning-free asymmetric 2bit quantization for kv cache
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024b. Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the stru...
-
[23]
Pointer sentinel mixture models
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843,
-
[24]
Can a suit of armor conduct electricity? a new dataset for open book question answering
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018a. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a ...
-
[25]
Don’t give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don’t give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745,
-
[27]
Rwkv: Reinventing rnns for the transformer era
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048,
-
[28]
Prosparse: Introducing and enhancing intrinsic activation sparsity within large language models
Chenyang Song, Xu Han, Zhengyan Zhang, Shengding Hu, Xiyu Shi, Kuai Li, Chen Chen, Zhiyuan Liu, Guangli Li, Tao Yang, et al. Prosparse: Introducing and enhancing intrinsic activation sparsity within large language models. arXiv preprint arXiv:2402.13516, 2024a. Yixin Song, Zey...
-
[29]
Turbo sparse: Achieving llm sota performance with minimal activated parameters
Yixin Song, Haotong Xie, Zhengyan Zhang, Bo Wen, Li Ma, Zeyu Mi, and Haibo Chen. Turbo sparse: Achieving llm sota performance with minimal activated parameters. arXiv preprint arXiv:2406.05955, 2024b. 12 Published as a conference paper at ICLR 2025 Sharath Turuvekere Sreenivas...
2025 arXiv
-
[30]
A simple and effective pruning approach for large language models
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695,
-
[31]
Llama 2: Open foundation and fine-tuned chat models
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288,
-
[32]
Crowdsourcing multiple choice science questions
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209,
-
[34]
Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity
Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. arXiv preprint arXiv:2310.05175,
-
[35]
Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,
1905 arXiv
-
[36]
Opt: Open pre-trained transformer language models
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068,
-
[37]
Relu2 wins: Discovering efficient activation functions for sparse llms
Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, and Maosong Sun. Relu2 wins: Discovering efficient activation functions for sparse llms. arXiv preprint arXiv:2402.03804, 2024a. Zhenyu Zhang, Ying Sheng, Tianyi Zho...
2025 arXiv
-
[38]
Additionally, to ensure data diversity, we evaluated different domains from the RedPajama dataset1, including GitHub, ArXiv, StackExchange, and Wikipedia
Across different numbers of training samples, the importance patterns consistently exhibit high sparsity. Additionally, to ensure data diversity, we evaluated different domains from the RedPajama dataset1, including GitHub, ArXiv, StackExchange, and Wikipedia. As shown in Figu...
2025
-
[2015]
From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients
Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients. arXiv preprint arXiv:2407.11239,
-
[2016]
Distilling the knowledge in a neural network
G Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531,
-
[2017]
Smoothquant: Accurate and efficient post-training quantization for large language models
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087–38099. PMLR, 2023a. Guangxuan Xiao, Yuandong Tian,...
-
[2018]
Nvidia a100 tensor core gpu architecture
N NVIDIA. Nvidia a100 tensor core gpu architecture. Volume 1.0: Whitepaper, Part, 1(2020):82,
2020
-
[2019]
Think you have solved question answering? try arc, the ai2 reasoning challenge
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018a. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashis...
-
[2020]
Mamba: Linear-time sequence modeling with selective state spaces
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,
-
[2021]
Prompt-prompted mixture of experts for efficient llm generation
Harry Dong, Beidi Chen, and Yuejie Chi. Prompt-prompted mixture of experts for efficient llm generation. arXiv preprint arXiv:2404.01365,
-
[2022]
10 Published as a conference paper at ICLR 2025 Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. corr abs/2208.07339 (2022),
2022 arXiv
-
[2023]
Transformers to ssms: Distilling quadratic knowledge to subquadratic models
Aviv Bick, Kevin Y Li, Eric P Xing, J Zico Kolter, and Albert Gu. Transformers to ssms: Distilling quadratic knowledge to subquadratic models. arXiv preprint arXiv:2408.10189,
-
[2024]
Boolq: Exploring the surprising difficulty of natural yes/no questions
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044,
1905 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.