REVIEW 4 major objections 4 minor 23 references
SWSC: Shared Weight for Similar Channel in LLM
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims SWSC, a channel-clustering plus truncated-SVD compression method, holds Llama-2-7B's WikiText-2 perplexity near 7-10 at 2 average bits for query and key projectors, while RTN quantization fails at the same budget.
desk verdict Reasonable compression recipe for Q/K projections with a clear narrow win over RTN, but the 'low-precision LLM' claim overreaches the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is channel-wise K-Means clustering of the weight matrix together with truncated SVD error compensation. A weight matrix $W \in \mathbb{R}^{m\times m}$ is clustered along its $m$ channels; each channel is assigned a cluster label, and every channel in a cluster is replaced by that cluster's mean vector, reducing storage to roughly $(k+1)m$ parameters when there are $k$ clusters. The error matrix $W_{\text{err}} = W - W'$ is then decomposed as $U\Sigma V^T$, and only the top $r$ singular values and vectors are kept, stored as $U_r\Sigma_r^{1/2}$ and $\Sigma_r^{1/2}V_r$; during inference these are added to $W'$. The cluster labels, representative vectors, and truncated factors together determine the advertised bit rate, with the paper reporting that each increment of 128 clusters or 64 retained singular ranks adds 0.5 average bits for Llama-2-7B's self-attention layer.
What would settle it
Run SWSC on the value projector of Llama-2-7B at 2 average bits and measure WikiText-2 perplexity; if it degrades as sharply as RTN (or worse), the method's success is limited to the projectors it already excludes. Alternatively, apply SWSC to every linear layer in the model and see whether the overall perplexity stays near the uncompressed value.
Extended reading notes
Core claim
The central discovery is that a channel-wise clustering plus error-SVD scheme can keep a large language model's behavior nearly intact at bit budgets where standard low-precision rounding disintegrates, at least for the query and key projections of the self-attention layer. The evidence is Table I: with 2 average bits, SWSC gives perplexities of 7.297 for the query projector, 7.148 for the key projector, and 10.886 for both together on WikiText-2, while RTN at the same budget reaches 4958.396 for the query projector and returns 'nan' for the key projector. The paper presents this as evidence that query and key projector weights are insensitive to their exact values, so shared cluster representatives are acceptable, whereas the value projector is deliberately excluded because it 'stores the specific features of the model' and requires higher accuracy.
Load-bearing premise
The load-bearing premise is that query and key projector weights are insensitive to their exact values, so replacing whole channels with cluster means plus a low-rank correction preserves model behavior, while value projectors are too sensitive and are excluded.
Editorial extensions
If this is right
- SWSC can be combined with existing quantization, pruning, knowledge distillation, or low-rank methods, since it is presented as an orthogonal weight-compression technique.
- At very low average bits, SWSC is more stable than RTN on attention-projector weights, so it offers an alternative deployment path for memory-constrained devices.
- The cluster count $k$ and retained rank $r$ give a direct control knob for trading compression ratio against model quality.
- The paper's stated next step is to combine SWSC with other orthogonal compression methods, which would test whether the gains stack on top of standard quantization.
- Because the current experiments cover only query and key projectors, the method's broader applicability to other weight matrices remains open even if the reported numbers hold.
Reading between the lines
- If the query/key insensitivity generalizes beyond Llama-2-7B, SWSC could be applied to attention projections in other transformer LLMs, but the paper does not test cross-model transfer.
- The success at 2 bits suggests attention-projector weights have a clustered latent structure that standard rounding ignores, which could motivate further analysis of when mean-replacement is safe.
- The error correction acts as a low-rank residual on a codebook approximation, so combining SWSC with low-rank adaptation or pruning may yield smaller footprints than either approach alone, though the paper does not investigate this.
- A head-to-head against stronger post-training quantizers such as GPTQ would be needed to show SWSC's advantage is not limited to the weak RTN baseline.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SWSC, a post-training compression scheme for LLM weight matrices. The method clusters the channels of a weight matrix with k-means, replaces each cluster by its mean, and stores a cluster-label vector plus the cluster representative vectors; to recover accuracy, it computes the residual matrix W - W' and stores a truncated SVD of that residual, which is added back at inference time. On Llama-2-7B, the authors compress the query and key projectors (separately and jointly) to what they call 2-3 average bits and report WikiText-2 perplexities of roughly 6.5-10.9, compared with RTN at the same bit budget, which gives values such as 4958.4 or NaN. The paper's central claim is that SWSC can keep the compressed LLM's performance largely intact even under low-precision conditions, and that it is a promising alternative to quantization.
Significance. If the per-matrix results are taken at face value, the paper demonstrates a simple and interesting compression idea: the method has no fitted constants or data-dependent normalizations, and the SVD error correction is computed directly from the actual compression error, so the comparison with RTN is not circular. This is a genuine strength. The reported numbers for Q/K compression at 2 bits are strikingly better than RTN and suggest that cluster-then-correct is worth exploring. However, the significance as a full-model LLM compression method is not yet established: only the Q and K projectors are compressed, the average-bit accounting does not state the storage precision of the stored values, and no state-of-the-art PTQ baseline is included. The paper's general claims in the Abstract and Section V go beyond the experimental scope.
major comments (4)
- [§IV-B, Table I, Abstract/V] The experimental protocol compresses only the Query and Key projectors, and Section IV-B explicitly states that the Value Projector is not compressed. In Llama-2-7B, Q and K together are roughly 1.07B of about 6.7B parameters, so the 'Avg. Bits' figures in Table I are per-matrix, not per-model. If Q/K are at 2 bits and all remaining weights stay at 16 bits, the whole model runs at roughly 13.8 average bits, not 2 bits. The Abstract and Section V claims that SWSC 'can effectively ensure that the performance of the compressed LLM is not significantly affected even under low-precision conditions' therefore outrun the evidence. Please either report whole-model bit rates and include an ablation that compresses V (and ideally the MLP and embedding matrices), or scale the claims to 'Q and K projectors at low precision.'
- [§III-C and §IV-C] The definition of 'Avg. Bits' is incomplete. Section IV-C gives only the cluster/rank increments in Table II, but it does not state the storage precision of the cluster representative vectors, the cluster-label vector, or the SVD factors U_rΣ^{1/2} and Σ^{1/2}V_r. The reported 0.5-bit increments correspond to storing 16-bit values: for m=4096, one additional 4096-dimensional representative per 128 clusters contributes 128×4096×16/4096² = 0.5 bits per original weight. If fp32 or int8 storage were used, all average-bit figures would change; label overhead also needs to be included. Because RTN is an integer-quantization method, the comparison requires a precise bit-accounting convention. Please state the storage format for every stored quantity and recompute Table I under that convention.
- [Table I] Table I contains no row for the unmodified Llama-2-7B perplexity on WikiText-2. The central claim that SWSC 'ensures that the performance of the compressed LLM is not significantly affected' cannot be evaluated without this baseline. For example, the SWSC 2-bit Q&K result of 10.886 may be far from the full-precision model's perplexity even though it is much better than RTN's 10490. Please add the uncompressed baseline perplexity and, if possible, a 4-bit RTN row so that the reader can see where the compressed model stands relative to the original.
- [§IV-A and §IV-B] The comparison is limited to RTN, and the paper does not compare with standard post-training quantization methods such as GPTQ, AWQ, or SmoothQuant, even though these are cited in the related-work section. This matters because the conclusion states that SWSC is 'a promising approach for LLM compression' in general, not merely better than RTN. Please add at least one state-of-the-art PTQ baseline at comparable bit rates, or temper the conclusion to a claim about outperforming RTN. In addition, the statement that the Value Projector 'stores the specific features' and therefore should not be compressed is asserted without quantitative support; an ablation showing the relative sensitivity of Q, K, and V under SWSC would make the selection criterion credible.
minor comments (4)
- [§III-B] The statement that 'a compression rate of over 90% can be achieved' for m=4096 and k=256 appears to count only the cluster-representative storage and not the SVD error-compensation factors; with nonzero retained rank the compression rate is lower. Please clarify the accounting in this sentence.
- [Table I] The 'nan' entry for RTN on the K projector at 2 bits is unexplained; state whether it is a numerical overflow, a divergence, or a failed evaluation.
- [§IV-C] The column header 'Avg. Bits' should specify 'average bits per original weight parameter,' and the sentence about increments of 0.5 bits should explicitly state the assumed bit width of the stored real-valued factors.
- [Various] There are several typographical and formatting issues: the 'Index Terms' line is broken, some inline math symbols are mangled in the extracted text, and the RTN rows in Table I are visually ambiguous. A careful proofread would improve readability.
Circularity Check
No circularity: SWSC reconstructs W from cluster means plus a truncated SVD of the true error W - W', and validates against an external RTN baseline on WikiText-2; no fitted quantity is relabeled as a prediction.
full rationale
SWSC's derivation chain is self-contained and externally benchmarked. The compressed representation is W' (cluster means) plus a truncated SVD reconstruction of the exact error matrix W - W', so every stored quantity is a function of the original matrix W and the chosen hyper-parameters (number of clusters k, retained rank r). The reported per-matrix average bit counts are accounting identities derived from the storage format, not fitted parameters, and the perplexity comparisons in Table I are against an external baseline (RTN) on an external benchmark (WikiText-2). The central claim is therefore not equivalent to its input by construction. There are no self-citations and no imported uniqueness or ansatz results: SVD and product quantization are standard external tools used to define the method, not to justify its validity. The notable limitation is that Section IV-B restricts compression to Query and Key projectors and explicitly leaves the Value Projector uncompressed, so the abstract's phrase 'compressed LLM' overstates the experimental scope; this is a generalization and external-validity concern, not a circularity, and it does not raise the circularity score.
Assumptions & free parameters
free parameters (3)
- number of K-means clusters k per weight matrix =
unreported per experiment
- SVD rank r for the error-compensation matrix =
unreported per experiment
- storage bit width of cluster representatives, labels, and SVD factors =
unstated
assumptions (5)
- domain assumption Channels that K-means groups together are similar enough that replacing each by the cluster mean preserves model behavior.
- domain assumption Query and key projections are insensitive to their exact values, while value projections are sensitive.
- domain assumption A low-rank SVD approximation of the compression error captures the information needed to restore LLM accuracy.
- domain assumption WikiText-2 perplexity is a sufficient measure of compressed LLM performance.
- domain assumption Equal average bits is a fair basis for comparing SWSC with RTN.
Cite this review
Pith. "Pith review of SWSC: Shared Weight for Similar Channel in LLM." pith.science (2026). https://pith.science/paper/KAQ2KDYI
@misc{pith2026250108631,
author = {Pith},
title = {Pith review of: SWSC: Shared Weight for Similar Channel in LLM},
year = {2026},
howpublished = {\url{https://pith.science/paper/KAQ2KDYI}},
note = {Machine review of arXiv:2501.08631}
}
read the original abstract
Large language models (LLMs) have spurred development in multiple industries. However, the growing number of their parameters brings substantial storage and computing burdens, making it essential to explore model compression techniques for parameter reduction and easier deployment. We propose SWSC, an LLM compression method based on the concept of Shared Weight for Similar Channel. It uses the K-Means clustering algorithm to cluster model weights channel-by-channel, generating clusters with highly similar vectors within each. A representative vector from each cluster is selected to approximately replace all vectors in the cluster, significantly reducing the number of model weight parameters. However, approximate restoration will inevitably cause damage to the performance of the model. To tackle this issue, we perform singular value decomposition on the weight error values before and after compression and retain the larger singular values and their corresponding singular vectors to compensate for the accuracy. The experimental results show that our method can effectively ensure the performance of the compressed LLM even under low-precision conditions.
Figures
Reference graph
Works this paper leans on
-
[1]
Revolutionizing finance with llms: An overview of applications and insights,
H. Zhao, Z. Liu, Z. Wu, Y . Li, T. Yang, P. Shu, S. Xu, H. Dai, L. Zhao, G. Mai et al. , “Revolutionizing finance with llms: An overview of applications and insights,” arXiv preprint arXiv:2401.11641 , 2024
arXiv 2024
-
[2]
Large language models for education: A survey and outlook,
S. Wang, T. Xu, H. Li, C. Zhang, J. Liang, J. Tang, P. S. Yu, and Q. Wen, “Large language models for education: A survey and outlook,” arXiv preprint arXiv:2403.18105 , 2024
arXiv 2024
-
[3]
A survey on large language models for code generation,
J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,” 2024. [Online]. Available: https://arxiv.org/abs/2406.00515
arXiv 2024
-
[4]
A survey on model compression for large language models,
X. Zhu, J. Li, Y . Liu, C. Ma, and W. Wang, “A survey on model compression for large language models,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 1556–1577, 2024
2024
-
[5]
Numerical inverting of matrices of high order,
J. V on Neumann and H. H. Goldstine, “Numerical inverting of matrices of high order,” Psychometrika, 1947
work page 1947
-
[6]
Llm-qat: Data-free quantization aware training for large language models,
Z. Liu, B. Oguz, C. Zhao, E. Chang, P. Stock, Y . Mehdad, Y . Shi, R. Krishnamoorthi, and V . Chandra, “Llm-qat: Data-free quantization aware training for large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2305.17888
arXiv 2023
-
[7]
Gptq: Accurate post-training quantization for generative pre-trained transformers,
E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323, 2022
arXiv 2022
-
[8]
Smoothquant: Accurate and efficient post-training quantization for large language models,
G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in International Conference on Machine Learning . PMLR, 2023, pp. 38 087–38 099
2023
Show all 23 references
-
[9]
Duquant: Distributing outliers via dual transformation makes stronger quantized llms,
H. Lin, H. Xu, Y . Wu, J. Cui, Y . Zhang, L. Mou, L. Song, Z. Sun, and Y . Wei, “Duquant: Distributing outliers via dual transformation makes stronger quantized llms,” 2024. [Online]. Available: https://arxiv.org/abs/2406.01721
2024 arXiv
-
[10]
Sparsegpt: Massive language models can be accurately pruned in one-shot,
E. Frantar and D. Alistarh, “Sparsegpt: Massive language models can be accurately pruned in one-shot,” in International Conference on Machine Learning. PMLR, 2023, pp. 10 323–10 337
2023
-
[11]
A simple and effective pruning approach for large language models,
M. Sun, Z. Liu, A. Bair, and J. Z. Kolter, “A simple and effective pruning approach for large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2306.11695
2024 arXiv
-
[12]
Llm-pruner: On the structural pruning of large language models,
X. Ma, G. Fang, and X. Wang, “Llm-pruner: On the structural pruning of large language models,” Advances in neural information processing systems, vol. 36, pp. 21 702–21 720, 2023
2023
-
[13]
Shortgpt: Layers in large language models are more redundant than you expect,
X. Men, M. Xu, Q. Zhang, B. Wang, H. Lin, Y . Lu, X. Han, and W. Chen, “Shortgpt: Layers in large language models are more redundant than you expect,” arXiv preprint arXiv:2403.03853 , 2024
2024 arXiv
-
[14]
In-context learning dis- tillation: Transferring few-shot learning ability of pre-trained language models,
Y . Huang, Y . Chen, Z. Yu, and K. McKeown, “In-context learning dis- tillation: Transferring few-shot learning ability of pre-trained language models,” arXiv preprint arXiv:2212.10670 , 2022
2022 arXiv
-
[15]
Explanations from large language models make small reasoners better,
S. Li, J. Chen, Y . Shen, Z. Chen, X. Zhang, Z. Li, H. Wang, J. Qian, B. Peng, Y . Mao, W. Chen, and X. Yan, “Explanations from large language models make small reasoners better,” 2022. [Online]. Available: https://arxiv.org/abs/2210.06726
2022 arXiv
-
[16]
Lion: Adversarial distillation of proprietary large language models,
Y . Jiang, C. Chan, M. Chen, and W. Wang, “Lion: Adversarial distillation of proprietary large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2305.12870
2023 arXiv
-
[17]
Minillm: Knowledge distillation of large language models,
Y . Gu, L. Dong, F. Wei, and M. Huang, “Minillm: Knowledge distillation of large language models,” in The Twelfth International Conference on Learning Representations, 2024
2024
-
[18]
On-policy distillation of language models: Learning from self-generated mistakes,
R. Agarwal, N. Vieillard, Y . Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem, “On-policy distillation of language models: Learning from self-generated mistakes,” 2024. [Online]. Available: https://arxiv.org/abs/2306.13649
2024 arXiv
-
[19]
Loraprune: Structured pruning meets low-rank parameter-efficient fine-tuning,
M. Zhang, H. Chen, C. Shen, Z. Yang, L. Ou, X. Yu, and B. Zhuang, “Loraprune: Structured pruning meets low-rank parameter-efficient fine-tuning,” 2024. [Online]. Available: https://arxiv.org/abs/2305.18403
2024 arXiv
-
[20]
Zeroquant-fp: A leap forward in llms post-training w4a8 quantization using floating-point formats,
X. Wu, Z. Yao, and Y . He, “Zeroquant-fp: A leap forward in llms post-training w4a8 quantization using floating-point formats,” 2023. [Online]. Available: https://arxiv.org/abs/2307.09782
2023 arXiv
-
[21]
Product quantization for nearest neighbor search,
H. J ´egou, M. Douze, and C. Schmid, “Product quantization for nearest neighbor search,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 1, pp. 117–128, 2011
2011
-
[22]
Llama 2: Open foundation and fine-tuned chat models,
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023
2023 arXiv
-
[23]
Pointer sentinel mixture models,
S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843 , 2016
2016 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.