Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Matryoshka Quantization

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Jointly training a single quantized model across int8, int4, and int2 slices of the same weights yields int2 models that beat independently trained int2 baselines by up to 4% with OmniQuant and 7% with QAT, while leaving int8 and int4…

desk verdict MatQuant's int2 gains are real, but the abstract attributes them to the wrong mechanism; the paper's own ablation points to the 8-bit container. read the letter →

arxiv 2502.06786 v3 pith:LGCTBA34 submitted 2025-02-10 cs.LG cs.AI

Pranav Nair , Puranjay Datta , Jeff Dean , Prateek Jain , Aditya Kusupati This is my paper · ORCID
classification cs.LGcs.AI
keywords modelquantizationmulti-scaletrainingnestedbit-widthsint2largelanguagemodelslow-bitinferenceweightcompressionMatryoshkastructure
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MatQuant trains one integer-quantized model that can be served at several precisions at once. The paper's central claim is that optimizing the losses for the int8, int4, and int2 slices of the same int8 weights jointly makes the extracted int2 model more accurate than a model trained for int2 alone, by up to 4% with OmniQuant and 7% with QAT, while int8 and int4 accuracy stay roughly unchanged. If true, practitioners would no longer need to keep separate checkpoints for different precision levels: a single checkpoint could serve whatever bit-width the deployment supports, and even bit-widths never trained for, such as int3 and int6, would come for free by slicing. The paper also shows that one extra bucket for outliers yields an effective 2.05-bit model with a further 6% improvement over plain MatQuant int2.

What carries the argument

The load-bearing object is the MSB slicing operator $$S(q_c,r) = \mathrm{clamp}(\mathrm{round}(q_c / $2^{{c-r}}$), 0, 2^r-1) \cdot $2^{{c-r}}$,$$ which extracts the $r$ most significant bits of a $c$-bit integer weight so that an int4 or int2 model is nested inside an int8 model. MatQuant's objective (Eq. 7) sums the base algorithm's loss over the bit-widths $R=\{8,4,2\}$ with per-width weights $\lambda_r$, using the same quantized weights and a straight-through estimator for gradients. This joint loss is what lets the low-precision slices pull on the shared high bits during training; it is also what produces the interpolative int3/int6 behavior and the layer-wise Mix'n'Match flexibility, since every layer's int8 weights already contain well-trained int4 and int2 slices.

What would settle it

Run MatQuant and the int2-only baseline on several independent C4 subsamples with different random seeds and evaluate on a held-out task suite beyond the six used; if the int2 accuracy advantage does not consistently clear seed-level noise (the reported margins are roughly 1 to 7 percentage points), the central claim fails.

Watch

Extended reading notes

Core claim

The discovery is that the nested bit structure of integer types is not just a storage convenience but a training signal. Because an int4 or int2 value is literally a slice of the most significant bits of an int8 value, MatQuant defines a slicing operator $S(q_c,r)$ and minimizes a weighted sum of the base quantizer's losses at all three precisions (Eq. 7), so gradient descent can adjust the shared high bits to serve the low-precision model without hurting the high-precision ones. Empirically, on Gemma-2 2B/9B and Mistral 7B, the int2 models extracted this way beat explicitly trained int2 baselines by 1 to 4 percentage points with OmniQuant and 4.5 to 7 percentage points with QAT, with int8 and int4 within about 0.5 percentage points of their baselines. The same joint training shifts the quantized weight distribution toward higher-valued buckets, which the paper identifies as the mechanism behind the int2 gain. As a byproduct, slicing the trained int8 model to int3 or int6 performs on par with models explicitly trained at those widths, and layer-wise mixes of precisions densely span the accuracy-versus-memory curve.

Load-bearing premise

The central comparison assumes that the C4 training subset and the six evaluation tasks are representative enough that the measured int2 gains generalize; since the paper reports single runs without seed variance, the headline improvements over standard int2 quantization may not hold under different data sampling or seeds.

Editorial extensions

If this is right

  • A single MatQuant checkpoint can be served at int8, int4, or int2, and at interpolated int3 or int6, without retraining, with int2 quality better than an independently trained int2 model.
  • Int2 accuracy gains reach up to 4 percentage points with OmniQuant and 7 percentage points with QAT on the tested LLMs, while int8 and int4 stay within roughly half a percentage point of their baselines.
  • Layer-wise Mix'n'Match of int2/int4/int8 across layers produces many models that densely span the accuracy-versus-memory trade-off, including sub-4-bit models that beat the plain 4-bit slice.
  • Giving the int2 model one extra bucket, for an effective precision of about 2.05 bits, adds roughly 6% more accuracy, indicating that a single extra bucket captures many outliers.
  • MatQuant-style training also yields Single Precision MatQuant, which improves standalone int2 quantization even when only the int2 loss is used, though int4 and int8 accuracy drop.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to apply MatQuant to other learning-based quantizers, such as coordinate-descent or learned-rotation methods; if the nested-bit training signal is the real cause, those bases should also show int2 gains.
  • The right-shifted weight distribution suggests standard quantizers underuse high-valued buckets, so co-training may act as a regularizer that redistributes weights without a dedicated outlier model; this could generalize beyond LLM feed-forward blocks to other architectures.
  • Because floating-point formats lack constant bucket sizes under MSB slicing, MatQuant-style nesting for FP8/FP4 would need a different nesting rule, and a natural next test is whether exponent-aware slicing with variable bucket sizes recovers the int2-style gains.
  • In deployment, a single checkpoint that serves many precisions could simplify serving stacks and enable dynamic precision switching based on load or latency, a practical consequence the paper gestures at but does not measure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MatQuant, a multi-scale weight quantization method that exploits the nested MSB structure of integer types: an int8-quantized weight can be sliced to yield int4 and int2 models. The method jointly optimizes losses over several precisions (Eq. 7) on top of base algorithms OmniQuant and QAT, and is evaluated on FFN weights of Gemma-2 2B/9B and Mistral 7B. The main empirical claims are that MatQuant matches the int8/int4 baselines, improves int2 accuracy by up to 3.11 points with OmniQuant and 7.02 points with QAT, supports untrained interpolated precisions (int3, int6), and enables layer-wise Mix'n'Match. An extended variant, Extra Precision MatQuant, adds one extra bucket to the 2-bit model and reports further gains at an effective 2.05 bits. The paper also includes an erratum section explaining a subtle slicing bug in an earlier draft and repurposing the bug as the Extra Precision method.

Significance. If the empirical results hold, MatQuant would be practically valuable: a single checkpoint could be served at multiple bit-widths with int2 quality better than training for int2 alone, and the sliced interpolation and Mix'n'Match results suggest a dense accuracy-versus-cost frontier. The core idea is simple and the implementation is transparent, with detailed per-task results in the appendix and a candid erratum. However, the paper's own ablation in Section 5.3 shows that the int2 gains are not caused by the multi-scale joint optimization in Eq. 7: a single-precision variant (R={2}) matches or beats MatQuant on int2 across all models. This undercuts the causal attribution in the abstract and introduction, although the empirical superiority over the baselines remains. The paper also lacks error bars or seed variance, and the abstract overstates the OmniQuant gains (3.11% max in Table 1, not 4%) and attributes the gains to co-training plus co-distillation even though co-distillation is only an ablation, not part of the main results. These issues are fixable with a revised framing and additional experiments, so the contribution is defensible after major revision.

major comments (4)
  1. [5.3, Table 5] The ablation in Table 5 contradicts the causal claim in the abstract and Section 1 that jointly optimizing the multi-precision loss in Eq. 7 drives the int2 improvements. Single Precision MatQuant, which sets R={2} and removes the int8 and int4 losses, achieves int2 Task Avg 64.02 on Gemma-2 9B with OmniQuant versus 63.35 for MatQuant, and is within 0.12–0.49 of MatQuant for QAT across all three models. The paper itself states that 'the int2 performance is slightly worse than Single Precison MatQuant' (Section 5.3). The empirical gains over the standard int2 baseline therefore appear to come from training within the overparameterized int8 container while optimizing only the int2 loss, not from the multi-scale co-training that the paper emphasizes. Please either revise the abstract and introduction to attribute the gain to the nested 8-bit representation (which is still a valid and interesting finding) or provide new evidence, such as a controlled comparison that fixes the int8 container and ablates the additional losses, that the joint optimization itself contributes to int2 quality.
  2. [Abstract; Table 1] The abstract claims int2 models extracted by MatQuant outperform standard int2 quantization by 'up to 4% and 7%' with OmniQuant and QAT, respectively. The QAT claim matches Table 2 (7.02% on Mistral 7B), but the OmniQuant claim is not supported: the largest OmniQuant int2 gain in Table 1 is 3.11% (Gemma-2 9B), with 1.04% and 3.01% for the other two models. Section 4.1 correctly reports these smaller numbers. In addition, the abstract attributes the gains to 'co-training and co-distillation regularization', but Tables 1 and 2 do not use co-distillation; co-distillation appears only in the Section 5.2 ablation on Gemma-2 9B. The abstract should be corrected to state the actual OmniQuant gains and should not present co-distillation as part of the main method, since the default MatQuant recipe in the main experiments does not include it.
  3. [4.2, Table 2; Appendix B] All results in the paper are single runs with no seed variance, and the QAT experiments are acknowledged to overfit to the C4 subset (Section 4.2). This is especially concerning for the headline QAT int2 gains: the QAT int2 baseline for Gemma-2 2B (Task Avg 47.74) is substantially lower than the OmniQuant int2 baseline (51.33) despite QAT using 10x more training tokens, and the authors attribute this to overfitting. Without multiple seeds or at least a repeated baseline run, it is difficult to rule out that part of the reported 4.46–7.02 point QAT gains comes from an unlucky baseline run. Please report variance over at least three seeds for the int2 baselines and MatQuant, or provide a clear justification for why the single-run numbers are stable.
  4. [7, Eq. 8; Table 7] The Extra Precision MatQuant result is obtained by reverting to the buggy slicing operator from the first draft (Eq. 8 without the clamp), which the authors corrected in Eq. 6. The paper is transparent about this in the Errata, but the main text and abstract describe the method as 'using an extra bit to represent outliers' without clarifying that this is an extra quantization bucket, not a per-outlier bit allocation, and that the effective bit count rises to 2.05 bits. The relationship between the corrected MatQuant (Eq. 6) and Extra Precision MatQuant should be moved or summarized in Section 3 so that the contribution is defined in the main body rather than only in the Errata; otherwise the reader must reverse-engineer the method from the bug report. The empirical finding is interesting, but the framing should distinguish the deliberate extra-bucket design from the accidental bug that motivated it.
minor comments (6)
  1. [Abstract] There is a typo 'up to to 4%' in the abstract; also the abstract's '4%' should be corrected to match Table 1 as noted in the major comments.
  2. [Section 3.2, Eq. 6; Appendix A] The text in Appendix A says the rounding is 'captured by the round function in Equation 6', but Eq. 6 uses a floor symbol with clamping. Please clarify whether the operator rounds to nearest (with ties handled as described in Appendix A) or floors before clamping, since the example (53 -> 1) implies round-to-nearest behavior.
  3. [Section 4.1; Figure 1b] Figure 1b shows 'especially > 8% for int2' on Gemma-2 9B with OmniQuant, but the text in Section 4.1 reports a 3.11% gain (63.35 vs 60.24) for that configuration. The figure annotation appears to refer to the QAT results or to the co-distillation ablation; please label the figure consistently with the reported numbers.
  4. [Section 5.3; Table 5] The abbreviation 'S.P. MatQuant' and 'S.P. E.P. MatQuant' are used without expansion; please define them in the caption or text. The heading 'Single Precison MatQuant' contains a typo ('Precison').
  5. [Appendix B] There is a typo 'consine decay' and the text switches between '10M tokens' and '8.3M tokens' for OmniQuant co-distillation without explaining the difference. Please state the token budget for each configuration in a consistent way.
  6. [Section 5.4] The sentence 'custom-implemented CUDA kernels can can support' has a doubled 'can'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MatQuant's int2 gains are empirical measurements from a multi-scale training objective, not quantities forced by the objective's own definition.

full rationale

MatQuant's objective (Eq. 7) is a weighted sum of the base algorithm's losses evaluated on MSB-sliced quantized weights; the int8/int4/int2 numbers in Tables 1-2, the int3/int6 interpolation results, and the Single Precision and Extra Precision variants are all reported measurements produced by that objective, not quantities solved for by Eq. 7. The lambda_r weights are searched over as hyperparameters and the paper reports the resulting downstream accuracy; no headline result is obtained by substituting a fitted value back into the same equation that defines it. The slicing operator (Eq. 6) is explicitly defined, and the interpolative-behavior claims compare sliced MatQuant models to independently trained baselines, so those comparisons are external rather than definitional. Citations to Kusupati et al. (2022) and Devvrit et al. (2023) supply motivation and a Mix'n'Match strategy, but they do not carry the load of the int2 improvement claim, and neither citation is invoked as a uniqueness theorem or as a substitute for the paper's own experiments. The Single Precision MatQuant ablation in Section 5.3 does raise a legitimate internal-validity concern: Table 5 shows that removing the int4 and int8 losses can match or beat the full MatQuant objective on int2, which weakens the abstract's causal attribution of the gains to 'co-training and co-distillation regularization.' That is a correctness or interpretation concern, however, not circularity: the improvement over the baseline is still an empirical fact, and no equation in the paper reduces by construction to another equation or to its own fitted input.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new entities. Its dependence on fitted hyperparameters is limited to the precision loss weights and the post-hoc Extra Precision bucket choice. The core assumption is that an int8 weight can be trained to serve multiple precisions simultaneously without hurting the higher precisions.

free parameters (3)
  • Loss reweighting factors lambda_r = (0.1,0.1,1.0) for Gemma models; (0.4,0.4,1.0) for Mistral OmniQuant; (0.2,0.2,1.0) for Mistral QAT
    The joint MatQuant loss (Eq 7) weights each target precision; values are chosen by linear search in Section 5.1, and the final selection differs by model and base algorithm, indicating sensitivity.
  • Co-distillation configuration = [8,4,2,8->4;2] selected for best int2
    Table 4 shows several co-distillation variants; the best int2 result adds a distillation loss from the int8 model to the lower precisions, a design choice not derived from first principles.
  • Effective bit-width of Extra Precision MatQuant = 2.050 avg bits for int2
    The extra bucket from Eq 8 adds an average of 0.05 bits per parameter; this is a structural modification chosen post-hoc after discovering the bug.
assumptions (3)
  • standard math Slicing the most significant bits of an int8-quantized weight yields a valid lower-precision integer representation (Eq 6).
    The nested structure of integer types is a mathematical fact; clamp and round are added to handle overflow and rounding.
  • domain assumption The straight-through estimator provides usable gradients through the quantization operator (Section 3.1.1).
    Standard practice in QAT; not proven but widely used.
  • domain assumption A single shared int8 weight can simultaneously satisfy optimization objectives for int8, int4, and int2 (Eq 7).
    This is the central modeling hypothesis; the paper provides empirical evidence such as Figure 1c, but no guarantee that int8 and int4 accuracy will not degrade.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Matryoshka Quantization." pith.science (2026). https://pith.science/paper/LGCTBA34

@misc{pith2026250206786,
  author       = {Pith},
  title        = {Pith review of: Matryoshka Quantization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LGCTBA34}},
  note         = {Machine review of arXiv:2502.06786}
}
read the original abstract

Quantizing model weights is critical for reducing the communication and inference costs of large models. However, quantizing models -- especially to low precisions like int4 or int2 -- requires a trade-off in model quality; int2, in particular, is known to severely degrade model quality. Consequently, practitioners are often forced to maintain multiple models with different quantization levels or serve a single model that best satisfies the quality-latency trade-off. On the other hand, integer data types, such as int8, inherently possess a nested (Matryoshka) structure where smaller bit-width integers, like int4 or int2, are nested within the most significant bits. Leveraging this insight, in this paper, we propose Matryoshka Quantization (MatQuant), a novel multi-scale quantization technique that alleviates the aforementioned challenge. This technique allows us to train and maintain a single quantized model but serve it with the precision demanded by the deployment. Furthermore, leveraging MatQuant's co-training and co-distillation regularization, int2 precision models extracted by MatQuant outperform standard int2 quantization by up to to 4% and 7% with OmniQuant and QAT as base algorithms respectively. Finally, we demonstrate that by using an extra bit to represent outliers, a model with an effective precision of 2.05-bit gives an additional 6% improvement with OmniQuant as the base algorithm.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BitNet Text Embeddings

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    BITEMBED trains 1.58-bit ternary-weight LLM embedders with contrastive pre-training, supervised distillation, and multi-precision output training, matching FP16 teachers within ~0.6 MMTEB points at ~2x CPU speed.

  2. Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A 2-bit base plus three 2-bit residual stages gives one checkpoint that runs at 2, 4, 6, or 8 bits, matching a prior multi-precision baseline at 6-8 bits in most tested models.

Reference graph

Works this paper leans on

40 extracted references · 12 canonical work pages · cited by 2 Pith papers

  1. [1]

    Abdolrashidi, L

    A. Abdolrashidi, L. Wang, S. Agrawal, J. Malmaud, O. Rybakov, C. Leichner, and L. Lew. Pareto-optimal quantized resnet is mostly 4-bit. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3091--3099, 2021

    work page 2021

  2. [2]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    E. H. Adelson, C. H. Anderson, J. R. Bergen, P. J. Burt, and J. M. Ogden. Pyramid methods in image processing. RCA engineer, 29 0 (6): 0 33--41, 1984

    work page 1984

  4. [4]

    Adepu, Z

    H. Adepu, Z. Zeng, L. Zhang, and V. Singh. Framequant: Flexible low-bit quantization for transformers. arXiv preprint arXiv:2403.06082, 2024

    arXiv 2024

  5. [5]

    Ashkboos, A

    S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. CoRR, abs/2404.00456, 2024. doi:10.48550/ARXIV.2404.00456. URL https://doi.org/10.48550/arXiv.2404.00456

    arXiv doi:10.48550/arxiv.2404.00456 2024

  6. [6]

    Bengio, N

    Y. Bengio, N. L \'e onard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013

    arXiv 2013

  7. [7]

    Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intellig...

    doi:10.1609/aaai.v34i05.6239 2020

  8. [8]

    J. Chee, Y. Cai, V. Kuleshov, and C. M. De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

Show all 40 references
  1. [9]

    M. Chen, W. Shao, P. Xu, J. Wang, P. Gao, K. Zhang, Y. Qiao, and P. Luo. Efficientqat: Efficient quantization-aware training for large language models. CoRR, abs/2407.11062, 2024. doi:10.48550/ARXIV.2407.11062. URL https://doi.org/10.48550/arXiv.2407.11062

    2024 doi arXiv

  2. [10]

    Clark, K

    C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Associat...

    2019

  3. [11]

    Clark, I

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457

    2018 arXiv

  4. [12]

    E. L. Denton, S. Chintala, R. Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial networks. Advances in neural information processing systems, 28, 2015

    2015

  5. [13]

    Dettmers, M

    T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 0 30318--30332, 2022

    2022

  6. [14]

    Dettmers, R

    T. Dettmers, R. Svirschevski, V. Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078, 2023

    2023 arXiv

  7. [15]

    Devvrit, S

    F. Devvrit, S. Kudugunta, A. Kusupati, T. Dettmers, K. Chen, I. Dhillon, Y. Tsvetkov, H. Hajishirzi, S. Kakade, A. Farhadi, P. Jain, et al. Matformer: Nested transformer for elastic inference. arXiv preprint arXiv:2310.07707, 2023

    arXiv

  8. [16]

    D. Du, Y. Zhang, S. Cao, J. Guo, T. Cao, X. Chu, and N. Xu. Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation. In L. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (...

    2024 doi

  9. [17]

    Dubey, A

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    Frantar, S

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  11. [19]

    G Team, P

    G. G Team, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  12. [20]

    Gemma 2: Improving open language models at a practical size

    Gemma-Team. Gemma 2: Improving open language models at a practical size. ArXiv, abs/2408.00118, 2024. URL https://api.semanticscholar.org/CorpusID:270843326

    2024 arXiv

  13. [21]

    Jacob, S

    B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704--2...

    2018

  14. [22]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de Las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7b. CoRR, abs/2310.06825, 2023. doi:10...

    doi arXiv

  15. [23]

    S. Kim, C. Hooper, A. Gholami, Z. Dong, X. Li, S. Shen, M. W. Mahoney, and K. Keutzer. Squeezellm: Dense-and-sparse quantization. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openr...

    2024

  16. [24]

    Kusupati, G

    A. Kusupati, G. Bhatt, A. Rege, M. Wallingford, A. Sinha, V. Ramanujan, W. Howard-Snyder, K. Chen, S. Kakade, P. Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35: 0 30233--30249, 2022

    2022

  17. [25]

    J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023

    2023 arXiv

  18. [26]

    T.-Y. Lin, P. Doll \'a r, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117--2125, 2017

    2017

  19. [27]

    Z. Liu, B. Oguz, C. Zhao, E. Chang, P. Stock, Y. Mehdad, Y. Shi, R. Krishnamoorthi, and V. Chandra. LLM-QAT: data-free quantization aware training for large language models. In L. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguisti...

    2024 doi

  20. [28]

    Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V. Chandra, Y. Tian, and T. Blankevoort. Spinquant: LLM quantization with learned rotations. CoRR, abs/2405.16406, 2024 b . doi:10.48550/ARXIV.2405.16406. URL https://doi.org/10.48550/arXiv.2405.16406

    2024 doi arXiv

  21. [29]

    Y. Ma, H. Li, X. Zheng, F. Ling, X. Xiao, R. Wang, S. Wen, F. Chao, and R. Ji. Affinequant: Affine transformation quantization for large language models. arXiv preprint arXiv:2403.12544, 2024

    2024 arXiv

  22. [30]

    P. A. Nair and A. S. Suggala. Cdquant: Accurate post-training weight quantization of large pre-trained models using greedy coordinate descent. CoRR, abs/2406.17542, 2024. doi:10.48550/ARXIV.2406.17542. URL https://doi.org/10.48550/arXiv.2406.17542

    2024 doi arXiv

  23. [31]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  24. [32]

    Rippel, M

    O. Rippel, M. Gelbart, and R. Adams. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pages 1746--1754. PMLR, 2014

    2014

  25. [33]

    Sakaguchi, R

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IA...

    2020 doi

  26. [34]

    W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y. Qiao, and P. Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137, 2023

    2023 arXiv

  27. [35]

    M. Sun, Z. Liu, A. Bair, and J. Z. Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/foru...

    2024

  28. [36]

    Vaswani, N

    A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017. URL https://api.semanticscholar.org/CorpusID:13756489

    2017

  29. [37]

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087--38099. PMLR, 2023

    2023

  30. [38]

    H. Yu, H. Li, H. Shi, T. S. Huang, and G. Hua. Any-precision deep neural networks. ArXiv, abs/1911.07346, 2019. URL https://api.semanticscholar.org/CorpusID:208138922

    1911 arXiv

  31. [39]

    J. Yu, L. Yang, N. Xu, J. Yang, and T. Huang. Slimmable neural networks. arXiv preprint arXiv:1812.08928, 2018

    2018 arXiv

  32. [40]

    Zellers, A

    R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. M \` a rquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, ...

    2019 doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.