Pith. sign in

REVIEW 3 major objections 5 minor 56 references

Spectra 1.1: Scaling Laws and Efficient Inference for Ternary Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that ternary language models should be scaled by feeding them more training tokens rather than adding parameters, and that their weights can be packed to 1.6 or 2 bits and run with a custom GPU kernel up to five times…

desk verdict The scaling-law motivation is flatly contradicted by the paper's own Eq. (2), but the packing schemes and TriRun kernel are real, well-engineered contributions that deserve a serious look. read the letter →

arxiv 2506.23025 v1 pith:GGLR7ENL submitted 2025-06-28 cs.LG cs.AI

Tejas Vaidhya , Ayush Kaushal , Vineet Jain , Francis Couture Harpin , Prashant Shishodia , Majid Behbahani , Yuriy Nevmyvaka , Irina Rish This is my paper
classification cs.LGcs.AI
keywords ternarylanguagemodelsscalinglawsquantization-awaretraining1.6-bitpacking2-bitGPUinferencekernelsTriRunefficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that ternary language models—transformers whose weights are restricted to the values -1, 0, and 1—benefit more from extra training data than from extra parameters. It fits a scaling law showing that validation loss falls much faster with training tokens than with model size, then trains a suite of ternary models on up to 1.2 trillion tokens and reports sustained gains. It also shows that ternary weights can be stored losslessly in about 1.6 or 2 bits per weight, and presents a GPU kernel, TriRun, that accelerates end-to-end generation by up to roughly five times versus a float16 baseline. A reader should care because memory bandwidth, not compute, is the main bottleneck during LLM inference, and ternary weights directly shrink the bytes that must move through that bottleneck.

What carries the argument

The central object is the ternary linear layer, a linear layer whose weight matrix is restricted to $\{-1,0,1\}$ times a shared floating-point scale, trained with latent full-precision weights and on-the-fly rounding in the forward pass. The fitted scaling law of Equation (2) is the mechanism that carries the resource-allocation claim: its data exponent $0.81$ exceeding its parameter exponent $0.32$ is what tells the authors to invest compute in tokens rather than parameters. For the efficiency claim, the key identity is the lossless-packing condition $2^p > 3^k$, which shows that $k=5$ ternary digits fit into $p=8$ bits, yielding 1.6 effective bits per weight; TriRun then exploits the 2-bit packing with an FP16-times-INT2 mixed-precision tensor-core kernel using asynchronous memory copies and fused dequantization.

What would settle it

Train two ternary models matched for total compute on the same data and tokenizer—for example, 3.6B parameters on 600B tokens versus 1.8B parameters on 1.2T tokens—and compare their validation losses; if the larger-parameter model matches or beats the larger-data model, the fitted exponents in Equation (2) are not transferable to the scale the paper claims.

Watch

Extended reading notes

Core claim

For ternary language models trained with quantization-aware forward passes, the paper derives a parametric scaling law, fitted on models from 99M to 1.1B parameters and 20B to 150B tokens: $\hat{L}(N,D) \approx 2.19 + 4.73/N^{0.32} + 5.18/D^{0.81}$. Because the exponent on training tokens, 0.81, is larger than the exponent on parameters, 0.32, each additional token reduces validation loss more than each additional parameter does. The paper contrasts this with a fitted float16 model law where the exponents are nearly equal, and concludes that compute spent on tokens is the better investment for ternary models. That conclusion motivates Spectra-1.1, a suite of 1.5B, 2.5B, and 3.6B parameter ternary models trained on 1.2 trillion tokens, with the 3B model still improving at the end of training and competitive with floating-point models of similar compute budget. The paper further claims that ternary weights can be packed losslessly at 2 bits via base-4 encoding and at 1.6 bits via base-3 encoding when $2^p > 3^k$, and that the TriRun GPU kernel, built on the 2-bit packing, accelerates the ternary linear layers and whole-model generation relative to float16 implementations.

Load-bearing premise

The load-bearing premise is that the power law fitted on models up to 1.1B parameters and 150B tokens on one data mixture continues to hold at 3.6B parameters and 1.2T tokens on a different corpus with a different tokenizer; if the data exponent shrinks in that regime, the entire 'tokens over parameters' conclusion loses its support.

Editorial extensions

If this is right

  • Ternary models should be trained at much higher token-to-parameter ratios than floating-point models, so compute-optimal recipe guides derived for float LMs do not directly transfer.
  • A 3B ternary model can keep improving past 1.2T tokens, suggesting the practical scale limit for ternary pretraining is farther out than for equivalently sized float models.
  • Ternary weights need only about 1.6 bits per weight for near-optimal storage, cutting model memory by roughly a factor of eight compared with 16-bit weights.
  • The TriRun kernel can serve a 70B ternary model on a single GPU with roughly 4.9x end-to-end generation speedup over a float16 PyTorch baseline running on four GPUs, with larger models seeing larger gains because a greater fraction of compute is in the ternary layers.
  • The released Spectra-1.1 models and TriRun kernels give downstream researchers an open base for building and studying sub-2-bit language models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the token exponent persists outside the fitted range, the optimal parameter count for a fixed compute budget shifts smaller for ternary models, making them especially attractive for deployment on memory-constrained hardware.
  • Because the 1.6-bit packing is already close to the information-theoretic limit of $\log_2(3) \approx 1.585$ bits per weight, further memory reductions would have to come from embeddings, KV caches, or activations rather than from the weights themselves.
  • The gap between TriRun's roughly 8x per-layer speedup and its roughly 5x end-to-end speedup identifies non-linear-layer overhead as the next target for ternary inference work.
  • The scaling law was fitted on a different data mixture and tokenizer than the 1.2T-token runs, so a direct refit that includes the largest models would test whether the data-over-parameters conclusion is a law or an artifact of the small-scale regime.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies ternary language models (TriLMs). It fits a Hoffmann-style scaling law L(N,D)=E+A/N^alpha+B/D^beta, reporting alpha=0.32 and beta=0.81, and concludes that increasing training tokens is more effective than increasing model parameters. Based on this conclusion, the authors train the Spectra-1.1 suite of TriLMs on up to 1.2T tokens and report improved MMLU scores over the earlier Spectra models. The paper also contributes a lossless 1.6-bit packing scheme with a correctness theorem, CPU inference kernels (TQ1/TQ2), and a GPU kernel called TriRun that achieves up to roughly 8x per-layer and 5x end-to-end speedups over FP16 baselines.

Significance. The engineering contributions are concrete and well benchmarked: the packing theorem in Appendix E is proved, the CPU and GPU kernel experiments cover multiple hardware platforms and model sizes, and the release of Spectra-1.1 and TriRun would be a useful resource for the community. However, the paper's central scientific claim—that TriLMs benefit more from scaling data than from scaling parameters—is not supported by the fitted scaling law. The conclusion is based on comparing exponents rather than marginal loss reductions, and it is contradicted by Eq. (2) even within the fitted range. The extrapolation to 1.2T tokens and 3.6B parameters is also unvalidated and confounded by data mixture and tokenizer changes. The core systems results may survive a major revision, but the scaling-law narrative needs substantial correction.

major comments (3)
  1. [§2.2, Eq. (2)] The statement that increasing tokens lowers validation loss more effectively than increasing parameters is not supported by Eq. (2), even within the fitted range. At the upper end of the fit (N=1100M, D=150B), A/N^0.32 ≈ 0.503 and B/D^0.81 ≈ 0.0895, so doubling N reduces the predicted loss by about 0.100 while doubling D reduces it by about 0.038. At the deployed Spectra-1.1-3B scale (N=3668M, D=1200B), the parameter term is ≈0.342 and the data term is ≈0.0166; doubling N gives ≈0.068 versus ≈0.007 for doubling D. A larger exponent beta=0.81 than alpha=0.32 does not by itself mean data is the better investment; under the standard compute-optimal interpretation of Eq. (1), the fitted exponents imply N_opt ∝ C^{0.72} and D_opt ∝ C^{0.28}, which is the opposite of the paper's data-heavy design choice.
  2. [§2.2–§2.3, data and tokenizer switch] The scaling law is fit on a SlimPajama subset with N ≤ 1.1B and D ≤ 150B, while the Spectra-1.1 models are trained on a different mixture (FineWeb-Edu is 80% of the 1.2T-token corpus in Table 1) with a different tokenizer and at N up to 3.7B and D = 1.2T. No held-out N or D points are used to test the law, and no measured validation loss from the 1.2T-token models is compared against the prediction of Eq. (2). The R² = 0.9921 reported in Appendix C.2 is an in-sample fit statistic; it does not establish predictive validity outside the fitted range. The extrapolation is load-bearing for the paper's central motivation and is currently unsupported.
  3. [§2.3, Figure 4] The comparison between Spectra-1.1 (1.2T tokens, new data, LLaMA tokenizer) and Spectra (300B tokens, old data, GPT-NeoX tokenizer) changes several variables at once. The MMLU gains shown in Figure 4 could be due to data quality, tokenizer, or training-configuration changes (for example, Table 2 introduces QK-Norm and z-loss), so the figure does not isolate the effect of training tokens and cannot validate the claim that TriLMs benefit more from data scaling than from parameter scaling.
minor comments (5)
  1. [§1] The word "refered" in the introduction should be "referred".
  2. [Table 5] The caption of Table 5 refers to an "AMD EPYC 750 laptop", but the benchmark is run on an AMD EPYC 7502 server CPU.
  3. [§2.2] The text refers to "Appendix 1" for dataset details; the actual dataset description and token counts are in Appendix B.2 and Table 1.
  4. [References] The reference list contains two entries for Dettmers et al. (2022a and 2022b) that appear to be the same arXiv paper; the duplicate should be removed or disambiguated.
  5. [§3.2] The decoding formulas for the 1.6-bit scheme use both b'_i and x_i without explicitly specifying the handling of a partial final block, which makes the pseudocode difficult to implement for sequences whose length is not a multiple of k.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the scaling-law conclusions are interpretations of a fitted curve, the 1.2T results are measured, and the packing proofs are self-contained.

full rationale

The paper's scaling-law claim ('From Equation (2), we observe that increasing the number of tokens lowers the validation loss more effectively than increasing the number of parameters') is a direct interpretation of the fitted exponents in Eq. (2), not a prediction generated by a separately fitted parameter and then reported as confirmation. The Spectra-1.1 results in Section 2.3 are empirical measurements on the 1.2T-token models, not values produced by Eq. (2), so no fitted input is being renamed as a prediction. The scaling-law form is adopted from Hoffmann et al. (2022) by explicit assumption and then fit to the paper's own validation losses; this is ordinary curve fitting and does not smuggle in the conclusion. Self-citations to Kaushal et al. (2024) supply the TriLM architecture and prior Spectra models, but the new scaling-law fit, the 1.2T-token training runs, and the packing/kernel benchmarks are independent of those citations. The 1.6-bit/2-bit packing theorems are proved from the stated definitions of P and U, with no circular appeal. The reader's and skeptic's concerns about extrapolating Eq. (2) to 1.2T tokens and about the internal direction of the exponent comparison are substantive correctness and validity risks, but they are not circularity: the paper does not define 'benefit more' as 'β > α' and then claim an empirical discovery, nor does it validate its central claim using the same fitted values from which the claim was derived.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the fitted scaling-law parameters (E, A, alpha, B, beta), which are free parameters fitted to the validation-loss data. The extrapolation of that law to 1.2T tokens and 3.6B parameters is an additional domain assumption. No new physical entities are introduced.

free parameters (5)
  • E = 2.19
    Irreducible loss term in the TriLM scaling law, fitted to the validation losses of the model suite.
  • A = 4.73
    Coefficient of the parameter term N^{-alpha} in Equation (2), fitted to training data.
  • alpha = 0.32
    Exponent for model parameters in the scaling law; its small value relative to beta is what drives the conclusion that data helps more.
  • B = 5.18
    Coefficient of the token term D^{-beta} in Equation (2), fitted to training data.
  • beta = 0.81
    Exponent for training tokens in the scaling law; this larger exponent is the basis for the main claim.
assumptions (5)
  • domain assumption Validation loss follows the separable parametric form L(N,D) = E + A N^{-alpha} + B D^{-beta} from Hoffmann et al.
    Assumed in Section 2.2, Equation (1). If the true loss is not separable, the fitted exponents and the resulting conclusion are not meaningful.
  • domain assumption The irreducible loss E is constant and includes a fixed quantization-error contribution independent of N and D.
    Stated in Section 2.2. If quantization error varies with model size or data, the fitted exponents would be biased.
  • domain assumption The fitted power law extrapolates from models up to 1.1B parameters and 150B tokens to 3.6B parameters and 1.2T tokens on a different data mixture with a different tokenizer.
    This extrapolation is the bridge from the scaling-law fit to the decision to train Spectra-1.1 at 1.2T tokens. It is stated implicitly in Sections 2.2 and 2.3 and is never validated with measured losses at the larger scale.
  • domain assumption Straight-through estimation and round-to-nearest ternary quantization in the forward pass produce a trainable and reliable model.
    Standard QAT practice, described in Appendix B.1, but not independently proven in this paper.
  • domain assumption Zero-shot benchmark scores (MMLU, commonsense, etc.) reflect the validation-loss improvements that the scaling law describes.
    The paper uses validation loss for the scaling law and MMLU for the model comparisons, but never establishes a quantitative connection between the two.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Spectra 1.1: Scaling Laws and Efficient Inference for Ternary Language Models." pith.science (2026). https://pith.science/paper/GGLR7ENL

@misc{pith2026250623025,
  author       = {Pith},
  title        = {Pith review of: Spectra 1.1: Scaling Laws and Efficient Inference for Ternary Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GGLR7ENL}},
  note         = {Machine review of arXiv:2506.23025}
}
read the original abstract

Large language models (LLMs) are increasingly used across research and industry applications, yet their inference efficiency remains a significant challenge. As the computational power of modern GPU architectures continuously improves, their memory bandwidth and capacity have not scaled proportionally, creating a critical bottleneck during inference. To address this, we investigate ternary language models (TriLMs) that employ quantization-aware training to significantly reduce memory requirements. We first analyze the scalability of TriLMs by conducting a scaling law analysis, revealing that TriLMs benefit more from increasing training data than from scaling model parameters. Based on this observation, we introduce Spectra-1.1, an open suite of TriLMs trained on up to 1.2 trillion tokens, demonstrating sustained performance gains at scale. Furthermore, to improve inference efficiency, we propose novel 2-bit and 1.6-bit packing schemes for ternary weights, which demonstrate accelerated inference across various CPU architectures. Also, building on the 2-bit packing, we develop a GPU kernel called TriRun that accelerates end-to-end model inference by up to 5 times compared to floating-point baselines. To encourage further exploration and development of TriLMs, we will release the Spectra-1.1 suite and TriRun inference kernels. Overall, our work lays the foundation for building and deploying efficient LLMs, providing a valuable resource for the research community.

Figures

Figures reproduced from arXiv: 2506.23025 by the authors.

Figure 1
Figure 1. Model performance (MMLU average accuracy) versus training FLOPs, considering only models with similar compute budgets and training tokens for a fair comparison (left); and end-to-end generation time speedup achieved by TriRun kernels over the PyTorch’s FP16 baseline (64 Input Tokens, 64 Output Tokens) on the NVIDIA L40S (right). absence of a comprehensive suite of strong open￾source models, suppressing innovation in… view at source ↗
Figure 2
Figure 2. Effect of scaling number of parameters (left) and number of training tokens (right) on final validation loss for TriLMs. The dotted lines show the power law derived in Equation (2). (Clement et al., 2019), Cosmopedia-v2 (Ben Al￾lal et al., 2024), PeS2o (Soldaini and Lo, 2023), Zyda-StarCoder-Git-Commits, Zyda-StarCoder￾Languages (Tokpanov et al., 2024), FineWeb-Edu (Lozhkov et al., 2024). The dataset details are sum… view at source ↗
Figure 3
Figure 3. Number of GPUs vs. Relative Speedup. 2.2 Scaling Laws for TriLMs Experimental Setup. For this study (≤150B to￾kens), we use a SlimPajama subset from Shen et al. (2024), while the 1.2T-token dataset incor￾porating additional sources (Appendix 1). All other aspects follow the procedures outlined in Sections 2.1 regarding the pretraining of the mod￾els. We train and evaluate a suite of TriLM models, conducting a series… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Average MMLU accuracy for Spectra-1.1 and Spec￾tra, with the dotted line representing LLaMA-1 7B (trained on 1.2T tokens). Note that LLaMA-3 (AI@Meta, 2024), trained on over 15 trillion tokens, is not included. benchmark scores with the Spectra suite of models, which h…
Figure 5
Figure 5. Figure 5: Comparison of output tokens for different model sizes running on a Mac M4 CPU laptop: (Left) Total output tokens (for a 256 prompt with 64 output tokens). (Center) Output tokens per second versus model size. (Right) Memory requirements by model size (in GB) with differ…
Figure 6
Figure 6. Figure 6: Performance evaluation of ternary layers in a transformer block, comparing TriRun with PyTorch FP16 (using CUTLASS), shows near-optimal inference speedup in high-batch settings for larger models. Each subplot corresponds to a specific Nvidia GPU. For additional results…
Figure 7
Figure 7. Figure 7: Comparison of TriRun kernels with the FP16 PyTorch baseline on NVIDIA L40S (for more details see Appendix G.6): (a) Left: Time to first token, (b) Center: Time per output token, (c) Right: Total time across different NVIDIA GPUs. these weights are used only once during…
Figure 8
Figure 8. Figure 8: Effect of scaling number of parameters (left) and number of training tokens (right) on final validation loss for FloatLMs. The dotted lines show the power law derived in Equation (3) [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Predicted versus actual values of the final validation loss based on the parametric fit of the scaling law for TriLMs (left) and FloatLMs (right). chines but not humans. Accuracy and normal￾ized accuracy are reported for this dataset. • WinoGrande: (Sakaguchi et al., 2…
Figure 10
Figure 10. Figure 10: Comparison of output tokens for different model sizes running on a AMD EPYC 750 laptop: (Left) Output tokens (for a 256 prompt with 64 output tokens). (Right) Output tokens per second versus model size. For more details, refer to [PITH_FULL_IMAGE:figures/full_fig_p02…
Figure 11
Figure 11. Figure 11: Speedup across hardware over the years using TriRun kernels G.4 Precision-Preserving Accumulation Intra-Warp Reduction Partial sums within a thread block are reduced across warps using shared memory. A tree-based summation merges per-warp FragC outputs, minimizing sha…
Figure 12
Figure 12. Figure 12: We evaluate the performance of ternary layers in transformer blocks, showing near-optimal speedup over [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 13
Figure 13. Figure 13: Comparison of TriRun kernels with the FP16 PyTorch baseline on NVIDIA L40S, L40, A40, and 4090 (top to bottom). More details in [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 5 canonical work pages

  1. [1]

    Advanced Micro Devices, Inc. 2022. https://www.amd.com/content/dam/amd/en/documents/instinct-business-docs/white-papers/amd-cdna2-white-paper.pdf AMD CDNA 2 ARCHITECTURE . White paper, Advanced Micro Devices, Inc

    work page 2022

  2. [2]

    Advanced Micro Devices, Inc. 2025. Amd instinct™ mi250x accelerators. https://www.amd.com/en/products/accelerators/instinct/mi200/mi250x.html. Accessed February 9, 2025

    work page 2025

  3. [3]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  4. [4]

    Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. https://arxiv.org/abs/2404.00456 Quarot: Outlier-free 4-bit inference in rotated llms . Preprint, arXiv:2404.00456

    arXiv 2024

  5. [5]

    Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. 2024. https://huggingface.co/datasets/HuggingFaceTB/cosmopedia Cosmopedia

    2024

  6. [6]

    Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. https://arxiv.org/abs/1308.3432 Estimating or propagating gradients through stochastic neurons for conditional computation . Preprint, arXiv:1308.3432

    arXiv 2013

  7. [7]

    Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. https://arxiv.org/abs/2304.01373 Pythia: A suite for analyzing large language models across training and scaling . Preprint...

    arXiv 2023

  8. [8]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. https://api.semanticscholar.org/CorpusID:208290939 Piqa: Reasoning about physical commonsense in natural language . In AAAI Conference on Artificial Intelligence

    2019

Show all 56 references
  1. [9]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1300 B ool Q : Exploring the surprising difficulty of natural yes/no questions . In Proceedings of the 2019 Conference of the North A ...

    2019 doi

  2. [10]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://api.semanticscholar.org/CorpusID:3922816 Think you have solved question answering? try arc, the ai2 reasoning challenge

    2018

  3. [11]

    Clement, Matthew Bierbaum, Kevin P

    Colin B. Clement, Matthew Bierbaum, Kevin P. O'Keeffe, and Alexander A. Alemi. 2019. https://arxiv.org/abs/1905.00075 On the use of arxiv as a dataset . Preprint, arXiv:1905.00075

    2019 arXiv

  4. [13]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022 b . https://arxiv.org/abs/2208.07339 Llm.int8(): 8-bit matrix multiplication for transformers at scale . Preprint, arXiv:2208.07339

    2022 arXiv

  5. [14]

    Tim Dettmers and Luke Zettlemoyer. 2023. https://arxiv.org/abs/2212.09720 The case for 4-bit precision: k-bit inference scaling laws . Preprint, arXiv:2212.09720

    2023 arXiv

  6. [15]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. https://arxiv.org/abs/2210.17323 Gptq: Accurate post-training quantization for generative pre-trained transformers . Preprint, arXiv:2210.17323

    2023 arXiv

  7. [16]

    Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh

    Elias Frantar, Roberto L. Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh. 2024. https://arxiv.org/abs/2408.11743 Marlin: Mixed-precision auto-regressive parallel inference on large language models . Preprint, arXiv:2408.11743

    2024 arXiv

  8. [17]

    Mahoney, and Kurt Keutzer

    Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, and Kurt Keutzer. 2024. https://arxiv.org/abs/2403.14123 Ai and memory wall . Preprint, arXiv:2403.14123

    2024 arXiv

  9. [18]

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack...

    2024 arXiv

  10. [19]

    Pujiang He, Shan Zhou, Wenhuan Huang, Changqing Li, Duyi Wang, Bin Guo, Chen Meng, Sheng Gui, Weifei Yu, and Yi Xie. 2024. https://arxiv.org/abs/2407.07304 Inference performance optimization for large language models on cpus . Preprint, arXiv:2407.07304

    2024 arXiv

  11. [20]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  12. [21]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  13. [22]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  14. [23]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  15. [24]

    Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kund...

    2019 arXiv

  16. [25]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://arxiv.org/abs/2001.08361 Scaling laws for neural language models . Preprint, arXiv:2001.08361

    2020 arXiv

  17. [26]

    Ayush Kaushal, Tejas Vaidhya, Arnab Kumar Mondal, Tejas Pandey, Aaryan Bhagat, and Irina Rish. 2024. https://arxiv.org/abs/2407.12327 Spectra: Surprising effectiveness of pretraining ternary language models at scale . Preprint, arXiv:2407.12327

    2024 arXiv

  18. [27]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. https://arxiv.org/abs/2306.00978 Awq: Activation-aware weight quantization for llm compression and acceleration . Preprint, arXiv:2306.00978

    2024 arXiv

  19. [28]

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2021. Logiqa: a challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI'20

    2021

  20. [29]

    Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. https://doi.org/10.18653/v1/2020.acl-main.447 S 2 ORC : The semantic scholar open research corpus . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969...

    2020 doi

  21. [30]

    Ilya Loshchilov and Frank Hutter. 2019. https://arxiv.org/abs/1711.05101 Decoupled weight decay regularization . Preprint, arXiv:1711.05101

    2019 arXiv

  22. [31]

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. https://doi.org/10.57967/hf/2497 Fineweb-edu: the finest collection of educational content

    2024 doi

  23. [32]

    Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. 2024. https://arxiv.org/abs/2402.17764 The era of 1-bit llms: All large language models are in 1.58 bits . Preprint, arXiv:2402.17764

    2024 arXiv

  24. [33]

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. https://arxiv.org/abs/1710.03740 Mixed precision training . Preprint, arXiv:1710.03740

    2018 arXiv

  25. [34]

    Denis Paperno, Germ \'a n Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern \'a ndez. 2016. https://doi.org/10.18653/v1/P16-1144 The LAMBADA dataset: Word prediction requiring a broad discourse cont...

    2016 doi

  26. [35]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://api.semanticscholar.org/CorpusID:160025533 Language models are unsupervised multitask learners

    2019

  27. [36]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. https://arxiv.org/abs/1910.02054 Zero: Memory optimizations toward training trillion parameter models . Preprint, arXiv:1910.02054

    2020 arXiv

  28. [37]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. https://doi.org/10.1145/3474381 Winogrande: an adversarial winograd schema challenge at scale . Commun. ACM, 64(9):99–106

    2021 doi

  29. [38]

    Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. 2024. https://arxiv.org/abs/2401.00448 Beyond chinchilla-optimal: Accounting for inference in language model scaling laws . Preprint, arXiv:2401.00448

    2024 arXiv

  30. [39]

    Noam Shazeer. 2020. https://arxiv.org/abs/2002.05202 Glu variants improve transformer . Preprint, arXiv:2002.05202

    2020 arXiv

  31. [40]

    Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric Xing. 2024. https://arxiv.org/abs/2309.10818 Slimpajama-dc: Understanding data combinations for llm training . Preprint,...

    2024 arXiv

  32. [41]

    Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. https://arxiv.org/abs/2303.06865 Flexgen: High-throughput generative infer...

    2023 arXiv

  33. [42]

    Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J

    Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor ...

    2024 arXiv

  34. [43]

    Luca Soldaini and Kyle Lo. 2023. peS2o (Pretraining Efficiently on S2ORC) Dataset . Technical report, Allen Institute for AI . ODC-By, https://github.com/allenai/pes2o

    2023

  35. [44]

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. https://arxiv.org/abs/2104.09864 Roformer: Enhanced transformer with rotary position embedding . Preprint, arXiv:2104.09864

    2023 arXiv

  36. [45]

    PNY Technologies. 2023. https://www.pny.com/en-eu/File Accessed: 2025-02-12

    2023

  37. [46]

    Yury Tokpanov, Beren Millidge, Paolo Glorioso, Jonathan Pilault, Adam Ibrahim, James Whittington, and Quentin Anthony. 2024. https://arxiv.org/abs/2406.01981 Zyda: A 1.3t dataset for open language modeling . Preprint, arXiv:2406.01981

    2024 arXiv

  38. [47]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama:...

    2023 arXiv

  39. [48]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. https://arxiv.org/abs/1706.03762 Attention is all you need . Preprint, arXiv:1706.03762

    2023 arXiv

  40. [49]

    Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. 2023. https://arxiv.org/abs/2310.11453 Bitnet: Scaling 1-bit transformers for large language models . Preprint, arXiv:2310.11453

    2023 arXiv

  41. [50]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. https://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models . Preprint, arXiv:2201.11903

    2023 arXiv

  42. [51]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. https://api.semanticscholar.org/CorpusID:1553193 Crowdsourcing multiple choice science questions . ArXiv, abs/1707.06209

    2017 arXiv

  43. [52]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2024. https://arxiv.org/abs/2211.10438 Smoothquant: Accurate and efficient post-training quantization for large language models . Preprint, arXiv:2211.10438

    2024 arXiv

  44. [53]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://doi.org/10.18653/v1/P19-1472 H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4...

    2019 doi

  45. [54]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. https://arxiv.org/abs/2210.03493 Automatic chain of thought prompting in large language models . Preprint, arXiv:2210.03493

    2022 arXiv

  46. [55]

    Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. 2024. https://arxiv.org/abs/2404.14294 A survey on efficient inference for large language ...

    2024 arXiv

  47. [56]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  48. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.