Pith. sign in

REVIEW 4 major objections 7 minor 6 cited by

The Zamba2 Suite: Technical Report

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Zamba2, a family of 1.2B-7.4B hybrid Mamba2-transformer models, achieves top-of-class accuracy among open models while cutting KV-cache memory by roughly sixfold.

desk verdict Zamba2 is a valuable open release of competitive hybrid SSM/transformer models and a 5T-token pretraining dataset, but the 'state of the art' claim rests on an uncontrolled cross-source evaluation comparison. read the letter →

arxiv 2411.15242 v1 pith:MD6LH35V submitted 2024-11-22 cs.LG cs.AIcs.CL

Paolo Glorioso , Quentin Anthony , Yury Tokpanov , Anna Golubeva , Vasudev Shyam , James Whittington , Jonathan Pilault , Beren Millidge This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords Mamba2hybridstate-spacemodelssharedattentionsmalllanguageinferenceefficiencyKVcachereductionZyda-2datasetopen-weights
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This report introduces Zamba2, a suite of 1.2B, 2.7B, and 7.4B parameter language models that combine a Mamba2 state-space backbone with a small number of shared transformer attention blocks. The authors' central claim is that, at each scale, these hybrids achieve state-of-the-art performance against leading open-weight transformer models on standard benchmarks such as MMLU and ARC, while also delivering 30–50% lower time-to-first-token and about a 6× reduction in KV-cache memory. The claim matters because it suggests that the efficiency advantages of recurrent state-space mixing can be obtained without sacrificing the quality that pure attention delivers, opening a more efficient route for small, deployable, openly available models. The report accompanies the claim by releasing model weights and the 5-trillion-token Zyda-2 pretraining dataset, so both the quality and the efficiency comparisons are checkable by others.

What carries the argument

The machinery is the Zamba2 hybrid layer stack: a Mamba2 backbone (a linear-time recurrent state-space layer with a fixed-size hidden state) interleaved with one or two globally shared transformer attention blocks, where non-shared low-rank adapters (LoRAs) let each attention invocation use different weights at small parameter cost and rotary position embeddings (RoPE) are applied in attention. The argument runs on the ratio: with only one shared attention block per several Mamba2 layers, the model stores a KV cache only for those few layers (a 1:6 ratio in the released models), which is what turns the 6× KV-cache reduction and the latency and throughput gains into concrete, measurable numbers.

What would settle it

Re-run every benchmark in Table I for Zamba2 and each listed competitor in a single evaluation harness using identical prompts, shots, and sampling; if any Zamba2 model no longer leads its weight class on the majority of these tasks, the paper's central quality claim is refuted. A narrower check: verify the 17k-token context extension for Zamba2-7B with a passkey-retrieval probe, since RoPE scaling without finetuning fails for many models.

Watch

Extended reading notes

Core claim

The paper's core discovery, stated on its own terms, is that a hybrid in which most sequence mixing is done by Mamba2 blocks and only a few alternating shared attention blocks provide full attention outperforms parameter-matched and FLOP-matched transformer baselines in head-to-head ablations, and that the released models then outperform the leading open models of their weight classes on the standard evaluation suite. The reported numbers (MMLU 43.1 at 1.2B scale versus 32.8 for Gemma2-2B; 55.97 versus 54.01 at 2.7B; 67.2 versus 65.18 at 7B) are the concrete form of the claim, and the efficiency figures (up to 4× Mamba2 throughput over a transformer block, a 1:6 attention-to-Mamba ratio cutting KV cache by 6×, and a 30–50% time-to-first-token reduction) are its practical consequence.

Load-bearing premise

The state-of-the-art claim assumes that the leaderboard scores of competitor models, drawn from their own published sources, are directly comparable to the authors' own evaluation runs, even though the evaluation harness, prompt templates, and sampling settings for Zamba2 are not specified.

Editorial extensions

If this is right

  • At each of the three scales, Zamba2 tops Table I on MMLU and most other listed benchmarks against open competitors including Gemma2, Llama3.2, and Mistral-7B.
  • Compared with a pure transformer, generation needs about 6× less KV-cache memory at long contexts, and time-to-first-token falls by 30–50%.
  • The 7.4B model's context window extends to roughly 17,000 tokens via NTK RoPE scaling without additional training, and the 2.7B model reaches 65,536-token passkey retrieval after curriculum finetuning.
  • Instruction-tuned variants are strongly competitive with official instruct finetunes on MT-Bench and IFEval using only open SFT/DPO data.
  • The open release of Zyda-2 and all weights means the reported quality and efficiency comparisons can be reproduced by the community.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • [Editorial inference] If the quality ranking survives re-evaluation in one shared harness, the practical import is that architecture choice, not just data quality, is a first-order lever for small-model deployment: hybrid SSM designs may become the default for on-device and long-context serving.
  • [Editorial inference] The shared-attention-plus-LoRA pattern suggests a general recipe for spending a parameter budget: buy full attention sparingly and differentiate its copies cheaply, a design that could transfer to mixture-of-experts or other recurrent layers.
  • [Editorial inference] The authors attribute most of their edge to architecture rather than data; a testable consequence of that attribution is that training a transformer with the same Zyda-2 tokens and annealing schedule should land below the Zamba2 curves in Figure 5.
  • [Editorial inference] Because Zyda-2 is open, the dataset-vs-architecture question can be settled by outsiders, which is unusual for leading small models and converts part of the paper's claim into an experiment anyone can run.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This technical report introduces the Zamba2 suite of hybrid Mamba2-transformer language models at 1.2B, 2.7B, and 7.4B parameters, trained for up to 3T tokens on the newly released Zyda-2 dataset. The paper claims state-of-the-art quality at each scale among open-weights models, substantial inference efficiency gains from the hybrid architecture, and strong instruct-tuned variants. It also releases model weights and the pretraining dataset. The central architecture is a Mamba2 backbone with shared attention blocks, LoRAs, RoPE, and an annealing phase with a 60/40 replay mix, with some model-specific heterogeneity.

Significance. If the central claims hold, this is a significant contribution to small-model and SSM-hybrid research. The release of open weights, a 5T-token dataset (Zyda-2), and detailed inference benchmarks would enable reproducibility and further study of hybrid architectures. The paper also provides a useful comparison of latency and memory efficiency, which is often missing from architecture papers. However, the strength of the quality claims is currently limited by the evaluation methodology and the absence of supporting ablation data, both of which are addressable.

major comments (4)
  1. [Section V.A, Table I] The central claim that Zamba2 models achieve state-of-the-art performance relies on comparing scores from heterogeneous public evaluations without a shared harness. The paper does not state which evaluation library, prompt templates, few-shot exemplar formats, sampling parameters, or answer extraction methods produced either the Zamba2 or the baseline numbers. Since prompt formatting alone can shift MMLU by several points, and several margins are small (e.g., Zamba2-7B MMLU 67.2 vs Llama3.2-8B 65.18; Zamba2-2.7B HellaSwag 76.35 vs Llama3.2-3B 73.6), a unified re-evaluation of all models under identical conditions is needed to substantiate the 'state of the art' and 'leading performance' claims.
  2. [Section II] The architecture section claims that 'rigorous ablations' led to the choices of Mamba2 over Mamba1, two alternating shared attention blocks, LoRAs on shared blocks, and RoPE, but no ablation results, parameter-matched baselines, or FLOP-matched comparisons are provided. Without these data, the attribution of performance gains to specific architectural components is unsupported. This is load-bearing because the architecture is the primary claimed contribution.
  3. [Section V.A, Fig. 5] The 'performance per training token' argument in Fig. 5 is used to conclude that 'Zamba2 outperforms due to its architecture,' but the comparison across models varies in dataset, tokenizer, training compute, and annealing protocol simultaneously. The paper's reasoning that because Zyda-2 is open, 'it is unlikely that our dataset alone gives us a strong advantage' is not a substitute for controlled experiments (e.g., training the same architecture on different datasets, or different architectures on Zyda-2). This conflation undermines the architectural attribution.
  4. [Section VI, Table II] The instruct-model comparison in Table II has the same methodology gap as Table I: no details are given for how MT-Bench and IFEval were run for Zamba2 or for the baselines, and the claim of being 'strongly competitive' is not quantified against evaluation variance. Since the abstract includes instruct variants as a contribution, this gap needs to be addressed as well.
minor comments (7)
  1. [Section I, Fig. 1] Fig. 1 is not referenced in the text; please add a reference in Section V.A or elsewhere where the quality-versus-latency trade-off is discussed.
  2. [Section III] The citation 'Adam optimizer (Kingma, 2014)' appears to refer to Kingma and Ba's Adam paper, which should be cited as (Kingma & Ba, 2015) with the correct year and author list.
  3. [Section IV, Fig. 4] The caption of Fig. 4 says 'Reported is the average score on a set of standard language modelling evaluation tasks,' but the figure appears to show individual bars; clarify the exact tasks and whether error bars or multiple runs are included.
  4. [Section V.A] The phrase 'standard language model evaluation metrics' is vague; specify the exact task versions, prompt templates, and the evaluation library used (e.g., lm-evaluation-harness, OpenCompass) for the Zamba2 numbers.
  5. [Section VI.A] The claim that 'Mamba2 blocks have approximately 4× the throughput of standard transformer blocks' is presented without a supporting benchmark; please provide a reference or a measurement in the paper.
  6. [Section VIII] The statement 'In many experiments we have observed...' refers to unpublished experiments; either move these observations to an appendix with full details or remove them, as they cannot be verified from the manuscript.
  7. [References] Several references are incomplete or have inconsistent formatting, e.g., 'Jamba Team. Jamba-1.5' lacks author names, and the Zyda-2 blog entry has an unusual author string; please ensure all entries are complete and consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the report is an empirical model release and its central claims rest on external evaluations rather than on self-referential derivation.

full rationale

The paper does not present a derivation chain in which a predicted quantity is equivalent to a fitted input. Its central claims (Section V.A, Table I) are empirical benchmark scores measured on standard external evaluations; the architecture ablations in Section II are described as empirical experiments. Self-citations to Zamba1, Zyda-1/Zyda-2, and the Zyphra cookbook are used as background and recipe references, not to prove the SOTA claim. The Section V inference that Zamba2's performance is 'largely due to the Zamba2 architecture' is an interpretation of Fig. 5 based on the openness of Zyda-2, not a mathematical reduction; its weakness is external validity (no controlled dataset swap), which is a correctness risk rather than circularity. Concerns about the absence of a shared evaluation harness for baseline scores in Table I are also external validity issues, not circular construction. Therefore no circular step can be exhibited under the stated standard.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The report is engineering-focused. The central predictions are benchmark numbers, not derived quantities. The hand-chosen hyperparameters above affect training and context extension, and the comparative claims rest on external evaluations and prior-work performance assertions. No new physical or mathematical entities are introduced.

free parameters (3)
  • NTK-aware RoPE scaling factor s = 16
    Chosen by hand to extend Zamba2-7B's context to 17000 tokens (Section VI, Eq. 1); no tuning procedure or ablation for this value is reported.
  • Annealing replay ratio (fraction of phase-1 data) = 60%
    Section III states a 60% replay of phase-1 data during annealing to mitigate catastrophic forgetting; no sensitivity study is given.
  • Annealing dataset budget = 100B tokens (50B dataset, 2 epochs)
    Section III states annealing ran for two epochs over a 50B-token mixture; duration is a design choice not justified by ablations in this report.
assumptions (3)
  • domain assumption Mamba2 blocks have roughly 4x the throughput of standard transformer blocks at equivalent quality.
    Used in Section V.B to explain why the hybrid can afford shared attention layers and still beat transformers on latency; the factor is imported from Dao & Gu (2024).
  • domain assumption Baseline scores in Table I are comparable despite heterogeneous public sources.
    Section V, Table I lists competitor scores without a unified evaluation harness or prompt specification; the SOTA comparison depends on this assumption.
  • ad hoc to paper The 'fairly clear sigmoidal curve' in Fig 5 is a reliable representation of the token-to-performance trend.
    Section V.A describes this trend to argue Zamba2 is an outlier; no curve fitting, error bars, or confidence intervals are shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Zamba2 Suite: Technical Report." pith.science (2026). https://pith.science/paper/MD6LH35V

@misc{pith2026241115242,
  author       = {Pith},
  title        = {Pith review of: The Zamba2 Suite: Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MD6LH35V}},
  note         = {Machine review of arXiv:2411.15242}
}
read the original abstract

In this technical report, we present the Zamba2 series -- a suite of 1.2B, 2.7B, and 7.4B parameter hybrid Mamba2-transformer models that achieve state of the art performance against the leading open-weights models of their class, while achieving substantial gains in inference latency, throughput, and memory efficiency. The Zamba2 series builds upon our initial work with Zamba1-7B, optimizing its architecture, training and annealing datasets, and training for up to three trillion tokens. We provide open-source weights for all models of the Zamba2 series as well as instruction-tuned variants that are strongly competitive against comparable instruct-tuned models of their class. We additionally open-source the pretraining dataset, which we call Zyda-2, used to train the Zamba2 series of models. The models and datasets used in this work are openly available at https://huggingface.co/Zyphra

Figures

Figures reproduced from arXiv: 2411.15242 by the authors.

Figure 1
Figure 1. Performance (MMLU 5-shot or 0-shot) vs time-to-first-token for the Zamba2 series models vs leading competing models. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Architecture diagrams for the 1.2B, 2.7B and 7.4B models. The 1.2B architecture differs in also including LoRAs on [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Pipeline for producing the Zyda-2 dataset. Zyda-2 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Performance (in 5-shot MMLU) vs the number of [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 4
Figure 4. Figure 4: The performance of Zyda-2 vs other leading language [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: Performance characteristics of the Zamba2-7B model vs competing models: time-to-first-token [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Performance characteristics of the Zamba2-2.7B model vs competing models: time-to-first-token [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Performance characteristics of the Zamba2-1.2B model vs competing models: time-to-first-token [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: MT Bench scores per subject of Zamba2 Instruct and [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Zamba2-2.7B’s performance on passkey retrieval. [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus

    cs.CL 2026-08 conditional novelty 7.0 of 10

    Massive activations in hybrid linear attention LLMs form pre-attention spikes before full attention layers and inter-spike plateaus that grow with full attention density.

  2. The Capability Convergence Hypothesis: Capability from Access Structure, Not Scale

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Long-context capability converges to hybrid architectures pairing O(1) state with a scalable index; pure scaling of SSMs or Transformers alone hits information, horizon, or circuit walls.

  3. SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot

    cs.LG 2025-06 conditional novelty 6.0 of 10

    SparseSSM extends OBS-style second-order pruning to Mamba's discretized, time-shared state-transition matrix, pruning 50% of its weights in one pass without fine-tuning.

  4. LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement

    cs.CL 2025-04 conditional novelty 6.0 of 10

    LongMamba identifies global channels in Mamba and skips low-importance tokens in them at inference, reducing hidden-state decay and improving long-context performance without training.

  5. ZUNA1.1: A more flexible EEG foundation model for Denoising and Super-resolution

    cs.LG 2026-07 conditional novelty 5.0 of 10

    ZUNA1.1, an open-source 380M EEG diffusion autoencoder, reconstructs variable-length, flexibly masked EEG at least as well as its predecessor and far better than spherical spline interpolation.

  6. Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Models

    cs.CL 2024-12 conditional novelty 5.0 of 10

    SE-Attn and HyLoRA fine-tune hybrid SSMs on sequences up to 8x the pre-training length, approaching full-attention performance at lower cost.

Reference graph

Works this paper leans on

60 extracted references · 11 canonical work pages · cited by 6 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    The case for co-designing model architectures with hardware, 2024 a

    Quentin Anthony, Jacob Hatef, Deepak Narayanan, Stella Biderman, Stas Bekman, Junqi Yin, Aamir Shafi, Hari Subramoni, and Dhabaleswar Panda. The case for co-designing model architectures with hardware, 2024 a . URL https://arxiv.org/abs/2401.14489

    arXiv 2024

  3. [3]

    The Zyphra Cookbook

    Quentin Anthony, Beren Millidge, Paolo Glorioso, and Yury Tokpanov. The Zyphra Cookbook . GitHub Repo, 2024 b . URL https://github.com/Zyphra/zcookbook

    work page 2024

  4. [4]

    Blackmamba: Mixture of experts for state-space models

    Quentin Anthony, Yury Tokpanov, Paolo Glorioso, and Beren Millidge. Blackmamba: Mixture of experts for state-space models. arXiv preprint arXiv:2402.01771, 2024 c

    arXiv 2024

  5. [5]

    Infinity instruct

    BAAI. Infinity instruct. https://huggingface.co/datasets/BAAI/Infinity-Instruct, 2024 a

    work page 2024

  6. [6]

    Orca dpo pairs

    BAAI. Orca dpo pairs. https://huggingface.co/datasets/Intel/orca\_dpo\_pairs, 2024 b

    work page 2024

  7. [7]

    Does your data spark joy? performance gains from domain upsampling at the end of training

    Cody Blakeney, Mansheej Paul, Brett W Larsen, Sean Owen, and Jonathan Frankle. Does your data spark joy? performance gains from domain upsampling at the end of training. arXiv preprint arXiv:2406.03476, 2024

    arXiv 2024

  8. [8]

    NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation

    bloc97. NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. , 2023. URL https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/

    2023

Show all 60 references
  1. [9]

    Empirical evaluation of gated recurrent neural networks on sequence modeling, 2014

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling, 2014. URL https://arxiv.org/abs/1412.3555

    2014 arXiv

  2. [10]

    Ultrafeedback: Boosting language models with scaled ai feedback, 2024

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback, 2024. URL https://arxiv.org/abs/2310.01377

    2024 arXiv

  3. [11]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. URL https://arxiv.org/abs/2405.21060

    2024 arXiv

  4. [12]

    Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022. URL https://arxiv.org/abs/2208.07339

    2022 arXiv

  5. [13]

    Qlora: Efficient finetuning of quantized llms, 2023

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. URL https://arxiv.org/abs/2305.14314

    2023 arXiv

  6. [14]

    Enhancing chat language models by scaling high-quality instructional conversations, 2023

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023. URL https://arxiv.org/abs/2305.14233

    2023 arXiv

  7. [15]

    Gemma 2: Improving open language models at a practical size, 2024 a

    Gemma Team . Gemma 2: Improving open language models at a practical size, 2024 a . URL https://arxiv.org/abs/2408.00118

    2024 arXiv

  8. [16]

    Gemma: Open models based on gemini research and technology

    Gemma Team . Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024 b

    2024 arXiv

  9. [17]

    Zamba: A Compact 7B SSM Hybrid Model , 2024

    Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A Compact 7B SSM Hybrid Model , 2024. URL https://arxiv.org/abs/2405.16712

    2024 arXiv

  10. [18]

    Is mamba capable of in-context learning?, 2024

    Riccardo Grazzi, Julien Siems, Simon Schrodi, Thomas Brox, and Frank Hutter. Is mamba capable of in-context learning?, 2024

    2024

  11. [19]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  12. [20]

    Deep learning scaling is predictable, empirically

    Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017

    2017 arXiv

  13. [21]

    Long short-term memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9 0 (8): 0 1735--1780, 1997

    1997

  14. [22]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022

  15. [23]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  16. [24]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024

    2024 arXiv

  17. [25]

    Open Hermes Preferences

    Shengyi Costa Huang, Agustín Piqueres, Kashif Rasul, Philipp Schmid, Daniel Vila, and Lewis Tunstall. Open Hermes Preferences . https://huggingface.co/datasets/argilla/OpenHermesPreferences, 2024

    2024

  18. [26]

    Simple and scalable strategies to continually pre-train large language models

    Adam Ibrahim, Benjamin Th \'e rien, Kshitij Gupta, Mats L Richter, Quentin Anthony, Timoth \'e e Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. arXiv preprint arXiv:2403.08763, 2024

    2024 arXiv

  19. [27]

    Jamba-1.5: Hybrid transformer-mamba models at scale, 2024

    Jamba Team . Jamba-1.5: Hybrid transformer-mamba models at scale, 2024. URL https://arxiv.org/abs/2408.12570

    2024 arXiv

  20. [28]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  21. [29]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. URL https://arxiv.org/abs/2001.08361

    2001 arXiv

  22. [30]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp.\ 5156--5165. PMLR, 2020

    2020

  23. [31]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  24. [32]

    Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan ...

    2024 arXiv

  25. [33]

    Starcoder: may the source be with you!, 2023 a

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, ...

    2023 arXiv

  26. [34]

    Textbooks are all you need ii: phi-1.5 technical report

    Yuanzhi Li, S \'e bastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023 b

    2023 arXiv

  27. [35]

    Openorca: An open dataset of gpt augmented flan reasoning traces

    Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and ``Teknium''. Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orca/OpenOrca, 2023

    2023

  28. [36]

    Jamba: A hybrid transformer-mamba language model, 2024

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Ere...

    2024

  29. [37]

    The llama 3 herd of models, 2024

    Llama3 Team . The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  30. [38]

    Peft: State-of-the-art parameter-efficient fine-tuning methods

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft, 2022

    2022

  31. [39]

    Introducing Meta Llama 3: The most capable openly available LLM to date

    Meta. Introducing Meta Llama 3: The most capable openly available LLM to date . https://ai.meta.com/blog/meta-llama-3/, 2024. Accessed:

    2024

  32. [40]

    Can mamba learn how to learn? a comparative study on in-context learning tasks

    Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? a comparative study on in-context learning tasks. arXiv preprint arXiv:2402.04248, 2024 a

    2024 arXiv

  33. [41]

    Can mamba learn how to learn? a comparative study on in-context learning tasks, 2024 b

    Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? a comparative study on in-context learning tasks, 2024 b

    2024

  34. [42]

    The fineweb datasets: Decanting the web for the finest text data at scale, 2024

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arxiv.org/abs/2406.17557

    2024 arXiv

  35. [43]

    Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Kris...

    2023

  36. [44]

    au2, Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Stanisław Woźniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, and Rui-Jie Zhu

    Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartłomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr. au2, Niklas Muennigho...

    2024

  37. [45]

    YaRN: Efficient Context Window Extension of Large Language Models

    Bowen Peng , Jeffrey Quesnelle , Honglu Fan , and Enrico Shippole . YaRN: Efficient Context Window Extension of Large Language Models . arXiv e-prints, art. arXiv:2309.00071, August 2023. doi:10.48550/arXiv.2309.00071

    2023 doi arXiv

  38. [46]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. URL https://arxiv.org/abs/2305.18290

    2024 arXiv

  39. [47]

    Zero: Memory optimizations toward training trillion parameter models, 2020

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. URL https://arxiv.org/abs/1910.02054

    2020 arXiv

  40. [48]

    Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Mu...

    2024 arXiv

  41. [49]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864

    2023 arXiv

  42. [50]

    Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023

    Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URL https://huggingface.co/datasets/teknium/OpenHermes-2.5

    2023

  43. [51]

    Zyda: A 1.3T Dataset for Open Language Modeling , 2024

    Yury Tokpanov, Beren Millidge, Paolo Glorioso, Jonathan Pilault, Adam Ibrahim, James Whittington, and Quentin Anthony. Zyda: A 1.3T Dataset for Open Language Modeling , 2024

    2024

  44. [52]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  45. [53]

    An empirical study of mamba-based language models

    Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba-based language models. arXiv preprint arXiv:2406.07887, 2024

    2024 arXiv

  46. [54]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464, 2024

    2024 arXiv

  47. [55]

    Gated linear attention transformers with hardware-efficient training, 2024

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training, 2024. URL https://arxiv.org/abs/2312.06635

    2024 arXiv

  48. [56]

    Building Zyda-2 , a 5 Trillion Token High-Quality Dataset , with NVIDIA NeMo Curator , October 2024

    Paolo Glorioso Yury Tokpanov, Ayush Dattagupta, Vibhu Jawa, Ryan Wolf, Vikranth Jeyakumar, Quentin Anthony Arham Mehta, and Beren Millidge. Building Zyda-2 , a 5 Trillion Token High-Quality Dataset , with NVIDIA NeMo Curator , October 2024. URL https://www.zyphra.com/post/buil...

    2024

  49. [57]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/abs/2306.05685

    2023 arXiv

  50. [58]

    Instruction-following evaluation for large language models, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/2311.07911

    2023 arXiv

  51. [59]

    Falcon mamba: The first competitive attention-free 7b language model, 2024

    Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, and Hakim Hacid. Falcon mamba: The first competitive attention-free 7b language model, 2024. URL https://arxiv.org/abs/2410.05355

    2024 arXiv

  52. [60]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.