{"id":"d996d932-6907-484f-b532-29cbbd0a9af5","arxiv_id":"2411.15242","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"This paper introduces Zamba2, a suite of 1.2B, 2.7B, and 7.4B hybrid Mamba2-transformer models that claims state-of-the-art small-model quality and 30-50% lower time-to-first-token, with open weights and a 5T-token pretraining dataset.","lead":"Zamba2 is a family of small language models that mix Mamba2 state-space layers with a few shared transformer attention blocks. The report shows these models match or beat comparable transformer models on standard benchmarks while generating text faster and using less memory.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I compares Zamba2 against baseline scores from heterogeneous public evaluations with no shared harness, so the 'state of the art' ranking is not yet established; a unified re-evaluation is required.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern: the SOTA claim in the abstract and Section V.A depends on Table I, where baseline numbers come from heterogeneous public sources and no shared evaluation methodology is reported. Without a unified harness, even plausible margins could invert, and the paper provides no way to rule this out. This is not an internal inconsistency; it is an external-validity gap that the released weights and dataset make directly testable. The same issue affects Fig 5's architecture-attribution argument, because comparing training tokens versus MMLU across models with different datasets and annealing protocols cannot separate architecture from data. The conditional verdict is appropriate: the claim is plausible and valuable, but it should be re-verified under controlled conditions before being taken as established. My read therefore does not change the reader's verdict.","tokens_in":13686,"tokens_out":4199,"duration_ms":87439,"concrete_test":"Download the released Zamba2 weights and the open baseline checkpoints (Llama3.2-1.2B/3B/8B, Gemma2-2B/2.6B, SmolLM-1.7B, StableLM-1.6B/3B, Mistral-7B, Gemma-7B, FalconMamba-7B, etc.) and evaluate all of them in a single framework, e.g., lm-evaluation-harness, using identical prompt templates, the same few-shot exemplars and order, greedy decoding, and identical answer normalization for the eight Table I tasks. Compute standard errors per task. If Zamba2 does not hold the maximum score on a majority of tasks in each scale band, the SOTA claim should be weakened to 'competitive on selected tasks'. Re-run the same protocol for MT-Bench and IFEval in Table II.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V.A and Table I are the sole support for the abstract's SOTA claim, yet the paper never states which evaluation harness, prompt templates, few-shot exemplars, sampling parameters, or answer extraction were used, for either Zamba2 or the listed baselines. The competitor scores appear to be copied from assorted public sources with varying methodologies; prompt formatting alone can shift MMLU by multiple points. Some per-scale margins are small (e.g., Zamba2-2.7B HellaSwag 76.35 vs Llama3.2-3B 73.6; Zamba2-7B MMLU 67.2 vs Llama3.2-8B 65.18), and a single standardized run could alter several maximum scores. Thus the central empirical claim is vulnerable to evaluation mismatch rather than to any evident internal contradiction. The same uncontrolled comparison underpins Fig 5's 'performance per training token' argument, which also conflates dataset quality with architecture. This is an addressable external-validity gap, not evidence of fraud; the released weights and dataset make a decisive check feasible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This technical report introduces the Zamba2 suite of hybrid Mamba2-transformer language models at 1.2B, 2.7B, and 7.4B parameters, trained for up to 3T tokens on the newly released Zyda-2 dataset. The paper claims state-of-the-art quality at each scale among open-weights models, substantial inference efficiency gains from the hybrid architecture, and strong instruct-tuned variants. It also releases model weights and the pretraining dataset. The central architecture is a Mamba2 backbone with shared attention blocks, LoRAs, RoPE, and an annealing phase with a 60/40 replay mix, with some model-specific heterogeneity.","tokens_in":13920,"tokens_out":4611,"duration_ms":44796,"significance":"If the central claims hold, this is a significant contribution to small-model and SSM-hybrid research. The release of open weights, a 5T-token dataset (Zyda-2), and detailed inference benchmarks would enable reproducibility and further study of hybrid architectures. The paper also provides a useful comparison of latency and memory efficiency, which is often missing from architecture papers. However, the strength of the quality claims is currently limited by the evaluation methodology and the absence of supporting ablation data, both of which are addressable.","major_comments":[{"comment":"The central claim that Zamba2 models achieve state-of-the-art performance relies on comparing scores from heterogeneous public evaluations without a shared harness. The paper does not state which evaluation library, prompt templates, few-shot exemplar formats, sampling parameters, or answer extraction methods produced either the Zamba2 or the baseline numbers. Since prompt formatting alone can shift MMLU by several points, and several margins are small (e.g., Zamba2-7B MMLU 67.2 vs Llama3.2-8B 65.18; Zamba2-2.7B HellaSwag 76.35 vs Llama3.2-3B 73.6), a unified re-evaluation of all models under identical conditions is needed to substantiate the 'state of the art' and 'leading performance' claims.","section":"Section V.A, Table I"},{"comment":"The architecture section claims that 'rigorous ablations' led to the choices of Mamba2 over Mamba1, two alternating shared attention blocks, LoRAs on shared blocks, and RoPE, but no ablation results, parameter-matched baselines, or FLOP-matched comparisons are provided. Without these data, the attribution of performance gains to specific architectural components is unsupported. This is load-bearing because the architecture is the primary claimed contribution.","section":"Section II"},{"comment":"The 'performance per training token' argument in Fig. 5 is used to conclude that 'Zamba2 outperforms due to its architecture,' but the comparison across models varies in dataset, tokenizer, training compute, and annealing protocol simultaneously. The paper's reasoning that because Zyda-2 is open, 'it is unlikely that our dataset alone gives us a strong advantage' is not a substitute for controlled experiments (e.g., training the same architecture on different datasets, or different architectures on Zyda-2). This conflation undermines the architectural attribution.","section":"Section V.A, Fig. 5"},{"comment":"The instruct-model comparison in Table II has the same methodology gap as Table I: no details are given for how MT-Bench and IFEval were run for Zamba2 or for the baselines, and the claim of being 'strongly competitive' is not quantified against evaluation variance. Since the abstract includes instruct variants as a contribution, this gap needs to be addressed as well.","section":"Section VI, Table II"}],"minor_comments":[{"comment":"Fig. 1 is not referenced in the text; please add a reference in Section V.A or elsewhere where the quality-versus-latency trade-off is discussed.","section":"Section I, Fig. 1"},{"comment":"The citation 'Adam optimizer (Kingma, 2014)' appears to refer to Kingma and Ba's Adam paper, which should be cited as (Kingma & Ba, 2015) with the correct year and author list.","section":"Section III"},{"comment":"The caption of Fig. 4 says 'Reported is the average score on a set of standard language modelling evaluation tasks,' but the figure appears to show individual bars; clarify the exact tasks and whether error bars or multiple runs are included.","section":"Section IV, Fig. 4"},{"comment":"The phrase 'standard language model evaluation metrics' is vague; specify the exact task versions, prompt templates, and the evaluation library used (e.g., lm-evaluation-harness, OpenCompass) for the Zamba2 numbers.","section":"Section V.A"},{"comment":"The claim that 'Mamba2 blocks have approximately 4× the throughput of standard transformer blocks' is presented without a supporting benchmark; please provide a reference or a measurement in the paper.","section":"Section VI.A"},{"comment":"The statement 'In many experiments we have observed...' refers to unpublished experiments; either move these observations to an appendix with full details or remove them, as they cannot be verified from the manuscript.","section":"Section VIII"},{"comment":"Several references are incomplete or have inconsistent formatting, e.g., 'Jamba Team. Jamba-1.5' lacks author names, and the Zyda-2 blog entry has an unusual author string; please ensure all entries are complete and consistent.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a technical report with a strong reproducibility angle (open weights, open dataset), but the empirical claims are currently under-supported. The evaluation methodology gap in Table I is the most serious issue and would require a unified re-evaluation or a clear statement of why the heterogeneous scores are directly comparable. The architecture ablations are also essential for an architecture paper and should be included. These are addressable within the scope of a revision. The reliance on self-cited resources (e.g., the Zyphra Cookbook) for critical training details should be reduced by adding the necessary specifics to the text or appendix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zamba2 is worth knowing about for two reasons: it ships open weights for three hybrid Mamba2/transformer models that look competitive at their scales, and it ships Zyda-2, a 5T-token pretraining dataset. The architecture work is incremental over Zamba1 — Mamba2 backbone, two alternating shared attention blocks, LoRAs on the shared blocks, RoPE in attention — but the ablations are described rather than shown, so the design story is credible but not fully evidenced. What is genuinely new is the combination of competitive checkpoints, an open dataset, and a clean report of inference latency, throughput, and KV-cache advantages.\n\nThe main soft spot is exactly what the stress-test note says: Table I is the only support for the \"state of the art\" claim, and the baseline numbers appear to be pulled from heterogeneous public evaluations. The paper doesn't say which harness, prompt templates, few-shot formatting, or post-processing were used for either Zamba2 or the competitors. Prompt format alone can move MMLU by several points, and some of the margins are thin (e.g., Zamba2-7B MMLU 67.2 vs Llama3.2-8B 65.18). That doesn't mean the models are bad, but the specific SOTA claim is not yet established by this report. The same issue affects Fig 5, which uses these same numbers to argue for superior performance-per-training-token, and the claim that because the dataset is open the advantage must be architectural doesn't hold up — open doesn't mean others have trained their models on it.\n\nThe other gaps are minor by comparison: no error bars or seed variance, ablation results summarized without tables, and the three models are not architecturally identical (the 1.2B has a single shared block, the 2.7B lacks RoPE), all of which they disclose honestly.\n\nWho is this for? Practitioners building small on-device models, and researchers working on hybrid SSM/transformer designs or open pretraining datasets. It deserves a serious referee because the artifacts are real and the central claim is checkable — someone can re-evaluate the released weights in a unified harness tomorrow. My recommendation: engage with it, but require a unified evaluation of baselines for the headline comparisons before calling it SOTA.","headline":"Zamba2 is a valuable open release of competitive hybrid SSM/transformer models and a 5T-token pretraining dataset, but the 'state of the art' claim rests on an uncontrolled cross-source evaluation comparison.","tokens_in":14463,"tokens_out":2609,"would_cite":true,"duration_ms":25748,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zamba2, a family of 1.2B-7.4B hybrid Mamba2-transformer models, achieves top-of-class accuracy among open models while cutting KV-cache memory by roughly sixfold.","keywords":["Mamba2","hybrid state-space models","shared attention","small language models","inference efficiency","KV cache reduction","Zyda-2 dataset","open-weights models"],"falsifier":"Re-run every benchmark in Table I for Zamba2 and each listed competitor in a single evaluation harness using identical prompts, shots, and sampling; if any Zamba2 model no longer leads its weight class on the majority of these tasks, the paper's central quality claim is refuted. A narrower check: verify the 17k-token context extension for Zamba2-7B with a passkey-retrieval probe, since RoPE scaling without finetuning fails for many models.","tokens_in":13493,"feed_emoji":"⚡","tokens_out":7936,"duration_ms":67090,"temperature":0.7,"pith_summary":"This report introduces Zamba2, a suite of 1.2B, 2.7B, and 7.4B parameter language models that combine a Mamba2 state-space backbone with a small number of shared transformer attention blocks. The authors' central claim is that, at each scale, these hybrids achieve state-of-the-art performance against leading open-weight transformer models on standard benchmarks such as MMLU and ARC, while also delivering 30–50% lower time-to-first-token and about a 6× reduction in KV-cache memory. The claim matters because it suggests that the efficiency advantages of recurrent state-space mixing can be obtained without sacrificing the quality that pure attention delivers, opening a more efficient route for small, deployable, openly available models. The report accompanies the claim by releasing model weights and the 5-trillion-token Zyda-2 pretraining dataset, so both the quality and the efficiency comparisons are checkable by others.","feed_headline":"Hybrid Mamba2 models outscore transformer rivals at small scale","feed_subtitle":"Attention is sparse and shared, so the same benchmarks come with roughly 6× less KV-cache memory.","key_machinery":"The machinery is the Zamba2 hybrid layer stack: a Mamba2 backbone (a linear-time recurrent state-space layer with a fixed-size hidden state) interleaved with one or two globally shared transformer attention blocks, where non-shared low-rank adapters (LoRAs) let each attention invocation use different weights at small parameter cost and rotary position embeddings (RoPE) are applied in attention. The argument runs on the ratio: with only one shared attention block per several Mamba2 layers, the model stores a KV cache only for those few layers (a 1:6 ratio in the released models), which is what turns the 6× KV-cache reduction and the latency and throughput gains into concrete, measurable numbers.","core_discovery":"The paper's core discovery, stated on its own terms, is that a hybrid in which most sequence mixing is done by Mamba2 blocks and only a few alternating shared attention blocks provide full attention outperforms parameter-matched and FLOP-matched transformer baselines in head-to-head ablations, and that the released models then outperform the leading open models of their weight classes on the standard evaluation suite. The reported numbers (MMLU 43.1 at 1.2B scale versus 32.8 for Gemma2-2B; 55.97 versus 54.01 at 2.7B; 67.2 versus 65.18 at 7B) are the concrete form of the claim, and the efficiency figures (up to 4× Mamba2 throughput over a transformer block, a 1:6 attention-to-Mamba ratio cutting KV cache by 6×, and a 30–50% time-to-first-token reduction) are its practical consequence.","pith_inferences":["[Editorial inference] If the quality ranking survives re-evaluation in one shared harness, the practical import is that architecture choice, not just data quality, is a first-order lever for small-model deployment: hybrid SSM designs may become the default for on-device and long-context serving.","[Editorial inference] The shared-attention-plus-LoRA pattern suggests a general recipe for spending a parameter budget: buy full attention sparingly and differentiate its copies cheaply, a design that could transfer to mixture-of-experts or other recurrent layers.","[Editorial inference] The authors attribute most of their edge to architecture rather than data; a testable consequence of that attribution is that training a transformer with the same Zyda-2 tokens and annealing schedule should land below the Zamba2 curves in Figure 5.","[Editorial inference] Because Zyda-2 is open, the dataset-vs-architecture question can be settled by outsiders, which is unusual for leading small models and converts part of the paper's claim into an experiment anyone can run."],"forward_implications":["At each of the three scales, Zamba2 tops Table I on MMLU and most other listed benchmarks against open competitors including Gemma2, Llama3.2, and Mistral-7B.","Compared with a pure transformer, generation needs about 6× less KV-cache memory at long contexts, and time-to-first-token falls by 30–50%.","The 7.4B model's context window extends to roughly 17,000 tokens via NTK RoPE scaling without additional training, and the 2.7B model reaches 65,536-token passkey retrieval after curriculum finetuning.","Instruction-tuned variants are strongly competitive with official instruct finetunes on MT-Bench and IFEval using only open SFT/DPO data.","The open release of Zyda-2 and all weights means the reported quality and efficiency comparisons can be reproduced by the community."],"supporting_citations":[{"why":"Supplies Mamba2, the structured state-space backbone whose linear time and memory behavior drives the efficiency claims.","marker":"(Dao & Gu, 2024)"},{"why":"Zamba1, the prior model family that introduced the shared-attention design this work optimizes and exceeds.","marker":"(Glorioso et al., 2024)"},{"why":"The empirical Mamba-hybrid study that frames the claim that hybrids can match transformers at scale.","marker":"(Waleffe et al., 2024)"},{"why":"LoRA, the non-shared adapters that give each shared attention invocation distinct weights at small parameter cost.","marker":"(Hu et al., 2021)"},{"why":"RoPE, the position embedding applied in shared attention and later rescaled by NTK for context extension.","marker":"(Su et al., 2023)"},{"why":"FineWeb-Edu, a core quality-filtered component of Zyda-2 and of the annealing data.","marker":"(Penedo et al., 2024)"},{"why":"DCLM, the second quality-filtered dataset blended into Zyda-2.","marker":"(Li et al., 2024)"},{"why":"The annealing-ablation protocol used to compare datasets in Figure 4.","marker":"(Blakeney et al., 2024)"}],"fun_headline_variants":["Zamba2 hybrids beat transformer rivals with 6x less KV cache","Mamba2 attention cuts memory 6x, tops open models","Small Zamba2 models outrank bigger transformers","Hybrid Mamba2: 4x throughput, 6x less KV, top scores","Zamba2 suite: state-of-the-art at 1B to 7B with Mamba2"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The state-of-the-art claim assumes that the leaderboard scores of competitor models, drawn from their own published sources, are directly comparable to the authors' own evaluation runs, even though the evaluation harness, prompt templates, and sampling settings for Zamba2 are not specified.","fun_headline_variants_meta":{"raw":{"variants":["Zamba2 hybrids beat transformer rivals with 6x less KV cache","Mamba2 attention cuts memory 6x, tops open models","Small Zamba2 models outrank bigger transformers","Hybrid Mamba2: 4x throughput, 6x less KV, top scores","Zamba2 suite: state-of-the-art at 1B to 7B with Mamba2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1585,"prompt_tokens":907,"completion_tokens":678,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":576}},"tokens_in":523,"tokens_out":678,"duration_ms":6845,"temperature":1.0,"reasoning_tokens":576,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:00:09.910348+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run every benchmark in Table I for Zamba2 and each listed competitor in a single evaluation harness using identical prompts, shots, and sampling; if any Zamba2 model no longer leads its weight class on the majority of these tasks, the paper's central quality claim is refuted. A narrower check: verify the 17k-token context extension for Zamba2-7B with a passkey-retrieval probe, since RoPE scaling without finetuning fails for many models.","supporting_citations":[],"review_version":1}