Pith. sign in

REVIEW 5 major objections 5 minor 4 cited by

Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper reports that a 34B hybrid attention-SSM model matches or beats 70B-class rivals on many reasoning, science, and code benchmarks while using about half the parameters and far less training data.

desk verdict Serious hybrid-architecture engineering with useful ablations, but the headline 'matches 70B' claim is not yet established by the paper's own evaluation evidence. read the letter →

arxiv 2507.22448 v1 pith:F7Z7MEBY submitted 2025-07-30 cs.CL

Jingwei Zuo , Maksim Velikanov , Ilyas Chahed , Younes Belkada , Dhia Eddine Rhayem , Guillaume Kunsch , Hakim Hacid , Hamza Yous
show 19 more authors
Brahim Farhat Ibrahim Khadraoui Mugariya Farooq Giulia Campesan Ruxandra Cojocaru Yasser Djilali Shi Hu Iheb Chaabane Puneesh Khanna Mohamed El Amine Seddik Ngoc Dung Huynh Phuc Le Khac Leen AlQadi Billel Mokeddem Mohamed Chami Abdalgader Abubaker Mikhail Lubinets Kacper Piskorski Slim Frikha
This is my paper
classification cs.CL
keywords Falcon-H1hybridarchitectureMamba-2statespacemodelsparameterefficiencylargelanguagelongcontextMaximalUpdateParametrization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents Falcon-H1, a family of large language models built from a parallel mix of Transformer attention and Mamba-2 state-space blocks, released at scales from 0.5B to 34B parameters. The central claim is that this hybrid design, combined with a heavily rewritten training corpus and a customized hyperparameter-transfer recipe, makes the models roughly twice as parameter-efficient as their dense Transformer counterparts. In particular, the paper reports that Falcon-H1-34B-Instruct matches or beats Qwen3-32B, Qwen2.5-72B, and LLaMA3.3-70B on many benchmarks, that the 1.5B-Deep variant rivals current 7B-10B models, and that the 0.5B model performs like typical 7B models from 2024. The paper also reports up to a 4x advantage in prefill throughput and an 8x advantage in generation throughput over a comparable Transformer at long contexts. The value of the claim, if true, is that strong reasoning and knowledge capability can be delivered with substantially less compute, memory, and data.

What carries the argument

The load-bearing object is the Falcon-H1 hybrid block: a residual block containing a parallel attention branch and a Mamba-2 SSM branch whose outputs are concatenated before the output projection, with the number of attention and SSM heads tunable independently. Around it sit three supporting mechanisms: the semi-parallel SA_M arrangement found optimal in channel-allocation sweeps; the unusually high RoPE base frequency b=1e11, which makes later sequence extension trivial because many frequency dimensions were never assigned during training; and the training recipe built on effective learning rate and effective weight decay, defined as sqrt(eta*lambda) and sqrt(lambda/eta), together with 35 tuned Maximal Update Parametrization forward multipliers that let all model sizes share a single learning rate and weight decay.

What would settle it

Run the same benchmark suite on the released Falcon-H1-34B-Instruct and Qwen3-32B with Qwen3 thinking mode enabled, all answers generated rather than logprob-scored under identical sampling; if Qwen3 wins on a majority of GSM8K, MATH-500, GPQA, MMLU-Pro, HumanEval, and HELMET-RAG, the headline claim of matching or outperforming 70B rivals fails under the paper's own protocol.

Watch

Extended reading notes

Core claim

The central discovery is that a parallel hybrid block, where attention and SSM run side-by-side and their outputs are concatenated, allows the attention fraction to be cut to about one-eighth of the mixer channels without hurting loss, while SSM channels do most of the sequence mixing. Systematic ablations at 300M to 1.5B scale led the authors to a semi-parallel block arrangement, a very high RoPE base frequency of about 1e11 that leaves many frequency dimensions effectively unassigned during training, and a training recipe with aggressive reuse of rewritten high-quality data. The paper argues these choices, rather than raw scale, explain why a 34B model can compete with 70B models and a 1.5B-deep model can compete with 7B-10B models.

Load-bearing premise

The load-bearing premise is that the standardized evaluation protocol, which disables thinking mode for Qwen3 and mixes logprobs and generation scoring, ranks the models the same way their creators' intended setups would.

Editorial extensions

If this is right

  • At the 34B scale, applications that currently require 70B-class models could run with roughly half the memory and up to 8x higher generation throughput at long context.
  • Edge deployments could use the 1.5B-Deep or 0.5B checkpoints for reasoning-heavy tasks at quality levels previously associated with 7B models from a year earlier.
  • If the channel-allocation result holds generally, future hybrid architectures can keep attention at a small fraction of channels and spend most capacity on SSM and MLP, reducing inference cost without sacrificing quality.
  • The training recipe, with anti-curriculum data mixing, heavy rewriting of web data, and an effective power scheduler, offers a path to reproducing strong small models on much smaller token budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implied next experiment is a pure-Transformer training run at RoPE base b=1e11 to see whether the high-base-frequency benefit is specific to hybrids or generalizes; the paper explicitly leaves this open.
  • The channel-allocation result suggests attention may act mainly as a precision correction over long-range content while SSM captures most local mixing; inspecting the learned attention patterns of Falcon-H1 could test that interpretation.
  • The reported memorization-window analysis implies that repeated epochs on high-quality tokens carry little risk at scale; a direct test would be measuring duplicate-token contamination on downstream benchmarks after multi-epoch training.
  • Comparisons with Qwen3 relied on disabling thinking mode, so the paper's efficiency claims would be sharpened by an evaluation where all models are allowed chain-of-thought generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents Falcon-H1, a family of hybrid attention-SSM language models at scales from 0.5B to 34B, with base and instruct variants. It describes a parallel hybrid architecture with tunable channel allocation, a series of proxy-model ablations for SSM hyperparameters, a custom Maximal Update Parametrization recipe with 35 tunable multipliers, a data strategy centered on rewritten and synthetic high-quality data, and training/inference infrastructure including Mixer Parallelism. The central empirical claims are that Falcon-H1-34B-Instruct rivals or outperforms open models up to 70B scale (Qwen3-32B, Qwen2.5-72B, Llama3.3-70B) at roughly half the parameter count, and that smaller variants (1.5B-Deep, 0.5B) match models several times their size. The evaluation is based on standard open benchmarks, with all numbers reported as point estimates.

Significance. If the headline claims are robust, this is a valuable contribution: it demonstrates that a hybrid architecture with a carefully tuned attention/SSM channel ratio and data strategy can achieve parameter- and compute-efficiency at the 34B scale, and it provides a rare systematic ablation of SSM design choices, effective learning-rate/weight-decay scaling, and training stability. The public release of over 30 checkpoints and the detailed reporting of hyperparameters, data mixtures, and infrastructure are concrete strengths that enable independent verification. The main risks are evaluative rather than architectural: the comparative protocol is not fully neutral, and several decisive margins are smaller than the expected evaluation noise.

major comments (5)
  1. [§5 Standardization and Reproducibility] The decision to disable Qwen3 thinking mode on all benchmarks is a load-bearing protocol choice for the flagship claim. Table 24 reports Falcon-H1-34B at 84.05 vs Qwen3-32B at 80.89 on MMLU and 49.66 vs 49.49 on GPQA_Diamond; these margins are within typical harness noise, and Qwen3's official scores are generally reported with thinking mode enabled. Please provide results with thinking mode enabled (or official reported scores) alongside the current numbers, and justify that disabling it does not systematically disadvantage Qwen3 on the math/science benchmarks that drive the comparison.
  2. [§5.2, Tables 19 and 24] The evaluation mixes lm-eval-harness (logprobs) with evalchemy (generation, 16 turns) and, for 'all other frameworks,' adheres to their default settings. Because different models are evaluated under different harnesses and defaults, the reported ordering can reflect implementation behavior rather than model capability. Concretely, Table 24's comparisons against Qwen3-32B, Qwen2.5-72B, and Llama3.3-70B are the sole evidence for the central 'rivals 70B' claim; please either run all models under a single standardized harness with identical prompts and decoding settings, or show that the framework-specific settings reproduce official published scores for the comparators.
  3. [§5, Tables 14-24] All results are point estimates with no error bars, repeated runs, or confidence intervals. Several comparisons that support the central claims are below 1-2 points (e.g., Table 24: MMLU 84.05 vs 84.42 for Qwen2.5-72B; GPQA_Diamond 49.66 vs 49.49 for Qwen3-32B). For a claim of 'rivals or outperforms,' report at least three repeated runs with standard deviations, or restrict the claim to 'comparable within noise' for those benchmarks.
  4. [Abstract and §1] The claim that Falcon-H1-0.5B 'performs comparably to typical 7B models from 2024' is not supported by the presented evidence: Table 14 compares the 0.5B model only against models up to 1.6B parameters. Please either add a comparison against representative 2024 7B models (e.g., Llama-3-8B, Mistral-7B, Qwen2-7B) on the same benchmarks, or temper the claim to what the table actually shows.
  5. [Abstract and §5.3] The abstract's 'trained on a fraction of the data' assertion is not quantified for the comparison models. Qwen2.5-72B, for example, is reported by its authors as trained on 18T tokens, which is the same order as Falcon-H1-34B's 18T budget. Please provide a table of training token counts for all models in Table 24 and state the comparison explicitly, or the data-efficiency claim will be hard to verify.
minor comments (5)
  1. [§3.3.1] Equation numbering restarts at (1) and (2) after equations (1)-(5) in Section 2; renumber to avoid confusion.
  2. [§5.3] The text 'As shown in Table 16' should read 'As shown in Figure 16'; the referenced item is a figure.
  3. [§2.4.1] There is a typo, 'acorss', in 'model performance acorss different model scales' in the opening paragraph.
  4. [§5.3] The efficiency comparison uses Qwen2.5-32B with the note that 'we anticipate no significant efficiency differences between Qwen2.5-32B and Qwen3-32B'; this is an unverified assumption and should be tested or removed.
  5. [Tables 13 and 19] The setting for MATH lvl5 is listed as 'math verify, logprobs, 4-shot'; clarify whether logprobs are used for answer selection or only for scoring, since this affects comparability with the evalchemy math results.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central benchmark claims rest on external evaluations and loss-based ablations, not on fitted parameters renamed as predictions.

full rationale

The paper's derivation chain is empirical rather than definitional. The central claims (e.g., Falcon-H1-34B-Instruct rivaling 70B-scale models) are supported by evaluations on external benchmark suites (MMLU, GPQA, GSM8k, HumanEval, etc.) run on released model checkpoints. The architectural choices—channel allocation, SSM hyperparameters, RoPE base, depth/width, and the 35 muP multipliers—are tuned on small proxy models using training loss or downstream validation, and the final large-model benchmark numbers are not defined as functions of those tuned values. No equation in the paper makes the 34B benchmark outcome equal to a fitted input by construction; the muP transfer from 0.5B/1.5B proxies to 34B is a scaling hypothesis, not a mathematical identity. Self-citations to Falcon-Mamba (for batch scaling and tokenizer experiments) and to Falcon3 (as a baseline) are methodological and comparative references, and they do not carry the load of the 'matches 70B' claim. The evaluation-protocol choices—disabling Qwen3 thinking mode, mixing logprobs and generation scoring across frameworks, and reporting single point estimates without error bars—are legitimate correctness and fairness concerns, but they are not circularity: they do not make the reported ordering equal to the paper's inputs by construction. Overall, minor self-citations are present but not load-bearing, so the circularity score is low.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a substantial set of hyperparameters tuned on proxy models and on domain assumptions about evaluation fairness and data decontamination. No new physical or conceptual entities are introduced; the 'invented' items are engineering methods rather than unobserved entities.

free parameters (5)
  • Attention channel fraction αA = 1/8 of total channels
    Selected from 21 admissible allocations on a 1.2B proxy model (Figure 2, Section 2.1). The central efficiency claim depends on keeping the attention share small.
  • RoPE base frequency b = 1e11
    Chosen from a sweep on the 0.5B model and confirmed on 7B/34B (Figure 5a, Section 2.3.1). Used to justify simple context extension to 256K.
  • 35 μP multipliers = Listed in Table 8 (e.g., memb=2^2.5, munemb=2^-5, mMLP=2^-2)
    Tuned via sensitivity analysis on a 1.5B base model and transferred to all sizes (Section 3.2.3). These 35 multipliers shape parameter norms and activations throughout the model and are tuned to proxy loss.
  • SSM state dimension dstate = 256
    Selected from an iso-parameter sweep balancing loss and throughput (Figure 3, Section 2.2). The long-context memory claims depend on this choice.
  • Data mixture percentages (e.g., rewritten data 52% to 75%) = Table 6
    Adjusted during pretraining based on 0.5B-scale validation and checkpoints (Section 3.1.2). The 34B and 7B mixtures were changed mid-training, so the final percentages are data-fitted.
assumptions (4)
  • domain assumption μP scaling rules for width transfer (Yang and Hu 2022) apply to the hybrid SSM/attention architecture
    Section 3.2.3 extends μP from Transformers to hybrid blocks; the validity of zero-shot hyperparameter transfer across model sizes is assumed without a formal proof for this architecture.
  • standard math The scalar stochastic process in Appendix B describes AdamW weight-norm dynamics well enough to define effective learning rate and effective weight decay
    The derivation of EWD and ELR relies on a toy model with i.i.d. gradient noise and stationarity, which the authors themselves call 'rough approximations' (end of Section 3.2.2).
  • domain assumption The evaluation benchmarks, prompts, and templates are a faithful measure of general capabilities and are not contaminated by the training data
    Section 5 standardizes settings but cannot guarantee decontamination of the unreleased training data; the math data decontamination is described in Section 3.1.1 but the full protocol is not given.
  • domain assumption The FIM and section-reordering long-context tasks improve long-context ability without degrading general performance
    Section 3.1.2 states these strategies were applied, but no ablation is shown isolating their effect on the final benchmark scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance." pith.science (2026). https://pith.science/paper/F7Z7MEBY

@misc{pith2026250722448,
  author       = {Pith},
  title        = {Pith review of: Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F7Z7MEBY}},
  note         = {Machine review of arXiv:2507.22448}
}
read the original abstract

In this report, we introduce Falcon-H1, a new series of large language models (LLMs) featuring hybrid architecture designs optimized for both high performance and efficiency across diverse use cases. Unlike earlier Falcon models built solely on Transformer or Mamba architectures, Falcon-H1 adopts a parallel hybrid approach that combines Transformer-based attention with State Space Models (SSMs), known for superior long-context memory and computational efficiency. We systematically revisited model design, data strategy, and training dynamics, challenging conventional practices in the field. Falcon-H1 is released in multiple configurations, including base and instruction-tuned variants at 0.5B, 1.5B, 1.5B-deep, 3B, 7B, and 34B parameters. Quantized instruction-tuned models are also available, totaling over 30 checkpoints on Hugging Face Hub. Falcon-H1 models demonstrate state-of-the-art performance and exceptional parameter and training efficiency. The flagship Falcon-H1-34B matches or outperforms models up to 70B scale, such as Qwen3-32B, Qwen2.5-72B, and Llama3.3-70B, while using fewer parameters and less data. Smaller models show similar trends: the Falcon-H1-1.5B-Deep rivals current leading 7B-10B models, and Falcon-H1-0.5B performs comparably to typical 7B models from 2024. These models excel across reasoning, mathematics, multilingual tasks, instruction following, and scientific knowledge. With support for up to 256K context tokens and 18 languages, Falcon-H1 is suitable for a wide range of applications. All models are released under a permissive open-source license, underscoring our commitment to accessible and impactful AI research.

Figures

Figures reproduced from arXiv: 2507.22448 by the authors.

Figure 1
Figure 1. Falcon-H1 architecture. Attention and SSM run in parallel within each block; their [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. (Left): The loss of fully parallel SAM hybrid block configuration for all possible (αS, αA, αM) channel allocations according to (1),(2). (Right) The loss of all 3 considered block configurations SAM (3), SA_M (4), and S_A_M(5) for fixed optimal attention allocation αA = 1 8 and varied SSM/MLP channel allocation. For the second part of the experiment, we compare all three SAM, SA_M, S_A_M block arrange￾ments while f… view at source ↗
Figure 3
Figure 3. Hyperparameter optimization landscapes for SSM number of groups and state dimension [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Model accuracy and computational efficiency across architectural dimensions. (a) Similar [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: (a) Dependence of the training loss on RoPE base frequency b. The dotted line shows the training sequence length for a reference. First, we tried many base frequencies and measured the loss early in the training at 20GT. Then, we picked 3 characteristic base frequency …
Figure 6
Figure 6. Figure 6: Model performance regarding different splitting strategies. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparison for a Chinese sentence, with or without punctuation splitting. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Performance on mathematical benchmarks for two 1B models during the training decay [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Model’s memorization window and loss trajectives [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Behavior of weight norms of embedding Wemb, unembedding Wunemb, and input/output projections Win, Wout layers of pure Mamba2 model. (Left) Evolution of weight norms during training of 1B model. In addition to no WD run, we show WD sweep (solid) and LR sweep (dashed), …
Figure 11
Figure 11. Figure 11: Effect of LR η and WD λ of the training loss curve of a 300M pure Mamba2 model. (Left) LR and WD sweeps on left and right subplots show that increasing (or decreasing) either LR or WD has a similar effect on the loss curve. (Right) Noise level measure as the loss gap …
Figure 12
Figure 12. Figure 12: Sensitivity of the loss with respect to all 35 [PITH_FULL_IMAGE:figures/full_fig_p031_12.png]
Figure 13
Figure 13. Figure 13: Rampup and warmup related observations on a 1.5B pure Mamba2 model. [PITH_FULL_IMAGE:figures/full_fig_p032_13.png]
Figure 14
Figure 14. Figure 14: Diagram illustrating the Mixer Parallelism (MP) strategies. Each row represents a full [PITH_FULL_IMAGE:figures/full_fig_p035_14.png]
Figure 15
Figure 15. Figure 15: Throughput comparison for Mixer Parallelism across different model sizes (3B and 7B). [PITH_FULL_IMAGE:figures/full_fig_p036_15.png]
Figure 16
Figure 16. Figure 16: Model efficiency comparison between Falcon-H1-34B and Qwen2.5-32B. [PITH_FULL_IMAGE:figures/full_fig_p052_16.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone

    cs.LG 2025-11 conditional novelty 6.0 of 10

    A masked diffusion language model built on a bidirectional Mamba backbone matches Transformer-based denoisers on quality while decoding with near-linear time scaling.

  2. Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Properly spaced multi-epoch reuse of high-quality data, guided by a measured memorization window, continues to improve LLM performance far beyond the common four-epoch heuristic.

  3. When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models

    cs.CL 2026-03 conditional novelty 5.0 of 10

    Perplexity-based scoring systematically understates teacher–student gaps in distilled hybrid LLMs and can reverse design-choice rankings; generation-based evaluation is the more reliable protocol.

  4. Memory for Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    The paper organizes LLM memory architectures along three axes—implicit vs explicit representation, offline vs online update, and short vs long-term persistence—plus finer-grained update rules.

Reference graph

Works this paper leans on

123 extracted references · 2 canonical work pages · cited by 4 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    https://www.statmt.org/europarl/

    E uropean P arliament P roceedings P arallel C orpus 1996-2011, R elease v7. https://www.statmt.org/europarl/

    1996

  3. [3]

    https://www.gutenberg.org/

    P roject G utenberg. https://www.gutenberg.org/

  4. [4]

    https://github.com/zeerakahmed/makhzan/

    Ma khzan D ataset. https://github.com/zeerakahmed/makhzan/

  5. [5]

    AIME problems and solutions, 2025

    AIME. AIME problems and solutions, 2025. URL https://https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions

    2025

  6. [6]

    GQA : Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA : Training generalized multi-query transformer models from multi-head checkpoints. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id=hmOwOZWzYE

    2023

  7. [7]

    M ath QA : Towards interpretable math word problem solving with operation-based formalisms

    Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. M ath QA : Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long...

    doi:10.18653/v1/n19-1245 2019

  8. [8]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

Show all 123 references
  1. [9]

    Llemma: An open language model for mathematics

    Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.10631, 2023

    arXiv

  2. [10]

    Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues

    Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, et al. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. arXiv preprint arXiv:2402.14762, 2024

    2024 arXiv

  3. [11]

    Titans: Learning to memorize at test time

    Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.00663, 2024

    2024 arXiv

  4. [12]

    Smollm-corpus, 7 2024

    Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Smollm-corpus, 7 2024. URL https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus

    2024

  5. [13]

    Scaling optimal LR across token horizons

    Johan Bjorck, Alon Benhaim, Vishrav Chaudhary, Furu Wei, and Xia Song. Scaling optimal LR across token horizons. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=WYL4eFLcxG

    2025

  6. [14]

    Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation, 2023 a

    bloc97. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation, 2023 a . URL https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_ scaled_rope_allows_llama_models_to_have/

    2023

  7. [15]

    by parts

    bloc97. Add ntk-aware interpolation "by parts" correction, 2023 b . URL https://github. com/jquesnelle/scaled-rope/pull/1

    2023

  8. [16]

    Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit, 2023

    Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit, 2023. URL https://arxiv.org/abs/2309.16620

    2023 arXiv

  9. [17]

    On the resemblance and containment of documents

    Andrei Broder. On the resemblance and containment of documents. 06 1997. doi:10.1109/SEQUEN.1997.666900

    1997

  10. [18]

    An investigation of incorporating mamba for speech enhancement

    Rong Chao, Wenze Ren, Wen-Yuan Ting, Hsin-Yi Lin, Yu Tsao, and Fan-Gang Zeng. An investigation of incorporating mamba for speech enhancement. arXiv preprint arXiv:2405.06573, 2024. URL https://arxiv.org/abs/2405.06573. Accepted to IEEE SLT 2024

    2024

  11. [19]

    Theoretical limitations of multi-layer transformer

    Lijie Chen, Binghui Peng, and Hongxun Wu. Theoretical limitations of multi-layer transformer. December 2024. URL https://arxiv.org/abs/2412.02975. arXiv:2412.02975v1 [cs.LG], 4 Dec 2024

    2024 arXiv

  12. [20]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021

  13. [21]

    Extending context window of large language models via positional interpolation, 2023

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023. URL https://arxiv.org/abs/2306.15595

    2023 arXiv

  14. [22]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vino...

    2022 arXiv

  15. [23]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  16. [25]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021 b

    arXiv

  17. [26]

    Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback

    Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. arXiv e-prints, pp.\ arXiv--2307, 2023

    2023

  18. [27]

    Unsloth, 2023

    Michael Han Daniel Han and Unsloth team. Unsloth, 2023. URL http://github.com/unslothai/unsloth

    2023

  19. [28]

    we also choose similar dimensions as modern Transformers, e.g. P =64 or P =128

    Tri Dao. State space duality (mamba-2) part i: The model. https://tridao.me/blog/2024/mamba2-part1-model/, 2024 a . “we also choose similar dimensions as modern Transformers, e.g. P =64 or P =128 .”

    2024

  20. [29]

    Flash A ttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024 b

    2024

  21. [30]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024

    2024 arXiv

  22. [31]

    causal-conv1d: Causal depthwise conv1d in cuda with a pytorch interface

    Dao-AILab . causal-conv1d: Causal depthwise conv1d in cuda with a pytorch interface. https://github.com/Dao-AILab/causal-conv1d, 2023. Features: kernel size 2, 3, 4; supports fp32/fp16/bf16

    2023

  23. [32]

    Griffin: Mixing gated linear recurrences with local attention for efficient language models

    Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint ...

    2024 arXiv

  24. [33]

    Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster, 2023

    Nolan Dey, Gurpreet Gosal, Zhiming, Chen, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, and Joel Hestness. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster, 2023. URL https://arxiv.org/abs/2304.03208

    2023 arXiv

  25. [34]

    Don't be lazy: Completep enables compute-efficient deep transformers, 2025

    Nolan Dey, Bin Claire Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don't be lazy: Completep enables compute-efficient deep transformers, 2025. URL https://arxiv.org/abs/2505.01618

    2025

  26. [35]

    Hymba: A hybrid-head architecture for small language models

    Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, et al. Hymba: A hybrid-head architecture for small language models. arXiv preprint arXiv:2411.13676, 2024

    2024 arXiv

  27. [36]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  28. [37]

    A curated research corpus for agricultural advisory ai applications, 2024

    Gardian. A curated research corpus for agricultural advisory ai applications, 2024. URL https://huggingface.co/datasets/CGIAR/gardian-ai-ready-docs

    2024

  29. [38]

    Zamba: A compact 7b ssm hybrid model

    Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7b ssm hybrid model. arXiv preprint arXiv:2405.16712, 2024

    2024 arXiv

  30. [39]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  31. [40]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  32. [41]

    Cruxeval: A benchmark for code reasoning, understanding and execution

    Alex Gu, Baptiste Rozi \`e re, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065, 2024

    2024 arXiv

  33. [42]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming -- the rise of code intelligence, 2024. URL https://arxiv....

    2024 arXiv

  34. [43]

    Infimm-webmath-40b: Advancing multimodal pre-training for enhanced mathematical reasoning

    Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang, Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, et al. Infimm-webmath-40b: Advancing multimodal pre-training for enhanced mathematical reasoning. arXiv preprint arXiv:2409.12568, 2024

    2024 arXiv

  35. [44]

    Simplifying transformer blocks

    Bobby He and Thomas Hofmann. Simplifying transformer blocks. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=RtDok9eS3s

    2024

  36. [45]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  37. [47]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021 b

    2021 arXiv

  38. [48]

    Clarification on how to interpret kernel size for conv1d (\#523)

    Scott Hoang and Mamba contributors . Clarification on how to interpret kernel size for conv1d (\#523). https://github.com/state-spaces/mamba/issues/523, 2024. GitHub issue discussing the meaning and limits of the conv1d kernel size

    2024

  39. [49]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024 a

    2024 arXiv

  40. [50]

    Yulan-mini: An open data-efficient language model, dec 2024 b

    Yiwen Hu, Huatong Song, Jia Deng, Jiapeng Wang, Jie Chen, Kun Zhou, Yutao Zhu, Jinhao Jiang, Zican Dong, Wayne Xin Zhao, and Ji-Rong Wen. Yulan-mini: An open data-efficient language model, dec 2024 b . URL https://arxiv.org/abs/2412.17743

    2024 arXiv

  41. [52]

    Opencoder: The open cookbook for top-tier code large language models

    Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, JH Liu, Chenchen Zhang, Linzheng Chai, et al. Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arXiv:2411.04905, 2024 b

    2024 arXiv

  42. [53]

    Qwen2.5-coder technical report, 2024

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou...

    2024 arXiv

  43. [54]

    Teknium"

    "Teknium" "interstellarninja". Hermes-function-calling-dataset-v1. URL https://huggingface.co/NousResearch/hermes-function-calling-v1

  44. [55]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  45. [56]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  46. [57]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  47. [58]

    Math-Verify: Math Verification Library

    Hynek Kydlíček. Math-Verify: Math Verification Library . URL https://github.com/huggingface/math-verify

  48. [59]

    Tulu 3: Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  49. [60]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023

    2023

  50. [61]

    Jamba: A hybrid transformer-mamba language model

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024

    2024 arXiv

  51. [62]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  52. [63]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021

    2021 arXiv

  53. [64]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024 a

    2024 arXiv

  54. [65]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024 b

    2024 arXiv

  55. [66]

    Ring attention with blockwise transformers for near-infinite context, 2023 a

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context, 2023 a . URL https://arxiv.org/abs/2310.01889

    2023 arXiv

  56. [67]

    Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023 b . URL https://open...

    2023

  57. [68]

    Finemath: A fine-grained mathematical evaluation benchmark for chinese large language models

    Yan Liu, Renren Jin, Ling Shi, Zheng Yao, and Deyi Xiong. Finemath: A fine-grained mathematical evaluation benchmark for chinese large language models. arXiv preprint arXiv:2403.07747, 2024 c

    2024 arXiv

  58. [69]

    Vmamba: Visual state space model

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024 d . doi:10.48550/arXiv.2401.10166. URL https://arxiv.org/abs/2401.10166. NeurIPS 2024 Spotlight

    2024 doi arXiv

  59. [70]

    Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Rob...

    2023 arXiv

  60. [71]

    Acemath: Advancing frontier math reasoning with post-training and reward modeling

    Zihan Liu, Yang Chen, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acemath: Advancing frontier math reasoning with post-training and reward modeling. arXiv preprint arXiv:2412.15084, 2024 e

    2024 arXiv

  61. [72]

    Neural thermodynamic laws for large language model training, 2025

    Ziming Liu, Yizhou Liu, Jeff Gore, and Max Tegmark. Neural thermodynamic laws for large language model training, 2025. URL https://arxiv.org/abs/2505.10559

    2025 arXiv

  62. [73]

    Starcoder 2 and the stack v2: The next generation

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024

    2024 arXiv

  63. [74]

    Finefineweb: A comprehensive study on fine-grained domain web corpus, December 2024

    M-A-P , Ge Zhang, Xinrun Du, Zhimiao Yu, Zili Wang, Zekun Wang, Shuyue Guo, Tianyu Zheng, Kang Zhu, Jerry Liu, Shawn Yue, Binbin Liu, Zhongyuan Peng, Yifan Yao, Jack Yang, Ziming Li, Bingni Zhang, Minghao Liu, Tianyu Liu, Yang Gao, Wenhu Chen, Xiaohuan Zhou, Qian Liu, Taifeng ...

    2024

  64. [75]

    Falcon2-11b technical report

    Quentin Malartic, Nilabhra Roy Chowdhury, Ruxandra Cojocaru, Mugariya Farooq, Giulia Campesan, Yasser Abdelaziz Dahou Djilali, Sanath Narayan, Ankit Singh, Maksim Velikanov, Basma El Amel Boussaha, et al. Falcon2-11b technical report. arXiv preprint arXiv:2407.14885, 2024

    2024 arXiv

  65. [76]

    On the SDE s and scaling rules for adaptive gradient algorithms

    Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the SDE s and scaling rules for adaptive gradient algorithms. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:...

    2022

  66. [77]

    Peft: State-of-the-art parameter-efficient fine-tuning methods

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft, 2022

    2022

  67. [78]

    Characterizing state space model (ssm) and ssm-transformer hybrid language model performance with long context length

    Saptarshi Mitra, Rachid Karami, Haocheng Xu, Sitao Huang, and Hyoukjun Kwon. Characterizing state space model (ssm) and ssm-transformer hybrid language model performance with long context length. arXiv preprint arXiv:2507.12442, 2025. doi:10.48550/arXiv.2507.12442. URL https:/...

    2025 doi

  68. [79]

    Gptqmodel

    ModelCloud.ai and qubitium@modelcloud.ai. Gptqmodel. https://github.com/modelcloud/gptqmodel, 2024. Contact: qubitium@modelcloud.ai

    2024

  69. [80]

    Oumi: an Open, End-to-end Platform for Building Large Foundation Models

    Oumi Community . Oumi: an Open, End-to-end Platform for Building Large Foundation Models . URL https://github.com/oumi-ai/oumi

  70. [81]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  71. [82]

    Openwebmath: An open dataset of high-quality mathematical web text

    Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. arXiv preprint arXiv:2310.06786, 2023

    2023 arXiv

  72. [83]

    Let ssms be convnets: State-space modeling with optimal tensor contractions

    Yan Ru Pei and others . Let ssms be convnets: State-space modeling with optimal tensor contractions. arXiv preprint arXiv:2501.13230, 2025. URL https://arxiv.org/abs/2501.13230

    2025 arXiv

  73. [84]

    The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv pr...

    2023 arXiv

  74. [85]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557, 2024 a

    2024 arXiv

  75. [86]

    Fineweb2: A sparkling update with 1000s of languages, 12 2024 b

    Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro von Werra, and Thomas Wolf. Fineweb2: A sparkling update with 1000s of languages, 12 2024 b . URL https://huggingface.co/datasets/HuggingFaceFW/fineweb-2

    2024

  76. [87]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023

    2023 arXiv

  77. [88]

    Evalchemy , 6 2025

    Negin Raoof, Etash Kumar Guha, Ryan Marten, Jean Mercat, Eric Frankel, Sedrick Keh, Hritik Bansal, Georgios Smyrnis, Marianna Nezhurina, Trung Vu, Zayne Rea Sprague, Mike A Merrill, Liangyu Chen, Caroline Choi, Zaid Khan, Sachin Grover, Benjamin Feuer, Ashima Suvarna, Shiye Su...

    2025

  78. [89]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  79. [90]

    Samba: Simple hybrid state space models for efficient unlimited context language modeling

    Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. arXiv preprint arXiv:2406.07522, 2024

    2024 arXiv

  80. [91]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  81. [92]

    Analysing mathematical reasoning abilities of neural models

    David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations, 2019

    2019

  82. [93]

    Neural machine translation of rare words with subword units, 2016

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units, 2016. URL https://arxiv.org/abs/1508.07909

    2016 arXiv

  83. [94]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  84. [95]

    Cox, and Rameswar Panda

    Yikang Shen, Matthew Stallone, Mayank Mishra, Gaoyuan Zhang, Shawn Tan, Aditya Prasad, Adriana Meza Soria, David D. Cox, and Rameswar Panda. Power scheduler: A batch size and token number agnostic learning rate scheduler, 2024. URL https://arxiv.org/abs/2408.13359

    2024 arXiv

  85. [96]

    Language models are multilingual chain-of-thought reasoners

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057, 2022

    arXiv

  86. [97]

    Singh and DJ Strouse

    Aaditya K. Singh and DJ Strouse. Tokenization counts: the impact of tokenization on arithmetic in frontier llms, 2024. URL https://arxiv.org/abs/2402.14903

    2024 arXiv

  87. [98]

    Learning long sequences in spiking neural networks

    Matei-Ioan Stan and Oliver Rhodes. Learning long sequences in spiking neural networks. Scientific Reports, 14 0 (1): 0 21957, 2024. doi:10.1038/s41598-024-71678-8. URL https://www.nature.com/articles/s41598-024-71678-8

    2024 doi

  88. [99]

    Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset, 2024

    Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset, 2024. URL https://arxiv.org/abs/2412.02595

    2024 arXiv

  89. [100]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    arXiv

  90. [101]

    Scaling laws with vocabulary: Larger models deserve larger vocabularies

    Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. arXiv preprint arXiv:2407.13623, 2024

    2024 arXiv

  91. [102]

    The falcon 3 family of open models, December 2024

    Falcon-LLM Team. The falcon 3 family of open models, December 2024

    2024

  92. [103]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram \'e , Morgane Rivi \`e re, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  93. [104]

    Jamba-1.5: Hybrid transformer-mamba models at scale

    Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al. Jamba-1.5: Hybrid transformer-mamba models at scale. arXiv preprint arXiv:2408.12570, 2024

    2024 arXiv

  94. [105]

    Culturay: A large cleaned multilingual dataset of 75 languages, 2024

    Huu Nguyen Thuat Nguyen and Thien Nguyen. Culturay: A large cleaned multilingual dataset of 75 languages, 2024

    2024

  95. [106]

    Finding alternative translations in a large corpus of movie subtitle

    J \"o rg Tiedemann. Finding alternative translations in a large corpus of movie subtitle. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pp.\ 3518--3522, 2016

    2016

  96. [107]

    Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data

    Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01560, 2024

    2024 arXiv

  97. [108]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  98. [109]

    TRL: Transformer Reinforcement Learning

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformer Reinforcement Learning . URL https://github.com/huggingface/trl

  99. [110]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024

    2024 arXiv

  100. [111]

    Livebench: A challenging, contamination-free llm benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314, 4, 2024

    2024 arXiv

  101. [112]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-Art Natural...

    2020

  102. [113]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024 a

    2024 arXiv

  103. [114]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024 b

    2024 arXiv

  104. [115]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  105. [116]

    Greg Yang and Edward J. Hu. Feature learning in infinite-width neural networks, 2022. URL https://arxiv.org/abs/2011.14522

    2022 arXiv

  106. [117]

    Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao

    Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer, 2022. URL https://arxiv.org/abs/2203.03466

    2022 arXiv

  107. [118]

    Tensor programs vi: Feature learning in infinite-depth neural networks, 2023

    Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs vi: Feature learning in infinite-depth neural networks, 2023. URL https://arxiv.org/abs/2310.02244

    2023 arXiv

  108. [119]

    Simon, and Jeremy Bernstein

    Greg Yang, James B. Simon, and Jeremy Bernstein. A spectral condition for feature learning, 2024 c . URL https://arxiv.org/abs/2310.17813

    2024 arXiv

  109. [120]

    Helmet: How to evaluate long-context language models effectively and thoroughly

    Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context language models effectively and thoroughly. In International Conference on Learning Representations (ICLR), 2025

    2025

  110. [121]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning. In Conference on Neural Information Processing Systems, 2022

    2022

  111. [122]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  112. [123]

    Muse: Parallel multi-scale attention for sequence to sequence learning, 2019

    Guangxiang Zhao, Xu Sun, Jingjing Xu, Zhiyuan Zhang, and Liangchen Luo. Muse: Parallel multi-scale attention for sequence to sequence learning, 2019. URL https://arxiv.org/abs/1911.09483

    2019 arXiv

  113. [124]

    LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models . Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System De...

    2024 arXiv

  114. [125]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023

    2023 arXiv

  115. [126]

    Falcon mamba: The first competitive attention-free 7b language model

    Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, and Hacid Hakim. Falcon mamba: The first competitive attention-free 7b language model. 2024. URL https://arxiv.org/abs/2410.05355

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.