Pith. sign in

REVIEW 10 cited by

Jamba-1.5: Hybrid Transformer-Mamba Models at Scale

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.12570 v1 pith:MJF5YUCT submitted 2024-08-22 cs.CL cs.LG

Jamba Team: Barak Lenz , Alan Arazi , Amir Bergman , Avshalom Manevich , Barak Peleg , Ben Aviram , Chen Almagor , Clara Fridman
show 52 more authors
Dan Padnos Daniel Gissin Daniel Jannai Dor Muhlgay Dor Zimberg Edden M Gerber Elad Dolev Eran Krakovsky Erez Safahi Erez Schwartz Gal Cohen Gal Shachaf Haim Rozenblum Hofit Bata Ido Blass Inbal Magar Itay Dalmedigos Jhonathan Osin Julie Fadlon Maria Rozman Matan Danos Michael Gokhman Mor Zusman Naama Gidron Nir Ratner Noam Gat Noam Rozen Oded Fried Ohad Leshno Omer Antverg Omri Abend Opher Lieber Or Dagan Orit Cohavi Raz Alon Ro'i Belson Roi Cohen Rom Gilad Roman Glozman Shahar Lev Shaked Meirom Tal Delbari Tal Ness Tomer Asida Tom Ben Gal Tom Braude Uriya Pumerantz Yehoshua Cohen Yonatan Belinkov Yuval Globerson Yuval Peleg Levy Yoav Shoham
This is my paper · ORCID
classification cs.CLcs.LG
keywords modelsjamba-1jambalargemodelactivearchitecturebenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present Jamba-1.5, new instruction-tuned large language models based on our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts architecture, providing high throughput and low memory usage across context lengths, while retaining the same or better quality as Transformer models. We release two model sizes: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-Mini, with 12B active parameters. Both models are fine-tuned for a variety of conversational and instruction-following capabilties, and have an effective context length of 256K tokens, the largest amongst open-weight models. To support cost-effective inference, we introduce ExpertsInt8, a novel quantization technique that allows fitting Jamba-1.5-Large on a machine with 8 80GB GPUs when processing 256K-token contexts without loss of quality. When evaluated on a battery of academic and chatbot benchmarks, Jamba-1.5 models achieve excellent results while providing high throughput and outperforming other open-weight models on long-context benchmarks. The model weights for both sizes are publicly available under the Jamba Open Model License and we release ExpertsInt8 as open source.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures

    cs.AI 2026-03 conditional novelty 6.0 of 10

    Hybrid Transformer-Mamba models beat pure transformers and SSMs on n-gram retrieval efficiency and length generalization, but transformers stay ahead on two-hop position lookup, with SSM-based models uniquely forming ...

  2. Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web

    cs.CY 2025-10 conditional novelty 6.0 of 10

    Reputable news sites block AI crawlers at roughly six times the rate of misinformation sites, and this gap is widening over time.

  3. Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Falcon-H1 reports competitive benchmark scores for a 0.5B to 34B family of parallel hybrid attention/Mamba-2 models, claiming 2x to 4x parameter efficiency versus dense transformers.

  4. Cartridges: Lightweight and general-purpose long context representations via self-study

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.

  5. Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networks

    cs.CL 2025-05 conditional novelty 6.0 of 10

    FlowHN splits input tokens between parallel attention and SSM branches to balance compute load, improving throughput and hardware utilization in small autoregressive language models.

  6. Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approaches

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Full-context and retrieval-based methods outperform hierarchical and incremental compression for large-scale multi-document summarization, though compression methods show strong intermediate information retention.

  7. NoLiMa: Long-Context Evaluation Beyond Literal Matching

    cs.CL 2025-02 conditional novelty 6.0 of 10

    NoLiMa shows that long-context language models' strong retrieval performance largely depends on literal matches, and that accuracy drops sharply once those matches are removed.

  8. GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A new synthetic benchmark reveals that LLM reasoning accuracy decays sigmoidally with problem complexity and that repeated sampling has poor scaling efficiency.

  9. Memory for Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    The paper organizes LLM memory architectures along three axes—implicit vs explicit representation, offline vs online update, and short vs long-term persistence—plus finer-grained update rules.

  10. CryptoX : Compositional Reasoning Evaluation of Large Language Models

    cs.CR 2025-02 conditional novelty 4.0 of 10

    A benchmark that encodes prompts in secret codes and measures how much accuracy models lose, showing most LLMs, especially open-source ones, struggle on this two-step compositional task.

Pith tools