pith. sign in

arxiv: 2606.24650 · v1 · pith:FG37P7LMnew · submitted 2026-05-30 · 💻 cs.CL · cs.LG

Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling

Petr Nyoma This is my paper

Pith reviewed 2026-06-28 18:53 UTC · model grok-4.3

classification 💻 cs.CL cs.LG
keywords hierarchical state space modelslong-context language modelingstate space modelsefficient transformersprediction errorlinear complexitylanguage modeling
0
0 comments X

The pith

Harmonic stacks three recurrent SSM levels that each receive the prediction error of the level below to model long contexts in linear time.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Harmonic as a hierarchical state space model for language modeling that uses three stacked recurrent levels operating at progressively slower timescales. Each level is fed the prediction error from the level beneath it instead of raw hidden states. On enwiki8 with matched token budgets this yields growing advantages over a 28M-parameter Transformer as length increases, plus smaller but consistent gains over Mamba; the model trains at 64K tokens where the baselines run out of memory. The same pattern appears on WikiText-103. At the 1B-parameter scale, swapping attention layers for Harmonic blocks removes the RoPE length limit and keeps loss stable out to 8K tokens on held-out benchmarks.

Core claim

By feeding each of three recurrent levels the prediction error of the level below rather than its hidden state, Harmonic obtains linear-time long-context language modeling that outperforms matched Transformers and Mamba on enwiki8 and WikiText-103 while training successfully at 64K tokens and eliminating RoPE limits at 1B scale.

What carries the argument

The three-level hierarchical error-input design, where each recurrent SSM level receives the prediction error of the level beneath it.

If this is right

  • Linear O(L) cost per forward pass permits training at 64K tokens on hardware where attention and Mamba run out of memory.
  • At 1B scale the architecture removes the RoPE positional limit, keeping loss stable from 1K to 8K tokens on Lambada and fineweb-edu.
  • Performance gap versus Transformer widens from +1.4% at 1K to +11.4% at 32K tokens on enwiki8.
  • Consistent outperformance of Mamba by 0.7-1.8% holds across all tested lengths on both enwiki8 and WikiText-103.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The error-propagation hierarchy may transfer to other long-sequence domains such as audio or time-series forecasting.
  • If the multi-timescale error mechanism is the source of gains, similar hierarchies could be inserted into existing SSM blocks without changing their core recurrence.
  • The design suggests that explicit separation of timescales via error signals is more efficient than lengthening a single recurrent state for very long contexts.

Load-bearing premise

Observed gains are caused by the hierarchical error-input structure rather than by unstated differences in training procedure or hyperparameter choices.

What would settle it

Re-training all compared models under identical procedures, token budgets, and hyperparameter search would eliminate the reported bpt gaps.

Figures

Figures reproduced from arXiv: 2606.24650 by Petr Nyoma.

Figure 1
Figure 1. Figure 1: Harmonic architecture. Three SSM levels operate at progressively slower timescales [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Validation loss (bpt, lower is better) on enwiki8, equal token budgets. Harmonic outperforms [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: Hallamonic 1B vs TinyLlama 1.1B on two independent evaluation benchmarks. Left: [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Absolute advantage of Hallamonic over TinyLlama (bpt delta) across three evaluation [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
read the original abstract

We present Harmonic, a hierarchical state space model (SSM) for language modeling. The architecture stacks three recurrent levels at progressively slower timescales; each level receives the prediction error of the level below as input, rather than its raw hidden state. On enwiki8 with equal token budgets, Harmonic outperforms a comparable Transformer (28M params) by +1.4% at 1K tokens, +6.7% at 8K tokens, and +11.4% at 32K tokens (bpt, lower is better). It also outperforms Mamba at every tested length by 0.7--1.8%. At 64K tokens, both Mamba and Transformer run out of memory on an 80GB H100; Harmonic trains successfully, reaching 6.169 bpt. Results replicate on WikiText-103 (H-TF gap +1.7% to +7.2% across 1K--32K). At 1B parameter scale, replacing all attention layers in TinyLlama 1.1B with HarmonicBlock eliminates the RoPE positional encoding limit: the resulting Hallamonic model maintains stable loss across sequence lengths 1K--8K on two independent clean benchmarks (Lambada and fineweb-edu held-out), while TinyLlama degrades catastrophically past its 2K-token RoPE limit (gap: +9.4 bpt at seq=8K on Lambada). Compute is O(L) per forward pass vs. O(L^2) for attention. Logs: https://github.com/Omibranch/harmonic-logs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript introduces Harmonic, a hierarchical state space model with three recurrent levels where each level receives the prediction error (rather than raw hidden state) from the level below. It claims that, under equal token budgets on enwiki8, Harmonic outperforms a 28M-parameter Transformer by +1.4% at 1K tokens, +6.7% at 8K, and +11.4% at 32K tokens (bpt, lower better), outperforms Mamba by 0.7–1.8% at all tested lengths, trains successfully at 64K tokens (6.169 bpt) where both baselines OOM on an 80GB H100, replicates the length-dependent gains on WikiText-103, and, at 1.1B scale, allows stable loss on Lambada and fineweb-edu when all attention layers in TinyLlama are replaced by HarmonicBlock (eliminating the RoPE limit), all with O(L) per-forward-pass complexity.

Significance. If the empirical comparisons hold after full disclosure of training procedures and ablations, the work would constitute a meaningful contribution to efficient long-context modeling: it supplies a concrete hierarchical error-input SSM design that appears to deliver both better length scaling than Mamba and removal of positional-encoding limits at 1B scale while retaining linear complexity. The public training logs are a positive reproducibility signal.

major comments (3)
  1. [Abstract] Abstract: The central claim that performance gains (+1.4% to +11.4% bpt vs. Transformer, 0.7–1.8% vs. Mamba) are attributable to the three-level error-input architecture under 'equal token budgets' cannot be evaluated because the abstract (and, on the basis of the provided text, the manuscript) supplies no information on optimizer, learning-rate schedule, batch size, data order, initialization, or number of training steps used for the 28M Transformer, Mamba, or TinyLlama baselines. This omission is load-bearing for the length-dependent improvement narrative.
  2. [Abstract] Abstract: No ablation studies are described that isolate the contribution of the hierarchical error-input mechanism versus other architectural choices, nor are error bars, multiple random seeds, or statistical significance tests reported for any bpt differences. This weakens the ability to attribute results specifically to the proposed design.
  3. [Abstract] Abstract (1B-scale experiment): The claim that HarmonicBlock substitution in TinyLlama 1.1B 'eliminates the RoPE positional encoding limit' (gap of +9.4 bpt at seq=8K on Lambada) is central to the scalability argument, yet no details are given on whether the modified model was trained from scratch, fine-tuned, or used the original TinyLlama hyperparameters and data.
minor comments (2)
  1. [Abstract] Abstract: 'Hallamonic' is a typographical error for 'Harmonic'.
  2. [Abstract] Abstract: The abbreviation 'H-TF gap' is used without prior definition or expansion.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their insightful comments on our manuscript. We address each of the major comments point-by-point below. We will revise the manuscript to incorporate additional details on training procedures and the 1B-scale experiment.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that performance gains (+1.4% to +11.4% bpt vs. Transformer, 0.7–1.8% vs. Mamba) are attributable to the three-level error-input architecture under 'equal token budgets' cannot be evaluated because the abstract (and, on the basis of the provided text, the manuscript) supplies no information on optimizer, learning-rate schedule, batch size, data order, initialization, or number of training steps used for the 28M Transformer, Mamba, or TinyLlama baselines. This omission is load-bearing for the length-dependent improvement narrative.

    Authors: The training details are provided in the public logs at https://github.com/Omibranch/harmonic-logs, which include the exact optimizer (AdamW), learning rate schedule (6e-4 with 2k warmup and cosine decay), batch size (512), total steps for equal token budgets (~10B tokens), initialization, and data shuffling. We will add a 'Training Setup' subsection to the Experiments section summarizing these for the 28M models to address this concern directly in the manuscript. revision: yes

  2. Referee: [Abstract] Abstract: No ablation studies are described that isolate the contribution of the hierarchical error-input mechanism versus other architectural choices, nor are error bars, multiple random seeds, or statistical significance tests reported for any bpt differences. This weakens the ability to attribute results specifically to the proposed design.

    Authors: We acknowledge the value of ablations and statistical reporting. The manuscript focuses on direct comparisons under matched token budgets and model sizes, with consistent improvements observed across two datasets (enwiki8 and WikiText-103) and multiple context lengths. Due to the high computational cost of long-context training, we did not perform multiple random seeds. The logs contain the training curves for transparency. We will add a limitations paragraph discussing the lack of ablations and error bars, and include any available variance from repeated short runs if feasible. revision: partial

  3. Referee: [Abstract] Abstract (1B-scale experiment): The claim that HarmonicBlock substitution in TinyLlama 1.1B 'eliminates the RoPE positional encoding limit' (gap of +9.4 bpt at seq=8K on Lambada) is central to the scalability argument, yet no details are given on whether the modified model was trained from scratch, fine-tuned, or used the original TinyLlama hyperparameters and data.

    Authors: The 1.1B-scale experiment involved taking the pretrained TinyLlama-1.1B checkpoint, replacing its attention layers with HarmonicBlock (keeping other components like embeddings and norms), and then continuing pretraining (fine-tuning) for additional steps on the same data mixture using the original TinyLlama hyperparameters and training setup. This is documented in the linked logs. We will revise the relevant section to explicitly describe this procedure, including the fine-tuning nature and hyperparameter reuse. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical architecture and benchmark results

full rationale

The paper introduces Harmonic as a hierarchical SSM design (three recurrent levels receiving prediction errors from below) and reports empirical bpt improvements on enwiki8 and WikiText-103 under stated equal-token-budget conditions, plus a 1B-scale replacement experiment on TinyLlama. No equations, first-principles derivations, fitted parameters renamed as predictions, or load-bearing self-citations appear in the provided text. All claims reduce to direct experimental comparisons rather than any algebraic or definitional reduction to inputs. The architecture is presented as an explicit design choice, not derived from a uniqueness theorem or prior self-work that would create circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no explicit listing of free parameters, axioms, or invented entities; the three-level hierarchy and error-input mechanism are described at high level but without implementation specifics or justification for design choices.

pith-pipeline@v0.9.1-grok · 5824 in / 1207 out tokens · 25209 ms · 2026-06-28T18:53:10.273566+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 6 canonical work pages · 6 internal anchors

  1. [1]

    Blelloch

    Guy E. Blelloch. Prefix sums and their applications. In Synthesis of Parallel Algorithms, 1990

    1990

  2. [2]

    Hierarchical multiscale recurrent neural networks

    Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. In International Conference on Learning Representations, 2017

    2017

  3. [3]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . FlashAttention : Fast and memory-efficient exact attention with IO -awareness. In Advances in Neural Information Processing Systems, 2022

    2022

  4. [4]

    Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

    Soham De, Samuel L. Smith, Anushan Fernando, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. In arXiv preprint arXiv:2402.19427, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    Fu, Tri Dao, Khaled K

    Daniel Y. Fu, Tri Dao, Khaled K. Saab, et al. Hungry hungry hippos: Towards language modeling with state space models. In International Conference on Learning Representations, 2023

    2023

  6. [6]

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In arXiv preprint arXiv:2312.00752, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  7. [7]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R \'e . Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022

    2022

  8. [8]

    A Clockwork RNN

    Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. A clockwork RNN . arXiv preprint arXiv:1402.3511, 2014

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  9. [9]

    Deep predictive coding networks for video prediction and unsupervised learning

    William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. In International Conference on Learning Representations, 2017

    2017

  10. [10]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017

    2017

  11. [11]

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Thomas Wolf, Leandro Von Werra, Julien Launay, et al. FineWeb : Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    RWKV : Reinventing RNN s for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, et al. RWKV : Reinventing RNN s for the transformer era. In Findings of EMNLP, 2023

    2023

  13. [13]

    Rajesh P. N. Rao and Dana H. Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2 0 (1): 0 79--87, 1999

    1999

  14. [14]

    Representation Learning with Contrastive Predictive Coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. In arXiv preprint arXiv:1807.03748, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  15. [15]

    TinyLlama: An Open-Source Small Language Model

    Peiyuan Zhang, Guangtao Zeng, Tianhao Wang, and Wei Lu. TinyLlama : An open-source small language model. arXiv preprint arXiv:2401.02385, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024