Pith. sign in

REVIEW 5 major objections 5 minor 17 references

WuNeng: Hybrid State with Attention

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims WuNeng, a hybrid of standard attention and RWKV-7 state heads with cross-head interactions and multi-token state processing, outperforms Qwen2.5-7B-Instruct by roughly 10–15% in preliminary benchmarks.

desk verdict Incomplete draft whose headline 10-15% gain is confounded by the training pipeline; the architectural sketch is plausible but under-specified. read the letter →

arxiv 2504.19191 v1 pith:L4VA76JS submitted 2025-04-27 cs.CL

Liu Xiao , Li Zhiyuan , Lin Yueyu This is my paper · ORCID
classification cs.CL
keywords hybridattentionRWKV-7state-drivenheadscross-headinteractionmulti-tokenstateprocessinglanguagemodelarchitectureknowledgedistillationbenchmarkevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes WuNeng, an LLM architecture that keeps standard multi-head attention and adds extra heads driven by the continuous recurrent state of RWKV-7, rather than replacing attention heads for efficiency. It claims these added state-driven heads, a cross-head interaction scheme, and a multi-token state-processing trick improve contextual coherence and reasoning at a cost of under 5% extra parameters. The reported evidence is a preliminary evaluation in which WuNeng-7B scores 80.33% on MMLU and 92.22% on GSM8K, roughly 10–15% above Qwen2.5-7B-Instruct's 71.72% and 82.34%. The paper argues that expressivity, not KV-cache reduction, is the main payoff of hybrid attention-state designs.

What carries the argument

The central object is the hybrid-head layer. Standard attention $A_h = \operatorname{softmax}(Q_hK_h^T/\sqrt{d_k})V_h$ is augmented with state-derived keys $\hat K_h = W_{\hat K,h}S_tX$, and middle heads $M_h = \sigma(W_h^{\mathrm{mid}}(A_h + \beta S_t^T\hat K_h))$ bridge attention and state through learnable scalars and gates. The RWKV-7 state $S_t$ is updated by a generalized delta rule, and multi-token state processing injects the state into queries via $Q_h = XW_h^Q + \lambda W_h^{\mathrm{state}}S_t$. This machinery is what the paper says carries the benchmark gains, since it is the architectural addition beyond the distillation pipeline.

What would settle it

Run the identical supervised fine-tuning and preference-optimization recipe on Qwen2.5-7B-Instruct and compare against WuNeng-7B under the same protocol; if the gap collapses, the architectural claim is unsupported. A second check is to ablate the state-injected query term by fixing $\lambda=0$ and measuring whether the MMLU and GSM8K margins survive.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that a hybrid-head layer can outperform a strong dense-transformer baseline by a wide margin without scaling parameters. Each layer augments standard multi-head attention with RWKV-7 state-driven heads, links the two through learnable middle heads using concatenation, additive modulation, or gated fusion, and feeds the recurrent state back into the query projection as $Q_h = XW_h^Q + \lambda W_h^{\mathrm{state}} S_t$. In the preliminary Stage 3 comparison, WuNeng-7B reaches 80.33% on MMLU, 92.22% on GSM8K, 55.12% on GPQA, and 82.45% on IFEval, against 71.72%, 82.34%, 49.0%, and 73.62% for Qwen2.5-7B-Instruct. The paper attributes this 10–15% gap to the hybrid attention mechanism and cross-head interactions, while stating that evaluation is ongoing and the results are preliminary.

Load-bearing premise

The load-bearing premise is that the reported 10–15% benchmark gap is caused by the WuNeng architecture itself, not by the extra distillation, supervised fine-tuning, and preference-optimization training that the baseline models did not receive.

Editorial extensions

If this is right

  • If the reported gains hold, adding recurrent state heads to existing transformer layers can improve reasoning benchmarks at the same parameter scale.
  • If the gains reproduce under controlled comparisons, the hybrid-head design gives a practical path to combining linear-time state summarization with high-resolution attention recall.
  • If the architecture is the cause, the under-5% parameter overhead means the approach could be applied to stronger base models without retraining from scratch.
  • If the mechanism works as described, multi-token state injection should also help long-context and state-tracking tasks, which the paper lists as future directions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The main reported comparison is not pipeline-matched: WuNeng-7B received distillation, supervised fine-tuning, and preference optimization on 770M tokens, while the Qwen2.5-7B-Instruct numbers are the public baseline without that extra alignment, so part of the 10–15% gap could be training, not architecture.
  • A controlled ablation that masks out the state heads and cross-head gating while keeping the same three-stage pipeline would isolate how much of the gain each component contributes.
  • A targeted test of the state-injection term is to set the learnable scalar $\lambda$ to zero and check whether the MMLU and GSM8K margins persist; if they do, the multi-token state injection is not the driver.
  • The design suggests a testable combination with existing sparse-attention models: adding state heads to a sparse-attention transformer could give both long-range state and local precision, something the paper only gestures at.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper introduces WuNeng, a hybrid large-language-model architecture that augments standard multi-head attention with RWKV-7 state-driven heads, adds cross-head interaction mechanisms (additive modulation, gated fusion, and a 'middle head' bridge), and conditions queries on a multi-token RWKV-7 state. The authors describe a three-stage ARWKV-based training pipeline (attention alignment, knowledge distillation, and SFT/DPO) and report preliminary benchmark numbers claiming WuNeng-7B outperforms Qwen2.5-7B-Instruct by 10-15% on MMLU, GSM8K, GPQA, and other tasks (Table 1). The paper explicitly states several times that evaluation is ongoing and contains unresolved placeholders, including an unresolved citation and a missing figure reference.

Significance. The architectural idea of keeping full attention while adding RWKV-7 state heads and explicit cross-head interactions is a reasonable direction to explore, and the paper correctly identifies that prior hybrid designs tend to prioritize efficiency over expressivity. If the reported gains were rigorously attributed to the architecture, this would be a useful contribution to the ongoing attention-versus-state-model discussion. However, the current manuscript does not support that attribution: the central comparison is confounded by unequal training, the architecture is under-specified to the point of non-reproducibility, and no ablation or proof substantiates the individual mechanisms. As submitted, the paper is closer to an extended abstract or position statement than a complete technical report.

major comments (5)
  1. [Sec 4.1 / Table 1] The headline claim of a 10-15% improvement over Qwen2.5-7B-Instruct is uninterpretable because WuNeng-7B was distilled from Qwen2.5-7B-Instruct and then trained with SFT and DPO on 770M tokens (Sec 4.1, 4.4), while the baselines (Qwen2.5-7B-Instruct, Hymba-1.5B, LLaMA3.2-3B) were evaluated off-the-shelf. There is no same-pipeline control (e.g., Qwen2.5-7B-Instruct trained on the same Stage 3 data and protocol). The text's statement that baselines were evaluated 'under identical conditions' refers only to inference settings, not to training, so the measured differences cannot be attributed to the WuNeng architecture rather than to the additional training itself. This missing control is load-bearing for the paper's central claim.
  2. [Sec 3.1-3.3, Eqs (2), (6), (11)] The architecture is insufficiently specified for reproduction. The operator F in Eq (2) is described only as a 'kernel combine mechanism, such as concatenation, summation, or a learned transformation,' but the paper never states which variant is used in the experiments or how the combination is computed when F takes multiple argument sets. Eq (6) uses F with three argument sets (including {M_h}), whereas Eq (2) uses two, and no rule is given for reducing these to the intermediate representation W_l_attn consumes. Eq (11) adds λ W_state^h S_t to X W_Q^h; since S_t is a d×d state matrix and Q_h is a length×d query matrix, the addition is not dimensionally coherent unless W_state^h maps S_t (or a function of it) to a sequence-length representation in a way that is not described. These gaps prevent the reader from connecting the experimental results to the proposed mechanism.
  3. [Sec 3.1] The statement that the RWKV-7 state enables 'complex state manipulations such as swapping entries, which enhances expressivity beyond the TC0 complexity class' is asserted without proof or a supporting citation. If this is a formal complexity claim, a proof sketch or a precise reduction must be supplied; if it is intended to echo results from RWKV-7 or other work, the relevant reference and precise statement are needed. As written, the sentence is a bare assertion in a section that otherwise defines the architecture.
  4. [Sec 4.2, 4.3, Figure 3] The convergence evidence is reported as single loss values without error bars, repeated runs, or a same-size RWKV-7-7B baseline. The text claims WuNeng-7B's hybrid attention 'converged to a loss of 0.15' in Stage 1 and 0.08 in Stage 2, but these numbers are not accompanied by variance or the number of seeds. Figure 3's caption says WuNeng-7B (green) and Qwen2.5-7B-Instruct (blue) are compared 'during knowledge distillation,' yet Qwen2.5 is a Transformer, not a RWKV-based model, making it unclear what the comparison is intended to show. These points must be clarified before the claimed convergence advantage can be assessed.
  5. [Sec 3.2-3.3 / Table 1] No ablation isolates the three proposed components (hybrid heads, cross-head interactions, multi-token state processing). The only reported comparison is the full WuNeng-7B (with cross-head interactions and active MLPs, plus SFT/DPO) against untrained baselines. Consequently, even if the same-pipeline control were added, the individual contributions claimed in Sections 3.2 and 3.3 would remain unsupported; the paper needs ablations removing each mechanism to allow attribution.
minor comments (5)
  1. [Sec 4.2] The citation 'as in ARWKV [?]' is unresolved; a proper author/year citation must be provided.
  2. [Sec 5] The text refers to 'Figure ??' for the alignment loss curves; the figure number is missing and must be fixed.
  3. [Abstract] The abstract claims WuNeng 'sets a new standard for balancing expressivity and computational efficiency'; given the explicitly preliminary evaluation and the unresolved comparison issues, this overclaims and should be tempered.
  4. [Sec 3.1] The notation for state-derived keys is inconsistent: the text introduces \\hat{K}_h and also calls it 'state-derived key,' but does not relate it to the previously defined K_h; consistent notation would improve reproducibility.
  5. [Figure 3] The caption lists the colors but the figure itself should include a legend and explicit axis labels with units; the y-axis (loss) and x-axis (steps) are never defined in the text.

Circularity Check

2 steps flagged · score 6.0 of 10

The central multi-token state processing and cross-head update equations are self-referential as written: S_t is updated from attention outputs that are themselves functions of the same S_t.

  1. self definitional [Section 3.3, Eqs. (10)-(12)]
    "The RWKV-7 state S_t is updated using the generalized delta rule as defined in RWKV-7 : S_t = S_{t−1} (diag(w_t)−κ_T_t (a_t⊗κ_t)) + v_T_t k_t, (10) where ... v_t = W_V F ({A_h}) ... The enriched state S_t is then used to modulate the attention mechanism ... Q_h = X W_Q^h + λW_h_state S_t, (11) ... A_h = softmax(Q_h K_h^T / sqrt(d_k)) V_h, (12)"

    Eq. (12) defines A_h from Q_h, and Eq. (11) defines Q_h by adding the state S_t to the query. Eq. (10) updates S_t using v_t = W_V F({A_h}), i.e., the attention output. Therefore the S_t on the left-hand side of Eq. (10) is defined through A_h, which is defined through the same S_t on the right-hand side. No temporal offset (e.g., S_{t-1}) or fixed-point specification is given, so the multi-token state processing mechanism is self-referential and not a well-defined sequential computation as written.

  2. self definitional [Section 3.2, Eqs. (6) and (9)]
    "M_h = σ(W_mid^h (A_h + β S_T_t ˆK_h)) is the middle head output, bridging attention and the RWKV-7 state S_t. ... S_t = S_{t−1}(diag(w_t)−κ_T_t(a_t⊗κ_t)) + v_T_t k_t, (9) where ... v_t = W_V F ({A_h},{M_h})"

    The state update in Eq. (9) uses v_t = W_V F({A_h},{M_h}), and M_h from Eq. (6) is a function of S_t via β S_T_t ˆK_h. Thus the state S_t being produced by the update appears on its own right-hand side through M_h. As with Eqs. (10)-(12), the same symbol S_t is used on both sides with no specified time offset, making the cross-head interaction update circular as written.

full rationale

The paper's headline 10-15% benchmark advantage is an empirical measurement, not a quantity derived from a fitted equation, so the benchmark comparison itself is not circular in the formal sense. The primary non-circularity concern there is a fairness confound: WuNeng-7B was distilled from Qwen2.5-7B-Instruct and then received additional SFT/DPO on 770M tokens, while Qwen2.5-7B-Instruct was evaluated off-the-shelf; Section 4.1 says baselines were evaluated under identical conditions, but that does not control for the extra training. This is a correctness-risk issue, not a construction-level circularity. The ARWKV self-citation ([Yueyu et al., 2025], sharing authors Lin Yueyu, Li Zhiyuan, and Liu Xiao) provides the training methodology, but the benchmark scores are not taken from that citation. However, the architecture's defining update rules are self-referential: Eq. (10) updates S_t using attention outputs A_h, while Eq. (11) makes Q_h depend on the same S_t; analogously, Eq. (9) updates S_t using M_h from Eq. (6), which depends on S_t. As written, S_t appears on both sides of its own update with no temporal offset or fixed-point specification, so the central 'multi-token state processing' and 'cross-head interaction' mechanisms are defined circularly. These equations are load-bearing for the paper's claim that the architecture's expressivity comes from state-augmented attention. Also flagged: Section 4.2 contains an unresolved citation 'as in ARWKV [?]', Section 5 refers to 'Figure ??', and Section 4 opens with 'As the WuNeng architecture is an ongoing work, its evaluation is currently in progress'; these are missing-support issues that do not themselves create an additional circular reduction beyond the equation-level self-reference.

Assumptions & free parameters 3 free parameters · 5 assumptions · 3 invented entities

The paper introduces several learnable scalars and projections without reporting their values or dimensions. The key combining operation F is left as an unspecified design choice. The RWKV-7 state update is imported as a black box, and the architecture adds ad hoc assumptions about dimensional consistency and a claimed complexity result with no proof. No independent evidence is provided for the new architectural components.

free parameters (3)
  • alpha, beta, gamma, lambda (learnable scalars) = not reported
    These are introduced in Eqs (2), (6), (7), (8), (11) to control inter-head modulation and state query mixing; their values are not reported, and they are architecture parameters rather than constants fit to benchmark data.
  • W_state, W_mid, W_gate, W_hatK projections = not reported
    Learnable projection matrices introduced for state modulation (Eq 11), middle heads (Eq 6-8), and state-derived keys; no initialization or dimensionality specified, making the architecture underdetermined.
  • F (kernel combine mechanism) = unspecified
    Eqs (2),(5),(6) define F only as 'concatenation, summation, or a learned transformation', a free design choice that critically affects the model and is not fixed or evaluated.
assumptions (5)
  • domain assumption RWKV-7 generalized delta rule (Eq 4) is used as the state update and is taken as correct.
    The paper imports RWKV-7's state update from Peng et al. 2025 without re-derivation; the state matrix S_t is then manipulated in ways (e.g., added to queries) whose dimensional consistency is not checked.
  • ad hoc to paper The query augmentation Q_h = XW_Q^h + λW_state^h S_t (Eq 11) is dimensionally valid.
    S_t is a recurrent state; its shape is never specified, so adding it to a projected query may be ill-defined. The paper assumes this operation is sound.
  • ad hoc to paper The F kernel combine operation exists and has the algebraic properties needed for Eqs (2)-(6).
    F is described only as concat/sum/learned, but the equations treat it as a well-behaved function without specifying shape or behavior.
  • domain assumption The claim that the architecture exceeds the TC0 complexity class is asserted without proof.
    Section 3.1 says 'enables complex state manipulations such as swapping entries, which enhances expressivity beyond the TC0 complexity class' but no derivation is given.
  • domain assumption The Stage 3 baseline comparison is valid: baselines were evaluated under identical conditions.
    Section 4.1 says baselines were evaluated under identical conditions, but no protocol, prompts, or evaluation code are given, and model sizes differ (7B vs 1.5B vs 3B).
invented entities (3)
  • Middle heads (M_h)
    purpose: Bridge standard attention heads and RWKV-7 state-driven heads in the cross-head interaction (Eqs 6-8).
    Middle heads are introduced as a new architectural component; no separate evidence or ablation isolates their contribution.
  • Cross-head interaction mechanism (gated fusion, additive modulation)
    purpose: Integrate attention and state information across heads (Eqs 6-8).
    The mechanism is proposed as a key innovation but is not compared against alternatives or validated independently.
  • Multi-token state processing
    purpose: Use the RWKV-7 state S_t to modulate queries to capture multi-token dependencies (Eq 11).
    Described as a novel mechanism, but it consists of adding a state projection to the query; no separate evidence shows it captures multi-token dependencies beyond standard state recurrence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WuNeng: Hybrid State with Attention." pith.science (2026). https://pith.science/paper/L4VA76JS

@misc{pith2026250419191,
  author       = {Pith},
  title        = {Pith review of: WuNeng: Hybrid State with Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L4VA76JS}},
  note         = {Machine review of arXiv:2504.19191}
}
read the original abstract

The WuNeng architecture introduces a novel approach to enhancing the expressivity and power of large language models by integrating recurrent neural network (RNN)-based RWKV-7 with advanced attention mechanisms, prioritizing heightened contextual coherence over reducing KV cache size. Building upon the hybrid-head concept from Hymba, WuNeng augments standard multi-head attention with additional RWKV-7 state-driven heads, rather than replacing existing heads, to enrich the model's representational capacity. A cross-head interaction technique fosters dynamic synergy among standard, state-driven, and newly introduced middle heads, leveraging concatenation, additive modulation, and gated fusion for robust information integration. Furthermore, a multi-token state processing mechanism harnesses the continuous RWKV-7 state to capture intricate, sequence-wide dependencies, significantly boosting expressivity. Remarkably, these enhancements are achieved with minimal additional parameters, ensuring efficiency while empowering the model to excel in complex reasoning and sequence generation tasks. WuNeng sets a new standard for balancing expressivity and computational efficiency in modern neural architectures.

Figures

Figures reproduced from arXiv: 2504.19191 by the authors.

Figure 1
Figure 1. Illustration of the WuNeng hybrid-head architec [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Visualization of the multi-token state processing [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Stage 2 alignment loss curves comparing WuNeng [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 3 canonical work pages

  1. [1]

    xlstm: Ex- tended long short-term memory

    [Beck et al., 2024] Maximilian Beck, Korbinian P¨oppel, Markus Spanring, Andreas Auer, Oleksandra Prud- nikova, Michael Kopp, G ¨unter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Ex- tended long short-term memory. arXiv preprint arXiv:2405.04517,

    arXiv 2024

  2. [3]

    Alphaedit: Null-space constrained knowledge editing for language models

    [Fang et al., 2024] Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiang- nan He, and Tat-Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. arXiv preprint arXiv:2410.02355,

    arXiv 2024

  3. [4]

    Better & faster large language models via multi-token prediction

    [Gloeckle et al., 2024] Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozi `ere, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404.19737,

    arXiv 2024

  4. [5]

    Multi- token attention

    [Golovneva et al., 2025] Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. Multi- token attention. arXiv preprint arXiv:2504.00927 ,

    arXiv 2025

  5. [7]

    Mamba: Linear-time sequence modeling with selective state spaces

    [Gu and Dao, 2023] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

    arXiv 2023

  6. [9]

    Moba: Mixture of block attention for long-context llms

    [Lu et al., 2025] Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture of block attention for long-context llms. arXiv preprint arXiv:2502.13189,

    arXiv 2025

  7. [10]

    Rwkv: Reinventing rnns for the transformer era,

    [Peng et al., 2023] Bo Peng, Bo Li, Wenhan Dai, Shu- jian Zhang, Jianzhong Qi, Wenjun Zeng, and Xuewei Li. Rwkv: Reinventing rnns for the transformer era,

    work page 2023

  8. [11]

    Rwkv-7” goose” with ex- pressive dynamic state evolution

    [Peng et al., 2025] Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Haowen Hou, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, et al. Rwkv-7” goose” with ex- pressive dynamic state evolution. arXiv preprint arXiv:2503.14456,

    arXiv 2025

Show all 17 references
  1. [12]

    Retentive network: A successor to transformer for large language models

    [Sun et al., 2023] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621,

    2023 arXiv

  2. [13]

    Atten- tion is all you need

    [Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Atten- tion is all you need. Advances in neural information processing systems, 30,

    2017

  3. [15]

    Gated delta networks: Improv- ing mamba2 with delta rule

    [Yang et al., 2024b] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improv- ing mamba2 with delta rule. arXiv preprint arXiv:2412.06464,

    arXiv

  4. [16]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention

    [Yuan et al., 2025] Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089,

    2025 arXiv

  5. [17]

    Arwkv: Pretrain is not what we need, an rnn-attention-based language model born from transformer

    [Yueyu et al., 2025] Lin Yueyu, Li Zhiyuan, Peter Yue, and Liu Xiao. Arwkv: Pretrain is not what we need, an rnn-attention-based language model born from transformer. arXiv preprint arXiv:2501.15570, 2025

    2025 arXiv

  6. [2017]

    [Yang et al., 2024a] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115,

    arXiv

  7. [2023]

    Minimax-01: Scaling foundation models with lightning attention

    [Li et al., 2025] Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Con- gchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.08313,

    2025 arXiv

  8. [2024]

    Hymba: A hybrid-head architecture for small language mod- els

    [Dong et al., 2024] Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Maha- baleshwarkar, Shih-Yang Liu, Matthijs Van Keirs- bilck, Min-Hung Chen, Yoshi Suhara, et al. Hymba: A hybrid-head architecture for small language mod- els. arXiv preprint arXiv:24...

    2024 arXiv

  9. [2025]

    The llama 3 herd of models

    [Grattafiori et al., 2024] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.