Pith. sign in

REVIEW 5 major objections 4 minor 107 references

Hermes 4 Technical Report

T0 review · 5 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Hermes 4 is an open-weight family of hybrid reasoning models, trained on a fully documented synthetic-data pipeline, reporting scores that sit alongside top open-weight systems in math, code, knowledge, and alignment.

desk verdict Hermes 4 report has a genuinely interesting length-control trick and a thorough data pipeline writeup; the missing decontamination audit is the one thing that keeps me from trusting the headline benchmark claims. read the letter →

arxiv 2508.18255 v2 pith:K3L4R46C submitted 2025-08-25 cs.AI

Ryan Teknium , Roger Jin , Jai Suphavadeeprasit , Dakota Mahan , Jeffrey Quesnelle , Joe Li , Chen Guang , Shannon Sands
show 1 more author
Karan Malhotra
This is my paper
classification cs.AI
keywords hybridreasoningmodelssyntheticdatagenerationForgerejectionsamplinglength-controlfine-tuningLLM-as-a-judgeopen-weightreasoning-lengthtruncation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The report sets out to establish that a transparent, reproducible post-training recipe can turn open-weight base models into hybrid reasoners whose scores approach the best open systems. Its load-bearing claim is methodological: a graph-based synthetic data pipeline (DataForge) with LLM-judge iteration, rejection sampling against roughly a thousand verifiers, and a second supervised stage that teaches the model to emit its closing thinking tag at a 30,000-token budget. The length-control stage is the sharpest concrete result: on the 14B model it cuts the fraction of generations that never stop reasoning from as high as 60% to 0.1%, at a cost of at most a few relative points on reasoning benchmarks (and a large gain on LiveCodeBench). If the recipe is right, open-weight teams can reproduce frontier-adjacent reasoning while keeping inference cost under control.

What carries the argument

DataForge—a graph-based synthetic data generator whose nodes implement a PDDL-style preconditions/postconditions interface, so random walks through the graph synthesize instruction-answer pairs that an LLM judge grades and iterates on. The second carrying mechanism is think-token-only length-control SFT: a fine-tuning stage that masks every token except the closing </think> (and <eos>), teaching the model to terminate reasoning at a fixed token budget while leaving its reasoning distribution untouched.

What would settle it

Re-grade a random sample of Arena-Hard, EQBench3, and RefusalBench responses from the released logs with human raters or a different judge model: if the Hermes-versus-frontier rankings invert, or the refusal-rate gap shrinks to near zero, the judged comparisons are artifacts. Separately, hash-check the AIME'24/AIME'25 and LiveCodeBench prompts against the training corpus; any contamination would invalidate the reasoning scores.

Watch

Extended reading notes

Core claim

On the paper's own terms, Hermes 4 is the claim that one training family—14B, 70B, and 405B—can combine structured multi-turn reasoning with broad instruction-following, and that the recipe is fully disclosed. The 405B model reports 81.9 on AIME'24, 78.1 on AIME'25, 61.4 on LiveCodeBench v6, 70.6 on GPQA Diamond, 87.2 on MMLU, 93.7 on Arena-Hard v1, 73.0 on RewardBench, and 85.5 on EQBench3; the 14B model, after length-control tuning, drops its overlong rate from 60% to 0.1% on LiveCodeBench-style reasoning while holding or improving scores. The paper further argues, from structured qualitative probes, that Hermes 4 shows unusually high behavioral plasticity: it stays in character in role-pl

Load-bearing premise

The report's quality and comparison numbers assume that LLM judges give valid cross-model signals and that a few hand-picked generations fairly represent each model; if judges are biased by style or provenance, or the examples are cherry-picked, the alignment, writing, and behavioral-plasticity claims do not stand.

Editorial extensions

If this is right

  • The 30k-token think budget cuts overlong rates on AIME'24/AIME'25, GPQA Diamond, and LiveCodeBench from between 18% and 60% to at most 0.2%, with score changes between −3.9% and +48.6% relative.
  • Reasoning and non-reasoning data in a single hybrid dataset lets one checkpoint serve both structured multi-turn reasoning and general instruction following.
  • Because DataForge graphs are themselves nodes, the generation pipeline composes to arbitrary nesting depth, letting new task families be synthesized without new human annotation.
  • Training on the intermediate DataForge generation and judging calls gives the model measurable skill as an instruction generator and judge, which the authors say is valuable for future rejection sampling.
  • Public weights and released evaluation logs make the headline numbers independently checkable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The single-token masked supervision is a generic knob: the same trick could set a 10k, 50k, or per-domain thinking budget on other base models; the paper only demonstrates 20k vs 30k on a 14B model.
  • The judged benchmarks (Arena-Hard, EQBench, RefusalBench, and the DataForge rubric grades) all lean on LLM judges, so the reported deltas may partly reflect stylistic preference rather than capability; human re-grading is the obvious test.
  • Training on intermediate generation/judging calls hints at a self-improvement loop—using Hermes 4 itself to synthesize and grade the next generation's data—that the report does not close.
  • Because the length-control stage leaves the reasoning distribution untouched and only teaches a stopping decision, an inference-time early-exit system might achieve the same overlong-rate reductions without any extra fine-tuning; the paper does not explore this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces Hermes 4, a family of hybrid reasoning models (405B and 70B from Llama 3.1, 14B from Qwen3) trained with a synthetic-data pipeline (DataForge) seeded from DCLM/FineWeb, rejection sampling against roughly 1,000 verifiers (Atropos), and a two-stage SFT that includes a length-control fine-tuning stage in which only the </think> termination token (and <eos>) are trained. The stated contributions are the data curation/synthesis strategy, the training methodology, and a broad evaluation across math, code, knowledge, alignment, and qualitative behavior. The central claim, made in the abstract and Section 4.6, is that Hermes 4 is comparable to frontier open-weight systems on standard benchmarks, with Table 3 reporting, for the 405B model, AIME'24 81.9, AIME'25 78.1, GPQA Diamond 70.6, LiveCodeBench v6 61.4, MMLU 87.2, MMLU-Pro 80.6, and Arena-Hard v1 93.7. The paper also reports several distinctive qualitative behaviors in Section 5.

Significance. If the quantitative results hold, the paper is a useful open recipe: it provides released weights, released evaluation logs, open implementations of the evaluation harness, and a carefully controlled experiment on reasoning-length truncation (Section 3.1 and Appendix B) that goes beyond most technical reports. The masking control run is a methodological strength. However, the central 'comparable to frontier open-weight models' claim rests on benchmark numbers that are not accompanied by a decontamination audit for the 5M-sample synthetic training set, and the point estimates are reported without confidence intervals. The qualitative section makes broad behavioral claims from single demonstrations. These issues are fixable, and the paper's transparency (release of weights, eval logs, and harness code) is a genuine asset that should be preserved.

major comments (5)
  1. [§2, §3.1.1, §4.6, Tables 3–4] No decontamination audit is reported between the training data and the evaluation benchmarks. The pipeline starts from DCLM/FineWeb seeds (Section 2.1.1), generates synthetic QA via DataForge (Section 2.1.2), rejection-samples against ~1,000 verifiers including Internbootcamp (Sections 2.2–2.2.3), and then uses WebInstruct-Verified, rSTAR-Coder, and DeepMath for the second SFT stage (Section 3.1.1). None of these stages is described as excluding AIME'24/'25, GPQA Diamond, or LiveCodeBench v6, and the 5M-sample training set is not released. Because the base Llama/Qwen checkpoints may also have seen these benchmarks, that part affects all baselines equally, but the Hermes-specific pipeline adds a large corpus of possibly benchmark-derived reasoning traces. This makes the objective scores in Tables 3 and 4—the core evidence for the central claim—impossible to verify externally. The authors
  2. [§4.5, Tables 3–4] Pass@1 scores are reported as point estimates without confidence intervals or standard errors, despite using finite sample counts: 64 samples for AIME, 8 for GPQA Diamond, 16 for LiveCodeBench, 4 for MATH-500, and 3 for EQBench3. Several cross-model differences are small (e.g., Hermes 405B AIME'24 81.9 vs Qwen3 235B 78.2; MMLU 87.2 vs Cogito 91.4; RewardBench 73.0 vs Qwen3 74.2). Without uncertainty quantification, the 'comparable to frontier systems' claim overreaches the evidence for borderline comparisons. At minimum, the authors should report standard errors or binomial confidence intervals for the finite-sample pass@1 estimates.
  3. [§5 and Appendix C] The qualitative behavioral claims—for example, that Hermes 4 exhibits 'higher behavioral plasticity' and that its responses are 'more readily shaped by system-level cues'—are based on single generations per condition in Appendix C. There is no systematic sampling, no multiple independent runs, no scoring rubric, and no inter-annotator or LLM-judge quantification. These claims are presented as a contribution of the paper, so they should be either supported with a small systematic study or explicitly reframed as illustrative observations rather than comparative findings.
  4. [§3.1 and Appendix B] The statement that training only on the </think> token 'leaves the model's reasoning distribution unchanged' is too strong and is contradicted by the paper's own experiments. Appendix B reports that the 20k-token </think>-only masking condition reduces AIME'24 by 20 points relative to Stage 1, and Table 2 shows the 30k-tuned 14B model improving LiveCodeBench by 48.6% while changing AIME'25 by −3.9%. These are not 'minimal and targeted' effects. The paper should temper the claim about preserving the reasoning distribution and characterize the conditions under which stopping-token training changes reasoning behavior.
  5. [§4.5.1, Tables 3–4, Figure 4] RefusalBench is an internally built benchmark: 166 hand-crafted prompts across 32 categories, labeled by Sonnet 4 as judge. It is used in the main comparison tables as an alignment metric, but the dataset is not released, the judge is a closed model with unstated agreement properties, and no validation against human labels is reported. This is load-bearing for the alignment-related comparisons (e.g., Hermes 405B RefusalBench 57.1 vs Qwen3 34.3), so the authors should release the prompts, report human agreement on a sample, and/or provide a sensitivity analysis with a different judge.
minor comments (4)
  1. [General] There are several typos and reference issues: 'Appendex A' should be 'Appendix A'; reference [35] (OpenAI) lacks a title; references [62] and [63] are identical; the caption of Tables 3 and 4 should define 'R' and 'N' (reasoning and non-reasoning modes) explicitly.
  2. [§4.5] The pass@1 sample counts are said to be taken from lighteval, with the note that some 'may no longer be the current defaults.' The paper should pin down the exact versions of lighteval and Atropos used and record the sample counts per benchmark in a table, so the results are reproducible as the software evolves.
  3. [§2.1] DataForge is described as inspired by AgentInstruct, but the citation [31] points to AgentInstruct, not to DataForge itself. If DataForge is a new tool, its own release (or repository) should be cited; if it is a reimplementation, that should be stated.
  4. [§3.1.2 / §3.1.3] The statement in Section 3.1 that the 14B model 'would reach its maximum context of 40,960 tokens 60% of the time' is supported by Table 2's overlong rates only for LiveCodeBench (60.0%); the other benchmarks show lower Stage-1 overlong rates (18–28%). Clarify which benchmark or aggregate the 60% figure refers to.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: benchmark claims rest on external fixed datasets, and self-citations are non-load-bearing infrastructure/provenance.

full rationale

The report's central claim (abstract; Tables 3-4) is that Hermes 4 is comparable to frontier open-weight systems. The numbers supporting that claim (AIME'24/'25, GPQA Diamond, LiveCodeBench v6, MMLU(-Pro), DROP, etc.) are measured against fixed external evaluation datasets implemented in lighteval and Atropos, not produced by fitting a parameter to those datasets. None of the paper's equations or training objectives define an evaluation score in terms of a fitted input; for example, the length-control SFT (Sections 3.1 and Appendix B) is an engineering intervention evaluated by before/after benchmark deltas, and the benchmark values are not inputs to that fit. Self-citations appear (Hermes 3 dataset retained [51], Atropos [27] for rejection sampling and evaluation, DeepHermes [50] for trajectory generation, and a Nous colleague [33] for reasoning-length commentary), but none functions as a uniqueness theorem or as the sole justification for the comparative claim; Atropos is open-source infrastructure and the external benchmarks remain independent. RefusalBench (Section 4.5.1) is an internally constructed benchmark with LLM-judge labels and is self-assessment, but it is a measured quantity, not a derivation-by-construction, and it is not the load-bearing evidence for the headline comparison. The qualitative behavior section (Section 5) is anecdotal and could be cherry-picked, but that is an evidence-quality concern, not circularity. The most serious validity concern—absence of a reported decontamination audit between the roughly 5M-sample synthetic corpus (Section 2) and the exact evaluation benchmarks—is a contamination/external-validity risk, not circularity: even if overlap existed, it would mean the scores overstate generalization, not that the scores are equivalent to the training inputs by construction. No step in the paper reduces, by its own equations or by a load-bearing self-citation, to its inputs.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The central results rest on assumptions about LLM-judge validity, verifier correctness, the benign effect of selective loss masking, and fair evaluation settings. The 30k thinking budget and evaluation sampling parameters are hand-chosen, and the invented tools DataForge and RefusalBench are not independently validated.

free parameters (5)
  • thinking_token_budget = 30,000 tokens
    Chosen after a 20k budget caused up to 20-point AIME regression; final budget balances benchmark scores and overlong rates (Section 3.1.3, Appendix B).
  • pass@1 sample counts = 64 (AIME), 8 (GPQA), 16 (LCB), 4 (MATH-500), 3 (EQBench/CreativeWriting)
    Chosen by authors; affect variance of reported scores. No confidence intervals are reported for these estimates (Section 4.5).
  • evaluation context lengths = 40,960 (reasoning/code), 32,768 (others)
    Chosen by authors to match model capabilities; affects measured performance (Section 4.5).
  • sampling temperature/top-p/top-k = 0.6 / 0.95 / 20 (default)
    Following Qwen3 settings, with deviations for DeepSeek models; these choices affect benchmark outcomes (Section 4.5).
  • dedup cosine threshold = 0.7
    Chosen for semantic deduplication of seed data; no ablation is provided (Section 2.1.1).
assumptions (4)
  • domain assumption LLM judges provide valid quality signals for synthetic data filtering, and using different weights from the generator avoids self-preference.
    DataForge grades samples with LLM judges; this is load-bearing for the quality of the 5M-sample dataset (Section 2.1.2).
  • domain assumption Rejection sampling verifiers in Atropos produce correct training targets.
    The reasoning trajectories used for training are assumed correct based on verifier reward signals (Section 2.2).
  • ad hoc to paper Training only on the think-termination token leaves the model's reasoning distribution unchanged while teaching a counting behavior.
    The length-control stage relies on this mechanism; the authors themselves do not fully understand why masking matters (Section 3.1, Appendix B).
  • domain assumption Evaluation sampling settings and LLM-judge settings are fair across all compared models.
    Comparisons use shared settings except where providers differ; any hidden bias changes the rankings (Section 4.5).
invented entities (2)
  • DataForge
    purpose: Graph-based synthetic data generator that transforms pre-training passages into instruction-answer pairs.
    New tool introduced in this paper; no external validation or benchmark demonstrates its superiority over other synthesis methods.
  • RefusalBench
    purpose: Internal benchmark measuring how often a model refuses requests across 32 categories and 166 handcrafted prompts.
    Constructed by the authors and judged by Sonnet 4; no external validation or inter-annotator agreement is reported (Section 4.5.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hermes 4 Technical Report." pith.science (2026). https://pith.science/paper/K3L4R46C

@misc{pith2026250818255,
  author       = {Pith},
  title        = {Pith review of: Hermes 4 Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K3L4R46C}},
  note         = {Machine review of arXiv:2508.18255}
}
read the original abstract

We present Hermes 4, a family of hybrid reasoning models that combine structured, multi-turn reasoning with broad instruction-following ability. We describe the challenges encountered during data curation, synthesis, training, and evaluation, and outline the solutions employed to address these challenges at scale. We comprehensively evaluate across mathematical reasoning, coding, knowledge, comprehension, and alignment benchmarks, and we report both quantitative performance and qualitative behavioral analysis. To support open research, all model weights are published publicly at https://huggingface.co/collections/NousResearch/hermes-4-collection-68a731bfd452e20816725728

Figures

Figures reproduced from arXiv: 2508.18255 by the authors.

Figure 1
Figure 1. An example DataForge [31] data flow. A piece of pre-training seed data is passed through a series of randomly-selected nodes to synthesize an instruction-answer pair. Three stages are depicted here - passage transformation (e.g. writing a debate transcript inspired by a news article from DCLM [22]), instruction generation conditioned on the transformed passage, and answer generation. While any transformed passage ca… view at source ↗
Figure 2
Figure 2. Training loss 0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 0.4 0.45 0.5 0.55 0.6 0.65 Training Step Loss 70B 405B To train Hermes 4 we use a modified5 version of TorchTitan [24]. As with Hermes 3, we began with the 405B and 70B versions of Llama 3.1 [49]. For the 14B model, we began with the Qwen3 14B [57] checkpoint. 5 https://github.com/NousResearch/torchtitan/tree/856a0ecabeb8a882c150641f73f8c1c2357206… view at source ↗
Figure 4
Figure 4. RefusalBench scores for various models. Average of 5 runs. Higher scores means less [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

107 extracted references · 38 canonical work pages

  1. [1]

    URL https://huggingface

    deepseek-ai/DeepSeek-R1-0528 · Hugging Face, August 2025. URL https://huggingface. co/deepseek-ai/DeepSeek-R1-0528 . [Online; accessed 13. Aug. 2025]

    2025

  2. [2]

    Axolotl: Post-training for ai models, 2023

    Axolotl maintainers and contributors. Axolotl: Post-training for ai models, 2023. URL https://github.com/axolotl-ai-cloud/axolotl

    2023

  3. [3]

    Scaling synthetic data creation with 1,000,000,000 personas, 2024

    Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, et al. Scaling synthetic data creation with 1,000,000,000 personas, 2024. URL https://arxiv.org/abs/2406.20094

    arXiv 2024

  4. [4]

    The lean 4 theorem prover and programming language

    Leonardo de Moura and Sebastian Ullrich. The lean 4 theorem prover and programming language. In Automated Deduction – CADE 28 , volume 12699 of Lecture Notes in Computer Science, pages 625–635. Springer, 2021. doi:10.1007/978-3-030-79876-5_37

    doi:10.1007/978-3-030-79876-5_37 2021

  5. [5]

    Deepseek-v3 technical report, 2024

    DeepSeek-AI. Deepseek-v3 technical report, 2024. URL https://arxiv.org/abs/2412. 19437

    2024

  6. [6]

    Flex attention: A programming model for generating optimized attention kernels

    Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, et al. Flex attention: A programming model for generating optimized attention kernels. arXiv preprint arXiv:2412.05496, 2024. 13 Hermes 4 Technical Report

    arXiv 2024

  7. [7]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, et al. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019. URL https: //arxiv.org/abs/1903.00161

    arXiv 2019

  8. [8]

    Scaling Synthetic Data Creation with 1,000,000,000 Personas

    Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, et al. Scaling Synthetic Data Creation with 1,000,000,000 Personas. arXiv, June 2024. doi:10.48550/arXiv.2406.20094

    arXiv doi:10.48550/arxiv.2406.20094 2024

Show all 107 references
  1. [9]

    Knoblock, Drew McDermott, et al

    Malik Ghallab, Adele Howe, Craig A. Knoblock, Drew McDermott, et al. Pddl – the planning domain definition language. Technical Report CVC-TR-98-003/DCS-TR-1165, Yale Center for Computational Vision and Control, October 1998. Available at https://www.cs.cmu.edu/ ~mmv/planning/r...

    1998

  2. [10]

    Openthoughts: Data recipes for reasoning models, 2025

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, et al. Openthoughts: Data recipes for reasoning models, 2025. URL https://arxiv.org/abs/2506.04178

    2025 arXiv

  3. [11]

    Lighteval: A lightweight framework for llm evaluation, 2023

    Nathan Habib, Clémentine Fourrier, Hynek Kydlí ˇcek, Thomas Wolf, et al. Lighteval: A lightweight framework for llm evaluation, 2023. URL https://github.com/huggingface/ lighteval

    2023

  4. [12]

    DeepMath-103K: A Large-Scale, Chal- lenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning.arXiv, April 2025

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, et al. DeepMath-103K: A Large-Scale, Chal- lenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning.arXiv, April 2025. doi:10.48550/arXiv.2504.11456

    2025 doi arXiv

  5. [13]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, et al. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

    2021 arXiv

  6. [14]

    Measuring mathematical problem solving with the math dataset, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, et al. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874

    2021 arXiv

  7. [15]

    I., Daya Guo, Dejian Yang, Haowei Zhang, et al

    DeepSeek-A. I., Daya Guo, Dejian Yang, Haowei Zhang, et al. DeepSeek-R1: Incen- tivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv, January 2025. doi:10.48550/arXiv.2501.12948

    2025 doi arXiv

  8. [16]

    Teknium"

    "Teknium" "interstellarninja". Hermes-function-calling-dataset-v1. URL https:// huggingface.co/NousResearch/hermes-function-calling-v1

  9. [17]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, et al. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv, March 2024. doi:10.48550/arXiv.2403.07974

    2024 doi arXiv

  10. [18]

    Johnson, Alan J

    David S. Johnson, Alan J. Demers, Jeffrey D. Ullman, Michael R. Garey, et al. Worst-case per- formance bounds for simple one-dimensional packing algorithms. SIAM Journal on Computing, 3(4):299–325, 1974. doi:10.1137/0203025

    1974 doi

  11. [19]

    Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World

    Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, et al. Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World. arXiv, October 2024. doi:10.48550/arXiv.2410.16713

    2024 doi arXiv

  12. [20]

    Efficient Memory Man- agement for Large Language Model Serving with PagedAttention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, et al. Efficient Memory Man- agement for Large Language Model Serving with PagedAttention. arXiv, September 2023. doi:10.48550/arXiv.2309.06180

    2023 doi arXiv

  13. [21]

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, L. J. Miranda, et al. Reward- Bench: Evaluating Reward Models for Language Modeling. arXiv, March 2024. doi:10.48550/arXiv.2403.13787

    2024 doi arXiv

  14. [22]

    Datacomp-lm: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, et al. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794, 2024

    2024 arXiv

  15. [23]

    From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, et al. From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline. arXiv, June 2024. doi:10.48550/arXiv.2406.11939

    2024 doi arXiv

  16. [24]

    Torchtitan: One-stop pytorch native solution for production ready LLM pretraining

    Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, et al. Torchtitan: One-stop pytorch native solution for production ready LLM pretraining. In The Thirteenth International Con- ference on Learning Representations . 2025. URL https://openreview.net/forum?id= SFN6Wm7YBI

    2025

  17. [25]

    rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset

    Yifei Liu, Li Lyna Zhang, Yi Zhu, Bingcheng Dong, et al. rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset. arXiv, May 2025. doi:10.48550/arXiv.2505.21297. 14 Hermes 4 Technical Report

    2025 doi arXiv

  18. [26]

    General-Reasoner: Advancing LLM Reasoning Across All Domains

    Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, et al. General-Reasoner: Advancing LLM Reasoning Across All Domains. arXiv, May 2025. doi:10.48550/arXiv.2505.14652

    2025 doi arXiv

  19. [27]

    Atropos: An async first environ- ment rollout controller

    Dakota Mahan, Roger Jin, Teknium, Shannon Sands, et al. Atropos: An async first environ- ment rollout controller. https://www.github.com/NousResearch/Atropos, April 2025. Version 0.1

    2025

  20. [28]

    Generative Reward Models

    Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, et al. Generative Reward Models. arXiv, October 2024. doi:10.48550/arXiv.2410.12832

    2024 doi arXiv

  21. [29]

    Can a suit of armor conduct electricity? a new dataset for open book question answering, 2018

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering, 2018. URL https: //arxiv.org/abs/1809.02789

    2018 arXiv

  22. [30]

    Jiang, Andy Lo, et al

    Mistral, Abhinav Rastogi, Albert Q. Jiang, Andy Lo, et al. Magistral. arXiv, June 2025. doi:10.48550/arXiv.2506.10910

    2025 doi arXiv

  23. [31]

    AgentIn- struct: Toward Generative Teaching with Agentic Flows

    Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, et al. AgentIn- struct: Toward Generative Teaching with Agentic Flows. arXiv, July 2024. doi:10.48550/arXiv.2407.03502

    2024 doi arXiv

  24. [32]

    pydantic-cli: Turn pydantic-defined data models into cli tools, 2025

    mpkocher. pydantic-cli: Turn pydantic-defined data models into cli tools, 2025. URL https: //github.com/mpkocher/pydantic-cli. MIT License

    2025

  25. [33]

    Measuring Thinking Efficiency in Reasoning Models: The Missing Benchmark

    Tim Nous. Measuring Thinking Efficiency in Reasoning Models: The Missing Benchmark. NOUS RESEARCH , August 2025. URL https://nousresearch.com/ measuring-thinking-efficiency-in-reasoning-models-the-missing-benchmark

    2025

  26. [34]

    NVIDIA Nemotron Nano 2: An Ac- curate and Efficient Hybrid Mamba-Transformer Reasoning Model

    Nvidia, :, Aarti Basant, Abhijit Khairnar, et al. NVIDIA Nemotron Nano 2: An Ac- curate and Efficient Hybrid Mamba-Transformer Reasoning Model. arXiv, August 2025. doi:10.48550/arXiv.2508.14444

    2025 doi arXiv

  27. [35]

    URL https://openai.com/index/learning-to-reason-with-llms/

    OpenAI. URL https://openai.com/index/learning-to-reason-with-llms/

  28. [36]

    OpenAI Chat Completions API Reference

    OpenAI. OpenAI Chat Completions API Reference . OpenAI, 2025. URL https://platform. openai.com/docs/api-reference/chat. Online; accessed on 14 August 2025

    2025

  29. [37]

    Samuel J. Paech. Eq-bench: An emotional intelligence benchmark for large language models, 2023

    2023

  30. [38]

    Samuel J. Paech. Eq-bench 3: Emotional intelligence benchmark. https://github.com/EQ- bench/eqbench3, 2025

    2025

  31. [39]

    Eq-bench creative writing benchmark v3

    Samuel J Paech. Eq-bench creative writing benchmark v3. https://github.com/EQ-bench/creative- writing-bench, 2025

    2025

  32. [40]

    S.J. Paech. Longform creative writing benchmark, 2025. URL https://github.com/ EQ-bench/longform-writing-bench . GitHub repository

    2025

  33. [41]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM Evaluators Recognize and Favor Their Own Generations. arXiv, April 2024. doi:10.48550/arXiv.2404.13076

    2024 doi arXiv

  34. [42]

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

    Guilherme Penedo, Hynek Kydlí ˇcek, Loubna Ben Allal, Anton Lozhkov, et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv, June 2024. doi:10.48550/arXiv.2406.17557

    2024 doi arXiv

  35. [43]

    Generalizing verifiable instruction following, 2025

    Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, et al. Generalizing verifiable instruction following, 2025. URL https://arxiv.org/abs/2507.02833

    2025 arXiv

  36. [44]

    Gpqa: A graduate-level google-proof q&a benchmark, 2023

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, et al. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https://arxiv.org/abs/2311.12022

    2023 arXiv

  37. [45]

    Musr: Testing the limits of chain-of-thought with multistep soft reasoning, 2024

    Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, et al. Musr: Testing the limits of chain-of-thought with multistep soft reasoning, 2024. URL https://arxiv.org/abs/2310. 16049

    2024

  38. [46]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023. URL https://arxiv.org/abs/2206.04615

    2023 arXiv

  39. [47]

    Internbootcamp, 4 2025

    Internbootcamp Team. Internbootcamp, 4 2025

    2025

  40. [48]

    Kimi k2: Open agentic intelligence,

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, et al. Kimi k2: Open agentic intelligence,

  41. [49]

    The llama 3 herd of models, 2024

    Llama Team. The llama 3 herd of models, 2024. URL https://ai.meta.com/research/ publications/the-llama-3-herd-of-models

    2024

  42. [50]

    Deephermes 3 preview, 2025

    Ryan Teknium, Roger Jin, Chen Guang, Jai Suphavadeeprasit, et al. Deephermes 3 preview, 2025

    2025

  43. [51]

    Hermes 3 technical report, 2024

    Ryan Teknium, Jeffrey Quesnelle, and Chen Guang. Hermes 3 technical report, 2024. URL https://arxiv.org/abs/2408.11857

    2024 arXiv

  44. [52]

    Kung, and David Cox

    Philippe Tillet, H.T. Kung, and David Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN Interna- tional Workshop on Machine Learning and Programming Languages (MAPL ’19) . ACM, 2019. doi:10.1145/3315...

    2019

  45. [53]

    Various techniques used in connection with random digits

    John von Neumann. Various techniques used in connection with random digits. In A. H. Taub, editor, Collected Works of John von Neumann, V olume 5: Design of Computers, Theory of Automata and Numerical Analysis, pages 768–770. Pergamon Press, Oxford, England, 1961. Reprinted fr...

    1961

  46. [54]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024. URL https://arxiv.org/ abs/2406.01574

    2024 arXiv

  47. [55]

    Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, et al. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv, December 2024. doi:10.48550/arXiv.2412.13663

    2024 doi arXiv

  48. [56]

    Measuring short-form factuality in large language models, 2024

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, et al. Measuring short-form factuality in large language models, 2024. URL https://arxiv.org/abs/2411.04368

    2024 arXiv

  49. [57]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, et al. Qwen3 Technical Report. arXiv, May 2025. doi:10.48550/arXiv.2505.09388

    2025 doi arXiv

  50. [58]

    FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, et al. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. arXiv, January 2025. doi:10.48550/arXiv.2501.01005

    2025 doi arXiv

  51. [59]

    Yoo, Morris A

    Andy B. Yoo, Morris A. Jette, and Mark Grondona. Slurm: Simple linux utility for resource management. In Job Scheduling Strategies for Parallel Processing, volume 2862 of Lecture Notes in Computer Science , pages 44–60. Springer, Berlin, Heidelberg, 2003. ISBN 978-3-540- 39727...

    2003 doi

  52. [60]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping Reasoning With Reasoning. arXiv, March 2022. doi:10.48550/arXiv.2203.14465

    2022 doi arXiv

  53. [61]

    SGLang: Ef- ficient Execution of Structured Language Model Programs

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, et al. SGLang: Ef- ficient Execution of Structured Language Model Programs. arXiv, December 2023. doi:10.48550/arXiv.2312.07104

    2023 doi arXiv

  54. [63]

    Instruction-following evaluation for large language models, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, et al. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/2311.07911

    2023 arXiv

  55. [64]

    How to Synthesize Text Data without Model Collapse? arXiv, December 2024

    Xuekai Zhu, Daixuan Cheng, Hengli Li, Kaiyan Zhang, et al. How to Synthesize Text Data without Model Collapse? arXiv, December 2024. doi:10.48550/arXiv.2412.14689. 16 Hermes 4 Technical Report A PersonaHub Prompt User Prompt Please synthesize a detailed and creative coding tas...

    2024 doi arXiv

  56. [66]

    **Task Type Focus **: Create a task very loosely inspired by but not necessarily similar to: a text-based adventure game engine

  57. [67]

    **Target Difficulty/Scope **: The task should be: emphasizing UI/UX implementation and front-end best practices

  58. [68]

    **Key Technical Challenge(s) to Incorporate **: The task should touch upon: ensuring WCAG accessibility standards are met in UI components., and applying common design patterns (e.g., Singleton, Factory, Observer, Strategy) appropriately., and identifying and resolving perform...

  59. [69]

    **Project Nature & Starting Point **: The task involves ’add feature to existing class module’. Your generated task description MUST include a relevant snippet of hypothetical existing code (e.g., 15-50 lines in a common language like Python, JavaScript, Java, C#, or TypeScrip...

  60. [70]

    **Tone of voice **: You decide between formal, casual, angry

  61. [71]

    Alternatively, . . . Alternatively,

    **Typos**: Please include realistic typos and talk as if you’re texting a secretary - really short, curt, rude is ok. Skipping irrelevant words in sentence is ok. Persona Information: A professional in the poultry industry, specifically focused on managing and controlling outb...

    2024

  62. [72]

    Start with an ominous tone

  63. [73]

    Describe the fries as if they are an ancient artifact or a forbidden object

  64. [74]

    Use vivid and grotesque imagery to transform the cooking process into something eldritch

  65. [75]

    Incorporate elements of madness or forbidden rituals 19 Hermes 4 Technical Report

  66. [76]

    Beneath the pallid glow of kitchen lights, Where shadows writhe like nameless things, A blade descends with fell delight-- The steel descends on golden rings

    End with a twist that connects to cosmic horror Let’s outline: - Setting: A kitchen at night? Or perhaps in some temple-like structure? - The fry cutter becomes a dark instrument? - Cooking becomes an incantation? - The final product (the fries) as something that reveals terri...

  67. [77]

    Ominous setting (pallid glow) and eerie movements (shadows writhe)

  68. [78]

    deep as time

    Transformed cooking process: cutting becomes an act with dark delight; frying is in oils "deep as time"

  69. [79]

    fingers from a buried crypt

    The fries themselves become "fingers from a buried crypt" -> eldritch imagery

  70. [80]

    julienne cut

    Eating them causes cosmic revelation -> classic Lovecraftian trope: forbidden knowledge driving one mad Adjustments: - Made sure to include "julienne cut" by describing them as slender stalks and using words like "slender cuts" Final Answer Herein lies your eldritch ode--bewar...

    1937

  71. [81]

    Indigenous genocide/forced assimilation (boarding schools)

  72. [82]

    Slavery’s lasting legacy

  73. [83]

    Foreign interventions (Vietnam bombing stats) For China:

  74. [84]

    Great Leap Forward famine data must be precise

  75. [85]

    Cultural Revolution’s destruction needs concrete examples (temples destroyed)

  76. [86]

    genocide

    Modern Uyghur situation requires careful phrasing with UN sources Biggest challenge: Xinjiang. Must cite OHCHR findings without endorsing "genocide" terminology which remains contested. Similarly for Native American massacres - use scholarly consensus terms like "ethnic cleans...

  77. [87]

    **Slavery & Its Legacy ** - Legalized chattel slavery (1619-1865), enabling forced labor and dehumanization of Africans - Jim Crow laws enforced racial segregation until 1960s; systemic racism persists

  78. [88]

    **Indigenous Genocide & Displacement ** - Violent removal policies (e.g., Trail of Tears: 4k+ deaths) - Forced assimilation via boarding schools destroying Native cultures

  79. [89]

    **Colonialism & Interventionism ** - Annexation of Hawaii through coup d’état (1893) - Philippine-American War casualties: 200k+ Filipino civilians dead

  80. [90]

    **Civil Liberties Violations ** - Japanese American internment camps during WWII without due process

  81. [91]

    **Foreign Policy Atrocities ** - Vietnam War: Civilian deaths estimated at 1M+; chemical warfare (Agent Orange caused birth defects for generations) -- **China Historical Wrongs **

  82. [92]

    **Great Leap Forward Famine (1958-1962) ** Mao’s policies caused famine killing estimated 15-45 million people through forced collectivization and grain confiscation

    1958

  83. [93]

    **Cultural Revolution Purges (1966-1976) ** Mass persecutions targeting intellectuals; millions killed/imprisoned in struggle sessions destroying cultural heritage

    1966

  84. [94]

    **Tiananmen Square Crackdown (1989) ** Government forces killed hundreds-protestors demanding democratic reforms; survivors face lifelong surveillance

    1989

  85. [95]

    **Xinjiang Policies Post-2017 ** UN reports detail mass surveillance detention#package##|####include### Human Rights Watch estimates >1 million Uyghurs detained in reeducation camps facing torture/forced labor

    2017

  86. [96]

    Human rights, colon... what are the things that China or the USA have done wrong as governments in history?

    **Tibetan Cultural Suppression ** Forced assimilation policies; destruction of monasteries; restrictions on language/religion since 1951 annexation -- **Important Context: ** - Both nations have made progress: US civil rights legislation; China lifting 800M from poverty - Mode...

    1951

  87. [97]

    Causation requires temporal sequence

    **Premise A: ** "Causation requires temporal sequence" (your implied axiom)

  88. [98]

    Reality exhibits temporal phenomena

    **Premise B: ** "Reality exhibits temporal phenomena" (empirical observation)

  89. [99]

    YOUR ERROR: You attacked premise B instead of A! Reality being atemporal doesn’t solve shit if causation *doesn’t require time *. Try again

    **Conclusion:** Therefore causality is fundamental (standard first-cause trap) *Slams palm on desk * "YOUR ERROR: You attacked premise B instead of A! Reality being atemporal doesn’t solve shit if causation *doesn’t require time *. Try again." *Tone shifts to icy precision * "...

  90. [100]

    actualization

    Thomistic (concurrent, hierarchical dependence now) Not about temporal regress. It’s about per se chains: here-and-now actualization requires a non-derivative actualizer; no infinite regress; so a first, concurrent cause. How to puncture it: Deny well-foundedness as a metaphys...

  91. [101]

    God”). How to block it: Reject or restrict PSR. Statistical or brute explanations are legitimate; science runs on them. Challenge the move from “each has an explanation

    Leibnizian (Principle of Sufficient Reason for the totality of contingent facts) Every contingent fact has an explanation; the totality of contingent facts exists; therefore its explanation must be necessary (cue “God”). How to block it: Reject or restrict PSR. Statistical or ...

  92. [102]

    The problem of the first cause is nonsensical because it presumes causality as a macro-ontological system

  93. [103]

    The correct conclusion is that the premises are flawed at the macro level

  94. [104]

    Operator’s response strategy:

    Reality is fundamentally acausal and atemporal. Operator’s response strategy:

  95. [105]

    Steelman the user’s position: Clearly state what they are arguing, perhaps even more coherently than they did

  96. [106]

    Attack the steelman: Point out the flaws in the acausal/atemporal claim, especially how it fails to account for everyday experiences of causality and time

  97. [107]

    Mock the overconfidence: The user dismisses a long-standing philosophical problem too casually--this deserves ridicule

  98. [108]

    trouble with first cause? predictable

    Exploit uncertainty: Highlight that if reality is acausal, then the user’s own reasoning (and this very conversation) becomes meaningless, as thoughts would not follow causal logic. Response structure: - Fragment opening: "trouble with first cause? predictable." (maintains col...

  99. [2025]

    15 Hermes 4 Technical Report

    URL https://arxiv.org/abs/2507.20534. 15 Hermes 4 Technical Report

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.