{"id":"a42b4cfe-bab1-4151-81d3-09adc8acf7e7","arxiv_id":"2607.20448","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Domyn-Small is a 10B reasoning LLM that claims to deliver roughly one-third the inference tokens of Qwen3.5-9B at competitive accuracy, though results are marked as preliminary.","lead":"This paper presents Domyn-Small, a 10-billion-parameter open-weight reasoning language model built by adapting an existing base model through continued pre-training, supervised fine-tuning, and multi-stage reinforcement learning. It claims a strong accuracy-per-token efficiency advantage over similar-sized peers, but the reported benchmark numbers are explicitly preliminary.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5.1 evaluates the 'latest checkpoint', not necessarily the released v1.0 weights; if Tables 7–8 describe unpublished weights, the headline efficiency claim is unverified.","rationale":"The reader's weakest_assumption—that benchmark numbers correspond to the actual released checkpoint—is exactly the load-bearing concern. The paper's Section 5.1 caveat directly undermines the central claim without the reader having to reach outside the text. I do not see a more fundamental issue: the token-efficiency methodology, while imperfect (provider-recommended decoding, no contamination analysis), is a legitimate empirical comparison; the strongest accuracy gap on hard math is honestly disclosed. The checkpoint mismatch is the single issue that would invalidate the headline claim. Because the reader already issued CONDITIONAL with this as the core assumption, my stress-test does not change the verdict. I recommend UNCHANGED, with the concrete re-evaluation step as the required condition.","tokens_in":29250,"tokens_out":3925,"duration_ms":41015,"concrete_test":"Download the released domyn/Domyn-Small-v1.0 weights and rerun the Section 5 evaluation for at least the reasoning benchmarks (MATH-500, AIME 2025, GPQA-Diamond) with the same Domyn Swarm/vLLM setup, provider-recommended sampling, and avg@48 for math. Compare token counts and accuracies against Table 7 and Table 8. If results match within sampling error (e.g., AIME 2025 has only 30 problems, so use 95% confidence intervals), the concern is resolved. If they shift materially, the paper must either release the evaluated checkpoint or re-run all tables on v1.0.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—Domyn-Small's accuracy/token-efficiency advantage—is only meaningful if the evaluated checkpoint is the artifact users can download. Section 5.1 explicitly states: 'The Domyn-Small results presented here correspond to the latest checkpoint of the multi-environment GRPO stage (Section 4.4); values may be refreshed upon final model release.' But Section 4.4 says the released v1.0 weights are the 'best-validation checkpoint' from that run, which is not necessarily the 'latest checkpoint' at the end of training. If these differ, every headline number (e.g., reasoning token budget 2,690 vs. Qwen3.5-9B's 8,440, GPQA-Diamond 50.0, IFEval 79.9) describes a different model than the one released on Hugging Face. The paper itself creates this ambiguity: the safety section (Section 6.1) claims evaluation on the 'released Domyn-Small-v1.0 checkpoint', while Section 5.1 does not. This is not an external attack; it is an internal inconsistency in the manuscript. Until resolved, the strongest_claim is conditional at best.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Domyn-Small, a 10B-parameter open-weight reasoning model obtained by adapting Italia 10B through a five-stage pipeline: continued pre-training on 503B tokens with context extension to 32K, SFT on a 3.85M-sample instruction mixture, math-only GRPO with verifiable rewards, DPO, and a final multi-environment GRPO across five task domains. The central claim is a strong accuracy-efficiency balance in the 7–10B class: the model is reported to produce roughly one-third the tokens of Qwen3.5-9B and about 35% of OLMo-3-7B-Think's token budget on core reasoning benchmarks, with competitive GPQA-Diamond (50.0) and IFEval (79.9). The paper also releases the post-training recipe, safety evaluation, EU AI Act documentation, and the Domyn Swarm inference framework. The headline efficiency/accuracy positioning is conditional on the evaluated checkpoint being the same as the released v1.0 weights, an issue the manuscript itself raises.","tokens_in":29524,"tokens_out":6613,"duration_ms":68352,"significance":"If the reported numbers are reproducible on the released weights, this is a useful contribution: a compact open-weight reasoning model with a fully documented post-training recipe, MIT-licensed weights, and an open-source HPC inference framework. The paper is unusually transparent about data mixtures, hyperparameters, and stage-wise evaluation, and it ships a concrete artifact. However, two load-bearing issues prevent immediate acceptance: the evaluation checkpoint may not be the released v1.0 checkpoint, and no contamination analysis is provided for benchmarks that overlap with the large public SFT/CPT mixtures. The efficiency claim, which is the paper's main novelty, is only as strong as the benchmark/token-count measurements, so these issues must be resolved in revision.","major_comments":[{"comment":"The manuscript is internally inconsistent about which weights were evaluated. Section 5.1 states: \"The Domyn-Small results presented here correspond to the latest checkpoint of the multi-environment GRPO stage (Section 4.4); values may be refreshed upon final model release.\" Section 4.4 says released v1.0 weights are \"the best-validation checkpoint\" from that run, which is not necessarily the latest checkpoint. Section 6.1, by contrast, says safety evaluations were run on \"the released Domyn-Small-v1.0 checkpoint.\" If the latest checkpoint differs from the best-validation checkpoint, then Tables 7–9, Figure 3, and the abstract's efficiency numbers describe an unreleased model, not the downloadable artifact. This must be fixed by either re-evaluating the released v1.0 weights for all headline results or explicitly confirming that the latest checkpoint is identical to the released best-val","section":"Section 5.1 vs Section 4.4 and Section 6.1"},{"comment":"No contamination analysis is reported for any evaluation benchmark. The SFT mixture is assembled from 40+ public instruction datasets (Section 4.1, Table 2), and the CPT corpus includes SFT-style instruction data, web crawl, and academic text (Figure 2). The evaluation suites include public benchmarks such as GPQA-Diamond, MMLU, MMLU-Pro, IFEval, HumanEval, and BFCL. Without an n-gram overlap or holdout analysis, the reported accuracy numbers—and therefore any accuracy-efficiency conclusion—may be inflated by training-set leakage. This is a missing load-bearing support, not a matter of style. Please report contamination checks for all headline benchmarks and, if needed, re-score on decontaminated subsets.","section":"Section 4.1 / 5.1"},{"comment":"The evaluation mixes sampling protocols in a way that undermines the precision of the efficiency claim. Section 5.1 says math benchmarks use avg@48 sampling, but Table 9's footnote says \"thinking-on AIME 2025 is reported as avg@48, all other thinking-on entries are single-pass.\" For Table 7, token counts are reported as \"mean generated tokens per problem,\" but if math token counts are collected under avg@48, it is unclear whether the denominator is a problem, a single rollout, or an averaged rollout. Token counts also depend on decoding parameters, which the paper states are \"provider-recommended\" and therefore differ across models. For a paper whose central claim is a 3x token-budget advantage, the token-count definition and decoding configuration must be unambiguous and comparable across models.","section":"Section 5.1 / Table 9"},{"comment":"Section 5.1 defines the relevant deployment cost as \"cost per correct answer: accuracy weighted by the number of tokens generated,\" but Section 5.2 and the abstract instead emphasize raw token budgets. The paper never reports cost-per-correct-answer values. Because Domyn-Small trails Qwen3.5-9B substantially on AIME 2025 (35.7 vs 90.0) and LiveCodeBench (55.0 vs 86.2), raw token savings alone do not establish the claimed \"accuracy-efficiency balance.\" Please report the cost-per-correct-answer metric explicitly, or soften the claim to a token-budget comparison with accuracy reported separately.","section":"Section 5.1 / Section 5.2"}],"minor_comments":[{"comment":"The OLS regression line is fit to only five model points and is labeled a \"frontier.\" This is at best suggestive; consider removing the line or adding a clear caveat that it is not a fitted Pareto frontier.","section":"Figure 3"},{"comment":"The table header says \"Mean generated tokens per problem,\" while Figure 3's axis says \"Mean avg output tokens per sample.\" Use consistent terminology and define whether averages are computed over all rollouts or over pass@k samples.","section":"Table 7"},{"comment":"The claim of AI Act Article 53 compliance mentions a companion training-data summary artifact, but no URL or appendix pointer is given. Add a reference or state where this artifact is published.","section":"Section 6.8"},{"comment":"The reward function is described with an unusual range notation (\"{1,0,−0.8}\") and a sign convention for verifier errors. Clarifying the reward computation with a formula would improve reproducibility.","section":"Section 4.2"},{"comment":"The abstract's \"roughly one-third as many tokens as Qwen3.5-9B\" is accurate only for the reasoning grand mean in Table 7, not for code or general knowledge. Qualify the claim as \"on the reasoning benchmarks evaluated here.\"","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The checkpoint ambiguity in Section 5.1 is the most serious issue and is entirely fixable: re-run the headline evaluations on the released v1.0 weights and state the relationship explicitly. The missing contamination analysis is also fixable. I do not think the paper should be rejected, but the central efficiency claim cannot be accepted with these open questions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my take on Domyn-Small. It's a system report, not a methods paper. The genuinely new pieces are the accuracy/token-budget Pareto framing and the stage-by-stage post-training tables — those are worth reading. The pipeline itself is assembled from known techniques: CPT, SFT, GRPO, DPO, multi-env RL. No new algorithm, and the paper doesn't claim one.\n\nWhat it does well: the recipe documentation is unusually thorough. Data mixes, hyperparameters, reward functions, rollout architecture — all specified. Tables 5 and 6 trace how each stage moves the benchmarks and where it regresses. That is honest and useful. The model weights are released, and Domyn Swarm is open-sourced. Credit where due.\n\nThe soft spot is load-bearing. Section 5.1 says the reported numbers come from the 'latest checkpoint' of multi-env GRPO and may be refreshed 'upon final model release.' Section 4.4 says the released v1.0 weights are the best-validation checkpoint from that run. Those are not obviously the same thing. The safety section claims evaluation on the released checkpoint. As written, the headline efficiency numbers might describe a different model than the one on Hugging Face. That's an internal inconsistency, not an external attack. It has to be fixed before the central claim is believable.\n\nOther concerns, in proportion. No contamination analysis, despite an SFT mixture of 40+ public instruction datasets — standard issue, but it matters for every accuracy claim. The evaluation mixes avg@48 with single-pass and uses provider-recommended sampling per model, so the token comparisons are slightly apples-to-oranges. Both are fixable. The model also trails its strongest peers on hard math and long context; the paper acknowledges this.\n\nWho's this for? Practitioners in cost-sensitive or regulated deployments, and people who want a detailed record of what a post-training pipeline actually does. Anyone looking for methodological novelty will be disappointed.\n\nMy verdict: this deserves a serious referee. The checkpoint-identity ambiguity is real and must be resolved — ideally by re-running all headline numbers on the released weights and reporting error bars or at least a contamination check. If that's done, the token-efficiency analysis is a useful contribution to the 7–10B deployment literature.","headline":"Solid system report with a genuinely useful token-efficiency analysis, but the headline numbers may describe a checkpoint other than the released weights — fix that and it earns its place.","tokens_in":30046,"tokens_out":2813,"would_cite":true,"duration_ms":27613,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 10-billion-parameter open-weight reasoning model claims the best accuracy-per-token balance in its class, producing roughly one-third as many reasoning tokens as Qwen3.5-9B and about 35% of OLMo-3-7B-Think's budget while staying competiti","keywords":["reasoning language model","token efficiency","GRPO","DPO","continued pre-training","instruction following","open weights","EU AI Act"],"falsifier":"Reproduce the evaluation on the publicly released v1.0 weights using the paper's stated harness and decoding parameters, and compare mean tokens per problem on MATH-500, AIME 2025, and GPQA-Diamond. If the released model generates substantially more than the reported 2,690-token reasoning grand mean or scores far from 93.2, 35.7, and 50.0, the central claim collapses.","tokens_in":1501,"feed_emoji":"🧠","tokens_out":1519,"duration_ms":44908,"temperature":0.7,"pith_summary":"The paper introduces Domyn-Small, a 10B open-weight reasoning model built by adapting a 2024-vintage foundation model through five post-training stages: continued pre-training, supervised fine-tuning with math annealing, GRPO with verifiable rewards, DPO, and multi-environment GRPO. Its central claim is that this pipeline achieves the strongest accuracy-efficiency balance in the 7–10B parameter class, measured as mean generated tokens per problem alongside accuracy. The model produces roughly one-third the reasoning tokens of Qwen3.5-9B and 35% of OLMo-3-7B-Think's budget, while posting competitive scores on IFEval (79.9) and GPQA-Diamond (50.0). If true, this matters because regulated enterprises seeking sovereign, cost-effective deployment could serve a capable reasoning model on a modest GPU allocation at substantially lower inference cost. The paper releases weights, a recipe-level post-training description, and an open-source inference framework.","feed_headline":"A 10B reasoning model needs one-third the tokens of bigger peers","feed_subtitle":"If its accuracy-per-token claims hold, regulated enterprises get cheap sovereign inference on a single GPU.","key_machinery":"The central evaluation construct is the accuracy/token-budget Pareto frontier, with token efficiency treated as a first-class deployment metric: mean generated tokens per problem is reported alongside accuracy, and the two are read jointly as cost per correct answer. The mechanisms that move the model along this frontier are the GRPO soft length penalty (linearly decreasing from 0 to −0.25 over response length) and the SFT mixture's bias toward concise reasoning traces, supplemented by a multi-environment GRPO stage that uses leave-one-out advantage normalization and dynamic sampling to discard zero-variance groups.","core_discovery":"The paper's central claim is that a carefully sequenced post-training pipeline can convert an existing 10B base model into a reasoning model that sits on the accuracy/token-budget Pareto frontier of its class. The authors report that Domyn-Small generates a weighted grand mean of 2,690 tokens per problem on reasoning benchmarks—approximately 32% of Qwen3.5-9B's 8,440 and 35% of OLMo-3-7B-Think's 7,641—while reaching 93.2 on MATH-500, 35.7 on AIME 2025, and 50.0 on GPQA-Diamond. They attribute this efficiency to the SFT stage biasing toward shorter validated reasoning traces and to a soft length penalty in the GRPO reward. The final multi-environment GRPO stage, spanning math, code, QA, instr","pith_inferences":["If the token-efficiency advantage generalizes beyond the ten reported benchmarks, token-budget-aware post-training could become a standard objective for small-model deployment, shifting competition from raw accuracy toward cost-per-correct-answer.","The claimed balance depends on deployment granularity: Qwen3.5-9B still leads heavily on hard math (AIME 2025: 90.0 vs 35.7), so enterprises that need frontier math may not accept the trade for token savings.","A direct testable extension is to measure whether the efficiency advantage persists on multi-turn tool-calling trajectories, where the paper reports strong single-turn BFCL scores but a weak Multi-Turn split (7.0).","The 2024-vintage base model likely caps the ceiling; a modern reasoning-optimized base combined with the same pipeline could potentially preserve the token efficiency while closing the hard-math gap."],"forward_implications":["At matched accuracy thresholds on reasoning tasks, Domyn-Small consumes roughly one-third the inference compute of Qwen3.5-9B and under 35% of OLMo-3-7B-Think's, reducing latency and leaving more headroom in fixed-context agentic loops.","The dual-mode reasoning toggle yields large within-model gains on code generation (+26.8 on HumanEval, +22.2 on MBPP) and science reasoning (+10.0 on GPQA-Diamond), confirming that explicit thinking traces matter most for multi-step search and synthesis.","The multi-environment GRPO stage recovers the instruction-following and tool-use regressions introduced by math-only GRPO, ending above the SFT baseline on IFEval and within one point on BFCL.","The 32K native context extends to 128K via YaRN, but 64K retrieval accuracy drops to 29.6 on RULER, indicating that training-free extrapolation underperforms native long-context training and pointing to a future long-context CPT phase.","The full post-training recipe and weights are released, allowing others to reproduce or adapt the pipeline on different base models and domains."],"fun_headline_variants":["10B reasoning model uses one-third the tokens of larger peers","Open 10B model matches big peers with a third of the tokens","10B reasoning model: 35% of the token budget, same science scores","10B open reasoning model: token-efficient, sovereign-inference ready","10B reasoning model cuts tokens 65%, keeps GPQA-Diamond 50"],"cache_read_input_tokens":31360,"weakest_assumption_plain":"The headline benchmark numbers are measured on the latest checkpoint of the multi-environment GRPO stage, which the paper itself says may differ from the final released weights, so if the released v1.0 checkpoint produces different token counts or accuracy, the central efficiency claim describes a different model.","fun_headline_variants_meta":{"raw":{"variants":["10B reasoning model uses one-third the tokens of larger peers","Open 10B model matches big peers with a third of the tokens","10B reasoning model: 35% of the token budget, same science scores","10B open reasoning model: token-efficient, sovereign-inference ready","10B reasoning model cuts tokens 65%, keeps GPQA-Diamond 50"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000919,"raw_usage":{"total_tokens":3852,"prompt_tokens":886,"completion_tokens":2966,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":2869}},"tokens_in":630,"tokens_out":2966,"duration_ms":22241,"temperature":1.0,"reasoning_tokens":2869,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T14:03:07.953244+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the evaluation on the publicly released v1.0 weights using the paper's stated harness and decoding parameters, and compare mean tokens per problem on MATH-500, AIME 2025, and GPQA-Diamond. If the released model generates substantially more than the reported 2,690-token reasoning grand mean or scores far from 93.2, 35.7, and 50.0, the central claim collapses.","supporting_citations":[],"review_version":1}