{"id":"47c3e181-57f8-477a-90fc-645c94361190","arxiv_id":"2506.22760","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A 4B model trained with reinforcement learning and no standard next-token fine-tuning reportedly reaches 83.2% on SimpleQA when given web-search tools, beating the 78.2% reported for DeepSeek-671B with the same tool setup.","lead":"Jan-nano, a 4B parameter model built from Qwen3-4B, is reported to score 83.2% on the SimpleQA factuality benchmark when connected to a Google search tool via MCP. The report argues that small, search-specialized models can outperform far larger models on tool-augmented question answering while running on consumer hardware.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 83.2% headline is not tied to a documented full-SimpleQA protocol; Table 4 labels only the 80.7% run as \"full,\" and the Qwen3-4B baseline has no same-setup evidence, so the central comparison is unverifiable.","rationale":"Load-bearing because the abstract and conclusion rest entirely on the 83.2% number and the 24-point delta over a 4B baseline; the novel training method is secondary to that empirical comparison. The manuscript's own tables create ambiguity: the only row explicitly marked \"Full\" is 80.7%, while the 83.2% headline appears without set specification in Table 1. The 10% subset in Table 2 is the only place a subset is mentioned, so a reader cannot rule out that the headline came from a favorable slice or from a run selected post hoc. The baseline comparison is equally fragile because MCP search behavior is known to shift scores by several points, as the paper itself says, and no protocol details are given for the 59.2% baseline. Agreement with the reader is partial: the reader focuses on subset representativeness, while I see the deeper issue as an unspecified evaluation protocol covering set labeling, run selection, and baseline comparability. The concrete reproduction test resolves the concern because the weights are public; if the number reproduces on the full benchmark with the same harness, the central claim becomes credible, and if it does not, the claim fails.","tokens_in":61580,"tokens_out":7591,"duration_ms":76126,"concrete_test":"Run a fixed public harness on the released Jan-nano-128k weights using the exact Appendix A.2 LangGraph MCP ReAct configuration and mcp-server-serper: greedy decoding, all SimpleQA questions, recording the Table 2 10% subset separately and reporting full-set and held-out-90% accuracy. Then run Qwen3-4B through the identical harness. If the full-set Jan-nano-128k score is not 83.2% within binomial sampling error, or if Qwen3-4B does not reproduce about 59.2%, the headline delta and the superiority claim fail.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is an empirical number: 83.2% SimpleQA with MCP from a 4B model. The report never states which set produced that number. Section 5.1/Table 2 introduces a 10% SimpleQA subset used to compare Think vs No-Think modes, but Section 2/Table 1 reports the 83.2% without saying whether it is full SimpleQA or that same subset, and Appendix A.2/Table 4 explicitly labels only Jan-nano (LangGraph MCP) at 80.7% as \"SimpleQA (Full).\" The same paragraph says the authors \"opted for the more conservative 83.2% result,\" which implies a selection among runs or implementations that is not specified. For the claimed 24-point jump over Qwen3-4B (59.2%), no evidence is given that the baseline was run with the identical mcp-server-serper, LangGraph ReAct template, grading, and temperature; Section 2 itself warns that different MCP implementations and evaluation setups affect direct comparisons, yet the baseline is not subjected to that caveat. If the 83.2% was measured on the 10% mode-selection slice, or if the baseline used a different tool stack, the headline conclusion—that a small specialized model beats much larger models on tool-augmented factuality—does not follow from the report. Public weights are a positive step, but they do not by themselves document the protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Jan-nano, a 4B-parameter model fine-tuned from Qwen3-4B using a three-stage Reinforcement Learning with Verifiable Rewards (RLVR) pipeline that the authors claim completely avoids SFT. The central empirical claim is that Jan-nano-128k with MCP achieves 83.2% on SimpleQA, a 24-point improvement over the Qwen3-4B MCP baseline (59.2%) and higher than DeepSeek-671B with MCP (78.2%). The report also claims that disabling the thinking mode (\"force non-thinking\") prevents overthinking and improves tool-use performance, and it releases several public model variants on Hugging Face.","tokens_in":61934,"tokens_out":5292,"duration_ms":103251,"significance":"If the headline result were fully supported, it would be a notable demonstration that a small, tool-specialized model can outperform models orders of magnitude larger on tool-augmented factuality, and the public release of four model variants is a practical strength. The authors' explicit choice of a conservative MCP setup, rather than a forked server they believe could exceed 85%, is also a positive sign. However, the evaluation protocol is not documented well enough to verify the central claim: the exact test set, grading rule, answer-matching procedure, run count, and baseline setup are missing or ambiguous. As written, the paper's significance cannot be assessed reliably.","major_comments":[{"comment":"The 83.2% headline in Table 1 is not tied to a documented evaluation protocol. Appendix A.2/Table 4 labels only Jan-nano (LangGraph MCP) at 80.7% as \"SimpleQA (Full)\", while Table 1 lists Jan-nano-128k (MCP) at 83.2% without stating whether it was measured on the full SimpleQA benchmark, the 10% subset introduced in Table 2, or some other slice. The text in §2 says the authors \"opted for the more conservative 83.2% result\", which implies a selection among runs or implementations that is never specified. Without a precise statement of the test set, grading rule, answer-matching criterion, temperature, and number of runs, the central result is not verifiable.","section":"§2, Table 1 and Appendix A.2, Table 4"},{"comment":"The force-non-thinking design choice was selected using a 10% SimpleQA subset, but the report does not describe how this subset was sampled, whether it is disjoint from the final evaluation, or whether the 83.2% figure was obtained on the full benchmark or on that same subset. Because the final evaluation uses the same benchmark family, the headline number is not an independent held-out evaluation unless overlap is explicitly ruled out. The authors should state the subset sampling procedure, report both full-set and subset results, and clarify all overlaps.","section":"§5.1, Table 2 and §2"},{"comment":"The Qwen3-4B baseline (59.2%) is central to the claimed 24-point improvement, but no evidence is given that this baseline was evaluated with the identical mcp-server-serper, LangGraph ReAct template, grading procedure, temperature, and test subset as Jan-nano. The report itself cautions in §2 that different MCP implementations and evaluation setups affect direct comparisons, yet the baseline is not subjected to that caveat. Without a same-setup comparison, the improvement claim does not follow from the data presented.","section":"§2, Table 1"},{"comment":"The claim that the RLVR system \"completely eliminates reliance on next token prediction training (SFT)\" is not substantiated. Section 4 gives no reward weights for the three stages, no DAPO hyperparameters, no number of RL steps, no convergence criteria, and no ablation comparing against an SFT-initialized control. The phrasing is also imprecise because the model is initialized from Qwen3-4B, which was itself trained with next-token prediction; the claim should be restated as \"no additional SFT\" and supported by the full training configuration.","section":"§4 and Abstract"},{"comment":"The overthinking conclusion is not directly supported by the reported numbers. Table 2 contains only two Think-mode rows (4B Think 71.0%, 8B Think 62.2%) on a 10% subset, with no No-Think accuracy or latency values, even though the text claims no-think variants deliver significantly faster response times while maintaining competitive accuracy. The qualitative conversation logs in Appendix A.1 illustrate a failure pattern but do not quantify the overthinking effect. Since the adoption of force non-thinking is a load-bearing design decision, this evidence gap should be filled.","section":"§5.1, Table 2 and Appendix A.1"}],"minor_comments":[{"comment":"The term \"RL VR\" appears with a space in the abstract and Section 4, while \"RLVR\" is used elsewhere; please standardize the notation.","section":"Abstract, §4, §6"},{"comment":"Figure 1 is never discussed in the main text; the caption says it visualizes \"focus during different training stages\", but the figure's axes, panels, and takeaway should be explained in a paragraph.","section":"Figure 1"},{"comment":"The conversation logs in Table 3 are extremely dense and difficult to read; consider separating the 4B and 8B traces into distinct columns or sub-tables, and fix the recurring \"ASSIST ANT\" typo.","section":"Appendix A.1, Table 3"},{"comment":"The SimpleQA grading protocol is not described: the paper does not state how answers are judged correct, whether \"not attempted\" answers are included, or whether an exact-match or LLM-based grader is used. This detail should be added in the evaluation section.","section":"§2"},{"comment":"The benchmark label is inconsistent: Table 1 says \"SimpleQA\" while Table 4 says \"SimpleQA (Full)\". These labels should be reconciled so that readers know which configuration produced which number.","section":"Tables 1 and 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is closer to a model card than a fully specified technical report. The decisive issue is the undocumented evaluation protocol for the headline 83.2% result and the baseline comparison. If the authors can provide the exact full-SimpleQA protocol, the 10% subset sampling procedure, and the baseline configuration, a major revision would be appropriate. If they cannot, the central claim should be withdrawn or substantially downgraded."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the thing about Jan-nano: the model is public, the training approach is a clean extension of Search-R1/ReSearch/DAPO-style RLVR to a 4B model, and the force-non-thinking observation (larger models over-filter and hallucinate) is worth taking seriously. That's the good part.\n\nThe bad part is that the central number, 83.2% on SimpleQA with MCP, is not tied to a documented protocol. The report says \"the more conservative 83.2% result\" after mentioning a forked MCP server that could beat 85%, which means the number is a selection among runs. Table 4 only labels the 80.7% run as \"SimpleQA (Full)\"; Table 1 gives 83.2% without saying full or subset. The 10% SimpleQA subset was used to decide Think vs No-Think, and there is no statement about whether the final eval used the full set or that same slice, or how the subset was sampled. So the headline may be the result of tuning on the test set.\n\nThe baseline comparison is also shaky. Qwen3-4B at 59.2% is presented without evidence that it ran with the same tool stack, prompts, grading, and temperature. The report itself warns that different MCP implementations affect direct comparisons, but doesn't apply that caveat to its own baseline. The no-SFT claim is asserted, not demonstrated—no ablation against an SFT baseline.\n\nI don't think the paper is a waste of time. The model is on Hugging Face, and the task design (RLVR with local RAG server, three-stage rewards, force non-thinking) is a reasonable recipe that someone could reproduce or extend. But the report as written fails to substantiate the one number that matters. A serious referee would need to see the evaluation harness, the answer-matching rule, the full test set details, and at least one ablation for the no-SFT claim.\n\nMy recommendation: don't desk-reject it outright—the claims are important enough and the model ships—but it needs heavy revision before it can be taken as evidence. If it's a technical report rather than a journal paper, then the bar is lower, and the public weights carry some weight. Still, the headline should be re-evaluated or clearly labeled as preliminary.\n\nWho this is for: practitioners doing tool-augmented small models, and people who care about honest benchmark reporting. I'd bring it to reading group to discuss what counts as a verifiable evaluation. I would not cite it in my own work yet.\n\nRecommendation: send it to peer review with a request for a revised version that documents the evaluation protocol and ablations.","headline":"Plausible 4B tool-augmented QA result, but the evaluation protocol is undocumented enough that the headline 83.2% SimpleQA is not verifiable.","tokens_in":62467,"tokens_out":3769,"would_cite":false,"duration_ms":36097,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 4B-parameter model trained without next-token prediction reaches 83.2% on SimpleQA with tool access, beating far larger models.","keywords":["small language models","tool-augmented factuality","SimpleQA","reinforcement learning with verifiable rewards","retrieval-augmented generation","model specialization","consumer hardware inference"],"falsifier":"Run the released model on a fresh, randomly drawn half of SimpleQA using the same minimal tool setup with no configuration changes; if the score falls back toward the 59–71% range instead of near 83.2%, the headline gain comes from tuning on the test slice.","tokens_in":61402,"feed_emoji":"🔍","tokens_out":6453,"duration_ms":67288,"temperature":0.7,"pith_summary":"The paper claims that a 4-billion-parameter model specialized for tool use can beat models orders of magnitude larger on factual question answering. Jan-nano, built on a 4B base model and trained with reinforcement learning from verifiable rewards, scores 83.2% on the SimpleQA benchmark when connected to search and scraping tools, a 24-point gain over the base model under the same tool setup and higher than a 671B-parameter model scored with tools. The training pipeline deliberately skips supervised next-token prediction, using reward signals for correct answers and successful tool calls across three stages. The point of the claim is that intelligence for knowledge tasks can come from strategy—knowing how to find information—rather than from parameter count.","feed_headline":"Small 4B model beats 671B giant on tool-augmented facts","feed_subtitle":"Jan-nano hits 83.2% on SimpleQA with search tools, outpacing far larger models on consumer hardware.","key_machinery":"The central mechanism is multi-stage reinforcement learning with verifiable rewards (RLVR), paired with a local retrieval server that simulates a search engine. The server exposes two tools—websearch(query) returning snippets and scrape(id) returning full documents—and training is conducted on MuSiQue multihop questions. Three stages progress from basic tool mechanics to answer correctness to context extension, with a force non-thinking regularization that suppresses chain-of-thought-style overplanning, and a JSON-in-XML tool format keeps rewards easy to parse.","core_discovery":"The report's central claim is that a 4B model specialized for tool use can outperform even 671B-parameter models on tool-augmented short-form factuality. Jan-nano reaches 83.2% on SimpleQA, compared with 78.2% for the 671B model and 59.2% for the originating 4B base model under the same tool setup. The authors attribute the result to a design that prioritizes finding information over storing it, supported by the overthinking analysis: in head-to-head traces, the 4B model's direct, unfiltered searches retrieve the correct document, while an 8B model's added year filters discard relevant results and lead to hallucinated answers. They further claim the entire skill was acquired through multi-stage RLVR without any supervised next-token prediction stage.","pith_inferences":["If these numbers replicate, the bottleneck for factuality shifts from model size to tool quality: a better search API and scraper could push the same 4B model toward or beyond 85%, as the authors themselves note.","The overthinking result suggests a general design rule for agent training: reward designs that penalize unnecessary actions may help small models in other tool-use domains, not just fact lookup.","A direct test of the no-SFT claim would be to train the same base model with standard supervised fine-tuning on the same MuSiQue tool traces and compare SimpleQA scores; the paper does not report this ablation.","Because the headline evaluation couples model and search tool, the claim is best read as 'small model plus good search beats giant model plus good search,' not as small models being intrinsically more factual."],"forward_implications":["Tool-augmented factuality can be decoupled from model scale: a 4B model can beat a 671B model on SimpleQA when both use the same search-tool setup.","A small model can learn complex tool use without any supervised next-token training, using only verifiable rewards.","Suppressing chain-of-thought overthinking improves both speed and answer accuracy for search-based question answering.","Longer context (128K versus 8K) contributes a small gain (80.7% to 83.2%), so context length appears secondary to tool strategy.","The model's skill transfers partially across tool-calling formats: 76.2% on a Python-code agent framework versus 80.7% on a JSON-based framework, indicating format alignment matters."],"supporting_citations":[{"why":"Defines SimpleQA, the benchmark whose exact accuracy (83.2%) is the paper's headline result.","marker":"Wei et al. [2024]"},{"why":"Supplies the Qwen3-4B base model that Jan-nano is fine-tuned from and the baseline scoring 59.2% with the same tool setup.","marker":"Yang et al. [2025]"},{"why":"Provides the MuSiQue-Ans multihop questions used in the controlled local search-and-scrape training environment.","marker":"Trivedi et al. [2022]"},{"why":"Gives the DAPO reinforcement-learning system used for the multi-stage RLVR training.","marker":"Yu et al. [2025]"}],"fun_headline_variants":["4B model beats 671B on tool-augmented facts","Tiny 4B model outdoes 671B with search tools","Jan-nano: 4B model beats 671B on SimpleQA","Tool-specialized 4B model tops 671B on facts","4B Jan-nano beats 671B models on tool facts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 10% subset of SimpleQA used to choose no-thinking mode represents the full benchmark well enough that the configuration decision does not inflate the reported 83.2%.","fun_headline_variants_meta":{"raw":{"variants":["4B model beats 671B on tool-augmented facts","Tiny 4B model outdoes 671B with search tools","Jan-nano: 4B model beats 671B on SimpleQA","Tool-specialized 4B model tops 671B on facts","4B Jan-nano beats 671B models on tool facts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2789,"prompt_tokens":824,"completion_tokens":1965,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":1870}},"tokens_in":440,"tokens_out":1965,"duration_ms":13501,"temperature":1.0,"reasoning_tokens":1870,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:58:29.022039+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released model on a fresh, randomly drawn half of SimpleQA using the same minimal tool setup with no configuration changes; if the score falls back toward the 59–71% range instead of near 83.2%, the headline gain comes from tuning on the test slice.","supporting_citations":[],"review_version":1}