{"id":"fa514fee-7782-454f-a710-7fdd391d3372","arxiv_id":"2602.01348","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A dual-reward RL framework (deterministic + judge-based faithfulness) improves both answer accuracy and trace faithfulness for RAG-based multi-hop QA, with 7B models matching or beating large closed APIs.","lead":"CRAFT is a reinforcement-learning training method that makes language models produce structured, checkable reasoning traces when answering multi-hop questions from retrieved documents. It reports that adding a faithfulness-checking reward improves answer accuracy rather than lowering it, across model sizes from 0.5B to 7B.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Faithfulness improvement may be an artifact of using the same judge for reward and evaluation; the human validation does not rule out reward hacking.","rationale":"The central claim has two components: improved answer accuracy and improved reasoning faithfulness. The accuracy component is supported by EM/F1, which are external and not affected by the judge. The judge-reward ablation (Table 2) shows consistent EM gains with the judge across all variants and benchmarks, making the 'accuracy is not hurt' conclusion reasonably robust, though error bars are absent. The faithfulness component, however, rests entirely on a judge model that also serves as the training reward. The paper's rebuttals to circularity are real but thin: human validation covers only one model, one dataset, and the same criteria; cross-judge checks only preserve aggregate rankings over four configurations. The most direct threat is reward hacking: the policy may optimize the judge's specific binary checks without achieving human-recognizable faithfulness. Because the title and abstract emphasize both accurate and faithful reasoning, a failure of the faithfulness metric would substantially reduce the contribution, leaving only the claim that a particular judge-based reward does not hurt EM. The reader's weakest assumption identifies exactly this concern, and I agree. The proposed concrete test—blind holistic human evaluation across conditions—would determine whether the judge's rankings reflect genuine faithfulness or an artifact of training to the judge. No other concern (lack of OOD evaluation, missing error bars, missing baselines) is as central or as potentially fatal to the paper's stated contribution.","tokens_in":22982,"tokens_out":6864,"duration_ms":79017,"concrete_test":"Conduct a blind human evaluation on 400 MuSiQue traces (100 each from Qwen2.5-7B Base, SFT, CRAFT w/o judge, CRAFT full). Two annotators rate each trace for holistic faithfulness using a Likert scale (e.g., each reasoning step is supported by the cited evidence; the answer is entailed by the reasoning), not the paper's four binary criteria. Compare human-rated faithfulness across conditions and per-trace agreement with Qwen3-Max's binary scores. If humans do not rank CRAFT full above CRAFT w/o judge and both above SFT/Base, or if humans judge a substantial fraction of high-scoring CRAFT traces as unfaithful, the faithfulness improvement is a judge artifact rather than genuine grounding. Include adversarial traces with unsupported leaps to check judge sensitivity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The claim that CRAFT improves reasoning faithfulness is measured exclusively by Qwen3-Max, which also supplies the Rfaith training reward (§3.3.2, §4.5). The two defenses against circularity are insufficient. First, the human validation (§4.5, Table 3) is restricted to 500 MuSiQue traces from CRAFT 7B only, and annotators use the same four binary criteria as the judge (Algorithm 1). This confirms that the judge applies these criteria consistently with humans, but it does not establish that satisfying the criteria entails genuine faithfulness. The model could exploit systematic judge leniency or blind spots (e.g., citing a document that mentions the right entity but not the required relation) while still passing all four checks. Second, the cross-judge evaluation (Appendix C, Table 12) only compares aggregate faithfulness scores over four configurations; it does not compare per-trace judgments. Two LLM judges can share the same biases, so a preserved ranking does not rule out gamesmanship. If the model has learned to satisfy Qwen3-Max's specific checklist without true grounding, the reported +22.1% Faithfulness gain (Table 1) is an artifact, and the broader interpretation that faithfulness-guided alignment improves accuracy is unsupported. The EM gains are externally measured and thus not threatened by this concern, which is why the accuracy half of the central claim remains credible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CRAFT, a GRPO-based reinforcement learning framework for the response-generation stage of retrieval-augmented multi-hop QA. The model is trained to emit structured XML-style traces (plan, gold-document citations, reasoning, answer) under a composite reward: deterministic rewards for format compliance, citation validity, and answer correctness, plus a judge-based reward (Rfaith) for semantic faithfulness. The faithfulness reward is computed by Qwen3-Max according to four binary checks (plan-to-reason, citation-boundary, reason-to-answer, and evidence grounding). Experiments on MuSiQue, HotpotQA, and 2WikiMHQA across Qwen2.5 models from 0.5B to 7B report simultaneous gains in EM/F1 and in a Faithfulness metric, with CRAFT 7B reportedly matching or exceeding several larger API models. The authors also ablate trace variants, remove the judge reward, validate the judge against 500 human-annotated MuSiQue traces, and report a cross-judge comparison with Gemini-2.5-Flash.","tokens_in":23246,"tokens_out":6834,"duration_ms":78789,"significance":"If the results hold, the framework is practically valuable for auditable RAG reasoning, and the finding that judge-based process rewards can improve rather than trade off answer accuracy is an interesting counterpoint to outcome-only RL. The paper has real strengths: a thorough ablation over five trace variants and four model scales, deterministic rewards that are directly verifiable, training-dynamics analysis, human validation of the judge, a cross-judge consistency check, and released code. The main weakness is that the Faithfulness metric used to validate the central claim is produced by the same judge model that supplies the Rfaith training reward. The two defenses—human agreement on 500 traces from one 7B model using the same criteria, and aggregate rank preservation over four configurations with a second judge—are not yet sufficient to rule out reward hacking. The accuracy half of the central claim is externally measured and therefore credible; the faithfulness half is currently over-strong.","major_comments":[{"comment":"Circularity of the faithfulness metric is load-bearing for the title claim. Qwen3-Max supplies Rfaith during training and also produces the reported Faithfulness numbers; the human validation covers only 500 MuSiQue traces from CRAFT 7B and uses the same four binary criteria as the judge. This establishes that the judge applies its checklist consistently with humans, not that satisfying the checklist entails genuine faithfulness or that the trained model did not exploit judge blind spots. The cross-judge check compares aggregate scores over only four configurations; rank preservation does not rule out shared biases. Please add per-trace agreement against an independent judge not used in training, human evaluation across model scales and at least one additional benchmark, or an objective grounding metric (e.g., claim-level precision against gold supporting documents).","section":"§3.3.2, §4.5, Appendix C (Table 3, Table 12)"},{"comment":"All evaluation is in-distribution: the 20K training samples and the 2K test samples come from the same three benchmarks (MuSiQue, HotpotQA, 2WikiMHQA). The Limitations section explicitly says the synergy is established 'within the training distribution first.' The abstract and conclusion, however, state the general result that faithfulness-guided alignment improves answer accuracy. Either temper the claims to the in-distribution setting or add a held-out multi-hop QA benchmark not seen in training (e.g., an additional RAG dataset) to test transfer.","section":"§4.1 and Limitations"},{"comment":"No repeated runs or statistical uncertainty are reported. Several load-bearing comparisons rely on small deltas—for example, adding the judge reward improves CRAFT v1 MuSiQue EM by only +3.0 points (51.37 vs. 54.35), and some cross-template differences are similar in size. Because GRPO is stochastic, single-run numbers are insufficient to establish the 'judge reward improves accuracy' claim. Report at least three seeds with mean and standard deviation (or confidence intervals) for the main tables and for the judge-reward ablation.","section":"§4.3, Table 2; §4.2, Table 1"}],"minor_comments":[{"comment":"The retrieval setup is underspecified: how are the candidate document sets D constructed, which retriever is used, and are gold supporting documents derived from the original dataset annotations? This is needed for reproducibility. Also specify the exact prompt templates for base and API models so readers know how unstructured baselines were mapped to the CRAFT trace format.","section":"§4.1"},{"comment":"The axes labels are unclear: the left y-axis is 'Total Reward' and the right is 'sub-rewards,' but it is not stated whether Total Reward is the normalized GRPO reward or the unnormalized sum, nor what the numerical range represents. Add a legend or caption note.","section":"Figure 3"},{"comment":"Line 5 refers to 'Faithfulness Judge Principles (A, B, C, D)' but A–D are not defined in the algorithm; either label the checks A, B, C, D or remove the parenthetical.","section":"Algorithm 1"},{"comment":"The comparison between training-time average Rfaith (0.11, from on-policy high-temperature sampling over all samples) and greedy test-time average (69.33%, computed only on correct-answer samples) is not apples-to-apples. Clarify or provide the test-time average over all samples as well.","section":"Appendix B.2"},{"comment":"The colored small delta numbers in Table 2 may not survive print; use explicit columns such as ΔEM/ΔF1/ΔFaith. Also, Table 10 uses '2WikiMQA' while elsewhere '2WikiMHQA' is used; standardize.","section":"Table 2 and Table 10"},{"comment":"Define w_c in the main text (the weights are only mentioned as equal to 1.0 in §4.1) and state explicitly that all reward components are in [0,1] so the normalization is meaningful.","section":"Eq. (3)"},{"comment":"Cohen's kappa is normally attributed to Cohen (1960); the Artstein (2017) handbook citation is appropriate but should be supplemented with the primary reference.","section":"§4.5"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern in Major Comment 1 is the main barrier: the paper's most novel claim—that faithfulness-guided alignment improves faithfulness—is measured by the same judge used as the training reward. The accuracy gains are externally measured and likely robust, but the current evidence does not yet support the title conclusion as stated. I would encourage a revision that adds a non-circular faithfulness evaluation or substantially narrows the claims; this is a fixable issue, not a fundamental flaw. I would also note the risk of benchmark contamination when using a closed API model (Qwen3-Max) as judge, since it may have seen the test sets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing you should know: the EM results are credible, and the judge-reward ablation is informative. The paper's central finding—that adding a semantic judge-based reward improves answer accuracy rather than hurting it—is supported by Table 2, where CRAFT with the judge reward matches or exceeds the no-judge version on EM across variants. That part is not circular, because EM is externally defined. The accuracy half of the claim holds up.\n\nWhat is actually new: CRAFT integrates structured trace templates with GRPO and a dual deterministic-plus-judge reward. The trace-variant ablation (v1–v5) is a real contribution—it shows that structure helps small models and that faithfulness supervision helps accuracy at 7B. The paper also reports training dynamics that reveal a capacity threshold for faithfulness learning, which is a genuinely interesting result. Code is released, and the human validation of the judge on 500 traces is a serious attempt, not a box-check.\n\nWhere it gets soft: the faithfulness metric is the same Qwen3-Max judge that provides the Rfaith training reward. The paper's defenses are not enough. Human validation on 500 MuSiQue traces from one 7B model confirms that the judge applies its own criteria consistently with humans, but it does not rule out the model learning to satisfy those criteria without genuine grounding. The cross-judge check with Gemini-2.5-Flash only compares aggregate scores over four configurations, which does not catch shared judge blind spots. Also, all evaluations are in-distribution on the training benchmarks, there are no repeated runs with different seeds, and the paper does not compare against closely related RL-RAG methods—including the authors' own OPERA, which is cited but never empirically tested. The paper honestly admits the in-distribution limitation, but that doesn't fix the problem.\n\nSo the picture is mixed but not damning. The accuracy claim is solid enough to justify publication once the faithfulness claim is treated with appropriate caution. The paper deserves a serious referee, with requested revisions: out-of-distribution evaluation, an independent or frozen local judge, multiple seeds, and a comparison to OPERA or R3-RAG. I would not desk-reject this.\n\nFor a reading group, I'd bring it—it's a good example of a paper with a strong empirical core and a measurement issue that is worth dissecting. I'd cite the EM ablation result if I were working on RAG or RL, but I'd be careful about citing the faithfulness numbers as ground truth.","headline":"CRAFT's accuracy gains are real and worth engaging with, but the faithfulness improvement claim rests on an evaluator that also provides the training reward—so treat the headline synergy as provisional.","tokens_in":23794,"tokens_out":1798,"would_cite":true,"duration_ms":21237,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"CRAFT shows that faithfulness-guided alignment improves, not sacrifices, answer accuracy in retrieval-augmented multi-hop QA.","keywords":["retrieval-augmented generation","multi-hop question answering","faithful reasoning","reinforcement learning","LLM-as-a-judge","process supervision","reasoning traces","answer accuracy"],"falsifier":"Take CRAFT-trained models to a held-out multi-hop dataset not in the training mix and have human annotators plus two independent judge models flag claims that are unsupported by the cited documents. If the faithfulness advantage disappears, or if the model's citations do not move when supporting passages are shuffled or replaced with near-duplicate distractors, the model is likely satisfying the judge's checklist rather than learning grounded reasoning.","tokens_in":22798,"feed_emoji":"🎯","tokens_out":6202,"duration_ms":70311,"temperature":0.7,"pith_summary":"The paper asks whether training a retrieval-augmented model to produce evidence-grounded, auditable reasoning hurts its answer accuracy. It answers no: CRAFT, a reinforcement-learning framework, trains models to emit structured traces—plan, cited document set, step-by-step reasoning, and answer—under four rewards covering format, citation validity, answer correctness, and a judge-based faithfulness audit. Across 0.5B to 7B models and three multi-hop benchmarks, CRAFT improves both Exact Match and Faithfulness, and the 7B model matches or exceeds much larger closed-source API models. The central, somewhat counterintuitive result is that the semantic judge reward raises answer accuracy rather than trading it off against faithfulness.","feed_headline":"Faithfulness training lifts RAG accuracy, not just reasoning quality","feed_subtitle":"CRAFT's four-reward RL raises both exact match and faithfulness from 1.5B to 7B models in multi-hop QA.","key_machinery":"The load-bearing mechanism is CRAFT's chain of faithfulness: a structured XML-style trace of plan, gold-document indices, reasoning steps, and answer, constrained so each component supervises the next. Training uses group-relative policy optimization with a decomposed reward: deterministic checks for format, citation F1, and answer soft-F1, plus an LLM judge reward that averages four binary faithfulness audits. This design turns open-ended semantic properties—plan adherence, evidence grounding, logical entailment—into differentiable process-level supervision, and makes unfaithful traces detectable at training time.","core_discovery":"The paper's central claim is that outcome-level reinforcement learning alone leaves reasoning traces weakly grounded, but adding a semantic faithfulness reward does not degrade accuracy—it improves it. CRAFT combines deterministic rewards (format compliance, citation-set F1 against gold documents, soft-F1 answer match) with a judge-based reward that audits four binary criteria: plan-to-reason consistency, evidence-boundary-to-reason consistency, reason-to-answer entailment, and claim-level grounding in the cited documents. Because the trace fields form a chain, any hallucination breaks at least one verifiable link, making unfaithful reasoning machine-checkable during training. Experiments ac","pith_inferences":["A testable extension the paper leaves implicit: if the judge model is swapped during training, the learned trace behavior should shift toward the new judge's notion of faithfulness; the paper's cross-judge check only re-evaluates, it does not re-train.","The accuracy–faithfulness synergy is demonstrated within the training distribution; the paper itself flags that transfer to out-of-distribution multi-hop QA remains open, so a useful stress test is applying CRAFT to a held-out benchmark not in the 20K mixture.","The 0.5B model's collapse on the answer-only template suggests structured traces act as a credit-assignment scaffold; this predicts that adding a lightweight intermediate reward to a small model should improve exploration, which could be tested directly.","If judge-scored faithful traces are genuinely grounded, they could serve as high-quality training data for downstream auditing or answer verification, producing a measurable downstream benefit beyond benchmark scores."],"forward_implications":["Faithfulness can be enforced as a training-time objective, not just measured after generation, without sacrificing accuracy.","Structured trace templates can be tuned to desired auditability: full plan-and-citation traces give high auditability, while simpler reason-and-answer traces give high raw faithfulness.","Small models (1.5B, 3B) gain substantially from trace structure, and the 0.5B model only learns when some reasoning structure is present.","Removing the judge reward costs faithfulness without an accuracy gain, supporting the claim that faithfulness and accuracy are not in tension.","A 7B model trained on 20K mixed multi-hop examples can reach competitive accuracy and faithfulness on these benchmarks, comparable to much larger closed-source APIs."],"fun_headline_variants":["RAG accuracy improves with reasoning faithfulness, no trade-off","Faithful reasoning boosts multi-hop QA accuracy and auditability","CRAFT: RL with faithfulness reward raises RAG accuracy and trust","Right answers and traceable reasoning go hand in hand in RAG"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the judge model's binary faithfulness audit captures genuine evidence grounding rather than a learnable checklist: the same high-capacity LLM supplies the training reward and the reported evaluation score, and the paper's own validation is limited to 500 human-annotated traces from a single 7B model plus a four-configuration cross-judge ranking check.","fun_headline_variants_meta":{"raw":{"variants":["RAG accuracy improves with reasoning faithfulness, no trade-off","Faithful reasoning boosts multi-hop QA accuracy and auditability","CRAFT: RL with faithfulness reward raises RAG accuracy and trust","Right answers and traceable reasoning go hand in hand in RAG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1447,"prompt_tokens":753,"completion_tokens":694,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":623}},"tokens_in":497,"tokens_out":694,"duration_ms":8164,"temperature":1.0,"reasoning_tokens":623,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:40:37.441541+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take CRAFT-trained models to a held-out multi-hop dataset not in the training mix and have human annotators plus two independent judge models flag claims that are unsupported by the cited documents. If the faithfulness advantage disappears, or if the model's citations do not move when supporting passages are shuffled or replaced with near-duplicate distractors, the model is likely satisfying the judge's checklist rather than learning grounded reasoning.","supporting_citations":[],"review_version":1}