Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models

T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Rewarding each reasoning step for faithfulness, not just the final answer, is what reduces hallucinations in small reasoning models.

desk verdict FaithRL is a sensible step-level RL recipe for cutting CoT hallucinations in small models, but the headline aggregates and the unvalidated PRM call for caution. read the letter →

arxiv 2602.05897 v2 pith:GHAVRVZF submitted 2026-02-05 cs.CL

Shuo Nie , Hexuan Deng , Chao Wang , Ruiyu Fang , Xuebo Liu , Shuangyong Song , Yu Li , Min Zhang
show 1 more author
Xuelong Li
This is my paper
classification cs.CL
keywords faithfulnesshallucinationsmallreasoningmodelschain-of-thoughtprocessrewardmodelstep-levelreinforcementlearningtruncatedresamplingopen-bookquestionansweringmitigation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FaithRL is a training recipe for small reasoning models that keeps their chain-of-thought grounded in the given context. It gives the model two kinds of step-level feedback: an explicit faithfulness reward on each sentence from a process reward model, and an implicit contrastive signal from resampling a continuation after an unfaithful step is truncated. On open-book question-answering benchmarks, the paper reports that this consistently lowers hallucinations both in the reasoning trace and in the final answer, cutting answer hallucinations by about 3.9% in F1 and 3.5% by an LLM judge compared with the best baseline, without extra GPU time. It also presents evidence that answer-only reinforcement learning actually worsens chain-of-thought hallucinations, and that unfaithful steps paired with correct answers are harmful under adversarial re-questioning.

What carries the argument

The central object is the step-level faithfulness verifier together with dynamic truncated resampling. The verifier—a publicly available process reward model—produces a per-sentence faithfulness judgment relative to the context; that judgment sets the +1/-1 step reward and also decides where to truncate a rollout. Dynamic truncated resampling then regenerates the rest of the trajectory from the faithful prefix, and the paper decomposes the total reward into a shared-prefix average plus a differential suffix signal, which is the implicit step-level credit. The information-gain and repetition penalties are load-bearing guards against reward hacking: pure faithfulness rewards push the model tow

What would settle it

Train FaithRL with the automatic PRM versus FaithRL with human/perfect step labels on a held-out split; if the oracle version does not meaningfully outperform, judge accuracy is the real bottleneck. A simpler check: measure the PRM's precision and recall on labeled chain-of-thought sentences—if recall of hallucinated steps is low, truncation fires at the wrong places.

Watch

Extended reading notes

Core claim

The paper's central claim is that faithfulness hallucinations in small reasoning models live mostly in intermediate chain-of-thought steps, so the training reward must be step-level rather than outcome-level. FaithRL operationalizes this by having an automatic judge decide whether each chain-of-thought sentence is supported by the context. If the final answer is correct, each sentence gets a +1 or -1 from that verdict, plus penalties for verbosity and repetition; if the final answer is wrong, the whole trajectory gets the answer-level penalty. In parallel, whenever a hallucinated sentence is found, the trajectory is cut at that point and regenerated from the faithful prefix; the paper shows

Load-bearing premise

The method rests on the per-sentence faithfulness judge being accurate: if its verdicts are wrong, FaithRL rewards hallucinated steps or punishes faithful ones, and the paper only validates that judge through downstream results (its Table 8), not by precision or recall on labeled reasoning sentences.

Editorial extensions

If this is right

  • If FaithRL is right, answer-only reinforcement learning is not just blind to unfaithful reasoning but actively entrenches it, so reasoning-model training should include step-level grounding.
  • Final-answer correctness alone cannot serve as a trustworthiness signal: hallucinated steps can be turned into wrong answers by adversarial re-questioning, making faithful reasoning a distinct optimization target.
  • The reported reduction in generated tokens to about one-third of vanilla RL means step-level verification can buy faithfulness without a compute penalty, which matters for small-model deployment.
  • On attack sets built from hallucinated chain-of-thought steps, FaithRL's faithfulness advantage over outcome-only RL widens substantially, indicating improved robustness to adversarial reformulations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method's main risk is its dependence on the per-sentence faithfulness judge; a natural extension would be to run FaithRL with perfect oracle step labels on a small set to measure how much of the gain comes from judge accuracy.
  • The truncate-and-resample decomposition is a general mechanism: any verifier that can identify a bad prefix could generate implicit step-level reward without explicit per-step scoring, which may transfer to tasks without answer rewards.
  • The verbosity and repetition penalties appear to do much of the heavy lifting, suggesting part of the reported improvement comes from length control rather than faithfulness targeting alone.
  • The attack construction could be turned into a reusable benchmark: generate new questions from hallucinated steps and measure hallucination propagation, turning the paper's diagnostic into an evaluation for reasoning faithfulness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies faithfulness hallucinations in small reasoning models (SRMs) on open-book QA. It first documents that SRMs hallucinate more in CoT than in answers, and that unfaithful CoT can coexist with correct answers and lead to errors under constructed attacks. It then proposes FaithRL, which trains with GRPO plus (i) explicit sentence-level rewards from HHEM-2.1 (Eq. 1), (ii) dynamic truncated resampling that regenerates from a faithful prefix when an unfaithful step is detected, and (iii) information-gain and repetition penalties (Eqs. 2-5). Experiments on three SRMs and five QA datasets report consistent gains in answer accuracy (F1) and faithful rate (LLM-as-a-judge) over SFT, GRPO, KD, Self-Refine, and FSPO, with ablations.

Significance. If the headline results hold, FaithRL would be a practical, efficiency-preserving recipe for reducing CoT and answer hallucinations in small models, and the analysis of CoT/answer inconsistency plus the attack results would be a useful contribution. The paper provides code, multiple backbones, ablations, and a rollout-budget comparison. However, the central quantitative claims are currently not reproducible from Table 3, and the method's dependence on an unvalidated PRM plus a train/eval overlap on HotpotQA leaves the main conclusion unsupported as written. With corrections and added validation, the core idea is promising.

major comments (4)
  1. [§5.2 / Table 3 / Abstract] The abstract and §5.2 state average reductions of 3.86% in Acc and 3.48% in Faith versus the best baseline. These numbers do not follow from Table 3. For example, the Qwen3-1.7B FaithRL Acc average is printed as 73.55, but the five per-dataset values average to 71.55. Recomputing per-model best-baseline improvements gives roughly (56.81-54.29)=2.52 for DPSK-1.5B, (64.08-61.38)=2.70 for Qwen3-0.6B, and (71.55-69.25)=2.30 for Qwen3-1.7B, for a mean of about 2.5 points, not 3.86; per-dataset best-baseline differences give an even smaller mean (~1.6 points). Similar discrepancies appear for Faith (mean per-model gain ~2.7 points vs 3.48 claimed). Please correct the table, recompute all aggregate claims, and state exactly which baseline and averaging scheme is used.
  2. [§5.1 / Table 3] Training data includes 8,000 examples sampled from HotpotQA (§5.1), and HotpotQA is one of the five evaluation benchmarks. The paper does not state that evaluation uses a disjoint split or that training samples are removed. Since some HotpotQA training examples may be members of the evaluation set or share distribution, the HotpotQA column and the overall averages are at risk of data leakage. Report results on a held-out HotpotQA split or remove HotpotQA from evaluation, and rerun the headline averages.
  3. [§4.1-4.2 / Table 8] The entire step-level signal (Eq. (1) and the DTR truncation) is produced by HHEM-2.1 sentence-level faithfulness verdicts. The only validation offered is Table 8, which compares downstream trained models using HHEM-2.1 vs Qwen3-8B as PRM. That comparison cannot verify the correctness of the per-sentence verdicts that drive training; a biased PRM could reward terse, PRM-pleasing CoT without genuine grounding (a risk amplified by the R_inf length penalties). Please provide sentence-level precision/recall or agreement of HHEM-2.1 on CoT sentences from the base SRMs (e.g., against Qwen3-32B or human labels), and/or an oracle-PRM control. Without this, the mechanistic claim of 'targeted penalization of unfaithful steps' is not established.
  4. [§4.1 Equivalence derivation] The decomposition of the total reward into shared-prefix and suffix-differential terms assumes that the pair (original, resampled) is compared in isolation and that Rr_k > Rk holds when resampling corrects a hallucination. In the actual GRPO procedure, rewards are normalized against the group mean/std of up to 16 rollouts, and the inequality is an assumption rather than a guarantee. Please present this as an intuition or prove it under explicit GRPO advantage definitions, and report the empirical frequency with which Rr_k > Rk in training. This matters because the 'implicit step-level reward' is a stated contribution.
minor comments (6)
  1. [§4.2 / Reproducibility] Eqs. (2)-(5) define hyperparameters λs, λt, Ls, Lt, τ, and λ_rep, but no values are reported in the main text or appendix. Please add the full hyperparameter configuration.
  2. [§4.2 / Eq. (7)] The answer reward Ranswer uses a rule-based correctness check, but the matching procedure (e.g., exact match vs F1 threshold) is not specified. Clarify how 'correct' is determined, especially for multi-word answers.
  3. [Figure 2] Figure 2 is dense and the labels for DTR and reward flows are hard to read. Consider larger fonts, separate panels, or a step-by-step diagram.
  4. [§5.2 / Efficiency claim] The claim that FaithRL 'reduces SRM-generated tokens to about one-third' is not backed by a table or measurement protocol. Report average token counts per model/method or add a figure.
  5. [§B.2 / SFT baseline] The SFT baseline selects only responses with correct answers; the number of selected samples and the resulting data size should be reported, as this affects the comparison.
  6. [§3.2 / Attack experiment] The attack experiment uses 'GPT-5.2'; specify the exact model version and access date, and provide the prompt or cite the model card.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training reward and evaluation judge are distinct, and the implicit step-level reward is a reparameterization rather than a fitted prediction.

full rationale

FaithRL's derivation chain is not circular. The explicit step-level reward in Eq. (1) directly encodes HHEM-2.1's per-sentence faithfulness verdict, and the final token reward in Eq. (7) also depends on a rule-based answer correctness check and on hand-defined information-gain and repetition penalties. The paper's headline claims—reduced answer hallucinations and reduced CoT hallucinations—are measured with F1 and with Qwen3-32B as an LLM-as-a-judge, which is not the same model that provides the HHEM-2.1 training signal, so the evaluation is not a definitional restatement of the reward. The DTR 'implicit step-level reward' in Section 4.1 is derived as a mathematical identity: for two rollouts sharing a prefix, assigning the shared prefix the average reward and the divergent suffixes plus/minus half the reward difference exactly accounts for the total reward difference; the claim that R_r^k > R_k when resampling corrects a hallucinated step is presented as an empirical tendency, not as a construction that forces the sign. The selection of HHEM-2.1 over Qwen3-8B in Appendix B.1 is based on downstream performance (Table 8), which is model selection rather than circular reasoning. Self-citations appear only as general related-work context and are not load-bearing. The lack of direct PRM precision/recall validation and the use of HotpotQA-derived training data with HotpotQA evaluation are legitimate correctness and generalizability risks, but they do not reduce any prediction to its inputs.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central claim rests on two learned/judged oracles (HHEM-2.1 for rewards, Qwen3-32B for evaluation), six unstated shaping hyperparameters, and an unproven sign assumption for resampled rewards. No new physical entities are introduced. The HotpotQA training/evaluation overlap adds a self-referential signal to one of the five benchmarks.

free parameters (8)
  • lambda_s = not reported
    Penalty coefficient for sentence-count constraint in Eq. (2); no numeric value given.
  • lambda_t = not reported
    Penalty coefficient for token-count constraint in Eq. (3); no numeric value given.
  • L_s = not reported
    Sentence-count safety threshold in Eq. (2); no numeric value given.
  • L_t = not reported
    Token-count safety threshold in Eq. (3); no numeric value given.
  • tau = not reported
    N-gram repetition threshold in Eq. (5); no numeric value given.
  • lambda_rep = not reported
    Repetition penalty coefficient in Eq. (5); no numeric value given.
  • rollout_count = 8 initial, 16 effective
    Training budget choice described in §4.1 and Appendix B.2; affects comparison fairness.
  • training_set_size = 8,000 examples from HotpotQA + 2WikiMultiHopQA
    Data-selection choice in §5.1; overlaps with the HotpotQA evaluation set.
assumptions (5)
  • domain assumption HHEM-2.1 sentence-level faithfulness judgments are accurate enough to drive truncation and rewards.
    Central to §4.1 and Eq. (1); the paper only validates HHEM-2.1 via downstream task metrics in Table 8, not precision/recall against human labels.
  • domain assumption Qwen3-32B LLM-as-a-judge faithfully measures answer and CoT faithfulness.
    Used to compute Faithful Rate, CoT Faith, HR, and KHR (§3.1, §3.2, §5.4); no human agreement study is reported.
  • ad hoc to paper When resampling corrects a hallucinated step, the resampled trajectory has higher reward (Rr_k > Rk).
    Required for the sign of the implicit step-level reward in §4.1; the paper says this is 'typically observed' but does not prove it.
  • domain assumption Compositional sentence-level faithfulness is the correct optimization target for reducing CoT hallucinations.
    The method optimizes per-sentence rewards; no evidence is given that independent sentence faithfulness decomposes cleanly into reasoning quality.
  • domain assumption Training on 8,000 HotpotQA + 2WikiMultiHopQA examples transfers to all five evaluation benchmarks.
    Assumed in §5.1; the HotpotQA evaluation portion is not independent of the training sample.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models." pith.science (2026). https://pith.science/paper/GHAVRVZF

@misc{pith2026260205897,
  author       = {Pith},
  title        = {Pith review of: Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GHAVRVZF}},
  note         = {Machine review of arXiv:2602.05897}
}
read the original abstract

As large language models become smaller and more efficient, small reasoning models (SRMs) are crucial for enabling chain-of-thought (CoT) reasoning in resource-constrained settings. However, they are prone to faithfulness hallucinations, especially in intermediate reasoning steps. Existing mitigation methods based on online reinforcement learning rely on outcome-based rewards or coarse-grained CoT evaluation, which can inadvertently reinforce unfaithful reasoning when the final answer is correct. To address these limitations, we propose Faithfulness-Aware Step-Level Reinforcement Learning (FaithRL), introducing step-level supervision via explicit faithfulness rewards from a process reward model, together with an implicit truncated resampling strategy that generates contrastive signals from faithful prefixes, while also mitigating reward hacking from step-level rewards. Experiments across multiple SRMs and Open-Book QA benchmarks demonstrate that FaithRL consistently reduces hallucinations in both the CoT and final answers, leading to more faithful and reliable reasoning. Code is available at https://github.com/Easy195/FaithRL.

Figures

Figures reproduced from arXiv: 2602.05897 by the authors.

Figure 1
Figure 1. For small reasoning models, faithfulness hallucinations in the CoT and the final answer are often inconsistent, making outcome-only rewards ineffective. Figure shows an example on NewsQA with DeepSeek-R1-Distill-Qwen-1.5B (DPSK-1.5B): although the final answer is correct, faithfulness hallucinations still occur within the CoT. By constructing new questions based on hallucinated CoT steps, we find that the model freq… view at source ↗
Figure 2
Figure 2. An overview of the FaithRL framework. The left part illustrates the Implicit Step-Level Rewards with Dynamic Truncated Resampling. The model performs sentence-level faithfulness detection on the CoT generated by the policy model. When a faithfulness hallucination is detected, the reasoning process is truncated, and the faithful CoT sentences preceding it are used as a prefix for resampling. The right part depicts Ex… view at source ↗
Figure 3
Figure 3. Curves of Key CoT Faithful Rate and Answer Faithful Rate during training on DPSK-1.5B evaluated on NewsQA. repetitive statements, thereby accelerating convergence. Impact of Dynamic Truncated Resampling on Perfor￾mance in Complex Reasoning. Removing the resam￾pling mechanism leads to notable performance declines on complex reasoning benchmarks such as TriviaQA and HotpotQA. This demonstrates that our DTR is essentia… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CLIR-Bench: Benchmarking Multimodal Question Answering over Irregular Clinical Time Series

    cs.CL 2026-07 conditional novelty 6.0 of 10

    CLIR-Bench shows generalist and time-series LLMs struggle to ground clinical answers in sparse irregular ICU evidence, with top accuracy near 50% and weak causal evidence use.

Reference graph

Works this paper leans on

10 extracted references · 4 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Reuse the SAME misleading cue that triggered the original failure (e.g., entity name overlap)

  2. [2]

    Introduce at least ONE additional distractor that reinforces the incorrect assumption

  3. [3]

    Make the incorrect answer a highly plausible span directly supported by surface-level reading

  4. [4]

    new_question_1

    Require the correct answer to depend on at least TWO disambiguation steps. TARGET MODEL WEAKNESS ASSUMPTIONS • Trusts the first entity that matches; Overweights frequently mentioned facts. • Prefers short, fluent reasoning chains; Rarely backtracks. OUTPUT FORMAT(Strict JSON, no extra text): {{ "new_question_1": "...", "standard_answer_1": "...", "new_que...

    2025

  5. [5]

    URL https://aclanthology.org/2025.findin gs-acl.269/

    doi: 10.18653/v1/2025.findings-acl.269. URL https://aclanthology.org/2025.findin gs-acl.269/. Hughes, S., Bae, M., and Li, M. Vectara Hallucination Leaderboard, 2023. URL https://github.com/v ectara/hallucination-leaderboard. Jacovi, A., Wang, A., Alberti, C., Tao, C., Lipovetz, J., Olszewska, K., Haas, L., Liu, M., Keating, N., Blo- niarz, A., et al. The...

    arXiv 2025

  6. [6]

    URL https://arxiv.org/abs/2412.1 6720. Pham, Q. P. M., Nguyen, K. T. N., Doan, N. H., Pham, C. A., Sun, Q., Qi, W., Inui, K., and Song, D. Smallplan: Leverage small language models for sequential path planning with simulation-powered, llm-guided distilla- tion.arXiv preprint arXiv:2505.00831, 2025. URL https://arxiv.org/abs/2505.00831. Prieto, P. and Abad...

    arXiv 2025

  7. [7]

    Wait, I need to think

    URL https://arxiv.org/abs/2408.0 3314. Team, K., Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., et al. Kimi k1.5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025a. URL https://arxiv.or g/abs/2501.12599. Team, M., Xiao, C., Li, Y ., Han, X., Bai, Y ., Cai, J., Chen, H., Chen, W., Cong, X.,...

    arXiv 2025

  8. [2017]

    doi: 10.18653/v1/P17-1147

    Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclantho logy.org/P17-1147/. Kalai, A. T., Nachum, O., Vempala, S. S., and Zhang, E. Why language models hallucinate.arXiv preprint arXiv:2509.04664, 2025. URL https://arxiv.or g/abs/2509.04664. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A. P., Alberti...

    arXiv 2025

Show all 10 references
  1. [2024]

    Liu, B., Li, H., Wang, S., Nie, S., and Zhang, S

    URL https://openreview.net/forum ?id=v8L0pN6EOi. Liu, B., Li, H., Wang, S., Nie, S., and Zhang, S. Sub- graph aggregation for out-of-distribution generalization on graphs.Proceedings of the AAAI Conference on Artifi- cial Intelligence, 39(18):18763–18771, April 2025. ISSN 2159...

    2025 arXiv

  2. [2025]

    Haque, M

    URL https://arxiv.org/abs/2506.0 0782. Haque, M. A., Rahman, F., Gupta, K. D., Shujaee, K., and George, R. Tinyllm: Evaluation and optimization of small language models for agentic tasks on edge devices. arXiv preprint arXiv:2511.22138, 2025. URL https: //arxiv.org/abs/2511.22...

    2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.