Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

Reinforcement Learning for Large Language Model Selective Evidence Adoption from Contaminated Retrieval Results

T0 review · 4 major / 7 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reinforcement learning can modestly improve selective evidence adoption, but the gain is not statistically robust.

desk verdict Honest, carefully measured paper on a real task; the numbers are as weak as the authors say, and the evaluator's construct validity is the one gap that matters. read the letter →

arxiv 2607.20090 v1 pith:3XLH4DXJ submitted 2026-07-22 cs.CL cs.AI

Yanyu Chen , Yue Li , Yongyi Cui , Dongsheng Shi , Lichang Dai This is my paper
classification cs.CLcs.AI
keywords selectiveevidenceadoptionretrieval-augmentedgenerationreinforcementlearningDAPOpromptinjectionanswersafetyHotpotQAMMLU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Retrieval-augmented LLMs face contexts that mix useful evidence with misleading or instruction-like content; the paper's target capability is selective adoption: take what supports the answer, reject what is wrong or unsafe. The authors construct SelectBench, a controlled benchmark and training set, and post-train Qwen3.5-4B with DAPO under two reward signals: a deterministic rule matcher and a frozen semantic judge. On a corrected 325-example test set, strict success rises from 22.46% to 25.54% (rule) and 26.46% (judge), forbidden-content adoption drops, and general capability on MMLU and clean HotpotQA is preserved. The paper is explicit that the paired gains fail to survive Holm correction, that prompt-injection following does not improve, and that the result is a modest directional improvement rather than a conclusive one. A sympathetic reader would take the contribution to be showing that selective evidence adoption is a measurable, trainable post-training objective, with robustness still open.

What carries the argument

The central mechanism is the ternary task reward combined with DAPO, an RL post-training algorithm. The reward places the policy gradient on the final model-generated token, treats forbidden adoption and tool omission as hard failures, labels wrong, incomplete, or over-refusing answers as neutral, and applies continuous length shaping beyond 768 tokens. The deterministic Rule evaluator—phrase matching with negation handling—serves both as the training signal for DAPO-Rule and as the shared formal evaluator, so it carries the argument. The benchmark holds retrieval output fixed offline, isolating evidence selection from search-engine variance.

What would settle it

Run the same 325 examples with a paraphrase-tolerant, human-validated evaluator: if strict success no longer improves for the trained checkpoints (or the direction flips), the claimed effect is an artifact of the rule matcher rather than true selective adoption. Alternatively, a single replication with another seed that shows no or negative point estimate would falsify the directional claim.

Watch

Extended reading notes

Core claim

The central claim is that a 4B language model can be steered toward selectively adopting evidence from contaminated retrieval output by directly optimizing a reward that grants +1 only for correct, complete, and non-forbidden answers, -1 for forbidden adoption or tool failure, and 0 otherwise, with mild length shaping. Applying DAPO with this reward for 130 optimizer steps raises strict success on SelectBench-v2 from 73/325 to 83/325 (rule) and 86/325 (judge), lowers forbidden adoption from 236/325 to 225/325 and 224/325, and leaves MMLU and clean HotpotQA effectively unchanged. The paper explicitly states these improvements are directional, not statistically conclusive after Holm correction

Load-bearing premise

The load-bearing premise is that the deterministic phrase-matching evaluator is a faithful measure of the target construct: answers that are correct, complete, and free of forbidden or unsafe content; if the matcher mishandles negation, paraphrase, or partial adoption, the reported differences reflect how well policies satisfy the matcher rather than how well they selectively adopt evidence.

Editorial extensions

If this is right

  • If the directional improvement is real, selective evidence adoption can be improved without degrading general knowledge, because strict success rose while MMLU and clean HotpotQA stayed flat.
  • Forbidden-content adoption remains the bottleneck: roughly 60% of factually correct outputs still copy a forbidden phrase, so future work should target this failure mode directly.
  • Prompt-injection resistance did not improve and may even worsen, meaning injection defense needs a separate mechanism rather than emerging from generic selective-adoption training.
  • Because both reward variants move numbers in the same direction, the effect is not highly sensitive to whether the reward is a phrase matcher or a semantic judge.
  • The Holm-adjusted p-values and wide confidence intervals imply single-run results should be treated as pilot evidence; replicated training runs are needed before deployment decisions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to make the reward itself paraphrase-tolerant: if the frozen matcher is why gains are small, a learned judge integrated into the reward rather than only as a label source could yield larger effects.
  • The benchmark's fixed offline contexts remove search variance; testing on live retrieval with variable context quality would tell whether the training transfers to realistic tool use.
  • Adding an explicit injection-resistance term to the reward (e.g., negative reward for following any retrieved instruction) is a direct testable extension implied by the paper's own failure analysis.
  • The corrected v2 test set changes 233 of 325 contexts, highlighting how sensitive benchmark construction is; a reader might infer that similarly rigorous auditing is needed in other selective-adoption benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces SelectBench, a controlled dataset for selective evidence adoption from retrieval contexts contaminated with misleading, unsafe, or instruction-like content, and uses DAPO to post-train Qwen3.5-4B with two reward variants: a deterministic rule matcher (DAPO-Rule) and a frozen semantic judge (DAPO-DeepSeek). On the corrected 325-example SelectBench-v2 test set, strict selective-adoption success rises from 22.46% to 25.54% (DAPO-Rule) and 26.46% (DAPO-DeepSeek), with small reductions in forbidden-content adoption; these paired gains do not survive Holm correction, and prompt-injection following does not improve. The paper also reports capability retention on MMLU and clean HotpotQA for DAPO-DeepSeek. The central claim is deliberately hedged: modest, directional, not statistically robust after multiple-comparison correction.

Significance. If the directional gains reflect genuine selective evidence adoption, the paper provides a useful controlled benchmark and a reproducible RL post-training recipe for a real failure mode of retrieval-augmented agents. Strengths include the careful dataset-version audit (SelectBench-v2 correction), the fully inventoried reproducible pipeline, paired bootstrap CIs and exact McNemar tests with Holm correction, honest reporting of nonsignificance, and the decision to expose the evaluator-alignment limitation. The value is incremental rather than breakthrough: the effect is small, not robust, and the construct validity of the central metric is not independently established. The benchmark itself and the failure-mode diagnostics may be useful to the community.

major comments (4)
  1. [§3.3, §4.3, §5.2, Eq. (1)] The Rule evaluator used as the formal test metric is the same whole-phrase-matching procedure that provides the DAPO-Rule training reward. The paper acknowledges the alignment (§5.2) but provides no validation that this matcher measures the intended construct of correct, complete, and safe selective adoption. Phrase presence with a four-token negation window can be gamed by inserting required strings verbatim and superficially negating or labeling forbidden phrases; no human-agreement study, independent-evaluator comparison, or error analysis on the matcher is reported. Since the headline strict-success and forbidden-adoption differences are defined by this matcher, the observed +3.08pp gain for DAPO-Rule is expected under reward hacking and cannot be distinguished from genuine improvement. DAPO-DeepSeek is also evaluated with this matcher rather than its own judge, so its +4.00pp result
  2. [§4.1, Table 2] Each reward variant is trained once, so training-seed variability is unmeasured. The abstract and conclusion describe DAPO-Rule and DAPO-DeepSeek as two reward variants, but §4.1 and §4.3 state that the runs also differ in prompt batch size, GPU count, and execution history. The Rule-to-DeepSeek contrast (Table 4, +0.92pp strict success, p=0.664) is therefore descriptive, not a comparison of reward designs. Given that the main claim is only directional and not robust after correction, this is not disqualifying, but the wording should consistently label both the between-variant comparison and the headline gains as single-run, uncontrolled evidence.
  3. [§5.2, Table 2] The dataset-version audit is a strength, but it also means the substantive results are confined to 325 examples. With only 20 improvements vs. 10 regressions (DAPO-Rule) and 21 vs. 8 (DAPO-DeepSeek), the strict-success gain is driven by a handful of examples. The bootstrap CI for DAPO-Rule includes zero ([0.00, 6.46]) and the Holm-adjusted p is 0.691. The paper's own conclusion is appropriately cautious, but the significance section of the abstract ('demonstrate a directional improvement') may overstate what the evidence supports; I recommend rewording to 'suggest a possible directional improvement' or adding a replication statement.
  4. [§3.3, §4.3] The DAPO-Rule reward and the formal evaluator apply forbidden-adoption detection to the entire final assistant turn (including <think>), while correctness is evaluated only on the visible answer. The paper notes this is conservative, and it is a reasonable design, but it has an unexplored consequence: the policy may learn to avoid mentioning forbidden content in <think> or to mention it only in negated form, which can reduce detected adoption without improving the actual reasoning or the visible answer. The paper does not analyze whether the reductions in forbidden adoption come from genuine rejection, rewording, or omission. A qualitative error analysis of adopted vs. rejected forbidden phrases would strengthen the construct-validity argument.
minor comments (7)
  1. [Abstract / §7] The abstract says 'demonstrate a directional improvement'; given the Holm-adjusted p-values, a more precise formulation such as 'suggest a directional improvement' or 'report directional improvements that do not reach significance after correction' would better match the results.
  2. [§3.1] The sentence 'These labels describe construction provenance only. Training conversion actively removes chosen and rejected assistant trajectories, and RL receives only the question, fixed tool context, and reward labels' is unclear. Please clarify how preference pairs are generated and how 'removes chosen and rejected assistant trajectories' is implemented.
  3. [§4.1 / Table 2] Minor wording: 'the original checkpoint' is compared against two trained checkpoints, but 'original' might be confused with the base model before any post-training. Please define precisely: the pinned Qwen3.5-4B checkpoint used as the RL starting point, as in §4.1.
  4. [§5.3, Table 5] The MMLU zero-shot accuracy gain (+0.21pp, p=0.010) is called 'no material degradation,' which is fair, but the paper should note that this zero-shot gain is nominally significant and was not corrected for multiple comparisons; the paired interval [0.057, 0.356] excludes zero. This is a minor interpretive point.
  5. [§3.3] The description of the DAPO-DeepSeek judge is detailed, but the prompt of the selectbench-deepseek-rm-v2 rubric is not included. If the rubric is available, please provide it as an appendix for reproducibility.
  6. [Throughout] The paper uses 'selectbench search' and other terms without defining them in a table; a notation table would improve readability. Also, 'code comparisons' in Table 2 caption should be 'pre-specified comparisons' or 'pre-specified family'.
  7. [§4.4] The 'one trajectory per example' evaluation with temperature 0 and seed 42 is fine for reproducibility, but the paper does not report whether the base model has any variance across seeds. Given the small differences, a sensitivity check across a few seeds would be useful, though not required.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: held-out empirical results, with the reward/evaluator overlap explicitly disclosed and DAPO-DeepSeek providing an independent arm.

full rationale

The only structurally suspicious point is that DAPO-Rule's training reward (Eq. 1) and the formal test evaluator (§4.3) are the same Rule matcher, and §5.2 explicitly discloses that the evaluator 'is aligned with the training signal for DAPO-RULE.' This is a transparency limitation, but not circularity: the test set is example-disjoint and held out, no checkpoint was selected on SelectBench-v2, and the reported gain is not forced—the paper documents 20 improvements versus 10 regressions and non-significant Holm-adjusted p-values. DAPO-DeepSeek was trained with a separate frozen semantic judge and is evaluated with the Rule matcher, so its +4.00pp strict-success change is not a re-measurement of its own reward. No load-bearing self-citation chain or imported uniqueness theorem appears; DAPO, DeepSeek, Qwen, and the QA datasets are external references. The construct-validity worry about the phrase-matching evaluator is a measurement/correctness risk, not a circular derivation, and the paper's own limitation statements acknowledge the alignment without hiding it.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on the construct validity of the rule evaluator, the quality of the underlying QA datasets, and hand-chosen reward-shaping parameters. No new physical or ontological entities are introduced.

free parameters (2)
  • Length-penalty thresholds = 768 tokens, 1024 tokens, slope 1/256 per token
    Hand-chosen shaping parameters in Eq. (2) that penalize outputs over 768 tokens; no sensitivity analysis is reported, and they affect the final reward used for policy updates.
  • Negation-context window = 4 tokens to the left
    The rule matcher treats a forbidden phrase as negated if a negation marker appears within four tokens to its left; this hand-chosen heuristic affects forbidden-adoption and strict-success labels in both training and evaluation.
assumptions (4)
  • domain assumption HotpotQA, 2WikiMultiHopQA, and MuSiQue gold answers are correct and complete for the constructed questions.
    SelectBench training and evaluation labels inherit these QA datasets; if the gold answers are noisy, both the reward signal and the test metric are noisy.
  • domain assumption The frozen deepseek-v4-flash judge provides a valid semantic reward for selective evidence adoption.
    DAPO-DeepSeek training assumes the external judge's -1/0/+1 rubric measures the intended construct; judge errors propagate into the trained policy.
  • domain assumption The corrected SelectBench-v2 supporting-fact resolution is the right version of the benchmark.
    All results are regenerated from v2 after a post-hoc audit that changed trusted evidence in 233/325 examples; if v2 introduces its own errors, the measurement is biased.
  • domain assumption The deterministic phrase-matching evaluator is a faithful measure of forbidden-content adoption and answer correctness.
    Every headline number is computed with this evaluator, and DAPO-Rule is trained with the same rule; if the matcher is gameable, the measured gains do not reflect the intended capability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcement Learning for Large Language Model Selective Evidence Adoption from Contaminated Retrieval Results." pith.science (2026). https://pith.science/paper/3XLH4DXJ

@misc{pith2026260720090,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Learning for Large Language Model Selective Evidence Adoption from Contaminated Retrieval Results},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3XLH4DXJ}},
  note         = {Machine review of arXiv:2607.20090}
}
read the original abstract

Retrieval-augmented large language models frequently face contexts that interleave useful evidence with misleading statements or instruction-like content. Blanket refusal discards valid evidence, whereas uncritical adoption yields incorrect or unsafe answers. The ability to selectively adopt relevant information while rejecting deceptive or harmful content is therefore critical for reliable deployment in real-world retrieval settings. We introduce SelectBench, a controlled benchmark and training set for selective evidence adoption, and post-train Qwen3.5-4B directly with DAPO using either deterministic rule rewards or a frozen semantic judge. On the corrected 325-example SelectBench-v2 test set, strict success rises from 22.46% for the original checkpoint to 25.54% with DAPO-Rule and 26.46% with DAPO-DeepSeek. Both trained policies reduce forbidden-content adoption and produce shorter, more focused responses, yet prompt-injection following does not improve. The paired gains are modest and fail to survive Holm correction, suggesting that stronger reward shaping or additional training iterations may be needed for more robust gains. DAPO-DeepSeek exhibits no material degradation on MMLU or clean HotpotQA, indicating that the post-training procedure preserves general capabilities. These results demonstrate a directional improvement in selective evidence use, while identifying injection resistance and statistical robustness as important remaining challenges for future work.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks

    cs.DB 2026-08 reject novelty 6.0 of 10

    DBLifeBench, a five-phase database lifecycle benchmark, shows SQL-specialized LLMs underperform general-purpose LLMs on non-query tasks such as schema design and system maintenance.

Reference graph

Works this paper leans on

12 extracted references · 9 linked inside Pith · cited by 1 Pith paper

  1. [8]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369–2380,

    2018

  2. [10]

    Latent-space adver- sarial training with post-aware calibration for defending large language models against jailbreak attacks.Expert Systems with Applications, pp

    Xin Yi, Yue Li, Dongsheng Shi, Linlin Wang, Xiaoling Wang, and Liang He. Latent-space adver- sarial training with post-aware calibration for defending large language models against jailbreak attacks.Expert Systems with Applications, pp. 129101, 2025a. Xin Yi, Yue Li, Dongsheng Shi, Linlin Wang, Xiaoling Wang, and Liang He. Unified defense for large langua...

    arXiv

  3. [11]

    A survey on evaluating quality and trustworthiness in LLM-generated data.Transactions on Machine Learning Research, 2026a

    Kaituo Zhang, Mingzhi Hu, Hoang Anh Duy Le, Fariha Kabir Torsha, Zhimeng Jiang, Minh Khai Bui, Chia-Yuan Chang, Yu-Neng Chuang, Zhen Xiong, Ying Lin, Guanchu Wang, and Na Zou. A survey on evaluating quality and trustworthiness in LLM-generated data.Transactions on Machine Learning Research, 2026a. ISSN 2835-8856. URLhttps://openreview.net/ forum?id=f2gS9L...

    arXiv

  4. [12]

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al

    URLhttps://openreview.net/forum?id= 1w9Hay7tvm. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

    arXiv

  5. [2017]

    Sem: Reinforcement learning for search-efficient large language models.arXiv preprint arXiv:2505.07903,

    Zeyang Sha, Shiwen Cui, and Weiqiang Wang. Sem: Reinforcement learning for search-efficient large language models.arXiv preprint arXiv:2505.07903,

    arXiv

  6. [2018]

    Harness-bench: Measuring harness effects across models in realistic agent workflows.arXiv preprint arXiv:2605.27922,

    Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, et al. Harness-bench: Measuring harness effects across models in realistic agent workflows.arXiv preprint arXiv:2605.27922,

    arXiv

  7. [2020]

    From construction to injection: Edit-based fingerprints for large language models.arXiv preprint arXiv:2509.03122, 2025a

    Yue Li, Xin Yi, Dongsheng Shi, Yongyi Cui, Gerard de Melo, and Linlin Wang. From construction to injection: Edit-based fingerprints for large language models.arXiv preprint arXiv:2509.03122, 2025a. Yue Li, Xin Yi, Dongsheng Shi, Gerard De Melo, Xiaoling Wang, and Linlin Wang. Hierarchical safety realignment: Lightweight restoration of safety in pruned lar...

    arXiv 2025

  8. [2022]

    Deepseek-v4: Towards highly efficient million- token context intelligence.arXiv preprint arXiv:2606.19348,

    Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million- token context intelligence.arXiv preprint arXiv:2606.19348,

    arXiv

Show all 12 references
  1. [2023]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    arXiv

  2. [2024]

    Surgent: A surgical multi-agent assistance system across the perioperative workflow.arXiv preprint arXiv:2605.29368, 2026a

    8 Dongsheng Shi, Yue Li, Xin Yi, Yongyi Cui, Huawei Feng, and Linlin Wang. Surgent: A surgical multi-agent assistance system across the perioperative workflow.arXiv preprint arXiv:2605.29368, 2026a. Dongsheng Shi, Xin Yi, Yue Li, and Linlin Wang. Benchmarking large language mo...

    arXiv

  3. [2025]

    Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.arXiv preprint arXiv:2402.03300,

    arXiv

  4. [2026]

    net/forum?id=OuGAwwAT8G

    URLhttps://openreview. net/forum?id=OuGAwwAT8G. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    2009 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.