Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

TaoSR-AGRL claims that dense, rule-aware rewards plus adaptive guided replay break the GRPO ceiling for e-commerce relevance prediction, yielding state-of-the-art results offline and online.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 10:48 UTC pith:5E22LMDZ

load-bearing objection A plausible engineering combination with a real evaluation flaw: the headline W-Eval gain is selected over a threshold sweep on the same test set, so the 'breaks the ceiling' claim is overstated as written. the 3 major comments →

arxiv 2510.08048 v4 pith:5E22LMDZ submitted 2025-10-09 cs.IR cs.AIcs.CL

TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance

Jianhui Yang , Yiming Jin , Pengkun Jiao , Chenhe Dong , Zerui Huang , Shaowei Yao , Xiaojiang Zhou , Dan Ou
show 1 more author
Haihong Tang
This is my paper
classification cs.IR cs.AIcs.CL
keywords e-commerce searchquery-product relevancelarge language modelsreinforcement learningreward shapingguided replayGRPOreward sparsity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that state-of-the-art LLM alignment methods for e-commerce relevance prediction—SFT/DPO and then GRPO—plateau because the final relevance label is a sparse, all-or-nothing reward: the model can stumble into the right label via a wrong reasoning chain, and genuinely hard long-tail cases rarely yield any reward at all. To break that plateau, the authors add two mechanisms to GRPO. Rule-aware Reward Shaping turns the single outcome reward into a dense, weighted combination of category, attribute, reasoning-fidelity, and format rewards, with a hard gate so that credit is earned only when the final label and format are correct. Adaptive Guided Replay then detects batches with low average reward, diagnoses which reasoning dimension is weak, and appends minimal ground-truth guidance to the prompt for a second rollout. On production search data, the resulting model improves Macro-F1 by 2.28 points over the best baseline on the in-the-wild test set and wins online side-by-side human evaluation by 7.11% GSB, and the model has been deployed.

Core claim

The central claim is that the bottleneck in GRPO-based relevance training is not model capacity but reward sparsity and the reward-hacking it permits. The paper's evidence: standard GRPO and a process-reward variant (GRPO-PR) both stall at roughly a 0.2-point Macro-F1 gain over DPO, while TaoSR-AGRL gains 2.28 points on the in-the-wild set, with the largest single-class improvement in Class-1 (Irrelevant), up 7.28 points. On the paper's own terms, the framework works by making the reward dense and conditional: a validity gate zeroes all fine-grained credit unless the final relevance label and the output format are clean, and a reasoning reward checks that the intermediate category and attrib

What carries the argument

Rule-aware Reward Shaping decomposes the final relevance judgment into dense, weighted rewards over three components—label (category, attribute, relevance), reasoning (rule adherence and self-consistency), and format—all multiplied by a hard gate that requires a correct final label and well-formed output. Adaptive Guided Replay is the second mechanism: for a batch whose average reward falls below a threshold, it computes per-dimension accuracy, generates a textual guidance for every dimension below threshold, appends it to the input, and resamples the policy. The two together recast GRPO's sparse reward into a dense, curriculum-like signal and target exploration toward the model's specific b

Load-bearing premise

The entire advantage rests on test sets built from the same four hand-picked query categories used for training, with no external benchmark or independent dataset, so the gains might reflect tuning to those categories rather than general reasoning ability.

What would settle it

Run the trained model on queries outside the four training categories (e.g., price-comparison or trending-item queries) with fresh human labels and compute Macro-F1 against the DPO baseline; if the advantage disappears or reverses on this held-out distribution, the claimed long-tail generalization collapses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the gains hold on other query distributions, e-commerce teams can improve LLM relevance beyond DPO/GRPO without training a separate reward model or judge.
  • The hard gate plus reasoning reward gives a concrete recipe against reward hacking: only reward fine-grained steps on trajectories whose final answer and format are already correct.
  • The large Class-1 (Irrelevant) improvement suggests better rejection of irrelevant items in production search, which should reduce over-matching and improve user trust.
  • The paper's deployment note that relevance-only optimization initially hurt commercial metrics implies that retrieval and ranking must be designed jointly with business objectives—a direct corollary of the reported online results.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because training deliberately excluded all-correct and all-wrong samples, the measured gains may be concentrated on medium-difficulty cases; whether the method lifts the truly extreme long tail is untested and is a natural next experiment.
  • The 'diagnose the failing dimension, reveal only that' replay scheme is a generalized sparse-reward cure that could transfer to other LLM reasoning tasks that have decomposable steps but no formal rule table.
  • The paper's silent assumption that the four hand-picked query categories define 'challenging' means the framework's value on unforeseen query types or on other markets is open; a cross-market transfer test would settle it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes TaoSR-AGRL, an RL post-training method for LLM-based e-commerce search relevance on Taobao. Starting from a DPO-aligned MoE model, it adds (1) Rule-aware Reward Shaping, which replaces sparse terminal rewards with a gated combination of label, reasoning, and format rewards, and (2) Adaptive Guided Replay, which identifies low-reward rollouts and regenerates them with ground-truth-based guidance. Offline experiments on private balanced and in-the-wild test sets report Macro-F1 improvements over DPO/GRPO baselines (e.g., 61.53 vs 59.25 on W-Eval), and an online side-by-side evaluation reports GSB +7.11%. The paper concludes that the framework alleviates reward sparsity and has been deployed on Taobao.

Significance. If the reported gains are unbiased, the paper offers a practical and interpretable solution to reward sparsity in industrial relevance models, with a potentially transferable combination of dense reward shaping and selective ground-truth guidance. The work is strengthened by the use of a real deployed system, honest reporting of an initial GMV decline and its remediation, and ablations that isolate the reasoning reward's effect on rule-adherence (RAR 87.25 vs 79.84). However, the evaluation protocol currently does not support the strength of the claims: the key hyperparameter τ is selected on the test set, no uncertainty estimates are provided, and the private test sets share the same hand-picked query categories as training. The central idea is plausible, but the evidence needs repair before the SOTA claim can be accepted.

major comments (3)
  1. [Section 4.4, Table 3 and Section 4.1] The guidance trigger threshold τ is tuned directly on the In-the-Wild Eval Set. Table 3 reports Macro-F1 for six τ values on W-Eval and identifies τ=0.1 as the peak (61.53); Table 1 then reports this same 61.53 as the headline result. This is test-set selection: the final number is a selected maximum, not an unbiased evaluation. Since GRPO-PR's coefficients were also optimized via 'extensive hyperparameter search' with no validation split described, the comparison may be unfair. Please use a held-out validation set for hyperparameter selection and report the corresponding test result, or report all τ values with a proper selection correction. The reward weights w_cate, w_attr, w_reason should also be justified on a validation set.
  2. [Section 4.3, Tables 1–2, Section 4.5] The paper claims 'significantly outperforms' and 'consistently improves', but no error bars, confidence intervals, or significance tests are reported. On B-Eval, TaoSR-AGRL's Macro-F1 is only 0.44 pt above GRPO (69.26 vs 68.82) and Class-3/Class-4 differences are 0.15 and 0.09 pt; on W-Eval, Class-4 is 0.03 pt above GRPO. With no variance estimates or multiple seeds, these differences may be noise. The online GSB result (+7.11% on 2,000 queries) likewise lacks a confidence interval. Please add at least multiple-seed means with standard deviations or significance tests for the main comparisons.
  3. [Section 3.1 and Section 4.1] Both training and test data are sampled from the same four hand-picked query categories (negation, affordable alternatives, Q&A, knowledge-based), and the training pipeline deliberately removes all-correct and all-wrong samples. The W-Eval and B-Eval are thus not independent of the training design choices, and no held-out query category or external corpus is used. The claims about 'generalization' and 'robustness' to long-tail/unseen distributions are not substantiated beyond the specific Taobao distribution. Adding a held-out category or an external relevance benchmark would materially strengthen the claims.
minor comments (5)
  1. [Figure 2 caption] Typo: 'Entrpoy' should be 'Entropy'. Also, Section 4.5 uses 'Negative' while Section 3.1 uses 'negation'; keep terminology consistent.
  2. [Equations (1)–(3) vs Section 3.3] Equation (3) triggers guided replay based on the mean reward relative to τ, whereas Section 3.3 says per-dimension in-batch accuracy is compared against τ. Please clarify the exact trigger condition to ensure reproducibility.
  3. [Table 8] The guidance text in the chiffon-dress case says 'with Attribute and Relevance as Mismatch', but the correct outcome is 4-Excellent and the response concludes 'Excellent'. The guidance appears to be mis-transcribed; it should presumably say 'Excellent'.
  4. [Section 4.1, Implementation Details] The reward weights w_cate=0.4, w_attr=0.4, w_reason=0.2 are said to be set by 'preliminary experiments', but no results of those experiments are given. Please provide a small sensitivity table or describe the selection protocol.
  5. [General formatting] The manuscript still contains ACM template placeholders (e.g., 'Conference acronym 'XX', 2018, Woodstock, NY', received dates) and inconsistent spacing like 'TaoSR -AGRL'. These should be cleaned before publication.

Circularity Check

0 steps flagged

No circularity: the claimed gains over DPO/GRPO are empirical comparisons on held-out and online data; no reward component or evaluation metric is defined as its own prediction.

full rationale

TaoSR-AGRL is a training procedure, not a closed-form derivation. Its reward shaping (Eqs. 4-5) and guided replay (Eqs. 1-3) use ground-truth labels and the Taobao relevance rules during training, which is standard supervised/RL signal; evaluation is on B-Eval, W-Eval, and online human side-by-side comparisons, none of which are used to construct the reward functions. The dependence on TaoSR1 is a prior system by overlapping authors, but it supplies the task setup and base model, not a uniqueness theorem or the AGRL result; the +2.28 pt W-Eval Macro-F1 and +7.11% GSB margins are independently reported comparisons. Section 4.4's tau sweep on W-Eval is an evaluation-protocol risk (test-set tuning) that could inflate the headline number, and the paper's own note about GMV decline in Section 4.5 is a business limitation; neither makes a derived quantity definitionally equal to its input, so the circularity score remains 0.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claim rests on a domain-specific rule table, human labels, and a fixed output format; the method itself introduces no new entities. Free parameters (reward weights, tau) are tuned through ablation on the test distribution without separate validation, which is a material limitation.

free parameters (3)
  • Reward weights w_cate, w_attr, w_reason = 0.4, 0.4, 0.2
    Weights in Eq. (4) for label and reasoning rewards, set through 'preliminary experiments' with no separate validation described.
  • Guidance trigger threshold tau = 0.1
    Selected by ablation over {0, 0.1, 0.2, 0.3, 0.4, 0.5} on the W-Eval test set (Table 3); tuning is performed on the evaluation distribution.
  • Difficulty filtering threshold range = [0.01, 0.9]
    Used in Difficulty-Aware Dynamic Sampling to remove easy and label-noisy samples; range is stated without justification.
axioms (5)
  • domain assumption The Relevance Derivation Rules (Table 6) correctly define query-item relevance in Taobao.
    The reward and the RAR evaluation metric both assume the mapping from (Category, Attribute) to Relevance in Table 6 is the correct business logic; if the rules are wrong or incomplete, the trained model inherits that error.
  • domain assumption Human annotations of category, attribute, and relevance labels are accurate and consistent.
    The reward, guidance, and evaluation all rely on these labels as ground truth.
  • domain assumption The four challenging query categories (negation, alternatives, Q&A, knowledge) are representative of the long-tail distribution.
    Training and test sets are constructed from these same categories; the paper does not test on queries outside them.
  • domain assumption The 'respond-then-think' format (final label first, then chain-of-thought) is a valid modeling choice.
    Format reward and reasoning structure depend on this fixed output schema.
  • ad hoc to paper The gating mechanism that nullifies all fine-grained rewards unless the final label and format are correct is beneficial.
    This is a design choice validated only by the paper's own ablations, not a standard RL principle.

pith-pipeline@v1.3.0-alltime-deepseek · 17648 in / 11975 out tokens · 116303 ms · 2026-08-04T10:48:37.606745+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance." pith.science (2026). https://pith.science/paper/5E22LMDZ

@misc{pith2026251008048,
  author       = {Pith},
  title        = {Pith review of: TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5E22LMDZ}},
  note         = {Machine review of arXiv:2510.08048}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Query-product relevance prediction is fundamental to e-commerce search and has become even more critical in the era of AI-powered shopping, where semantic understanding and complex reasoning directly shape the user experience and business conversion. Large Language Models (LLMs) enable generative, reasoning-based approaches, typically aligned via supervised fine-tuning (SFT) or preference optimization methods like Direct Preference Optimization (DPO). However, the increasing complexity of business rules and user queries exposes the inability of existing methods to endow models with robust reasoning capacity for long-tail and challenging cases. Efforts to address this via reinforcement learning strategies like Group Relative Policy Optimization (GRPO) often suffer from sparse terminal rewards, offering insufficient guidance for multi-step reasoning and slowing convergence. To address these challenges, we propose TaoSR-AGRL, an Adaptive Guided Reinforcement Learning framework for LLM-based relevance prediction in Taobao Search Relevance. TaoSR-AGRL introduces two key innovations: (1) Rule-aware Reward Shaping, which decomposes the final relevance judgment into dense, structured rewards aligned with domain-specific relevance criteria; and (2) Adaptive Guided Replay, which identifies low-accuracy rollouts during training and injects targeted ground-truth guidance to steer the policy away from stagnant, rule-violating reasoning patterns toward compliant trajectories. TaoSR-AGRL was evaluated on large-scale real-world datasets and through online side-by-side human evaluations on Taobao Search. It consistently outperforms DPO and standard GRPO baselines in offline experiments, improving relevance accuracy, rule adherence, and training stability. The model trained with TaoSR-AGRL has been successfully deployed in the main search scenario on Taobao, serving hundreds of millions of users.

Figures

Figures reproduced from arXiv: 2510.08048 by Chenhe Dong, Dan Ou, Haihong Tang, Jianhui Yang, Pengkun Jiao, Shaowei Yao, Xiaojiang Zhou, Yiming Jin, Zerui Huang.

Figure 1
Figure 1. Figure 1: Overview of the proposed TaoSR-AGRL for e-commerce search relevance. (1) Data Construction builds a balanced and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Ablation on training dynamics: Compared to its variants, TaoSR-AGRL exhibits (a) higher sample efficiency, (b) greater [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Synthetic Data Powers Product Retrieval for Long-tail Knowledge-Intensive Queries in E-commerce Search

    cs.IR 2026-02 unverdicted novelty 5.0

    Synthetic data generated via LLM query rewriting improves retrieval recall and user experience for long-tail knowledge-intensive queries in e-commerce search.

  2. K-CARE: Knowledge-driven Symmetrical Contextual Anchoring and Analogical Prototype Reasoning for E-commerce Relevance

    cs.IR 2026-04 unverdicted novelty 4.0

    K-CARE uses behavior-derived anchoring and expert prototype analogies to ground LLMs and improve relevance on knowledge-intensive e-commerce cases.

Reference graph

Works this paper leans on

64 extracted references · 31 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Akiko Aizawa. 2003. An information-theoretic perspective of tf–idf measures. Information Processing & Management 39, 1 (2003), 45–65. https://doi.org/10. 1016/S0306-4573(02)00021-3

    2003

  2. [2]

    Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled Sampling for Sequence Prediction with Recurrent Neural Net- works. In Advances in Neural Information Processing Systems , C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2015/fi...

    2015

  3. [3]

    Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. InProceedings of the 22nd International Conference on Machine Learning (Bonn, Germany) (ICML ’05). Association for Computing Machinery, New York, NY, USA, 89–96. https: //doi.org/10.1145/1102351.1102363

    arXiv 2005

  4. [4]

    Christopher J. C. Burges. 2010. From RankNet to LambdaRank to LambdaMART: An Overview. https://api.semanticscholar.org/CorpusID:397316

    2010

  5. [5]

    Le, Sergey Levine, and Yi Ma

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. 2025. SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training. arXiv:2501.17161 [cs.AI] https://arxiv.org/abs/2501.17161

    Pith/arXiv arXiv 2025

  6. [6]

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. 2025. Process Reinforcement through Implicit Rewards. arXiv:2502.01456 [cs.LG] https://...

    Pith/arXiv arXiv 2025

  7. [7]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, and Ruoyu Zhang et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capabil- ity in LLMs via Reinforcement Learning. arXiv:2501.12948 [cs.CL] https: //arxiv.org/abs/2501.12948

    Pith/arXiv arXiv 2025

  8. [8]

    Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. 2022. Exploration- guided reward shaping for reinforcement learning under sparse rewards. In Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’22). Curran Associates Inc., Red Hook, NY, USA, Article 422, 14 pages

    2022

  9. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy...

    doi:10.18653/v1/n 2019

  10. [10]

    Chenhe Dong, Shaowei Yao, Pengkun Jiao, Jianhui Yang, Yiming Jin, Zerui Huang, Xiaojiang Zhou, Dan Ou, and Haihong Tang. 2025. TaoSR1: The Thinking Model for E-commerce Relevance Search. arXiv:2508.12365 [cs.IR] https://arxiv.org/ abs/2508.12365

    arXiv 2025

  11. [11]

    Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune

  12. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, and Ahmad Al-Dahle et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  13. [13]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024. A Survey on LLM-as-a-Judge. ArXiv abs/2411.15594 (2024). https: //api.semanticscholar.org/CorpusID:274234014

    Pith/arXiv arXiv 2024

  14. [14]

    Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. 2024. Teaching Large Language Models to Reason with Reinforcement Learning. arXiv:2403.04642 [cs.LG] https://arxiv.org/abs/2403. 04642

    Pith/arXiv arXiv 2024

  15. [15]

    Courville, Alessandro Sor- doni, and Rishabh Agarwal

    Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron C. Courville, Alessandro Sor- doni, and Rishabh Agarwal. 2024. V-STaR: Training Verifiers for Self-Taught Rea- soners. ArXiv abs/2402.06457 (2024). https://api.semanticscholar.org/CorpusID: 267617275

    Pith/arXiv arXiv 2024

  16. [16]

    Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management . 2333–2338

    2013

  17. [17]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. Qwen2.5-Coder Technical Report. arXiv:2409.12186 [cs.CL...

    Pith/arXiv arXiv 2024

  18. [18]

    Arthur Juliani and Jordan T. Ash. 2024. A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning. arXiv:2405.19153 [cs.LG] https://arxiv.org/abs/ 2405.19153

    Pith/arXiv arXiv 2024

  19. [19]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open- Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for C...

    doi:10.18653/v1/2020.emnlp-main.550 2020

  20. [20]

    Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia

  21. [21]

    Shuangtao Li, Shuaihao Dong, Kexin Luan, Xinhan Di, and Chaofan Ding. 2025. Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search. arXiv:2501.01478 [cs.AI] https://arxiv.org/abs/2501.01478

    Pith/arXiv arXiv 2025

  22. [22]

    Gongrui Nan, Siye Chen, Jing Huang, Mengyu Lu, Dexun Wang, Chunmei Xie, Weiqi Xiong, Xianzhou Zeng, Qixuan Zhou, Yadong Li, and Xingzhong Xu. 2025. NGRPO: Negative-enhanced Group Relative Policy Optimization. arXiv:2509.18851 [cs.LG] https://arxiv.org/abs/2509.18851

    arXiv 2025

  23. [23]

    Vaskar Nath, Elaine Lau, Anisha Gunjal, Manasi Sharma, Nikhil Baharte, and Sean Hendryx. 2025. Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models. arXiv:2506.13923 [cs.LG] https://arxiv.org/abs/2506.13923

    Pith/arXiv arXiv 2025

  24. [24]

    Rodrigo Nogueira and Kyunghyun Cho. 2020. Passage Re-ranking with BERT. arXiv:1901.04085 [cs.IR] https://arxiv.org/abs/1901.04085

    Pith/arXiv arXiv 2020

  25. [25]

    Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Ja- cob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show Your Work: Scratchpads for Intermediate Computation with Language Models. arXiv:2112.00114 [cs.LG] https://arxiv.org/abs/2112.00114

    Pith/arXiv arXiv 2021

  26. [26]

    OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, and Ahmed El-Kishky et al. 2024. OpenAI o1 System Card. arXiv:2412.16720 [cs.AI] https: //arxiv.org/abs/2412.16720

    Pith/arXiv arXiv 2024

  27. [27]

    André Quadros, Cassio Silva, and Ronnie Alves. 2025. LLM-Driven Intrinsic Motivation for Sparse Reward Reinforcement Learning. arXiv:2508.18420 [cs.LG] https://arxiv.org/abs/2508.18420

    Pith/arXiv arXiv 2025

  28. [28]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290 [cs.LG] https://arxiv.org/ abs/2305.18290

    Pith/arXiv arXiv 2024

  29. [29]

    Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, and Tianlu Wang. 2025. Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a- Judge. ArXiv abs/2501.18099 (2025). https://api.semanticscholar.org/CorpusID: 275993427

    Pith/arXiv arXiv 2025

  30. [30]

    ByteDance Seed, Yuyu Zhang, Jing Su, Yifan Sun, Chenguang Xi, Xia Xiao, Shen Zheng, Anxiang Zhang, Kaibo Liu, Daoguang Zan, Tao Sun, Jinhua Zhu, Shulin Xin, Dong Huang, Yetao Bai, Lixin Dong, Chao Li, Jianchong Chen, Hanzhi Zhou, Yifan Huang, Guanghan Ning, Xierui Song, Jiaze Chen, Siyao Liu, Kai Shen, Liang Xiang, and Yonghui Wu. 2025. Seed-Coder: Let th...

    Pith/arXiv arXiv 2025

  31. [31]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeek- Math: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 [cs.CL] https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  32. [32]

    Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger

  33. [33]

    Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. 2022. Curriculum learning: A survey. International Journal of Computer Vision 130, 6 (2022), 1526– 1565

    2022

  34. [34]

    Svore and Christopher J.C

    Krysta M. Svore and Christopher J.C. Burges. 2009. A machine learning ap- proach for improved BM25 retrieval. In Proceedings of the 18th ACM Confer- ence on Information and Knowledge Management (Hong Kong, China) (CIKM ’09). Association for Computing Machinery, New York, NY, USA, 1811–1814. https://doi.org/10.1145/1645953.1646237

    arXiv 2009

  35. [35]

    Tian Tang, Zhixing Tian, Zhenyu Zhu, Chenyang Wang, Haiqing Hu, Guoyu Tang, Lin Liu, and Sulong Xu. 2025. LREF: A Novel LLM-based Relevance Framework for E-commerce Search. In Companion Proceedings of the ACM on Web Conference 2025 (WWW ’25) . ACM, 468–475. https://doi.org/10.1145/3701716.3715246

    arXiv 2025

  36. [36]

    Hieu Tran, Zonghai Yao, and Hong Yu. 2025. Exploiting Tree Structure for Credit Assignment in RL Training of LLMs. arXiv:2509.18314 [cs.CL] https: //arxiv.org/abs/2509.18314

    arXiv 2025

  37. [37]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL] https://arxiv.org/abs/1706.03762

    Pith/arXiv arXiv 2023

  38. [38]

    Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, Zichen Liu, Haizhou Zhao, Dakai An, Lunxi Cao, Qiyang Cao, Wanxi Deng, Feilei Du, Yiliang Gu, Jiahe Li, Xiang Li, Mingjie Liu, Yijia Luo, Zihe Liu, Yadao Wang, Pei Wang, Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Yang et ...

    Pith/arXiv arXiv 2018

  39. [39]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171 [cs.CL] https://arxiv.org/abs/2203.11171

    Pith/arXiv arXiv 2023

  40. [40]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    Pith/arXiv arXiv 2025

  41. [41]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, W...

  42. [42]

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. 2025. SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild. arXiv:2503.18892 [cs.LG] https: //arxiv.org/abs/2503.18892

    Pith/arXiv arXiv 2025

  43. [43]

    Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, Wanli Ouyang, and Dongzhan Zhou

  44. [44]

    Jixiao Zhang and Chunsheng Zuo. 2025. GRPO-LEAD: A Difficulty-Aware Rein- forcement Learning Approach for Concise Mathematical Reasoning in Language Models. ArXiv abs/2504.09696 (2025). https://api.semanticscholar.org/CorpusID: 277780631

    arXiv 2025

  45. [45]

    arXiv:2503.14476 [cs.LG] https://arxiv.org/abs/2503.14476

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476 [cs.LG] https://arxiv.org/abs/2503.14476

    Pith/arXiv arXiv

  46. [46]

    Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. 2025. On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting. arXiv:2508.11408 [cs.LG] https://arxiv.org/abs/2508.11408

    arXiv 2025

  47. [47]

    cashmere clothing

    Xiaoying Zhang, Hao Sun, Yipeng Zhang, Kaituo Feng, Chaochao Lu, Chao Yang, and Helen Meng. 2025. Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback. arXiv:2506.03106 [cs.CL] https: //arxiv.org/abs/2506.03106 TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance Conference acronym ’XX...

    Pith/arXiv arXiv 2025

  48. [48]

    arXiv:2410.02884 [cs.AI] https://arxiv.org/abs/2410.02884

    LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathe- matical Reasoning. arXiv:2410.02884 [cs.AI] https://arxiv.org/abs/2410.02884

    Pith/arXiv arXiv

  49. [50]

    Kaiyi Zhang, Ang Lv, Jinpeng Li, Yongbo Wang, Feng Wang, Haoyuan Hu, and Rui Yan. 2025. StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason. arXiv:2507.02841 [cs.AI] https://arxiv.org/abs/2507.02841

    Pith/arXiv arXiv 2025

  50. [56]

    Undyed Cashmere Coat contains cashmere

    Attribution Match: ... Undyed Cashmere Coat contains cashmere ... The conclusion is Excellent

  51. [57]

    Relevance label is 4-Excellent

    Judgement: ... Relevance label is 4-Excellent. Prompt with Guidance Response Query: cashmere clothing. Item: Women’s Undyed Cashmere Coat ... Guidance: Perform relevance reasoning per the guid- ance below, with Attribute and Relevance as Mis- match. 2-Mismatch

  52. [59]

    Item: This cashmere coat

  53. [61]

    Undyed Cashmere Coat contains cashmere, but the content is below 50%

    Attribution Match: ... Undyed Cashmere Coat contains cashmere, but the content is below 50% ... The conclusion is Mismatch

  54. [62]

    chiffon dress

    Judgement: ... Relevance label is 2-Mismatch. Table 8: Impact of Adaptive Guided Replay Case Study: Impact of Adaptive Guided Replay on a query for “chiffon dress”. Original Prompt Response Query: chiffon dress. Item: French-style Floral Vacation Dress ... 2-Mismatch

  55. [66]

    The dress contains chiffon, but the content is less than 50%

    Attribution Match: ... The dress contains chiffon, but the content is less than 50% ... The conclusion is Mismatch

  56. [67]

    Relevance label is 2-Mismatch

    Judgement: ... Relevance label is 2-Mismatch. Prompt with Guidance Response Query: chiffon dress. Item: French-style Floral Vacation Dress ... Guidance: Perform relevance reasoning per the guid- ance below, with Attribute and Relevance as Mis- match. 4-Excellent

  57. [68]

    Query: The user is looking for

  58. [69]

    Item: This floral vacation dress

  59. [70]

    The conclusion is Excellent

    Category Match: ... The conclusion is Excellent

  60. [71]

    The dress contains chiffon

    Attribution Match: ... The dress contains chiffon ... The conclusion is Excellent

  61. [72]

    Relevance label is 4-Excellent

    Judgement: ... Relevance label is 4-Excellent. Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

    2007

  62. [2019]

    arXiv preprint arXiv:1901.10995 (2019)

    Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995 (2019)

    Pith/arXiv arXiv 1901

  63. [2024]

    arXiv:2406.18629 [cs.LG] https://arxiv.org/abs/2406.18629

    Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs. arXiv:2406.18629 [cs.LG] https://arxiv.org/abs/2406.18629

    Pith/arXiv arXiv

  64. [2025]

    arXiv:2209.13085 [cs.LG] https://arxiv.org/abs/2209.13085

    Defining and Characterizing Reward Hacking. arXiv:2209.13085 [cs.LG] https://arxiv.org/abs/2209.13085

    Pith/arXiv arXiv