{"id":"ad86c290-07f1-4d6f-9e3c-283f498c254f","arxiv_id":"2608.03527","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RubricRanker is a reranker trained with LLM-generated search rubrics and reinforcement learning to select high-quality document sets, outperforming baselines on deep research and RAG benchmarks.","lead":"This paper trains a document reranker, RubricRanker, that selects sets of documents for deep research agents using query-specific rubrics generated by an LLM. It reports gains over existing rerankers on deep research and RAG benchmarks, but the training signal and evaluation both depend heavily on LLM judgments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"WebWalkerQA contamination risk: training queries come from WebWalker-Silver, evaluation uses WebWalkerQA, and no deduplication is reported; this benchmark drives most of the claimed 2.6-point gain.","rationale":"The reader's weakest_assumption focused on the reliability of GPT-5.1-generated rubrics and rewards, which is a legitimate concern about the self-referential LLM supervision loop. However, the RAG results with exact-match metrics provide an objective sanity check that the trained reranker does improve downstream generation, partially mitigating the concern that the method only games LLM judges. The more load-bearing issue is the likely train/test overlap on WebWalkerQA: the training data explicitly include WebWalker-Silver, while WebWalkerQA is one of the four deep research test sets, and the paper reports no leakage control. The claimed 2.6-point gain is concentrated on WebWalkerQA, so if leakage is present, the headline number collapses. This is a concrete, checkable methodological flaw that directly affects the central claim, and it was not the reader's primary identified weakness. The right verdict remains CONDITIONAL, because the paper may still be sound if the authors can demonstrate decontamination and report statistical significance; no change from the reader's conditional recommendation is needed, but the condition should now include a decontamination check.","tokens_in":23513,"tokens_out":3486,"duration_ms":41691,"concrete_test":"Run a deduplication check between the WebWalker-Silver training trajectories (queries, sub-queries, and visited URLs) and the 200 randomly sampled WebWalkerQA test queries. Re-evaluate RubricRanker and all baselines after removing any overlapping test queries, and recompute the Table 1 average with a bootstrapped confidence interval or paired significance test. If the WebWalkerQA gain shrinks or the average advantage over Rank4Gen drops below ~1 point, the headline claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result—a 2.6-point average improvement on deep research benchmarks—is driven predominantly by WebWalkerQA (Table 1: 58.0 vs. 52.0, a 6-point jump), while gains on the other three benchmarks are 1.9, 0.1, and 1.4 points. However, the training set includes sub-queries extracted from WebWalker-Silver (§4.1.2, Table 4), which belongs to the same WebWalker dataset family as the WebWalkerQA test benchmark. The paper does not report any deduplication between the sampled WebWalkerQA test queries (200 queries, §5.1) and the WebWalker-Silver training trajectories. If the same or similar user questions, sub-queries, or URLs appear in both, the headline gain is inflated by direct train/test leakage. Moreover, WebWalkerQA is the only deep research benchmark where RubricRanker beats the best baseline by a large margin; without it, the average advantage falls to roughly 1.1 points, which is within the range of noise given no error bars or significance tests are reported. If this contamination is confirmed, the central claim of consistent superiority on deep research benchmarks would not survive.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RubricRanker, a document-set reranker for deep research agents and RAG. Search-oriented rubrics are constructed by prompting GPT-5.1 to expand a fixed meta-rubric framework (set-level relevance, conciseness, consistency; document-level source authority and timeliness) into query-specific rubrics with weights. A two-stage training pipeline is used: rubrics-guided SFT with GPT-5.1 silver labels, followed by GRPO RL with a rubric-based reward computed by a GPT-5.1 judge. At inference, RubricRanker selects and ranks a subset of retrieved documents without access to rubrics. Experiments report an average gain of 2.6 points over the strongest baseline on four deep research benchmarks and about 2 EM points over Rank4Gen on five RAG benchmarks, plus ablations and a search-call analysis.","tokens_in":23829,"tokens_out":3746,"duration_ms":45692,"significance":"If the reported gains are robust, the paper makes a useful contribution by moving reranking supervision from single-document relevance to explicit set-level criteria—relevance, conciseness, consistency, authority, and timeliness—and by showing that such rubrics can be distilled into a small reranker. The RAG exact-match results provide an external, non-LLM-judge validation that partially supports the method. The authors also include ablations isolating the SFT and RL stages, a sensitivity analysis over the number of reranked documents, and a search-call efficiency analysis, and they release code. These are strengths. However, the deep-research claim rests on small samples, no variance reporting, a likely train/test overlap in the largest-gain benchmark, and an output-budget asymmetry between RubricRanker and vanilla baselines; these issues need to be resolved before the central claim can be accepted.","major_comments":[{"comment":"WebWalkerQA contamination risk is load-bearing. Training sub-queries are extracted from WebWalker-Silver (§4.1.2, Table 4), and evaluation uses WebWalkerQA from the same dataset family (§5.1). No deduplication between the 200 sampled WebWalkerQA test queries and the WebWalker-Silver training trajectories is reported. WebWalkerQA contributes the largest single gain in Table 1 (58.0 vs. 52.0, +6.0), while the other three benchmarks show gains of +1.9, +0.1, and +0.9 over the strongest baselines. The paper should report overlap at the question, sub-query, and URL levels, and should present deep-research averages with WebWalkerQA excluded. Without this, the headline 2.6-point improvement cannot be interpreted as evidence of generalization.","section":"§4.1.2 / Table 4 / Table 1"},{"comment":"The comparison is not budget-matched. Section 5.1 states that vanilla rerankers 'output a full ranking' and the top five documents are used, while RubricRanker's inference prompt (Figure 12, and Figure 13 for RAG) allows selection of 'at most 10' documents. If RubricRanker returns more documents on average than the baselines, the observed gains could partly reflect a larger evidence budget rather than better set selection, especially on EM-based RAG benchmarks. Please report the average/median number of selected documents per query for each method and add a controlled condition in which all rerankers return exactly five documents, or match budgets otherwise.","section":"§5.1 / Figures 12–13"},{"comment":"The central 'consistent gains' claim lacks statistical support. Deep-research test sets are only 100 or 200 queries, no seeds are reported for sampling, and no variance, confidence intervals, or significance tests are given. Removing WebWalkerQA, the average deep-research gain over the strongest baseline drops to roughly +1.0 point (HealthBench +1.9, DRB +0.1, ResearchQA +0.9), which is within plausible noise for these sample sizes. The authors should report multiple evaluation runs with different samples, or bootstrap confidence intervals/significance tests, particularly for the non-WebWalkerQA benchmarks where the improvements are small.","section":"§5.1 / Tables 1–2"},{"comment":"There is a self-referential supervision concern that should be addressed. The rubric generator, the SFT teacher, and the RL reward judge are all GPT-5.1, and the deep-research evaluation also uses LLM judges (GPT-4.1, Gemini 2.5 Flash, GPT-4.1-mini). This creates a risk that RubricRanker learns to exploit the preferences of LLM judges rather than intrinsic document-set quality. The RAG exact-match results are reassuring, but they do not validate the deep-research claim. Please add at least one of: (i) an independent human-judged or expert-judged evaluation on a subset of deep-research queries; (ii) an analysis of agreement between the GPT-5.1 teacher/judge and a different model family; or (iii) a direct test of whether the selected sets improve factuality or answer completeness under an objective metric.","section":"§4.2.1 / §4.2.2 / Eq. (1)–(3)"}],"minor_comments":[{"comment":"Typographical issue: 'may not form asetthat' is missing a space; similar spacing errors occur elsewhere in the text.","section":"Abstract / §1"},{"comment":"The text says RubricRanker 'consistently leads to fewer search calls' and 'significantly reduces' them, but no variance or significance test is reported for the search-call counts. Please soften the wording or add error bars.","section":"§5.5"},{"comment":"The meta-rubric framework is asserted as a sufficient decomposition of document-set quality, but this assumption is not tested. A brief validation (e.g., expert or LLM agreement that the generated rubrics cover the information need) would strengthen the methodology.","section":"§4.1.1"},{"comment":"For DRB, the evaluation mixes 50 English and 50 Chinese questions with a macro average. Please clarify whether the reported DRB score is comparable across languages and whether the sampled 100 questions are stratified by language.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper is competently written and the rubric-based idea is timely, but the main quantitative claim currently rests on a small number of evaluator samples, an unequal output budget, and a likely train/test overlap in the benchmark that contributes most of the gain. These are fixable with additional experiments and reporting, so I do not recommend rejection, but the authors should be asked to address all four major comments before publication. I would also encourage the editor to ensure that the reviewers' requests for deduplication analysis and budget-matched comparisons are explicitly answered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's the read.\n\nThe contribution is real as a system paper: RubricRanker is a new combination of LLM-generated hierarchical per-query rubrics with SFT plus rubric-rewarded RL for document-set selection, and at inference it drops the rubrics entirely. The design is coherent — meta-rubrics, teacher silver labels, GRPO with weighted set-level and doc-level rewards. The RAG results, measured by exact match on five closed benchmarks, give credible external support, and the ablations show that SFT, rubrics, and RL each add something. The search-call reduction is a nice practical signal. Citation pattern looks fine; prior set-ranking and rubric-reward work is acknowledged.\n\nThe soft spots are real and mostly in the open. Deep-research evaluation uses only 100–200 sampled queries per benchmark with no seeds, no error bars, and no significance tests. The supervision is largely self-referential: GPT-5.1 generates rubrics, acts as SFT teacher, and scores RL rewards; the deep-research judges are different models, which helps, but the loop is still there. The paper's own Limitations section concedes cost, small test samples, and dependence on agent generation quality. That honesty counts.\n\nThe bigger issue is the WebWalkerQA leak risk. Training queries include sub-queries from WebWalker-Silver, the same dataset family as the WebWalkerQA test set, and no deduplication is reported. WebWalkerQA alone contributes 6.0 points of the headline 2.6-point average gain; without it the average gain over the other three benchmarks is about 1.1 points, comfortably inside noise given no variance. That does not prove leakage, but it makes the \"consistently superior\" claim unsafe as written. A referee should ask for dedup details and pointwise gains with variance.\n\nWho is this for? People working on retrieval and reranking for agentic search and RAG. The method is worth knowing even if the headline number needs verification. Bottom line: this deserves peer review, not desk rejection. The paper should either fix the leak analysis or soften the claim, and it needs error bars or significance tests before acceptance.","headline":"Contributions are real, but the evidence is thinner than the abstract implies: a worthwhile system paper with a WebWalkerQA contamination risk and no variance reporting.","tokens_in":24306,"tokens_out":2495,"would_cite":false,"duration_ms":31165,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reranker trained on query-specific search rubrics selects document sets that better support deep research agents, outperforming the strongest baseline by 2.6 points and generalizing to RAG.","keywords":["deep research agents","document reranking","search rubrics","set-level relevance","reinforcement learning","retrieval-augmented generation","LLM judge","document set selection"],"falsifier":"Replace GPT-5.1 as teacher, reward judge, and rubric generator with a different, independently calibrated judge (e.g., human rubric annotations on a sample of document sets, or a smaller open-source judge), retrain RubricRanker, and check whether the reported gains on deep research and RAG benchmarks survive; if the gains disappear or reverse, the central claim that rubric-based training teaches set-level quality would be falsified.","tokens_in":23422,"feed_emoji":"🔍","tokens_out":5975,"duration_ms":49701,"temperature":0.7,"pith_summary":"This paper argues that the right supervision signal for a reranker serving a deep research agent is not single-document relevance but explicit, query-specific rubrics describing what a good document set must jointly cover. To test this, the authors build RubricRanker, trained in two stages: supervised fine-tuning on silver document-set labels produced by a strong LLM guided by hierarchical rubrics, then reinforcement learning with a rubric-weighted reward over set-level and document-level scores. On four deep research benchmarks it outperforms the strongest prior reranker by 2.6 average points, and it transfers to five RAG benchmarks with about a 2-point exact-match gain. If the claim holds, rerankers for agentic search should be optimized for set-level properties—diversity, conciseness, consistency, authority—rather than relevance alone.","feed_headline":"Reranker trained on search rubrics beats top baseline by 2.6 points","feed_subtitle":"Document-set reranking with explicit rubrics cuts deep-research agent search calls and generalizes to RAG benchmarks.","key_machinery":"Hierarchical search rubrics: a two-level rubric structure that starts from fixed meta rubrics (set-level Relevance, Conciseness, Consistency; document-level Source Authority, Timeliness) and expands them per query with query-specific descriptions and 1–5 importance weights, using GPT-5.1 with a synthesized reference answer as the information-need blueprint. These rubrics do two jobs: they prompt a teacher LLM to emit silver document-set labels for SFT, and they are turned into a scalar reward via rubric-weighted aggregation (set-level LLM judge scores plus averaged document-level scores) used in GRPO reinforcement learning. RubricRanker is a Qwen3-8B model fine-tuned this way; at inference i","core_discovery":"The central claim is that document-set quality for agent queries can be captured by hierarchical search rubrics—set-level rubrics for relevance (coverage of key aspects), conciseness (low redundancy), and consistency (compatible facts), plus document-level rubrics for source authority and timeliness—and that training a reranker against such rubrics yields document selections that better support downstream LLM generation than relevance-ranking or answer-preference training alone. RubricRanker is trained without needing rubrics at inference: it takes a query and candidate list and directly outputs a ranked subset. The experimental claim is an average score of 60.1 on four deep research benchma","pith_inferences":["One implication the authors leave implicit: the same rubric construction could be turned into a reusable reward model, which the limitations section names as future work; if that works, rubric-based training need not pay GPT-5.1 evaluation costs at scale.","A testable extension would apply the rubric-generation step to domain-specific agent queries (e.g., legal or financial research) where the authority dimension carries more weight, and measure whether RubricRanker's advantage grows in high-stakes domains.","Because the evaluation itself uses LLM judges for deep research answers, an independent check with human-annotated answer quality would tell whether the 2.6-point gain reflects real evidence quality or shared rubric-format bias between training and evaluation."],"forward_implications":["Rerankers trained on rubric-guided set selection generalize across deep research and RAG tasks, suggesting set-level criteria transfer rather than overfitting to one benchmark family.","Equipping a deep research agent with RubricRanker reduces the number of search actions needed (9–17% fewer calls on evaluated sets), implying better evidence per step and lower latency.","Rubric-based rewards can be computed for queries without verifiable answers, so the training recipe applies to open-ended domains where relevance labels are insufficient.","Generation-oriented rerankers trained on closed-form RAG tasks improve little on open-ended deep research benchmarks, whereas rubric-based training improves both, pointing to a gap in current training signal design."],"supporting_citations":[{"why":"Dr-Tulu-8B supplies the deep research agent used to generate training trajectories and to evaluate final answers.","marker":"(Shao et al., 2025)"},{"why":"Rank4Gen is the strongest generation-oriented baseline that RubricRanker must beat, and the contrast for answer-preference supervision.","marker":"(Fan et al., 2026)"},{"why":"SetR is the set-selection baseline that motivates selecting whole document sets rather than individual relevance rankings.","marker":"(Lee et al., 2025)"},{"why":"RAR provides the rubrics-as-rewards reinforcement learning idea used for rubric-based RL.","marker":"(Gunjal et al., 2025)"},{"why":"GRPO is the reinforcement learning algorithm used in the rubric-based RL stage.","marker":"(Shao et al., 2024)"},{"why":"HotpotQA supplies RAG training queries and serves as one of the five RAG evaluation benchmarks.","marker":"(Yang et al., 2018)"},{"why":"Natural Questions supplies RAG training queries and serves as one of the five RAG evaluation benchmarks.","marker":"(Kwiatkowski et al., 2019)"},{"why":"WebWalker-Silver and WebWalkerQA provide deep research training queries and an evaluation benchmark.","marker":"(Wu et al., 2025)"},{"why":"OpenScholar is a source of open-ended questions from which agent sub-queries are extracted for training.","marker":"(Asai et al., 2024)"}],"fun_headline_variants":["Set-level rubrics train reranker to pick better doc sets for agents","RubricRanker: document-set reranking beats relevance-only baselines","Search rubrics teach rerankers what good doc sets look like","Deep research agents get better docs via set-level rubric training","Explicit set rubrics improve reranking for agent queries and RAG"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that GPT-5.1's rubric-guided labels and rubric-based rewards faithfully measure the true quality of a selected document set; if those LLM judgments are biased or noisy, the training signal passes that bias into the reranker, and the LLM-judged evaluation could hide the problem.","fun_headline_variants_meta":{"raw":{"variants":["Set-level rubrics train reranker to pick better doc sets for agents","RubricRanker: document-set reranking beats relevance-only baselines","Search rubrics teach rerankers what good doc sets look like","Deep research agents get better docs via set-level rubric training","Explicit set rubrics improve reranking for agent queries and RAG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001291,"raw_usage":{"total_tokens":5088,"prompt_tokens":704,"completion_tokens":4384,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":4292}},"tokens_in":448,"tokens_out":4384,"duration_ms":34423,"temperature":1.0,"reasoning_tokens":4292,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:15:51.363992+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace GPT-5.1 as teacher, reward judge, and rubric generator with a different, independently calibrated judge (e.g., human rubric annotations on a sample of document sets, or a smaller open-source judge), retrain RubricRanker, and check whether the reported gains on deep research and RAG benchmarks survive; if the gains disappear or reverse, the central claim that rubric-based training teaches set-level quality would be falsified.","supporting_citations":[],"review_version":1}