{"id":"4d600ab8-88d9-47d3-8806-45b74252821b","arxiv_id":"2501.09186","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SlideGar adapts graph-based adaptive retrieval to listwise LLM rerankers, improving nDCG@10 by up to 13.23% and recall by up to 28.02% without extra LLM inferences.","lead":"A search reranking method called SlideGar feeds the ranked output of an LLM back into a graph of similar documents to pull in relevant results missed by the first retrieval step. It reports gains in recall and nDCG on TREC Deep Learning benchmarks while using the same number of LLM calls.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1 never specifies the final ordering of R1 when computing nDCG@10; if R1 is not sorted by reciprocal-rank pseudo-scores, headline nDCG gains may be an artifact of insertion order.","rationale":"The paper's central claim is that SlideGar improves nDCG@10 while keeping LLM inference count fixed. The graph-neighborhood assumption is empirically supported by the k-ablation and by consistent recall gains, so I do not press that point. The weakest link is the definition of the final ranked list: Algorithm 1 describes a pool R1 assembled from window chunks but says nothing about how that pool is ordered for nDCG@10. Since listwise rankers output only rankings and the paper explicitly does not use scores, ordering must be recovered from window-level ranks. The text mentions reciprocal-rank pseudo-scores only for neighbor prioritization, not for the final output. If the implementation sorts by pseudo-score, the method is complete and the reported numbers are plausible. If it does not, the nDCG improvements could reflect window and frontier alternation rather than the adaptive retrieval mechanism. This is a reproducibility and correctness risk that the released code can settle, so the existing CONDITIONAL verdict remains appropriate; no adjustment is needed.","tokens_in":14287,"tokens_out":6002,"duration_ms":63715,"concrete_test":"Inspect the released code at github.com/Mandeep-Rathee/llmgar to determine how R1 is ordered before metric computation. Then recompute the Table 1 entry BM25»RankZephyr, DL19, c=50 under two orderings: (a) insertion order as written in Algorithm 1, and (b) descending reciprocal-rank pseudo-score. If nDCG@10 differs by more than about 0.01 between the two, the reported 0.747 result depends on an unspecified ordering choice and the headline gain is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2, Algorithm 1 constructs the output pool R1 by line 9 (R1 <- R1 ∪ B[b+1:|B|]) and line 15 (R1 <- R1 ∪ L1), but never defines an ordering for R1. The accompanying text introduces reciprocal-rank pseudo-scores only to prioritize graph-neighbor selection, not to sort R1 itself. nDCG@10, the primary effectiveness metric, is order-dependent. If the released implementation sorts R1 by pseudo-score, the method is well-defined; if it evaluates R1 in insertion order, then the relative ordering of documents across windows—and SlideGar's alternation between R0 and the graph frontier—determines nDCG@10 independently of the adaptive-feedback claim. The reader's clustering-assumption concern is largely addressed by the k-ablation in Fig. 2; the unspecified output ordering is the least secure condition for the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SlideGar, an adaptive-retrieval method that extends graph-based adaptive reranking (Gar/Quam) to listwise LLM rerankers. SlideGar processes a sliding window of w documents, carries the top-b documents into the next window, adds the remaining documents to an output pool R1, and alternates the source of the other b documents in the next window between the initial retrieval list and a corpus-graph frontier. The authors evaluate SlideGar with BM25 and TCT first-stage retrieval, four listwise rankers (RankGPT-4o, RankGPT-3.5, RankZephyr, RankVicuna), and the TREC DL19-22 test collections, reporting Recall@c and nDCG@10 gains (up to 28.02% and 13.23% respectively) at a fixed number of LLM calls, together with a graph-depth ablation and a latency measurement.","tokens_in":14532,"tokens_out":12793,"duration_ms":122361,"significance":"If the effectiveness results are reproduced as stated, this is a useful and timely extension of adaptive retrieval to listwise rerankers: it addresses the bounded-recall problem without increasing the number of LLM calls, and the experimental coverage is broad (two corpora, four test sets, three graph substrates, four rankers, two budgets). The paper is also careful in several respects: hyperparameters w=20, b=10, k=16 are taken from prior work rather than tuned on the test sets; significance tests with Bonferroni correction are reported; the graph-depth ablation addresses the dependence on the clustering hypothesis; and the authors release code, indexes, and corpus graphs. The central caveat is that nDCG@10, one of the two headline metrics, depends on the order of the output pool R1, and Algorithm 1 does not specify that order.","major_comments":[{"comment":"The output R1 is defined through set unions, so its order is unspecified. nDCG@10 is order-dependent, and the claimed nDCG improvements (up to 13.23%, abstract and Table 1) therefore are not well-defined by the paper as written. Please state explicitly whether R1 is an ordered list; if so, specify exactly how B[b+1:|B|] is appended for each window, how the final L1 is placed on top, and whether any final ordering (e.g., by reciprocal-rank pseudo-scores) is applied. If the released code uses a particular ordering, the algorithm description must match it. Without this, the nDCG@10 results cannot be verified.","section":"Section 3.2, Algorithm 1 (lines 9 and 15)"},{"comment":"Line 10 computes F <- Neighbours(B,G) \\ (R1 union L1) over all documents B in the window, whereas the abstract and Figure 1 describe feedback from the most relevant documents seen up to that point, i.e., the top-b carried documents L1. This is a substantive discrepancy: including neighbors of low-ranked documents that were just added to R1 changes which documents can enter the frontier. Please either restrict the frontier update to L1 or redefine Neighbours(B,G) as the neighbors of all documents in the window, and in the latter case specify how neighbors from multiple source documents are prioritized and tie-broken.","section":"Section 3.2, Algorithm 1 line 10 vs. Figure 1 and abstract"},{"comment":"The statement that with the dense first-stage retriever we continue to see improvements is not supported by the table. For example, for TCT»RankGPT-3.5 on DL20, SlideGarTCT lowers nDCG@10 from 0.736 to 0.697 at c=50 and from 0.738 to 0.680 at c=100; similar degradations appear for TCT»RankZephyr and TCT»RankVicuna in several cells. The paper should quantify how often SlideGar helps versus hurts nDCG@10, report significance of degradations, and discuss the conditions under which adaptive expansion is detrimental when the first stage is already strong.","section":"Section 5.1, Table 1"}],"minor_comments":[{"comment":"The phrase subject to w and b is vague; please state that B is the permutation of L produced by the listwise ranker under the given window and step parameters.","section":"Algorithm 1, line 6"},{"comment":"Significance markers are placed inconsistently (e.g., 0.755 * 0.733 and 0.680 *0.729); please format markers so each metric's significance is unambiguous.","section":"Table 1"},{"comment":"Clarify whether the ± values are standard deviations over the five runs; the units and computation should be stated.","section":"Section 5.4"},{"comment":"Change de-duplicate to deduplicate, and fix the typo perfomring in Section 2.2; a general language-polish pass is needed.","section":"Section 4.1 and Section 2.2"},{"comment":"Specify the behavior when the frontier F contains fewer than b neighbors, since Select(top b from P) would otherwise be undefined.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the main idea is publishable after the output-ordering issue is resolved. The reported gains are large and the experiments are broad, but the nDCG numbers currently rest on an implementation detail that is not specified; I would like to see the corrected algorithm and a statement of which ordering the released code uses. The novelty relative to the authors' prior Gar/Quam is incremental, but the listwise adaptation and the constant-LLM-call property are a sufficient contribution for a full paper if the claims are stated accurately."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: SlideGar is a genuine and sensible adaptation of Gar/Quam to listwise LLM rerankers, and the experiments are broad and mostly well-executed. The main hang-up is that Algorithm 1 never specifies how the final pool R1 is ordered when computing nDCG@10. If the code sorts by the reciprocal-rank pseudo-scores, fine; if it uses insertion order from the sliding-window alternation, the headline nDCG gains might be an artifact of window scheduling rather than the adaptive feedback itself.\n\nWhat's new: prior adaptive retrieval required pointwise scores; here they convert listwise output to reciprocal-rank pseudo-scores to prioritize graph-neighbor exploration, and they carry the top-b documents of each window forward as seeds. That combination is, as far as I can tell, unexplored in the cited literature. The empirical coverage is strong: 4 listwise rankers, 2 initial retrievers, 2 corpora, 4 TREC Deep Learning test sets, with significance tests and a k-ablation (Fig. 2) that gives a reasonable check on the clustering hypothesis. The released code, corpus graphs, and de-duplicated indexes are a real plus.\n\nThe soft spots, in proportion: the biggest is the unresolved ordering of R1. The stress-test note is right. The paper describes pseudo-scores only for neighbor selection, not for constructing the final ranked list. nDCG@10 is order-dependent, so this is a load-bearing omission for the primary reported metric. Also, the abstract says SlideGar \"overcoming the bounded recall problem for LLM rerankers,\" which is too strong for dense first-stage setups: on TCT-initiated runs the recall gains are smaller and sometimes not significant (e.g., DL20 TCT»RankVicuna). The RM3 baseline is reasonable but not a strong PRF comparator; it mostly serves to show graph-based feedback behaves differently from query expansion.\n\nWho this is for: people working on LLM-based reranking or adaptive retrieval. It deserves a serious referee. I would send it to review, but the authors must clarify the final ranking construction, ideally by pointing to the exact lines in their released code. If the code sorts R1 by reciprocal-rank pseudo-scores, the central claim is likely sound; until then, the paper is incomplete as written.","headline":"Sensible extension of graph-based adaptive retrieval to listwise LLM rankers with solid experiments, but the final output ordering is underspecified and needs confirmation from the released code.","tokens_in":15027,"tokens_out":3147,"would_cite":false,"duration_ms":31448,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes SlideGar, an adaptive-retrieval loop that lets LLM listwise rerankers pull in documents beyond the initial retrieval pool, improving recall and ranking quality with no increase in the number of LLM inference calls.","keywords":["Reranking","Adaptive Retrieval","LLM rankers","Listwise ranking","Bounded recall","Corpus graph","Pseudo-relevance feedback","Sliding window"],"falsifier":"Take a query set and remove from the corpus graph every edge that connects a relevant document to any document in the initial retrieval pool, then run SlideGar: if recall@c still improves over the non-adaptive baseline, the reported gains do not come from the graph edges the paper says they come from. A cheaper check is to replace the graph neighbours with randomly chosen documents and measure whether the recall improvement collapses.","tokens_in":14070,"feed_emoji":"🔍","tokens_out":7047,"duration_ms":63161,"temperature":0.7,"pith_summary":"Cascading search pipelines rank whatever the first-stage retriever returns, so relevant documents that miss that cut are gone for good. The paper claims this bounded recall problem can be overcome for LLM listwise rerankers, which output only a relative order and cannot be treated as pointwise scorers. SlideGar adapts graph-based adaptive retrieval to this setting by using each ranked window as feedback: the top documents of a window seed a search of a precomputed corpus graph, and the next window mixes the initial ranking with the graph frontier. On TREC Deep Learning benchmarks, the authors report nDCG@10 gains up to 13.23 percent and recall gains up to 28 percent, at the same LLM inference budget and with about 0.02 percent added latency.","feed_headline":"LLM rerankers recover missed results at zero extra LLM calls","feed_subtitle":"SlideGar pulls graph neighbors of top-ranked items into the window, lifting recall by up to 28 percent on TREC DL benchmarks.","key_machinery":"The central mechanism is the sliding window with graph feedback, given in Algorithm 1. A window of size w is reranked by the listwise LLM; the top b documents are carried into the next window, the remaining documents are appended to the reranked pool, and the graph frontier is replaced with the k-nearest neighbours of the ranked batch, excluding documents already in the pool. The algorithm then alternates the source of the next b documents between the initial ranking and the frontier, so the LLM always sees a mixture of first-stage results and graph-discovered candidates. The corpus graph, built offline from sparse or dense document similarities, supplies the feedback in constant time, and reciprocal rank acts as the pseudo-score that orders the neighbours. This loop keeps the number of LLM calls identical to standard windowed listwise reranking: (c−w)/b + 1 windows for a reranking budget c.","core_discovery":"SlideGar is a sliding-window variant of graph-based adaptive retrieval designed for listwise LLM rankers. In each step, the LLM reranks a window of w documents; the top b become the seed for the next window, the remainder join the final pool, and the graph frontier is updated with the neighbors of the ranked batch. The algorithm alternates between the initial retrieval pool and the frontier, so the final pool contains documents that never appeared in the first-stage results. Because online listwise rankers give no scores, SlideGar uses reciprocal rank as a pseudo-score to prioritise graph neighbours. The authors demonstrate across four LLM rankers, two first-stage retrievers, and four TREC Deep Learning collections that this loop improves recall and nDCG, and attribute the gains to the Clustering Hypothesis encoded in the corpus graph.","pith_inferences":["Because SlideGar uses only the LLM's relative order within each window, its seed selection might be improved by exploiting confidence or generation scores from the ranker, a direction the paper does not explore.","The method should degrade gracefully with graph quality, so replacing the static sparse/dense graph with a graph built from the LLM's own document representations could push recall further, at the cost of building that graph.","A stress test for the Clustering Hypothesis: on queries whose relevant documents are topically scattered, such as multi-faceted information needs, the graph-neighbour feedback should contribute little, and the recall gain should shrink accordingly."],"forward_implications":["First-stage retrieval no longer sets a hard ceiling on the final ranking: documents the LLM finds relevant can enter the reranked pool via the corpus graph, so weaker lexical first stages can still feed strong LLM rankers.","The recall gain comes at no extra LLM inference cost: the total number of windows, and therefore LLM calls, is unchanged, and the adaptive overhead is a constant-time graph lookup plus a few list operations.","The improvement is consistent across zero-shot and fine-tuned listwise rankers, across sparse and dense first-stage retrievers, and across two different corpora (MSMARCO v1 and v2 with TREC DL19/20/21/22).","A dense-similarity corpus graph consistently outperforms both a sparse-similarity graph and an RM3-based feedback baseline, indicating that the adaptive loop benefits most from complementary dense signals."],"supporting_citations":[{"why":"Supplies the graph-based adaptive retrieval method and the corpus graphs that SlideGar builds on.","marker":"[12]"},{"why":"Provides the sliding-window listwise reranking strategy and the LLM ranker baselines that SlideGar extends.","marker":"[34]"},{"why":"Supplies RankZephyr, a fine-tuned listwise ranker used as a primary testbed.","marker":"[27]"},{"why":"Supplies RankVicuna, the other fine-tuned listwise ranker in the experiments.","marker":"[26]"},{"why":"Gives the Clustering Hypothesis that motivates exploring corpus-graph neighbours of top-ranked documents.","marker":"[5]"},{"why":"Provides the dense retriever whose document similarities are used to build one of the corpus graphs.","marker":"[8]"},{"why":"Provides the sparse retriever used both as initial retrieval and as the basis of the other corpus graph.","marker":"[31]"},{"why":"Defines RM3 pseudo-relevance feedback, used for the SlideGarRM3 baseline that SlideGar is compared against.","marker":"[1]"}],"fun_headline_variants":["SlideGar lets listwise LLM rankers fix their own recall","SlideGar boosts LLM reranker recall by 28% at no extra cost","Sliding-window adaptation guides LLM rerankers past recall ceiling","Graph-based sliding windows fix listwise reranker recall gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The corpus graph must connect documents the LLM ranks near the top of a window to relevant documents that the initial retrieval missed; if those edges are absent, the feedback loop cannot recover lost relevant documents and the reported recall gains disappear.","fun_headline_variants_meta":{"raw":{"variants":["SlideGar lets listwise LLM rankers fix their own recall","SlideGar boosts LLM reranker recall by 28% at no extra cost","Sliding-window adaptation guides LLM rerankers past recall ceiling","Graph-based sliding windows fix listwise reranker recall gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000612,"raw_usage":{"total_tokens":2858,"prompt_tokens":971,"completion_tokens":1887,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":1810}},"tokens_in":587,"tokens_out":1887,"duration_ms":14771,"temperature":1.0,"reasoning_tokens":1810,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:10:47.733526+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a query set and remove from the corpus graph every edge that connects a relevant document to any document in the initial retrieval pool, then run SlideGar: if recall@c still improves over the non-adaptive baseline, the reported gains do not come from the graph edges the paper says they come from. A cheaper check is to replace the graph neighbours with randomly chosen documents and measure whether the recall improvement collapses.","supporting_citations":[{"cited_title":"Computer Science Department Faculty Publication Series p","cited_arxiv_id":null,"evidence_quote":"Defines RM3 pseudo-relevance feedback, used for the SlideGarRM3 baseline that SlideGar is compared against."}],"review_version":1}