{"id":"f25eac7b-a8fb-4f39-8c91-1d52f47d864e","arxiv_id":"2505.10413","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LongRefiner refines long RAG contexts by building an XML tree of the document, scoring nodes with local and global relevance, and selecting a compact subset that improves QA accuracy on seven datasets while cutting token usage versus full-document input.","lead":"A new RAG component, LongRefiner, organizes long retrieved documents into a compressed outline, scores each part for the query, and sends only the best parts to the generator, reporting higher QA accuracy with far fewer tokens than reading entire documents. The paper's headline '10x cheaper' comparison is actually against the full-content setting, not against the best baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 1's Rq is softmax of Local/Global probabilities, compressing every query's information-scope weight into [0.269,0.731]; the adaptive refinement mechanism is therefore nearly constant, and the additive LS + Rq*GS combination is unvalidated.","rationale":"The paper's central claim is that a query-adaptive hierarchical refiner can match or beat full-document and compression baselines under a 2k-token budget. I focused on the step on which all downstream decisions depend: the node score. The reader flagged LS + Rq*GS as weakly supported; my reading of Eq. 1 reveals a sharper, more concrete problem. Softmaxing probabilities instead of logits compresses Rq into a narrow band, so the model cannot genuinely interpolate between local and global information needs. This makes the claimed adaptive behavior nearly a fixed blend, which undermines the mechanism story and the interpretation of the query-analysis ablation, even if the overall ranking in Table 2 is reproducible. The proposed lambda sweep separates these questions: it tests whether the exact Rq formula matters and whether the additive combination is calibrated. I do not see a basis to reject the empirical results outright, but the paper should not be accepted without either correcting Eq. 1 or showing that the same results hold when the weight is free. Since the reader already set CONDITIONAL, I keep the verdict unchanged.","tokens_in":18930,"tokens_out":12580,"duration_ms":129131,"concrete_test":"Run the Table 2 evaluation with Score(ni) = LS(ni) + lambda * GS(ni) for lambda in {0.269, 0.5, 0.731, 0, 1, raw-logit Rq}, keeping all trained components unchanged, on a held-out subset of 500 queries per dataset. If the six variants are statistically indistinguishable, the current Rq is not load-bearing and the adaptive-refinement claim should be downgraded. If the raw-logit variant meaningfully changes results, Eq. 1 is miscalibrated and the reported rankings need re-checking with a corrected weight.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines Rq = Softmax(Pl, Pg)_g with Pl and Pg described as generation probabilities of the special Local/Global tokens. As written, both lie in [0,1], so the second softmax maps every possible query to Rq in [1/(1+e), e/(1+e)] ≈ [0.269, 0.731]. A strongly Local query still receives about 27% of the global score, and a strongly Global query only about 73%. The claimed continuous information scope thus spans less than half of the unit interval, making the adaptive mechanism in Eq. 1 nearly constant across query types. This undermines the attribution of Table 3's w/o Query Analysis degradation to query adaptivity, since the weight cannot vary widely enough to produce the claimed interpolation. Additionally, Score(ni) = LS(ni) + Rq * GS(ni) adds an unnormalized reranker similarity (averaged upward over the tree) to a binary outline-selection score diluted by child counts; no calibration or independent validation of this additive sum is reported. Every selected-node ranking that feeds the 2k-token budget and all downstream QA numbers in Table 2 depends on this untested commensurability assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LongRefiner, a plug-and-play document refiner for long-context retrieval-augmented generation. The method has three components: dual-level query analysis, which classifies a query as local or global and derives a continuous scope weight; hierarchical document structuring, which converts plain retrieved documents into an XML-like document tree using trainable section/subsection/paragraph tags and skip placeholders; and adaptive refinement, which scores tree nodes by combining a local reranker score with a global outline-based selection score and selects nodes until a token budget is reached. Training uses three LoRA adapters on a single Qwen2.5-3B backbone. Experiments on NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, ASQA, and ELI5 under a 2k-token budget report that LongRefiner outperforms all compared baselines on all datasets, with a token count near 1,933 and an online latency of 10.8 seconds, and the paper claims an order-of-magnitude efficiency advantage over full-content input.","tokens_in":19191,"tokens_out":5699,"duration_ms":58268,"significance":"If the central results hold, LongRefiner is a practically useful contribution: it shows that structured, hierarchy-aware refinement can beat perplexity-based and retrieval-based compression methods while keeping generator input near 2k tokens, and the ablations, scaling analyses, and robustness checks across generator backbones and scoring models are valuable. The release of code and the use of external QA benchmarks for the headline accuracy numbers are strengths. However, the paper overstates the efficiency claim in the abstract, the query-adaptivity mechanism in Eq. (1) is far less adaptive than advertised, and the evaluation lacks variance reporting, so the strength of the claims exceeds what the evidence currently supports.","major_comments":[{"comment":"The abstract's claim of \"10x fewer computational costs and latency compared to the best baseline\" is not supported by Table 2. LongRefiner uses 1,933 tokens, whereas the strongest accuracy baseline, LongLLMLingua, uses 1,976 tokens, so the token reduction relative to the best baseline is about 2%, not 10x; the 10x figure only holds when comparing to the Full Content row (19,567 tokens), which is not the best baseline. The stated online latency advantage is also computed while excluding the offline hierarchical structuring stage described in Section 3.4, so the end-to-end computational cost of LongRefiner is not reported. Please rephrase the efficiency claims to specify the comparison point and report the total compute including the offline stage.","section":"Abstract; §4.4, Table 2"},{"comment":"The adaptive query-scope weight Rq is nearly constant by construction. Since Pl and Pg in Eq. (1) are generation probabilities in [0,1], Softmax(Pl, Pg)_g lies in [1/(1+e), e/(1+e)] ≈ [0.269, 0.731] for every possible query; a strongly local query still receives at least 27% global weight and a strongly global query at most 73%. Therefore the \"continuous information scope\" the authors claim is compressed into less than half of the unit interval, and the degradation of the \"w/o Query Analysis\" ablation in Table 3 cannot be attributed to meaningful query adaptivity unless additional evidence is provided. The additive combination Score(ni) = LS(ni) + Rq * GS(ni) also adds an unnormalized reranker similarity to a diluted outline-selection score without any calibration or independent validation; this is load-bearing because every downstream selection in Table 2 depends on this sum. Please either correct the formula (e.g., using log-probabilities before softmax) or demonstrate that the resulting Rq variation actually changes node rankings in a meaningful way.","section":"§3.1, Eq. (1); §3.3; Table 3"},{"comment":"The headline claim of outperforming all baselines on all seven datasets is reported as a single run with no error bars, confidence intervals, or significance tests. Greedy decoding reduces sampling variance, but not evaluation variance or sensitivity to the choice of test subset; moreover, some differences in Table 2 are very small, such as ELI5 F1 of 23.9 versus 23.7 for LongLLMLingua, and the \"w/o Query Analysis\" ablation in Table 3 differs by only 0.6-2.0 points depending on dataset. Since the central contribution is the claimed universal improvement, please report variance over multiple runs or a statistical significance test, and state how many examples are evaluated per dataset.","section":"§4.4, Tables 2 and 3"},{"comment":"The global-selection training labels are self-referential: as Appendix D states, the abstracts and outlines used to annotate the selected titles are generated by the authors' own trained structuring model rather than derived from the golden Wikipedia structures. This means the global-selection module may learn to compensate for the structuring model's systematic errors, which could inflate the module's measured contribution in the ablation and the end-to-end results. Because the main accuracy numbers are measured on external QA benchmarks, this is not a fatal circularity, but it is an unquantified bias in a load-bearing training stage. Please report an ablation in which the global-selection labels are created from gold or human-readable outlines, or provide an agreement analysis between the model-generated outlines and the ground-truth Wikipedia structure.","section":"§3.3; Appendix D"}],"minor_comments":[{"comment":"Typo: \"Correpsonding author\" should be \"Corresponding author.\"","section":"Footnote 1"},{"comment":"Figure 1 plots \"Latency\" against \"Performance,\" but the axes have no units and the identity of the plotted methods is not fully clear; please state the metric definitions and the experimental setting.","section":"Figure 1"},{"comment":"The text says that each section's content is enclosed within <section: {title}> and </subsection: {title}>, but the closing tag should be </section: {title}> according to Table 1; this appears to be a typo.","section":"§3.2"},{"comment":"The equation defines Rq, but the discussion refers to \"a continuous representation rq\"; please standardize the symbol to one or the other.","section":"§3.1 after Eq. (1)"},{"comment":"The sentence \"As shown in Figure 1, the XML-based Dxml reduces the token count to approximately 1/10 of the original\" cites a latency-performance scatter plot that does not show token counts; please correct the reference to the relevant table or plot.","section":"§3.2"},{"comment":"The row \"Best Baseline\" is not defined; please specify which method it denotes and how it was selected.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The empirical core of the paper is a solid systems contribution, but the abstract and Section 4.4 overclaim the efficiency advantage, and the query-adaptivity mechanism in Eq. (1) is mathematically too constrained to support the interpretation given to the ablations. The paper would be acceptable after a major revision that corrects the efficiency framing, reports uncertainty, and either fixes or reinterprets the Rq formula. The comparison point for 'best baseline' should also be made consistent throughout, since the text shifts between LongLLMLingua and Full Content."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious look. LongRefiner is a real system: hierarchical XML-style document restructuring with skip tokens, dual-level query analysis, and adaptive local/global node scoring, all trained on one LoRA-adapted model. As a combination it's new, and the empirical picture is unusually clean: under a 2k-token budget it beats every baseline on all seven QA datasets, with a second generator in the appendix confirming the trend. Ablations show each component contributes, and the code is out. That is a valuable contribution for anyone working on RAG efficiency.\n\nNow the caveats. The abstract says '10x fewer computational costs and latency compared to the best baseline.' That is not what Table 2 shows. The 10x token reduction is against 'Full Content' (~19.5k vs ~1.9k), not against the best baseline; all compared methods, including LongRefiner, sit at roughly 2k tokens. Latency is also reported only for the online stage, with the offline structuring cost excluded. For a static corpus that's fair amortization, but for dynamic corpora the real cost is higher. Please fix the abstract.\n\nThe more technical concern is Eq. 1. Rq is defined as a softmax over two generation probabilities, Pl and Pg. Since both are probabilities in [0,1], Rq is mathematically restricted to about [0.27, 0.73]. That means the 'continuous information scope' is nearly constant; a strongly local query still gets 27% of the global score, and a strongly global query only 73%. The adaptive mechanism, as written, cannot produce the wide interpolation the paper claims. The ablation shows query analysis helps, but the benefit may come from the binary Local/Global signal or the training objective, not from the smooth weight. Similarly, the additive combination of an unnormalized reranker similarity and a binary outline score is never calibrated or independently validated. These issues are fixable, but they should be addressed before the 'adaptive refinement' story is taken at face value.\n\nAlso missing: error bars or significance tests (greedy decoding helps, but doesn't eliminate variance), and at least one free parameter, k, is unreported.\n\nNone of this kills the paper. The central empirical result—that structure-aware, query-aware document refinement outperforms chunking, reranking, and perplexity compression under a strict 2k budget—is credible and useful. I'd send it to referees, with a request to correct the overclaims and either fix the Rq mechanism or soften its role.","headline":"A solid, useful RAG refiner whose abstract overclaims a 10x win, and whose adaptive query weight may be nearly constant as written—but the core empirical result is credible and deserves refereeing.","tokens_in":19696,"tokens_out":4487,"would_cite":true,"duration_ms":43159,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LongRefiner shows that turning long documents into a section tree lets a refiner keep only query-relevant nodes, matching or beating full-document RAG with roughly one-tenth the tokens.","keywords":["retrieval-augmented generation","long-context documents","document refinement","hierarchical document tree","prompt compression","query-aware node selection","multi-task LoRA","open-domain question answering"],"falsifier":"Construct a set of long documents with gold-annotated answer paragraphs, run LongRefiner twice per query—once with the predicted scope weight and once with an oracle weight chosen to maximize answer recall—and compare selection recall; if oracle-weighted selection recovers clearly more gold paragraphs, the query-scope mechanism is the bottleneck and the adaptive-refinement claim is falsified for those queries.","tokens_in":18759,"feed_emoji":"🌳","tokens_out":14235,"duration_ms":125782,"temperature":0.7,"pith_summary":"Long-context RAG is held back by retrieved documents that are far longer than the relevant parts: feeding everything to the generator is slow and costly, while chunk- or perplexity-based compression drops evidence. LongRefiner proposes turning each retrieved document into an XML-style tree of sections, subsections, and paragraphs, then selecting the tree nodes the query actually needs. The paper's claim is that this structure-aware selection, run within a 2k-token budget, outperforms existing refinement methods on all seven QA datasets tested and beats the full-content setting on six of them, while using roughly one-tenth the generator tokens and about one-quarter of the online latency. If that is right, long-context RAG becomes a cheap preprocessing problem rather than an expensive generation problem.","feed_headline":"A document tree refiner beats full-text RAG with one-tenth the tokens","feed_subtitle":"LongRefiner builds a section tree per document, keeps only query-relevant nodes, and wins on six of seven QA sets.","key_machinery":"The central object is the structured document tree $D_{\\text{str}} = (N,R)$, represented in the paper's own flat XML syntax: `<section: {title}>` and `<subsection: {title}>` mark the hierarchy, `<br>` marks paragraph breaks, and `<skip>` compresses each block to its first and last $k$ tokens so the learning target stays short. This representation is what turns refinement into node selection. The selection score is $\\text{Score}(n_i) = \\text{LS}(n_i) + R_q \\cdot \\text{GS}(n_i)$: local scores are averaged from leaves to roots, global scores are split uniformly from roots to leaves, and $R_q = \\text{Softmax}(P_{\\text{Local}}, P_{\\text{Global}})_g$ is the continuous query-scope weight. The machinery's role is to make the chosen nodes adapt to the query while keeping the tree itself reusable across all queries.","core_discovery":"At the center of the method is a document tree: each retrieved document is rewritten as a flat XML representation with section and subsection tags, a paragraph marker, and a skip token that keeps only the first and last k tokens of each block. A single 3B foundation model, trained with task-specific LoRA adapters, learns to build this tree from plain text, to classify a query as Local or Global, and to pick the globally relevant sections from the tree's outline. The paper claims that combining these three capabilities produces the best results among all compared refinement methods on seven QA datasets at a 2k-token budget, reaching or exceeding full-content accuracy on six datasets with ten times fewer tokens. The node selection rule is $\\text{Score}(n_i) = \\text{LS}(n_i) + R_q \\cdot \\text{GS}(n_i)$, where the local score propagates query-paragraph similarities upward through the tree and the global score propagates outline-based section relevance downward, with the query's predicted scope $R_q$ deciding how much weight each perspective gets.","pith_inferences":["The paper does not close the loop back to retrieval: the selected nodes are a natural source of positive training evidence for the retriever, and using them that way is a testable next step that follows directly from the tree-based selection.","Because the XML labels are derived from Wikipedia's section layout, the same pipeline on corporate or web documents with tables, navigation, and heterogeneous markup would need retraining; this is the boundary the paper itself names as its main limitation.","The additive weighting of local and global scores is one design choice among many; a learned gating function that takes the query text, not just a two-token softmax, might improve queries that need both scopes simultaneously, such as comparative or multi-hop questions."],"forward_implications":["With the 2k-token budget, LongRefiner is the top-scoring method on all seven QA datasets, including multi-hop and long-form QA, while consuming roughly one-tenth the generator tokens of the full-content setting.","The refinement transfers across generators: switching from Llama3.1-8B-Instruct to Qwen2.5-7B-Instruct preserves the advantage over baselines, so the selected nodes are not tuned to one generator's quirks.","Because hierarchical structuring runs offline on the corpus, a document tree is built once and then reused for every query; the online stage reads only a few hundred input tokens and reports about 25% of the full-pipeline latency.","Scaling the refiner backbone from 0.5B to 7B parameters and increasing training data both improve refinement recall and downstream accuracy, meaning the approach has a predictable scaling path rather than a fixed ceiling."],"supporting_citations":[{"why":"Defines the top-8 full-document retrieval setup that creates the long-context refinement task.","marker":"(Ziyan Jiang, 2024)"},{"why":"The strongest perplexity-based compression baseline that LongRefiner claims to beat on all seven datasets.","marker":"(Jiang et al., 2023a)"},{"why":"Supplies LLMLingua2, the token-level prompt-compression baseline in the 2k-token comparison.","marker":"(Pan et al., 2024)"},{"why":"Supplies Recomp, the extractive compression baseline whose token-budget comparison regime is used.","marker":"(Xu et al., 2023)"},{"why":"Supplies the LoRA method that lets the three refinement tasks share a single backbone.","marker":"(Hu et al., 2022)"},{"why":"Supplies the bge-reranker used to compute local leaf-level relevance scores.","marker":"(Xiao et al., 2024)"},{"why":"Supplies the KILT Wikipedia preprocessing from which the document-structure training labels are derived.","marker":"(Petroni et al., 2021)"},{"why":"Supplies the Wikipedia 2018 dump used as the retrieval corpus.","marker":"(Karpukhin et al., 2020)"}],"fun_headline_variants":["LongRefiner tree cuts RAG tokens 10x, matches full-text on six QA sets","Document-tree refiner matches full-text RAG with 10% of the tokens","LongRefiner: 10x cheaper RAG, equal or better on six of seven QA","Tree-structured RAG refinement: 2k tokens beat full-text on 6/7 datasets","LongRefiner builds a document tree to cut RAG cost 10x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a query's predicted Local-versus-Global scope can be turned into a single numeric weight that correctly balances paragraph-level and section-level evidence; if that weight is wrong for a query, the selected nodes can omit the answer even though the document contains it.","fun_headline_variants_meta":{"raw":{"variants":["LongRefiner tree cuts RAG tokens 10x, matches full-text on six QA sets","Document-tree refiner matches full-text RAG with 10% of the tokens","LongRefiner: 10x cheaper RAG, equal or better on six of seven QA","Tree-structured RAG refinement: 2k tokens beat full-text on 6/7 datasets","LongRefiner builds a document tree to cut RAG cost 10x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001462,"raw_usage":{"total_tokens":5860,"prompt_tokens":899,"completion_tokens":4961,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":4847}},"tokens_in":515,"tokens_out":4961,"duration_ms":34917,"temperature":1.0,"reasoning_tokens":4847,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:08:42.049568+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a set of long documents with gold-annotated answer paragraphs, run LongRefiner twice per query—once with the predicted scope weight and once with an oracle weight chosen to maximize answer recall—and compare selection recall; if oracle-weighted selection recovers clearly more gold paragraphs, the query-scope mechanism is the bottleneck and the adaptive-refinement claim is falsified for those queries.","supporting_citations":[],"review_version":1}