{"id":"83a2a76c-97e5-44e6-a21a-7875184976be","arxiv_id":"2504.13054","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SARESG prunes documents to aspect-relevant sentences via embedding similarity before LLM summarization, reporting gains over selective-context baselines on three datasets.","lead":"The paper introduces SARESG, a retrieval-and-prune pipeline that uses an embedding model to select aspect-relevant sentences before feeding them to a large language model for aspect-based summarization. It reports improved ROUGE, METEOR, and BERTScore results over selective-context baselines on three summarization datasets while saving token budget.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The token-limit claim is untested: all ICL samples are filtered to <1024 tokens and no token budgets are reported, so the headline 'mitigates token limitation' lacks experimental support.","rationale":"I considered the reader's weakest assumption, that cosine-similarity ranking preserves aspect-relevant sentences. That assumption matters, but the downstream tables provide indirect evidence for it: SARESG beats Original and Selective Context on most metrics. The more vulnerable point is that the paper's second headline contribution, token-limit mitigation, is never placed under token pressure. The authors explicitly filter ICL samples to under 1024 tokens, W is undefined, and no token counts are reported. These are mechanical, addressable issues, so the appropriate verdict remains CONDITIONAL rather than REJECT; hence I recommend UNCHANGED relative to the reader's verdict. The OAsum ICL anomaly is a related symptom, but the token-budget gap is the single most load-bearing concern because it targets a stated contribution rather than one dataset's noise. The proposed length-stratified experiment would settle whether the token-limitation claim holds.","tokens_in":12062,"tokens_out":7134,"duration_ms":67215,"concrete_test":"On held-out documents stratified by length (e.g., 1k/2k/4k/8k/16k tokens), run Truncated_ICL and SARESG_ICL with a fixed 4096-token inference budget, recording W and the actual token count of the pruned prompt for every input. Report ROUGE-1/2/L, METEOR, and BERTScore by length bucket, along with a retrieval-recall measure against reference summaries. If SARESG_ICL only matches or beats Truncated_ICL in the <1024 bucket, or if its input token count systematically exceeds the baseline's, the token-limitation claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim has two parts: superior summarization quality and mitigation of the token-limitation problem. The second part is not actually evaluated. In Section IV.A the authors state that 'due to the token limit in the ICL setting... we only retain samples with a length of less than 1024.' Thus every ICL comparison in Tables I-III is run on short documents that already fit the 4096-token budget; the retrieval-based pruning is never tested under the token pressure it is designed to relieve. Moreover, the pruning threshold W in Eq. (3) is never specified, and no table reports the actual number of input tokens consumed by SARESG_ICL versus Truncated_ICL. The reader therefore cannot tell whether any gain comes from better content selection or from a different or larger token allocation. The Introduction also describes the retrieval process as recursive, but no recursive algorithm or ablation is provided. Consequently, even if the quality results are accepted, the headline claim that the framework 'effectively mitigates the token limitation problem' is not supported by the reported experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SARESG, a retrieve-and-prune framework for aspect-based summarization with large language models. The input document is split into fixed-size chunks (256 words), each sentence is scored by cosine similarity to the aspect embedding, the top-scoring sentences are kept up to an unspecified word threshold W, and the pruned document is fed to an LLM, optionally with one in-context learning (ICL) demonstration. Experiments on Ma-news, OAsum, and USB with Mistral (Mixtral) 8x7b, Llama3-8b, and Llama3-70b compare SARESG and SARESG_ICL against Original, Selective Context, and Truncated_ICL baselines using METEOR, ROUGE, and BERTScore. The paper claims that the framework achieves superior summarization quality and effectively mitigates the token-limitation problem, and it includes ablations on chunk size and on sentence- versus chunk-level retrieval.","tokens_in":12271,"tokens_out":5173,"duration_ms":43399,"significance":"If the reported results hold with proper controls, SARESG is a simple and practical contribution to aspect-based summarization: it offers a lightweight retrieval-based pruning mechanism that can free token space for ICL while improving aspect alignment. The paper's strengths are its breadth of models and datasets, the inclusion of a context-compression baseline (Selective Context), and the ablation studies on chunk size and retrieval granularity. However, the central token-mitigation claim is not experimentally tested, the pruning hyperparameters are undisclosed, and the quality comparisons lack statistical rigor; the paper's value is therefore conditional on additional controlled experiments.","major_comments":[{"comment":"The claim that the framework 'effectively mitigates the token limitation problem' is not supported by the reported experiments. Section IV.A states that 'due to the token limit in the ICL setting... we only retain samples with a length of less than 1024.' Consequently, every ICL comparison in Tables I-III is performed on documents that already fit comfortably within the 4096-token budget, and the retrieval-based pruning is never tested under the token pressure it is designed to relieve. No table or figure reports the actual number of input tokens consumed by SARESG_ICL versus Truncated_ICL. Please add long-document experiments (or at least report token counts and the distribution of input lengths) to substantiate the token-usage claim.","section":"Section IV.A, Tables I-III, Abstract"},{"comment":"The pruning threshold W in Eq. (3) is never specified, and Section IV.A only says that input lengths were 'adjusted to ensure consistency with the baseline' without giving the resulting lengths. Without W or token counts, the gains of SARESG over Truncated_ICL could be due to a different effective context length rather than to better content selection. The Limitations section also acknowledges that 'the best pruning parameters will be varied as the length of articles varied.' Please report W, the resulting input lengths for each dataset, and a sensitivity analysis over W.","section":"Section III.A, Eq. (3); Section IV.A"},{"comment":"The OAsum ICL failure is explained post hoc without reporting corrected numbers. In Table III, Truncated_ICL and SARESG_ICL score below the zero-shot Original and SARESG (e.g., METEOR 24.18 and 23.72 vs 24.34 and 25.38). Section V.D attributes this to an unrepresentative random ICL sample and says additional experiments with different samples were conducted, but the corrected results do not appear in any table. This leaves the main results table inconsistent with the text and prevents the reader from assessing the ICL claim for OAsum. Please report the corrected OAsum ICL numbers (or a separate table) and state the sample-selection protocol before the experiments.","section":"Table III (OAsum rows); Section V.D"},{"comment":"Tables I-III list methods 1, 2, 3, 5, and 6 but not row 4; if a fourth baseline was intended (e.g., Selective Context with ICL), it should be included for a complete comparison. More importantly, no error bars, variance, or significance tests are reported on any metric. Differences such as Ma-news METEOR 24.07 (SARESG) vs 22.91 (Original) in Table I may be within sampling variation; please add standard errors or paired significance tests across the evaluation samples.","section":"Tables I-III; Section IV.A"},{"comment":"The Introduction states that the retrieval process is 'recursive, continuing until the text is pruned to the desired length,' but Section III describes only a single pass: chunk, score, select Top-W, concatenate. No recursive algorithm, termination criterion, or ablation is presented. Please either remove the word 'recursive' from the Introduction or specify and evaluate the recursive procedure.","section":"Section I (Introduction); Section III"},{"comment":"The chunk size of 256 appears to be selected based on ROUGE scores on the Ma-news evaluation set, with no indication of a held-out validation split. Since chunk size is a hyperparameter of SARESG, tuning it on the test set risks inflating the reported gains over baselines. Please describe how hyperparameters (chunk size, W) were chosen, ideally on a validation split.","section":"Section V.C; Section IV.A"}],"minor_comments":[{"comment":"The sentence 'the jasper model on top of the stella A model called stella en 1.5b' is unclear; please give the exact model name and version used for retrieval.","section":"Section IV.A"},{"comment":"The caption contains a typo: 'Retrival' should be 'Retrieval', and the second row should be 'Chunk Retrieval' rather than 'Chunk Retrival'.","section":"Table IV caption"},{"comment":"References [18] and [40] are the same paper (Frermann and Klementiev, ACL 2019); please merge them into a single citation.","section":"References"},{"comment":"The figure caption contains a stray space in 'V onderplanitz's', and the diagram text appears garbled; please regenerate the figure for clarity.","section":"Figure 1"},{"comment":"The model name 'Mistral 8×7b' should be 'Mixtral 8x7b', and the truncation baseline is inconsistently named 'Truncated' and 'Truncated_ICL' across the text and tables.","section":"Section IV.A"},{"comment":"The filtering criterion 'we only retain samples with a length of less than 1024' is not tied to the earlier statement that the first 2000 rows of Ma-news and OAsum were selected; please report the resulting dataset sizes and the length statistics for each dataset.","section":"Section IV.A"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful core idea and a reasonable set of initial experiments, but the main headline claim about token limitation is untested, and the evaluation has several gaps (missing baseline row, no significance tests, post hoc replacement of an ICL sample). These are fixable within the scope of a revision. I would lean toward reject if the authors do not add genuine long-document experiments or token-budget reporting, since the token-mitigation claim is central to the paper's framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is a new combination, not a new paradigm: use the aspect embedding as a query, score sentences inside fixed-size chunks by cosine similarity, prune each chunk to a word budget W, reassemble in original order, and feed the result to an LLM, optionally with one ICL example. That is a legitimate extension of Selective Context and query-focused summarization, and the paper describes it clearly. The zero-shot comparisons across three datasets and three LLMs consistently favor SARESG over the original text and Selective Context, and the ablations on chunk size and sentence-vs-chunk retrieval are useful. I also credit the limitations section: the authors admit ICL instability and that pruning parameters vary with article length.\n\nThe soft spots are real but mostly addressable. The biggest one is the token-limit claim. The stress-test is right: every ICL sample is filtered to under 1024 tokens, so the pruning never operates under the 4096-token constraint it is supposed to relieve. No table reports actual token counts or token budgets for SARESG_ICL versus Truncated_ICL, so you cannot tell whether gains come from better content selection or merely from a different token allocation. The pruning threshold W in Eq. (3) is never specified, and the introduction promises a recursive retrieval process that never appears in the method. That part of the headline is simply unsupported.\n\nOther issues are mechanical but should be fixed: Tables I-III skip method row 4, there are no error bars or significance tests, the OAsum ICL failure is explained post hoc by a bad sample without reporting the corrected numbers in the main tables, and the hallucination-mitigation claim has no hallucination metric behind it. The chunk-size ablation covers only one dataset and one model, and Figure 2 shows just three sizes.\n\nNone of this kills the method. The zero-shot results are plausible, and the pipeline is simple enough to be useful in token-constrained LLM applications. But as it stands, the paper overclaims and under-measures. A serious referee should engage with it, because the core idea is worth testing properly, but acceptance should require: specifying W, reporting token consumption, running ICL on genuinely long documents, adding error bars or significance tests, completing the tables, and softening the token-limit and hallucination claims to match the evidence.\n\nI would bring it to a reading group as an example of a promising pruning approach with an evaluation that falls short of its claims.","headline":"A sensible retrieval-pruning pipeline for aspect-based summarization that mostly supports its quality claims, but the token-limit headline is untested because the ICL evaluation never exercises long documents.","tokens_in":12777,"tokens_out":1854,"would_cite":false,"duration_ms":19378,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pruning a document to aspect-relevant sentences before LLM summarization improves the resulting summaries and leaves room for in-context examples.","keywords":["aspect-based summarization","large language models","in-context learning","retrieval-enhanced generation","dense retrieval","token limits","text pruning","summarization"],"falsifier":"Take a document with human-annotated sentences that appear in the gold aspect summary, run SARESG's pruning, and check how many gold-summary sentences fall below the Top-W cutoff; if a substantial fraction of the gold content is discarded while irrelevant sentences survive, the retrieval-ranking assumption is false. A cheaper version is an ablation that replaces the embedding scores with random sentence rankings: if random pruning performs as well as SARESG on the same metrics, the retrieval signal is not doing the work.","tokens_in":11886,"feed_emoji":"✂️","tokens_out":5081,"duration_ms":43025,"temperature":0.7,"pith_summary":"This paper tries to establish that before asking an LLM to write an aspect-specific summary, you can improve both quality and efficiency by first retrieving the sentences most related to that aspect and pruning everything else. The proposed framework, SARESG, splits a long document into 256-word chunks, scores each sentence with an embedding model by cosine similarity to the aspect, keeps only the top-scoring sentences up to a word budget, and reorders them before generation. Across three benchmark datasets and three LLMs, the pruned-input summaries score higher on METEOR, ROUGE, and BERTScore than full-text prompting and than an existing selective-context baseline, and the token savings leave room for in-context examples. The paper argues that this reduces hallucination risk and lets smaller models approach larger zero-shot models.","feed_headline":"Prune text by aspect before LLM summarization, scores rise","feed_subtitle":"A sentence-embedding filter keeps only aspect-relevant text, freeing tokens for in-context examples and lifting scores.","key_machinery":"The load-bearing object is the self-aspect retrieval score of Eq. (2), $S_{i,j} = \\cos(f(s_{i,j}), f(A))$, where $f$ is a pretrained embedding model, $s_{i,j}$ is a sentence inside a 256-word chunk, and $A$ is the aspect. That score drives the Top-W selection of Eq. (3): within each chunk the sentences are sorted by this score and kept until their cumulative word count reaches $W$, then the survivors are glued back in their original order. This mechanism converts a long-document summarization problem into a short-document one without retraining the LLM, and it is what frees token space for in-context learning.","core_discovery":"On the paper's own terms, the discovery is that relevance-based pruning of the input, driven by the aspect itself, is a sufficient alternative to either full-text prompting or length-based truncation for LLM aspect-based summarization. Rather than relying on prompt engineering or fitting the whole document into the context window, the system defines an aspect-aware selection step: embed each sentence and the aspect, rank sentences by cosine similarity, select the highest-scoring sentences per chunk until a word threshold is met, and concatenate the survivors in their original order. The experiments report consistent gains across Mistral and two Llama3 sizes, and the gains persist when an in-context example is added. The paper also claims the pruned input leaves token budget for such examples, which is why SARESG_ICL often beats truncated ICL.","pith_inferences":["The paper does not directly measure hallucination; I infer that the claimed hallucination reduction is a plausible consequence of a shorter, focused input, but it needs a factuality-oriented evaluation to be established.","Because the pruning threshold $W$ and chunk size are dataset-dependent, the framework's practical value depends on having a cheap way to tune them per corpus; the paper's own limitations say the best pruning parameters vary with article length.","The embedding model's ranking quality is the hidden bottleneck; any improvement in sentence-aspect similarity, such as learned relevance models or aspect-aware query expansion, should transfer directly to better summaries.","A natural extension is to make retrieval iterative or hierarchical, first pruning at chunk level and then at sentence level, which could handle even longer inputs than the current single-pass scheme."],"forward_implications":["If the retrieval-based pruning claim holds, long documents can be summarized by LLMs with small context windows without manually engineered prompts: the aspect name itself supplies the retrieval query.","Token savings from pruning allow one-shot in-context examples, and the tables suggest that an 8-billion-parameter model with SARESG_ICL can match a 70-billion-parameter zero-shot model on several metrics.","Because irrelevant sentences are removed before generation, the input the LLM sees is shorter and more topic-concentrated, which the paper argues lowers the chance of hallucinated or off-aspect content.","The chunk-size finding (256 words optimal in their test) implies that the method needs a coarse-enough context unit to judge relevance, and that sentence-level retrieval alone is weaker."],"supporting_citations":[{"why":"Supplies the USB benchmark dataset used as one of the three evaluation corpora.","marker":"[38]"},{"why":"Supplies the OAsum dataset, the second evaluation corpus with many aspects.","marker":"[39]"},{"why":"Supplies the Ma-news dataset, the third evaluation corpus for the experiments.","marker":"[40]"},{"why":"Defines the Llama3 model family used for Llama3-8b and Llama3-70b evaluations.","marker":"[44]"},{"why":"Defines the Mistral/Mixtral 8x7b model used as the second LLM family.","marker":"[45]"},{"why":"Defines the Selective Context baseline that the method is compared against in zero-shot and ICL settings.","marker":"[46]"},{"why":"Defines the METEOR metric used to evaluate generated summaries.","marker":"[41]"},{"why":"Defines the ROUGE family of metrics used for n-gram and longest-common-subsequence overlap.","marker":"[42]"},{"why":"Defines BERTScore, the embedding-based metric used for precision, recall, and F1 evaluation.","marker":"[43]"}],"fun_headline_variants":["Aspect-based sentence selection improves LLM summarization","Self-aspect retrieval enhances aspect-specific summaries","Relevance pruning frees tokens, lifts summarization scores","Aspect-aware embedding filter boosts summary quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that carries the whole method is that an embedding model's cosine similarity between a sentence and the aspect phrase ranks sentences by how much they matter for the final summary, so pruning to the top-scoring sentences never throws away content the summary needs.","fun_headline_variants_meta":{"raw":{"variants":["Aspect-based sentence selection improves LLM summarization","Self-aspect retrieval enhances aspect-specific summaries","Relevance pruning frees tokens, lifts summarization scores","Aspect-aware embedding filter boosts summary quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1195,"prompt_tokens":882,"completion_tokens":313,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":253}},"tokens_in":498,"tokens_out":313,"duration_ms":3734,"temperature":1.0,"reasoning_tokens":253,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:15:00.313712+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a document with human-annotated sentences that appear in the gold aspect summary, run SARESG's pruning, and check how many gold-summary sentences fall below the Top-W cutoff; if a substantial fraction of the gold content is discarded while irrelevant sentences survive, the retrieval-ranking assumption is false. A cheaper version is an ablation that replaces the embedding scores with random sentence rankings: if random pruning performs as well as SARESG on the same metrics, the retrieval signal is not doing the work.","supporting_citations":[{"cited_title":"Usb: A unified summarization benchmark across tasks and domains,","cited_arxiv_id":null,"evidence_quote":"Supplies the USB benchmark dataset used as one of the three evaluation corpora."},{"cited_title":"Oasum: Large-scale open domain aspect-based summarization,","cited_arxiv_id":null,"evidence_quote":"Supplies the OAsum dataset, the second evaluation corpus with many aspects."},{"cited_title":"Inducing document structure for aspect-based summarization,","cited_arxiv_id":null,"evidence_quote":"Supplies the Ma-news dataset, the third evaluation corpus for the experiments."},{"cited_title":"Compressing context to enhance inference efficiency of large language models,","cited_arxiv_id":null,"evidence_quote":"Defines the Selective Context baseline that the method is compared against in zero-shot and ICL settings."},{"cited_title":"ROUGE: A package for automatic evaluation of summaries,","cited_arxiv_id":null,"evidence_quote":"Defines the ROUGE family of metrics used for n-gram and longest-common-subsequence overlap."}],"review_version":1}