{"id":"fd0fe063-0b90-4db9-8dba-5ba729dfec66","arxiv_id":"1908.11152","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A section-based extractive summarization system for scientific papers was rated better than a whole-paper version in a small author-run evaluation, though the test had notable biases.","lead":"This paper describes IBM Science Summarizer, a search tool that automatically summarizes each section of a computer science paper so researchers can scan many papers quickly. The authors report that section-by-section summaries were rated more highly than whole-paper summaries in a small study where the papers' own authors judged the output.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation of the central paradigm is undermined by a coverage task that is satisfied by construction and by unblinded self-raters; the reported 'all 3 tasks' win is not yet evidence that section-based summarization is intrinsically better.","rationale":"Reader's weakest_assumption pointed to structural bias in the coverage task and the absence of neutral raters; I agree with that diagnosis and single it out as the load-bearing issue. The paper's central claim depends on a three-task comparison, and Task (2) is not a fair test because it measures exactly the property the section-based method hard-codes. The other two tasks are weakened by the rater pool and presentation differences, but even if they were clean, the headline would still be 'better on precision and global quality' rather than 'better on all 3 tasks.' A simple blinded re-run with independent raters would settle whether the paradigm itself, not the evaluation structure, drives the reported wins. Code/data release and external baselines are important but secondary; they do not affect the logical validity of the comparison as directly as the measurement artifact does. The overclaim of being the 'first tool' is a scope issue, not evidence about the paradigm. Because the concern is concrete and addressable, and because the paper does show a plausible and useful system, the conditional verdict stands.","tokens_in":6959,"tokens_out":5529,"duration_ms":52316,"concrete_test":"Recruit three independent NLP researchers per paper (72 ratings total), blind to the summary-generation condition and to the paper's author list, and present both summaries in identical flat formatting without section headers or any structural cues. Ask raters to (1) mark precision per sentence, (2) list which of a fixed set of section names they believe are covered, and (3) give a global quality score. Compare the two conditions with a paired Wilcoxon test; if Task (2) no longer shows a significant advantage and Task (3) falls below significance, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that section-based summaries 'significantly outperform' section-agnostic summaries on all three tasks (Section 6, Results) is load-bearing on the design of Task (2). Raters were asked to score 'how well each of the sections of the paper is covered in the summary.' The section-based summary is constructed from exactly 10 sentences per section, so every section is guaranteed representation; the section-agnostic summary is a single flat extract of the same total length, with no such guarantee. Task (2) therefore measures the method's defining property, not an independent aspect of quality; the 68% vs 22% win is nearly a tautology. The remaining tasks are also compromised: the raters are the papers' own authors, each paper is rated by a single person, no inter-annotator agreement is reported, and summaries are presented in a visibly different format (headers vs flat block), so the significant global-quality difference (Task 3) and small precision difference (Task 1) could reflect presentation and confirmation bias rather than the paradigm. Without a blinded, neutral evaluation and a coverage measure that does not presuppose section structure, the evidence does not establish the paradigm's superiority.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes IBM Science Summarizer, a system for automatic summarization and exploration of Computer Science publications. The system ingests roughly 270,000 papers from arXiv and the ACL Anthology, extracts text, tables, figures, and task/dataset/metric entities, and produces extractive summaries using an unsupervised cross-entropy optimization method. A distinctive design choice is that each section of a paper is summarized independently and the section summaries are then concatenated into the paper summary. The paper reports a human evaluation in which 12 authors of NLP papers rated summaries of 24 of their own papers, comparing this section-based summary against a section-agnostic flat summary of the same total length. The authors claim that the section-based summaries significantly outperform the section-agnostic summaries on all three evaluation tasks: sentence-level precision, section coverage, and global quality.","tokens_in":7186,"tokens_out":4910,"duration_ms":50952,"significance":"If the central comparative claim were supported, the section-based summarization paradigm would be a useful and practical contribution for scientific document exploration, and the system itself is a substantial engineering effort: it includes a large-scale ingestion pipeline, entity extraction with reported quantitative performance, a qualitative user study motivating the use cases, and a deployed user interface. The entity-extraction results (macro-F1 56.6, micro-F1 66.0) and the qualitative scenario analysis are concrete strengths. However, the reported validation is not currently convincing: the coverage task is partly satisfied by construction for the section-based method, the raters are the papers' own authors with no blinding or inter-annotator agreement, and the statistical reporting is incomplete. As presented, the evidence does not establish that the section-based paradigm is intrinsically superior to a section-agnostic approach.","major_comments":[{"comment":"The coverage task is confounded with the definition of the section-based summary. The section-based summary is constructed by selecting exactly 10 sentences from each section, so every section is guaranteed representation by construction. The section-agnostic summary is a single flat extract of the same total length with no per-section allocation. Asking raters to score 'how well each of the sections of the paper is covered in the summary' therefore measures whether the summary has the section structure, not whether the section-based approach produces better summaries. The reported 68% vs. 22% win on this task is close to a design artifact. A valid comparison would need a coverage measure that does not presuppose the section structure, for example asking annotators to identify key content units independently of section boundaries and measuring recall of those units.","section":"Section 6, Tasks (2)"},{"comment":"The human evaluation uses the authors of the summarized papers as raters, with a single rater per paper and no inter-annotator agreement reported. Authors are not neutral evaluators of summaries of their own work, and the summaries are presented in visibly different forms (a structured section-based summary versus a flat block), so the paper does not establish that raters were blinded to summary type. Consequently, the global-quality result (Task 3) and the precision result (Task 1) could reflect presentation effects and rater bias rather than the summarization paradigm. The evaluation needs neutral expert raters, multiple raters per summary, and a presentation format that does not reveal which method produced each summary.","section":"Section 6, Evaluators and Procedure"},{"comment":"The claim that section-based summaries 'significantly outperform' on all three tasks is not supported by the statistics as reported. No significance test is named, no test statistic or effect size is given, and the exact p-values are not reported. The percentages in Table 1 are also internally inconsistent with the stated sample size of 24 papers: for example, 68%, 22%, and 10% of 24 are not integers, and 4.5% wins for the section-agnostic summary on Task 3 is not attainable with 24 papers. The authors should report exact win/tie/loss counts or percentages consistent with the sample size, alongside the statistical procedure used (e.g., a paired Wilcoxon signed-rank test).","section":"Section 6, Results and Table 1"}],"minor_comments":[{"comment":"The statement that this is 'the first tool for automatic summarization and exploration of scientific documents' is too strong given the cited systems such as Surveyor, CL Scholar, TutorialBank, and ScisummNet; the claim should be qualified or removed.","section":"Section 2, Related Work"},{"comment":"The system is described as query-focused, but the human evaluation does not state whether a query was provided during summary generation or whether the summaries were query-agnostic. The evaluation conditions should be specified so the reader can tell what capability is being tested.","section":"Section 5 and Section 6"},{"comment":"The phrase '3 means good' for the 1-5 scale is confusing; if 1 is 'very bad' and 5 is 'excellent', then 3 should be neutral. Please clarify the scale anchors.","section":"Section 6, Task descriptions"},{"comment":"The footnote markers '†' and '‡' are used in the table but the legend appears after the table; consider placing it directly below the table for readability.","section":"Table 1"},{"comment":"The entity extraction module is evaluated with macro-F1 and micro-F1 scores, but no baseline comparison is provided, so it is difficult to assess whether the proposed textual-entailment formulation improves over existing approaches.","section":"Section 4, Entity Extraction"}],"recommendation":"major_revision","confidential_remarks":"The system is a real engineering contribution and the qualitative user study is a useful design input, but the validation section does not currently support the paper's main comparative claim. The coverage task is circular, the raters are self-interested, and the statistical reporting is incomplete. I would be willing to see a revised version that replaces this evaluation with a blinded study using neutral raters and a coverage measure that does not presuppose section structure. The 'first tool' claim should also be moderated. This is not a reject because the underlying system and the section-based idea are plausible and potentially publishable after a proper evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper you'd want to know about: it's a system paper for IBM Science Summarizer, and its real contribution is the section-based paradigm for summarizing CS papers, not the evaluation that claims the paradigm wins. The system itself is concrete—270k papers ingested, entity extraction over tasks/datasets/metrics, query-focused extractive summaries using their CE method—and the qualitative user-study framing (keep-up-to-date scenario) is sensible. If you work on scientific summarization, the section-based idea is worth engaging with.\n\nWhat the paper does well: it identifies a real pain point and builds a complete pipeline, which is more than most summarization papers ship. The related-work placement is fine, and they are candid that the production version currently serves query-agnostic summaries, so the query-focused machinery is present but not yet the deployed experience.\n\nThe soft spot is the human evaluation, and it is a load-bearing one. Task (2), the coverage task, asks raters to score how well each section of the paper is covered. The section-based summary is defined as 10 sentences per section; the section-agnostic summary is a flat block with the same total length. That task therefore rewards the defining property of the section-based method by construction. Calling the 68% vs 22% win an empirical validation of the paradigm is close to circular. The other two tasks are less structurally biased, but they are not clean either: the raters are the papers' own authors, each paper seems to get one rater, there is no inter-annotator agreement, the two summary types are shown in different visual formats (section headers vs flat block), and there is no external baseline such as LexRank or a ScisummNet-based method. The global quality difference (Task 3) could reflect format and confirmation bias as easily as the paradigm. So the sentence 'the quality of the section-based summaries significantly outperforms the section-agnostic summaries on all 3 tasks, supporting our proposed paradigm' goes beyond the evidence. Also, calling it the 'first tool' for automatic summarization and exploration is an overclaim; CL Scholar and others explore, and there are prior summarization tools.\n\nNone of this means the system is bad. It means the central claim is under-supported as reported. The evaluation is fixable: neutral, blinded raters, a coverage measure that does not presuppose section structure, a baseline or two, and released code/data or at least a public demo (the URL is cited but as of reading I can't verify it still works).\n\nWho is this for? System builders and researchers interested in scientific-document summarization. Worth a reading group to talk about evaluation design. I'd send it to peer review, but with a request for major revision focused on the evaluation, not on the system.","headline":"A useful system paper whose section-based paradigm is undercut by an evaluation task that rewards the method by construction; the paradigm may be right, but the paper doesn't show it.","tokens_in":7787,"tokens_out":4162,"would_cite":true,"duration_ms":36207,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that summarizing each section of a scientific paper separately produces better summaries than flattening the whole text.","keywords":["scientific document summarization","section-based summarization","query-focused summarization","extractive summarization","cross entropy method","entity extraction","human evaluation","academic search"],"falsifier":"Re-run the same three-task evaluation with twelve external researchers who did not write the papers, blinded to which summary is section-based and which is section-agnostic, and using a coverage question that does not presuppose section boundaries; if the section-agnostic summaries score as high or higher, the paper's central claim would be falsified. A quicker check is to compute inter-annotator agreement among the current author raters.","tokens_in":6787,"feed_emoji":"📄","tokens_out":9749,"duration_ms":82661,"temperature":0.7,"pith_summary":"The paper claims that the right way to summarize a scientific article is to summarize each section independently and then combine those section summaries into one paper summary. The authors built a retrieval-and-summarization system that ingests 270,000 computer science papers, extracts tasks, datasets, and metrics from the full text, and produces query-focused or generic summaries. To test the section-based idea, twelve authors rated summaries of their own papers on sentence-level precision, section coverage, and overall quality; the section-based summaries scored higher on all three tasks, with the coverage and quality gaps reported as statistically significant. If the claim is right, researchers can use section-level summaries to decide whether to read a paper and to jump directly to the parts they need, reducing information overload.","feed_headline":"Section-by-section summaries beat flat paper summaries","feed_subtitle":"Authors rated the section-based version higher on precision, coverage, and overall quality in a 24-paper study.","key_machinery":"The engine is an unsupervised, extractive, query-focused summarizer built on the Cross Entropy method, applied independently to each section of the paper. For a given section, it searches for a 10-sentence subset that maximizes a product of five quality predictors: query saliency (cosine similarity between the summary and query unigram bags-of-words), entity coverage (Jaccard similarity between the summary's task/dataset/metric entities and the target entities), diversity (entropy of the summary's unigram language model), text coverage (cosine similarity between the summary and the section bigram bags-of-words), and sentence length. Entity extraction, which identifies task-dataset-metric triples from the full paper using a textual entailment model, feeds the entity-coverage objective and supports filtering by those entity types. The same algorithm, run once per section and then concatenated, is the section-based mechanism that the human evaluation compares against a flat, section-agnostic run.","core_discovery":"The central claim is that section-based summarization—generating a separate extractive summary of each section and composing them into one paper summary—produces better summaries of scientific papers than applying the same algorithm to the paper as a single flat text. The proposed system is described as the first tool for automatic summarization and exploration of scientific documents. The human validation asked twelve authors to evaluate summaries of 24 co-authored papers on three tasks: per-sentence precision, per-section coverage, and global quality. The section-based summaries won on all three tasks, with more papers preferring the section-based summary in each task (63%, 68%, and 36% wins respectively), and the coverage and quality differences are reported as statistically significant. On the paper's own terms, this supports the section-based paradigm.","pith_inferences":["A neutral, double-blind replication with raters who did not author the papers would be the natural next test; until then, part of the measured advantage may reflect the evaluation's section-coverage question, which rewards the section-based design by construction.","If the advantage is real and structural, the same section-by-section approach should transfer to other structured document genres, such as medical articles or legal decisions, where section boundaries carry meaning.","Varying the fixed 10-sentence budget per section—for example, proportional to section length—could change the precision/coverage balance and is a direct extension the paper leaves open."],"forward_implications":["Readers can jump to the section summary that matches their information need, such as the experiments or related work, instead of reading the full paper.","Query-focused section summaries make it feasible to scan a large corpus for a specific need, like the state of the art on a benchmark or dataset, without opening the PDF.","Because the extraction of tasks, datasets, and metrics runs on the full paper rather than only the abstract, the system can support filtering and leaderboard-style exploration across 270,000 papers.","The unsupervised, extractive design can summarize newly published papers immediately, without waiting for citation data to accumulate."],"supporting_citations":[{"why":"Supplies the unsupervised query-focused extractive summarization algorithm that the system adapts to individual paper sections.","marker":"(Feigenblat et al., 2017)"},{"why":"Provides the Cross Entropy optimization method used to select the sentence subset for each section summary.","marker":"(Rubinstein and Kroese, 2004)"},{"why":"Supplies the pseudo-relevance query expansion that turns short queries into a 100-term profile.","marker":"(Xu et al., 2009)"},{"why":"Provides the fixed-point term weighting schema used for verbose queries.","marker":"(Paik and Oard, 2014)"},{"why":"Supplies the textual-entailment model that extracts task-dataset-metric triples from full papers, feeding the entity coverage objective.","marker":"(Hou et al., 2019)"},{"why":"Defines a citation-based summarization dataset and approach that the paper explicitly contrasts with its citation-free, more detailed summaries.","marker":"Yasunaga et al. (2019)"}],"fun_headline_variants":["Section-based summaries beat flat in 24-paper study","Summarize papers by section for better results","Section-wise summaries win on precision, coverage, quality","Splitting papers into sections improves summaries","Human experts prefer section-based paper summaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the twelve authors' self-ratings, especially the question of how well each paper section is covered, give an unbiased comparison even though the section-based summary is constructed one block per section while the section-agnostic summary is a single flat block.","fun_headline_variants_meta":{"raw":{"variants":["Section-based summaries beat flat in 24-paper study","Summarize papers by section for better results","Section-wise summaries win on precision, coverage, quality","Splitting papers into sections improves summaries","Human experts prefer section-based paper summaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000372,"raw_usage":{"total_tokens":1902,"prompt_tokens":770,"completion_tokens":1132,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":386,"completion_tokens_details":{"reasoning_tokens":1062}},"tokens_in":386,"tokens_out":1132,"duration_ms":7552,"temperature":1.0,"reasoning_tokens":1062,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:23:05.526817+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same three-task evaluation with twelve external researchers who did not write the papers, blinded to which summary is section-based and which is section-agnostic, and using a coverage question that does not presuppose section boundaries; if the section-agnostic summaries score as high or higher, the paper's central claim would be falsified. A quicker check is to compute inter-annotator agreement among the current author raters.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the unsupervised query-focused extractive summarization algorithm that the system adapts to individual paper sections."},{"cited_title":"Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction","cited_arxiv_id":"1906.09317","evidence_quote":"Supplies the textual-entailment model that extracts task-dataset-metric triples from full papers, feeding the entity coverage objective."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines a citation-based summarization dataset and approach that the paper explicitly contrasts with its citation-free, more detailed summaries."}],"review_version":1}