Pith. sign in

REVIEW 3 major objections 5 minor 19 references

A Summarization System for Scientific Documents

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that summarizing each section of a scientific paper separately produces better summaries than flattening the whole text.

desk verdict A useful system paper whose section-based paradigm is undercut by an evaluation task that rewards the method by construction; the paradigm may be right, but the paper doesn't show it. read the letter →

arxiv 1908.11152 v1 pith:CMC7643A submitted 2019-08-29 cs.CL

Shai Erera , Michal Shmueli-Scheuer , Guy Feigenblat , Ora Peled Nakash , Odellia Boni , Haggai Roitman , Doron Cohen , Bar Weiner
show 10 more authors
Yosi Mass Or Rivlin Guy Lev Achiya Jerbi Jonathan Herzig Yufang Hou Charles Jochim Martin Gleize Francesca Bonin David Konopnicki
This is my paper · ORCID
classification cs.CL
keywords scientificdocumentsummarizationsection-basedquery-focusedextractivecrossentropymethodentityextractionhumanevaluationacademicsearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the right way to summarize a scientific article is to summarize each section independently and then combine those section summaries into one paper summary. The authors built a retrieval-and-summarization system that ingests 270,000 computer science papers, extracts tasks, datasets, and metrics from the full text, and produces query-focused or generic summaries. To test the section-based idea, twelve authors rated summaries of their own papers on sentence-level precision, section coverage, and overall quality; the section-based summaries scored higher on all three tasks, with the coverage and quality gaps reported as statistically significant. If the claim is right, researchers can use section-level summaries to decide whether to read a paper and to jump directly to the parts they need, reducing information overload.

What carries the argument

The engine is an unsupervised, extractive, query-focused summarizer built on the Cross Entropy method, applied independently to each section of the paper. For a given section, it searches for a 10-sentence subset that maximizes a product of five quality predictors: query saliency (cosine similarity between the summary and query unigram bags-of-words), entity coverage (Jaccard similarity between the summary's task/dataset/metric entities and the target entities), diversity (entropy of the summary's unigram language model), text coverage (cosine similarity between the summary and the section bigram bags-of-words), and sentence length. Entity extraction, which identifies task-dataset-metric triples from the full paper using a textual entailment model, feeds the entity-coverage objective and supports filtering by those entity types. The same algorithm, run once per section and then concatenated, is the section-based mechanism that the human evaluation compares against a flat, section-agnostic run.

What would settle it

Re-run the same three-task evaluation with twelve external researchers who did not write the papers, blinded to which summary is section-based and which is section-agnostic, and using a coverage question that does not presuppose section boundaries; if the section-agnostic summaries score as high or higher, the paper's central claim would be falsified. A quicker check is to compute inter-annotator agreement among the current author raters.

Watch

Extended reading notes

Core claim

The central claim is that section-based summarization—generating a separate extractive summary of each section and composing them into one paper summary—produces better summaries of scientific papers than applying the same algorithm to the paper as a single flat text. The proposed system is described as the first tool for automatic summarization and exploration of scientific documents. The human validation asked twelve authors to evaluate summaries of 24 co-authored papers on three tasks: per-sentence precision, per-section coverage, and global quality. The section-based summaries won on all three tasks, with more papers preferring the section-based summary in each task (63%, 68%, and 36% wins respectively), and the coverage and quality differences are reported as statistically significant. On the paper's own terms, this supports the section-based paradigm.

Load-bearing premise

The load-bearing premise is that the twelve authors' self-ratings, especially the question of how well each paper section is covered, give an unbiased comparison even though the section-based summary is constructed one block per section while the section-agnostic summary is a single flat block.

Editorial extensions

If this is right

  • Readers can jump to the section summary that matches their information need, such as the experiments or related work, instead of reading the full paper.
  • Query-focused section summaries make it feasible to scan a large corpus for a specific need, like the state of the art on a benchmark or dataset, without opening the PDF.
  • Because the extraction of tasks, datasets, and metrics runs on the full paper rather than only the abstract, the system can support filtering and leaderboard-style exploration across 270,000 papers.
  • The unsupervised, extractive design can summarize newly published papers immediately, without waiting for citation data to accumulate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A neutral, double-blind replication with raters who did not author the papers would be the natural next test; until then, part of the measured advantage may reflect the evaluation's section-coverage question, which rewards the section-based design by construction.
  • If the advantage is real and structural, the same section-by-section approach should transfer to other structured document genres, such as medical articles or legal decisions, where section boundaries carry meaning.
  • Varying the fixed 10-sentence budget per section—for example, proportional to section length—could change the precision/coverage balance and is a direct extension the paper leaves open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper describes IBM Science Summarizer, a system for automatic summarization and exploration of Computer Science publications. The system ingests roughly 270,000 papers from arXiv and the ACL Anthology, extracts text, tables, figures, and task/dataset/metric entities, and produces extractive summaries using an unsupervised cross-entropy optimization method. A distinctive design choice is that each section of a paper is summarized independently and the section summaries are then concatenated into the paper summary. The paper reports a human evaluation in which 12 authors of NLP papers rated summaries of 24 of their own papers, comparing this section-based summary against a section-agnostic flat summary of the same total length. The authors claim that the section-based summaries significantly outperform the section-agnostic summaries on all three evaluation tasks: sentence-level precision, section coverage, and global quality.

Significance. If the central comparative claim were supported, the section-based summarization paradigm would be a useful and practical contribution for scientific document exploration, and the system itself is a substantial engineering effort: it includes a large-scale ingestion pipeline, entity extraction with reported quantitative performance, a qualitative user study motivating the use cases, and a deployed user interface. The entity-extraction results (macro-F1 56.6, micro-F1 66.0) and the qualitative scenario analysis are concrete strengths. However, the reported validation is not currently convincing: the coverage task is partly satisfied by construction for the section-based method, the raters are the papers' own authors with no blinding or inter-annotator agreement, and the statistical reporting is incomplete. As presented, the evidence does not establish that the section-based paradigm is intrinsically superior to a section-agnostic approach.

major comments (3)
  1. [Section 6, Tasks (2)] The coverage task is confounded with the definition of the section-based summary. The section-based summary is constructed by selecting exactly 10 sentences from each section, so every section is guaranteed representation by construction. The section-agnostic summary is a single flat extract of the same total length with no per-section allocation. Asking raters to score 'how well each of the sections of the paper is covered in the summary' therefore measures whether the summary has the section structure, not whether the section-based approach produces better summaries. The reported 68% vs. 22% win on this task is close to a design artifact. A valid comparison would need a coverage measure that does not presuppose the section structure, for example asking annotators to identify key content units independently of section boundaries and measuring recall of those units.
  2. [Section 6, Evaluators and Procedure] The human evaluation uses the authors of the summarized papers as raters, with a single rater per paper and no inter-annotator agreement reported. Authors are not neutral evaluators of summaries of their own work, and the summaries are presented in visibly different forms (a structured section-based summary versus a flat block), so the paper does not establish that raters were blinded to summary type. Consequently, the global-quality result (Task 3) and the precision result (Task 1) could reflect presentation effects and rater bias rather than the summarization paradigm. The evaluation needs neutral expert raters, multiple raters per summary, and a presentation format that does not reveal which method produced each summary.
  3. [Section 6, Results and Table 1] The claim that section-based summaries 'significantly outperform' on all three tasks is not supported by the statistics as reported. No significance test is named, no test statistic or effect size is given, and the exact p-values are not reported. The percentages in Table 1 are also internally inconsistent with the stated sample size of 24 papers: for example, 68%, 22%, and 10% of 24 are not integers, and 4.5% wins for the section-agnostic summary on Task 3 is not attainable with 24 papers. The authors should report exact win/tie/loss counts or percentages consistent with the sample size, alongside the statistical procedure used (e.g., a paired Wilcoxon signed-rank test).
minor comments (5)
  1. [Section 2, Related Work] The statement that this is 'the first tool for automatic summarization and exploration of scientific documents' is too strong given the cited systems such as Surveyor, CL Scholar, TutorialBank, and ScisummNet; the claim should be qualified or removed.
  2. [Section 5 and Section 6] The system is described as query-focused, but the human evaluation does not state whether a query was provided during summary generation or whether the summaries were query-agnostic. The evaluation conditions should be specified so the reader can tell what capability is being tested.
  3. [Section 6, Task descriptions] The phrase '3 means good' for the 1-5 scale is confusing; if 1 is 'very bad' and 5 is 'excellent', then 3 should be neutral. Please clarify the scale anchors.
  4. [Table 1] The footnote markers '†' and '‡' are used in the table but the legend appears after the table; consider placing it directly below the table for readability.
  5. [Section 4, Entity Extraction] The entity extraction module is evaluated with macro-F1 and micro-F1 scores, but no baseline comparison is provided, so it is difficult to assess whether the proposed textual-entailment formulation improves over existing approaches.

Circularity Check

1 steps flagged · score 4.0 of 10

The section-based paradigm's win on the coverage task is substantially built into the evaluation design, since the method is defined as one summary per section and the metric asks how well each section is covered.

  1. self definitional [Section 5 (Summarization) and Section 6 (Human Evaluation, Tasks and Results)]
    "to ensure our summarizer assigns sufficient attention to each of these aspects we have opted to generate a standalone summary for each section... For the section-based summary, each section's summary length was fixed to 10 sentences... (2) we asked them to evaluate how well each of the sections of the paper is covered in the summary (i.e., coverage/recall)... the quality of the section-based summaries significantly outperforms the section-agnostic summaries on all 3 tasks, supporting our proposed paradigm."

    The section-based method is defined as producing a separate 10-sentence summary for every section, so every section necessarily has a dedicated block of extracted sentences. The coverage/recall task then asks raters to score how well each section of the paper is covered in the summary. For the section-based summary, the coverage question is answered partly by construction: a section block exists for every section. The section-agnostic summary is a single flat extraction with no such per-section guarantee. Comparing the two on per-section coverage therefore measures the method's defining property rather than an independent quality dimension, making the observed 68% vs 22% win for Task (2) substantially a restatement of how the two summary types were generated.

full rationale

Most of the paper is system building rather than a mathematical derivation: PDF parsing, entity extraction, indexing, and the extractive summarization algorithm are either described in the text or cited as prior work, and the summarization module itself is not claimed to derive from a fitted parameter. The central comparative claim, however, rests on a human evaluation whose Task (2) mirrors the method's own definition: section-based summaries are constructed as one block per section, and the raters are asked how well each section is covered. This gives the section-based method a structural advantage that is close to a tautology for that task. Tasks (1) and (3) are less definitionally tied, which is why the circularity is only partial. Additional validity concerns, such as the raters being the papers' own authors with no reported inter-annotator agreement, further weaken the evidence but are not themselves circularity. Overall, the paper does contain independent technical content, but the headline evaluation of the proposed paradigm is partly circular in its coverage metric, so a moderate score of 4 is appropriate.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

This paper does not rest on a derivation, so the ledger records the modeling choices, borrowed components, and evaluation assumptions that the central claim depends on.

free parameters (4)
  • Section summary length = 10 sentences
    Chosen by hand and fixed for all sections; affects the total summary length and section coverage, and the paper leaves variable length for future work (Section 5).
  • Query expansion profile size = 100 unigram terms
    Used when the query is short; the number 100 is set without a sensitivity analysis (Section 5, Query Handling).
  • Entity extraction model parameters = Macro-F1 56.6, micro-F1 66.0
    Trained on 332 NLP papers and evaluated on 162; the trained weights come from the self-cited Hou et al. (2019) work and are not specified here.
  • Quality predictor combination = Product of five predictors
    The target function multiplies query saliency, entities coverage, diversity, text coverage, and sentence length, based on Feigenblat et al. (2017), without evaluating each predictor's contribution.
assumptions (6)
  • domain assumption The Cross Entropy method from Feigenblat et al. (2017) is a state-of-the-art query-focused summarization algorithm.
    Used as given in Section 5; no baseline comparison in this paper's setting.
  • domain assumption The five quality predictors multiplied together define summary quality.
    Stated in Section 5, but the relative weights and individual contributions are not validated.
  • domain assumption Entity dictionaries from paperswithcode and the textual entailment model from Hou et al. (2019) correctly identify tasks, datasets, and metrics.
    The entity extraction module is described in Section 4; no error analysis is provided in this paper.
  • domain assumption Paper authors can reliably judge whether individual sentences belong in a summary of their own paper and can rate section coverage and overall quality on a 1-5 scale.
    Section 6 uses 12 co-authors as evaluators without any neutral judges or inter-annotator agreement.
  • domain assumption Science-Parse accurately extracts text, tables, and figures from PDFs.
    The ingestion pipeline in Section 4 depends on this external tool with no quality check reported.
  • domain assumption A qualitative study with 6 NLP users identifies the most valuable scenarios for all CS researchers.
    The use-cases in Section 1 drive the system design but come from a small, domain-specific sample.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Summarization System for Scientific Documents." pith.science (2026). https://pith.science/paper/CMC7643A

@misc{pith2026190811152,
  author       = {Pith},
  title        = {Pith review of: A Summarization System for Scientific Documents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CMC7643A}},
  note         = {Machine review of arXiv:1908.11152}
}
read the original abstract

We present a novel system providing summaries for Computer Science publications. Through a qualitative user study, we identified the most valuable scenarios for discovery, exploration and understanding of scientific documents. Based on these findings, we built a system that retrieves and summarizes scientific documents for a given information need, either in form of a free-text query or by choosing categorized values such as scientific tasks, datasets and more. Our system ingested 270,000 papers, and its summarization module aims to generate concise yet detailed summaries. We validated our approach with human experts.

Figures

Figures reproduced from arXiv: 1908.11152 by the authors.

Figure 1
Figure 1. IBM Science Summarizer Framework. tool for automatic summarization and exploration of scientific documents.5 3 System Overview IBM Science Summarizer’s main purpose is to support discovery, exploration and understand￾ing of scientific papers by providing summaries. The system has two parts. First, an ingestion pipeline parses and indexes papers’ content from arXiv.com and ACL anthology, as depicted in Fig￾ure 1(a). … view at source ↗
Figure 2
Figure 2. IBM Science Summarizer UI. ACL anthology7 . The ingestion pipeline consists of paper acquisition, extracting the paper’s text, tables and figures and enriching the paper’s data with various annotations and entities. Paper Parsing. We use Science-Parse8 to ex￾tract the PDF text, tables and figures. Science￾Parse outputs a JSON record for each PDF, which among other fields, contains the title, abstract text, metadata … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 18 canonical work pages

  1. [1]

    Amjad Abu-Jbara and Dragomir Radev. 2011. http://dl.acm.org/citation.cfm?id=2002472.2002536 Coherent citation-based summarization of scientific papers . In Proceedings of the 49th Annual HLT, HLT '11, pages 500--509. Association for Computational Linguistics

    work page arXiv 2011

  2. [2]

    Aaron Elkiss, Siwei Shen, Anthony Fader, G\" u ne s Erkan, David States, and Dragomir Radev. 2008. https://doi.org/10.1002/asi.v59:1 Blind men and elephants: What do citation summaries tell us about a research article? J. Am. Soc. Inf. Sci. Technol., 59(1):51--62

    work page doi:10.1002/asi.v59:1 2008

  3. [3]

    Alexander Fabbri, Irene Li, Prawat Trairatvorakul, Yijiao He, Weitai Ting, Robert Tung, Caitlin Westerfield, and Dragomir Radev. 2018. Tutorialbank: A manually-collected corpus for prerequisite chains, survey extraction and resource recommendation. In Proceedings of the 56th ACL, pages 611--620

    work page 2018

  4. [4]

    Guy Feigenblat, Haggai Roitman, Odellia Boni, and David Konopnicki. 2017. Unsupervised query-focused multi-document summarization using the cross entropy method. In Proceedings of the 40th International ACM SIGIR , pages 961--964

    work page 2017

  5. [5]

    Carlos Flavian, Raquel Gurrea, and Carlos Orus. 2009. Web design: a key factor for the website success. Journal of Systems and Information Technology, 11(2):168--184

    work page 2009

  6. [6]

    Kata G \' a bor, Davide Buscaldi, Anne - Kathrin Schumann, Behrang QasemiZadeh, Ha \" fa Zargayouna, and Thierry Charnois. 2018. Semeval-2018 task 7: Semantic relation extraction and classification in scientific papers. In Proceedings SemEval@NAACL-HLT 2018, pages 679--688

    work page 2018

  7. [7]

    Mahak Gambhir and Vishal Gupta. 2017. https://doi.org/10.1007/s10462-016-9475-9 Recent automatic text summarization techniques: A survey . Artif. Intell. Rev., 47(1):1--66

    doi:10.1007/s10462-016-9475-9 2017

  8. [8]

    Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2019. https://arxiv.org/abs/1906.09317 Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction . volume arXiv:1906.09317

    work page Pith review arXiv 2019

Show all 19 references
  1. [9]

    Rahul Jha, Reed Coke, and Dragomir Radev. 2015. Surveyor: A system for generating coherent survey articles for scientific topics. In Proceedings of the Twenty-Ninth AAAI, AAAI'15, pages 2167--2173

    2015

  2. [10]

    Wang Jie, Zhang Chengzhi, Zhang Mengying, and Deng Sanhong. 2018. Citationas: A tool of automatic survey generation based on citation content*. Journal of Data and Information Science, 3(2)

    2018

  3. [11]

    Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of EMNLP 2018, pages 3219--3232

    2018

  4. [12]

    C. D. Paice. 1981. The automatic generation of literature abstracts: An approach based on the identification of self-indicating phrases. In Proceedings of the 3rd Annual ACM SIGIR, SIGIR '80, pages 172--191

    1981

  5. [13]

    Paice and Paul A

    Chris D. Paice and Paul A. Jones. 1993. https://doi.org/10.1145/160688.160696 The identification of important concepts in highly structured technical papers . In Proceedings of the 16th Annual International ACM SIGIR, SIGIR '93, pages 69--78, New York, NY, USA. ACM

    1993

  6. [14]

    Paik and Douglas W

    Jiaul H. Paik and Douglas W. Oard. 2014. A fixed-point method for weighting terms in verbose informational queries. CIKM '14, pages 131--140, New York, NY, USA. ACM

    2014

  7. [15]

    Vahed Qazvinian and Dragomir R. Radev. 2008. http://dl.acm.org/citation.cfm?id=1599081.1599168 Scientific paper summarization using citation summary networks . In Proceedings of the 22Nd International Conference on Computational Linguistics - Volume 1, COLING '08, pages 689--696

    2008

  8. [16]

    Rubinstein and Dirk P

    Reuven Y. Rubinstein and Dirk P. Kroese. 2004. The Cross Entropy Method: A Unified Approach To Combinatorial Optimization, Monte-carlo Simulation. Springer-Verlag, Berlin, Heidelberg

    2004

  9. [17]

    Mayank Singh, Pradeep Dogga, Sohan Patro, Dhiraj Barnwal, Ritam Dutt, Rajarshi Haldar, Pawan Goyal, and Animesh Mukherjee. 2018. Cl scholar: The acl anthology knowledge graph miner. In Proceedings of the NAACL 2018

    2018

  10. [18]

    Jones, and Bin Wang

    Yang Xu, Gareth J.F. Jones, and Bin Wang. 2009. Query dependent pseudo-relevance feedback based on wikipedia. In Proceedings of the 32Nd International ACM SIGIR, pages 59--66

    2009

  11. [19]

    Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander Richard Fabbri, Irene Li, Dan Friedman, and Dragomir R. Radev. 2019. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In AAAI 2019

    2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.