REVIEW 4 major objections 5 minor 1 cited by
Exploring Domain Shift in Extractive Text Summarization
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Extractive summarization models trained on one news publication lose accuracy on unseen ones, and domain-aware training partially closes the gap.
desk verdict A useful empirical testbed and a mostly sound demonstration that publication identity is a real domain axis, but the strategy-ranking claims rest on small ROUGE gaps without significance tests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is MULTI-SUM, a re-purposed subset of Newsroom containing ten publications as domains, with extractive labels generated by a greedy oracle. The central mechanism is the verification matrix: on the diagonal sits the ROUGE score of a model trained and tested on the same publication, and each off-diagonal cell is the difference obtained by training on another publication; negative off-diagonal entries quantify domain shift. The paper's second mechanism is the family of four learning strategies built on a CNN-Transformer extractive encoder: a shared-parameter joint model, a feature-based BERT variant, a domain-tag embedding that includes an ‘X’ tag for unseen domains, and a meta-learning gradient-communication rule that purifies shared features.
What would settle it
Train the same extractive model on equal-sized, length- and position-matched samples from each of the ten publications and rerun the cross-domain matrix; if the off-diagonal ROUGE drops largely disappear, the “serious domain shift” finding is an artifact of data-quantity or label-bias confounds rather than a genuine publication-domain effect.
Extended reading notes
Core claim
The central claim is that publication identity defines a meaningful domain axis in extractive summarization and that ignoring it seriously harms generalization. In the verification experiment, for every ordered pair of the ten publications, training on the source and testing on the target scores below training and testing on the target itself, so every off-diagonal entry of the ROUGE-1 matrix is negative. The paper interprets this as direct evidence of domain shift. On the learning-strategy comparison, the domain-tag model (Model III) outperforms the joint basic model on both in-domain and out-of-domain averages, and the meta-learning model (Model IV) cuts the ROUGE-1 gap between in-domain and out-of-domain performance from 1.47 to 0.71, making it the most transferable strategy even though it loses a little in-domain accuracy. BERT features improve in-domain scores but generalize less to out-of-domain or cross-dataset settings.
Load-bearing premise
The claim that publication identity itself causes the performance drops rests on the assumption that the gap is not actually produced by confounds such as unequal training-set sizes or differing sentence-position bias across publications, which the verification experiment does not control for.
Editorial extensions
If this is right
- Summarization models should be evaluated on unseen outlets, since in-domain scores systematically overstate their ability.
- A publication tag is a low-cost improvement: it helps both source domains and zero-shot target domains in MULTI-SUM.
- Meta-learning is the best choice when transfer matters most, but it trades away in-domain accuracy, so it is not optimal for users who care only about known domains.
- Feature-based BERT is a strong in-domain feature extractor but not a substitute for explicit domain-aware training when the test distribution shifts.
- On CNN/DailyMail, publication tags improve ROUGE over the base model, indicating that the source-identity signal also helps in a standard mixed-outlet benchmark.
Reading between the lines
- If outlet identity is a domain axis, the same verification logic could be applied within a single outlet to test finer-grained shifts (by section, author, or time period), which the paper does not do.
- The unknown-tag trick suggests a practical recipe for real-world corpora without outlet metadata: reserve a learned tag for examples whose source is unknown and let the model fall back on shared features.
- Because position bias differs strongly across the ten publications, part of the measured domain shift is probably stylistic (where summaries place their lead sentence), so a position-normalized or position-calibrated model might transfer even better than the reported ROUGE gaps suggest.
- The ranking of the four strategies on this testbed might change if source domains were balanced by size; reweighting training data by publication is a direct, testable extension.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes that the source publication, rather than the topical category, is a meaningful 'domain' for extractive summarization. It re-purposes a subset of Newsroom into MULTI-SUM with ten publication domains, split into five source and five target domains. Section 3 reports a verification experiment in which a basic CNN-Transformer extractive model trained on each domain is evaluated on all domains; all off-diagonal ROUGE-1 differences relative to the in-domain model are negative. Sections 4 and 5 then compare four multi-domain learning strategies—joint training, feature-based BERT initialization, domain-tag conditioning, and a meta-learning gradient-communication objective—under in-domain, out-of-domain, and cross-dataset settings. The paper concludes that domain shift is serious in extractive summarization and that domain-aware strategies, especially tags and meta-learning, improve zero-shot generalization, while BERT mainly improves in-domain feature extraction.
Significance. If the empirical claims held, the paper would make a useful contribution: a publication-based domain definition, a reusable multi-domain testbed for summarization, and one of the first systematic comparisons of domain-aware training strategies for extractive summarization. The qualitative analysis showing that label-position distributions differ across publications and affect model behavior is also interesting. However, the quantitative support for the headline strategy-level claims is currently too weak: most reported advantages are fractions of a ROUGE point without error bars, multiple seeds, or significance tests, and the domain-shift verification does not control for the very different training-set sizes across domains. With additional controls, multiple seeds, and significance testing, the testbed and the domain-shift observation could become a solid empirical contribution; as submitted, the strongest conclusions outrun the evidence.
major comments (4)
- [Section 3, Table 2] The verification experiment does not control for the large differences in training-set size across domains (31,896 to 152,959 documents in Table 1), nor for differences in lead bias or sentence-length statistics, so the all-negative off-diagonal matrix is consistent with a data-quantity or difficulty artifact as well as with genuine domain shift. Some cells survive this concern—for example, NYT (152,959 training documents) still underperforms the WTP-trained model on WTP by 0.61 ROUGE-1—but the blanket statement that the 'significant performance drops demonstrate that the domain shift problem is quite serious' (Section 3) needs controls such as subsampling all source domains to a common training size or matching lead-bias characteristics. This is load-bearing for the paper's first central claim.
- [Section 5.2, Tables 3 and 4] All strategy comparisons are single-run ROUGE scores without error bars, multiple seeds, or significance tests. The decisive improvements are small: ModelIII Tag over ModelI Basic is 0.39 ROUGE-1 in OUT-OF-DOMAIN (33.17 vs 32.78) and 0.17 in CROSS-DATASET (40.28 vs 40.11); ModelIV Meta is 0.46 over ModelI in OUT-OF-DOMAIN; Table 4 reports a 0.13 gain for tags on CNN/DailyMail and calls it significant. These differences are within the range where run-to-run variance can change the conclusion, so the second central claim, that domain-aware training improves zero-shot generalization, is not yet established. The authors should report variance across seeds and a paired significance test (for example, bootstrap over documents) for each ROUGE comparison.
- [Section 4.1, Eq. (4), Table 3, Figure 3] The gamma weight in Eq. (4) is not reported for the ModelIV Meta results in Table 3, even though Figure 3 shows that ROUGE scores under IN-DOMAIN, OUT-OF-DOMAIN, and CROSS-DATASET vary with gamma and trade off against each other. Without knowing the chosen gamma and how sensitive the reported numbers are to it, the reader cannot determine whether the claimed meta-learning generalization benefit is a representative property of the method or a consequence of a particular hyperparameter choice. The paper should state the gamma value used and report results across a small range of gamma values in the main text.
- [Section 4.1, ModelIII Tag] The unknown-domain tag 'X' is 'randomly relabeled' into training examples, and the resulting 0.1–0.4 ROUGE gains over the no-tag baseline could therefore be due to the regularization effect of injected label noise rather than to learning a meaningful domain embedding. An ablation that replaces the domain tag with a random categorical feature, or that varies the proportion of X-labeled examples, is needed to attribute the improvement to publication identity. This is particularly important because the tagging strategy is one of the two methods the paper recommends for unseen domains.
minor comments (5)
- [Section 2.2, Table 1] The 'randomly divide ten domains into two groups' step is not reported with a seed or any stability check; the top five/bottom five presentation suggests the split may be by convenience rather than by a realized random draw. The authors should state the random seed and, ideally, show that the main conclusions do not depend on one particular split.
- [Section 3, Table 2] The caption says ROUGE-2 and ROUGE-L results are in the Appendix, but the main text never returns to those tables; a pointer and at least a one-sentence summary of whether the trend holds would help the reader assess the robustness of the verification experiment.
- [Table 3] The OUT-OF-DOMAIN block contains the typo 'NTDN' instead of 'NYDN', and the caption describes red and bold formatting that is not reproducible in monochrome print; please use symbols or explicit annotations instead.
- [Section 4.1, Eq. (3)] The notation L(S(B),Y(B),∇θL_A) is ambiguous because the loss is written as a function of gradients but the mechanism by which ∇θL_A is fed into the loss is only described in prose; a precise definition would improve reproducibility.
- [Section 5.4] The notation alternates between subscripted forms such as ModelI and hyphenated forms such as Model-I; please choose one consistent notation throughout.
Circularity Check
No significant circularity: the domain-shift verification and strategy comparisons are empirical, and self-citations appear only in background or methodological roles.
full rationale
The paper's central claim—that models trained and tested on the same news publication outperform models trained on other publications—is an empirical result, not a construction. In the verification experiment (Section 3, Table 2), the diagonal Rii is defined as the same-domain benchmark and off-diagonal cells are differences Vij = Rij − Rjj; although the table is presented as deviations from the diagonal, the all-negative pattern is an observed outcome rather than a logical consequence, since a cross-domain model trained on a much larger corpus could in principle beat the target-domain model. The strategy comparisons in Section 5.2 are similarly independent: ModelI, ModelII, ModelIII, and ModelIV are distinct learning algorithms, and their relative ROUGE scores on IN-DOMAIN, OUT-OF-DOMAIN, and CROSS-DATASET are measured, not derived from the method definitions. The only self-citations (Liu and Huang 2018 for the meta-learning communication protocol; Zhong et al. 2019 for interpretable analysis; Liu et al. 2017 for feature entanglement) are used as background or as algorithmic inspiration; Eqns. 3–4 state the meta-objective explicitly, and the evaluation against CNN/DailyMail with comparisons to prior published systems provides external anchoring. The paper does not fit a parameter to the quantity it then predicts, and no uniqueness theorem or prior result by the same authors is invoked to rule out alternatives. Statistical-significance and data-size-matching concerns are validity weaknesses, not circularity, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (2)
- gamma (loss weight coefficient in Eq. 4) =
not stated; evaluated over {0.0, 0.1, 0.3, 0.5} in Figure 3
- summary length (number of extracted sentences) =
2 for MULTI-SUM, 3 for CNN/DailyMail
assumptions (4)
- domain assumption Each news publication forms a distinct domain with distributional differences that matter for summarization
- domain assumption Greedy extractive labels derived from Newsroom reference summaries are valid supervision for extractive summarization
- ad hoc to paper The random split of ten publications into five source and five target domains is representative and stable
- ad hoc to paper The meta-learning objective in Eq. (3) correctly implements the described gradient communication protocol
Cite this review
Pith. "Pith review of Exploring Domain Shift in Extractive Text Summarization." pith.science (2026). https://pith.science/paper/UVNSFLKP
@misc{pith2026190811664,
author = {Pith},
title = {Pith review of: Exploring Domain Shift in Extractive Text Summarization},
year = {2026},
howpublished = {\url{https://pith.science/paper/UVNSFLKP}},
note = {Machine review of arXiv:1908.11664}
}
read the original abstract
Although domain shift has been well explored in many NLP applications, it still has received little attention in the domain of extractive text summarization. As a result, the model is under-utilizing the nature of the training data due to ignoring the difference in the distribution of training sets and shows poor generalization on the unseen domain. With the above limitation in mind, in this paper, we first extend the conventional definition of the domain from categories into data sources for the text summarization task. Then we re-purpose a multi-domain summarization dataset and verify how the gap between different domains influences the performance of neural summarization models. Furthermore, we investigate four learning strategies and examine their abilities to deal with the domain shift problem. Experimental results on three different settings show their different characteristics in our new testbed. Our source code including \textit{BERT-based}, \textit{meta-learning} methods for multi-domain summarization learning and the re-purposed dataset \textsc{Multi-SUM} will be available on our project: \url{http://pfliu.com/TransferSum/}.
Figures
Forward citations
Cited by 1 Pith paper
-
Enhancing Health Mention Classification Performance: A Study on Advancements in Parameter Efficient Tuning
Applying prompt tuning and POS tagger features to health mention classification yields small F1 improvements over plain fine-tuning, but the paper does not compare with actual state-of-the-art systems.
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993--1022
2003
-
[4]
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 440--447
work page 2007
-
[5]
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. 2016. Domain separation networks. In Advances in Neural Information Processing Systems, pages 343--351
work page 2016
-
[6]
Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2017. http://arxiv.org/abs/1611.09238 Improving Multi-Document Summarization via Text Classification . Proceedings of the 31th Conference on Artificial Intelligence (AAAI 2017)
work page Pith review arXiv 2017
-
[7]
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 675--686
work page 2018
-
[8]
Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 484--494
work page 2016
Show all 45 references
-
[9]
Jackie Chi Kit Cheung and Gerald Penn. 2013 a . Probabilistic domain modelling with contextualized distributional semantic vectors. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 392--401
2013
-
[10]
Jackie Chi Kit Cheung and Gerald Penn. 2013 b . Towards robust abstractive multi-document summarization: A caseframe analysis of centrality and domain. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1,...
2013
-
[11]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
2018 arXiv
-
[12]
Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. http://arxiv.org/abs/1809.09672 BanditSum: Extractive Summarization as a Contextual Bandit . In Empirical Methods in Natural Language Processing (EMNLP)
2018 arXiv
-
[13]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126--1135
2017
-
[14]
Sebastian Gehrmann, Yuntian Deng, and Alexander M. Rush. 2018. http://arxiv.org/abs/1808.10792 Bottom-Up Abstractive Summarization . In Empirical Methods in Natural Language Processing (EMNLP)
2018 arXiv
-
[15]
Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. 2011. Domain adaptation for object recognition: An unsupervised approach. In 2011 international conference on computer vision, pages 999--1006. IEEE
2011
-
[16]
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol...
2018
-
[17]
Aria Haghighi and Lucy Vanderwende. 2009. Exploring content models for multi-document summarization. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 362--370. Associ...
2009
-
[18]
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693--1701
2015
-
[19]
Xinyu Hua and Lu Wang. 2017. A pilot study of domain adaptation effect for neural abstractive summarization. arXiv preprint arXiv:1707.07062
2017 arXiv
-
[20]
Masaru Isonuma, Toru Fujino, Junichiro Mori, Yutaka Matsuo, and Ichiro Sakata. 2017. Extractive summarization using multi-task learning with document classification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2101--2110
2017
-
[21]
Mahesh Joshi, William W Cohen, Mark Dredze, and Carolyn P Ros \'e . 2012. Multi-domain learning: when do domains matter? In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1302--131...
2012
-
[22]
Chris Kedzie, Kathleen Mckeown, and Hal Daum. 2018. Content Selection in Deep Learning Models of Summarization . In Empirical Methods in Natural Language Processing (EMNLP)
2018
-
[23]
Le and Tomas Mikolov
Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of ICML
2014
-
[24]
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. 2017. Learning to generalize: Meta-learning for domain generalization. arXiv preprint arXiv:1710.03463
2017 arXiv
-
[25]
Shoushan Li and Chengqing Zong. 2008. Multi-domain sentiment classification. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers, pages 257--260. Association for Computational Linguistics
2008
-
[26]
Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics
2003
-
[27]
Pengfei Liu and Xuanjing Huang. 2018. Meta-learning multi-task communication. arXiv preprint arXiv:1810.09988
2018 arXiv
-
[28]
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1--10
2017
-
[29]
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Thirty-First AAAI Conference on Artificial Intelligence
2017
-
[30]
Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction, pages 95--100. Association for Computational Linguistics
2012
-
[31]
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018 a . Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745
2018 arXiv
-
[32]
Cohen, and Mirella Lapata
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018 b . http://arxiv.org/abs/1802.08636 Ranking Sentences for Extractive Summarization with Reinforcement Learning
2018 arXiv
-
[33]
Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304
2017 arXiv
-
[34]
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: H...
2018
-
[35]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners
2019
-
[36]
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. 2010. Adapting visual category models to new domains. In European conference on computer vision, pages 213--226. Springer
2010
-
[37]
Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752
2008
-
[38]
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1073--1083
2017
-
[39]
Antonio Torralba, Alexei A Efros, et al. 2011. Unbiased look at dataset bias. In CVPR, volume 1, page 7. Citeseer
2011
-
[40]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998--6008
2017
-
[41]
Li Wang, Junlin Yao, Yunzhe Tao, Li Zhong, Wei Liu, and Qiang Du. 2018. A reinforced topic-aware convolutional sequence-to-sequence model for abstractive text summarization. arXiv preprint arXiv:1805.03616
2018 arXiv
-
[42]
Yuxiang Wu and Baotian Hu. 2018. Learning to extract coherent summary via deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence
2018
-
[43]
Xingxing Zhang, Mirella Lapata, Furu Wei, and Ming Zhou. 2018. http://arxiv.org/abs/1808.07187 Neural Latent Extractive Document Summarization
2018 arXiv
-
[44]
Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuan-Jing Huang. 2019. Searching for effective neural extractive summarization: What works and what’s next. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 1049--1058
2019
-
[45]
Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. Neural document summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)...
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.