Pith. sign in

REVIEW 4 major objections 4 minor 27 references

DocSum: Domain-Adaptive Pre-training for Document Abstractive Summarization

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DocSum claims that continuing BART-base's pre-training on OCR-transcribed administrative text, then fine-tuning it on LLM-generated references with question-answer pairs appended, improves abstractive summarization of noisy administrative…

desk verdict The evaluation is circular—same LLM makes the training targets, test references, and filters—so the summarization gains in Table 5 are not evidence of absolute quality, but the DTC result and the honest writing give the paper enough substance to send back for major revision. read the letter →

arxiv 2412.08196 v1 pith:H3PRPCNS submitted 2024-12-11 cs.CL cs.CV

Phan Phuong Mai Chau , Souhail Bakkali , Antoine Doucet This is my paper · ORCID
classification cs.CLcs.CV
keywords abstractivesummarizationdomain-adaptivepre-trainingOCRnoiseadministrativedocumentsquestion-answerpairsLLMdataannotationBARTRVL-CDIP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a small, deployable summarizer can be built for administrative documents despite three obstacles: OCR noise, domain-specific vocabulary, and the absence of human-labeled summarization data. Its recipe is to have Mistral-7B-Instruct write gold summaries and question-answer pairs, filter out low-confidence outputs, continue pre-training BART-base on 100,000 OCR-transcribed pages, and then fine-tune on the filtered documents. The paper reports that domain pre-training alone lifts ROUGE-1 from 49.52 to 50.72 and that appending question-answer pairs to the input lifts it further to 52.21, with BERTScore reaching 90.35. If true, this shows a path to domain-specific summarization without hiring human annotators, at the inference cost of a 139M-parameter model rather than a large LLM.

What carries the argument

The load-bearing mechanism is domain-adaptive pre-training with a token-masking denoising objective: BART-base is further trained to reconstruct token-masked OCR text, letting the model absorb both administrative vocabulary and the characteristic noise of OCR transcription. The second mechanism is the input format: appending a generated question-answer pair to the document before summarization, which the paper says focuses the decoder on key content. The third is the data-creation loop: Mistral-7B-Instruct generates summaries, question-answer pairs, and confidence scores, and outputs scoring below 0.9 are discarded before fine-tuning.

What would settle it

Take a random sample of the RVL-CDIP test pages, have human annotators write summaries, and compute ROUGE and BERTScore of DocSum against those human references instead of Mistral references. If the scores collapse while Mistral-reference scores stay high, the reported gains measure imitation of Mistral rather than improved summarization; if they track human references, the claim is supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that a 139M-parameter BART-base model, after continued pre-training on OCR-transcribed administrative text and fine-tuning on LLM-generated references, produces more accurate summaries of administrative documents than the same model fine-tuned without that adaptation. The paper further claims that adding a Mistral-generated question-answer pair to the input improves summary accuracy, and that filtering generated references by a self-reported confidence score above 0.9 keeps the training data reliable. The evidence is the RVL-CDIP test set, where every variant of the proposed pipeline outperforms the baseline across ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-Lsum, and BERTScore, and where the same pre-training also improves document classification accuracy from 88.21 to 89.52.

Load-bearing premise

The entire evaluation rests on assuming that summaries written by Mistral-7B-Instruct are a valid stand-in for human-quality summaries: the same model supplies the gold references the fine-tuned BART is scored against, and the filtering step that keeps data 'reliable' uses the same model's own confidence scores.

Editorial extensions

If this is right

  • Domain-adaptive pre-training on OCR text improves both summarization and document classification from the same checkpoint.
  • Appending LLM-generated answers or question-answer pairs to the input improves summary scores over OCR text alone, with question-answer pairs giving the best BERTScore.
  • A 139M-parameter model can reach these scores, so the approach is deployable on low-resource devices without running a large LLM at inference time.
  • The confidence filter at 0.9 keeps only 29,444 of the original documents yet leaves enough data for effective fine-tuning.
  • Heavily degraded OCR remains a hard failure mode even after domain pre-training, so the method does not fully solve noise robustness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same LLM-annotation-plus-filtering loop could be applied to other OCR-heavy domains, such as historical newspapers or medical records, wherever a small inference model is preferred; the paper only demonstrates administrative documents.
  • Editorial inference: because answer-only input scores slightly higher on ROUGE while question-answer input achieves the best BERTScore, the practical trade-off the paper notes suggests a testable design: train on answers, query with questions, and measure whether summary quality tracks the training format.
  • Editorial inference: the claim of robustness to OCR noise could be probed directly by adding controlled synthetic OCR corruptions at test time and measuring whether DocSum degrades more gracefully than the baseline; the paper only observes behavior on naturally noisy pages.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces DocSum, a 139M-parameter BART-base model adapted to administrative document summarization. The pipeline has three stages: (i) domain-adaptive pre-training on 100,000 OCR-transcribed IIT-CDIP pages with masked-language modeling, optionally augmented by Mistral-7B-Instruct-generated QA pairs; (ii) fine-tuning on 29,444 RVL-CDIP documents (after confidence filtering) with Mistral-generated 'gold' summaries as targets (Eq. 2); and (iii) evaluation with ROUGE and BERTScore on a held-out portion of the same Mistral-labeled data, plus document classification accuracy. The paper claims in Section 5 that domain-adaptive pre-training significantly improves robustness to noisy OCR and that adding QA pairs further boosts summary accuracy.

Significance. If the claims were established, DocSum would be a practical recipe for distilling an LLM into a small summarizer for OCR-degraded administrative documents, and the use of public datasets (IIT-CDIP, RVL-CDIP) is a strength. The DTC experiment (Table 6) provides one independent, human-labeled validation that pre-training helps. However, the paper's main summarization results rest entirely on references generated by the same LLM used to create training targets, and no human evaluation or teacher-model comparison is provided; in its current form the evidence does not support the central claims.

major comments (4)
  1. [Section 3.3 and Eq. (2)] The summarization evaluation is circular. The gold summaries used as fine-tuning targets are generated by Mistral-7B-Instruct, and the same prompt/process produces the reference summaries for the held-out test portion (Section 4.2). ROUGE and BERTScore in Table 5 therefore measure how closely DocSum reproduces Mistral's lexical and semantic choices, not whether summaries are accurate or useful. Because the model can be trained to mimic the reference generator, the reported gains over S1 may reflect style matching rather than quality. The manuscript needs an independent reference set or human evaluation of summary faithfulness/informativeness, and it should report agreement with human judgments before claiming 'summary accuracy' in Section 5.
  2. [Section 4.2] The confidence-score filtering introduces selection bias into the test set. Only 29,444 of 40,000 documents with confidence >0.9 remain, and the test split is drawn from this filtered set. The evaluation is therefore on a subset that Mistral itself judged easy, which can inflate the absolute scores and mask failures on harder OCR inputs. Please evaluate on an unfiltered or independently constructed test set, or at least report results separately for low-confidence documents.
  3. [Section 4.4.2 and Table 6] The DTC result is the only evaluation not tied to Mistral-generated references and it supports the pre-training benefit, but it does not cover the summarization or QA-pair claims. Moreover, the 1.31% accuracy increase is reported without variance or significance testing, so it should be treated cautiously. Please provide confidence intervals or multiple seeds.
  4. [Section 4.4.1 and Section 5] A direct baseline of Mistral-7B-Instruct zero-shot summaries is missing. Since Mistral generates the gold summaries, comparing DocSum's ROUGE/BERTScore to Mistral's own outputs on the same test set is needed to show that the distillation is worthwhile and to calibrate how much score gap is due to mimicking. Without it, the claim that the QA-pair input 'boosts summary accuracy' over the teacher's ability is not established.
minor comments (4)
  1. [Section 1] The word 'summarzation' is a typo for 'summarization', and the model name is spelled 'Mistal-7B-Instruct' instead of 'Mistral-7B-Instruct'.
  2. [Sections 1, 3.2, and 4.4.2] The task acronym is introduced as DAS in Section 1, but the classification task is called DTS in the Introduction and DTC in Section 3.2 and Section 4.4.2; please standardize the terminology.
  3. [Figure 2] Figure 2 is referenced as illustrating the overall pipeline, but the text does not explain all annotation elements in the caption, such as 'context' versus 'key information'; consider clarifying.
  4. [Table 5 and Section 4.3] Table 5 reports ROUGE-Lsum, but the metric is not defined in the metrics paragraph of Section 4.3; please define it or explain its relation to ROUGE-L.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported summarization results come from a standard held-out supervised pipeline, and the use of Mistral-generated references is a label-validity limitation rather than a by-construction equivalence.

full rationale

The paper's derivation chain is a standard supervised learning pipeline: BART-base is domain-adaptively pre-trained with an MLM objective, then fine-tuned to maximize the likelihood of Mistral-generated summaries (Eq. 2), and evaluated on a 70/15/15 split of the filtered RVL-CDIP subset using ROUGE and BERTScore against held-out references generated by Mistral. The test references are produced from test documents and are not mathematically constructed from the model's training labels, fitted parameters, or input question-answer pairs, so no reported metric reduces to its own input by construction. The stronger concern, that Mistral serves as both label generator and evaluation reference, is a genuine construct-validity issue: the ROUGE/BERTScore deltas measure fidelity to Mistral's summary behavior rather than independently established human quality. The paper itself acknowledges the absence of ground-truth summaries in Section 3.3, and it relies on external evidence ([14], [26]) that LLM-generated summaries can align with human judgments; it also reports an independent human-labeled classification result (Section 4.4.2, Table 6) that supports the pre-training benefit. No load-bearing self-citation, uniqueness import, or ansatz-smuggling-via-citation is present. The lack of human evaluation of summaries should be treated as a correctness/validity risk, but under the specified circularity criteria it does not constitute circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the untested assumption that Mistral-generated summaries are valid gold standards, and on several arbitrary thresholds (0.9 confidence, 100-word cutoff, 100k pre-training samples). No new entities are postulated. The circularity of using the same LLM for training labels, filtering, and evaluation is the dominant burden.

free parameters (4)
  • confidence threshold for data filtering = 0.9
    Section 3.4.2: 'selecting only documents with confidence scores exceeding 0.9'. This threshold is chosen by hand and no sensitivity analysis is provided, yet it determines which documents enter the fine-tuning and test sets.
  • minimum document length for fine-tuning = 100 words
    Section 4.1: fine-tuning uses RVL-CDIP 'samples with more than 100 words'. This filtering is arbitrary and could bias the evaluation toward longer, easier documents.
  • pre-training subset size = 100,000 samples
    Section 4.1: 'We pre-train our model on a subset of 100,000 samples from the IIT-CDIP collection'. The choice of size is not motivated, and using a different subset size could change the measured gains.
  • LLM choice and generation settings = Mistral-7B-Instruct-v0.3, max 128 tokens
    Sections 3.1 and 3.3. The entire synthetic data pipeline depends on this specific LLM and its decoding parameters; no alternative LLMs or settings are tested, so the results may not generalize.
assumptions (4)
  • domain assumption LLM-generated summaries are of comparable quality to human-written summaries and can serve as ground truth for summarization evaluation.
    Section 3.3: gold summaries are generated by Mistral because 'there is no readily available ground truth'. This assumption is load-bearing but unverified; the paper itself cites Zhang et al. [26] that low-quality references can mislead evaluation.
  • domain assumption Question-answer pairs generated by Mistral provide a faithful and useful signal for training a summarization model.
    Section 3.2: 'By integrating these question-answer pairs, the model is exposed to a richer context that allows it to better understand...'. This is asserted without a mechanism study or counterfactual control (e.g., random Q&A pairs).
  • domain assumption ROUGE and BERTScore, computed against LLM references, reliably indicate summary quality for administrative documents.
    Section 4.3 adopts ROUGE and BERTScore as the primary metrics. This assumes they correlate with human judgment in the OCR-noise domain, but no human correlation is provided.
  • domain assumption The OCR text in IIT-CDIP and RVL-CDIP is representative of administrative documents and the observed OCR noise patterns will transfer.
    Section 4.1 uses IIT-CDIP for pre-training and RVL-CDIP for fine-tuning without verifying that the noise distribution across the two datasets is compatible; the qualitative example in Figure 5 shows the model still fails on heavily degraded input.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DocSum: Domain-Adaptive Pre-training for Document Abstractive Summarization." pith.science (2026). https://pith.science/paper/H3PRPCNS

@misc{pith2026241208196,
  author       = {Pith},
  title        = {Pith review of: DocSum: Domain-Adaptive Pre-training for Document Abstractive Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H3PRPCNS}},
  note         = {Machine review of arXiv:2412.08196}
}
read the original abstract

Abstractive summarization has made significant strides in condensing and rephrasing large volumes of text into coherent summaries. However, summarizing administrative documents presents unique challenges due to domain-specific terminology, OCR-generated errors, and the scarcity of annotated datasets for model fine-tuning. Existing models often struggle to adapt to the intricate structure and specialized content of such documents. To address these limitations, we introduce DocSum, a domain-adaptive abstractive summarization framework tailored for administrative documents. Leveraging pre-training on OCR-transcribed text and fine-tuning with an innovative integration of question-answer pairs, DocSum enhances summary accuracy and relevance. This approach tackles the complexities inherent in administrative content, ensuring outputs that align with real-world business needs. To evaluate its capabilities, we define a novel downstream task setting-Document Abstractive Summarization-which reflects the practical requirements of business and organizational settings. Comprehensive experiments demonstrate DocSum's effectiveness in producing high-quality summaries, showcasing its potential to improve decision-making and operational workflows across the public and private sectors.

Figures

Figures reproduced from arXiv: 2412.08196 by the authors.

Figure 1
Figure 1. Example output summary. The first box displays text ex￾tracted from the left image. The second box presents a question￾answer pair that highlights key information from the document. The third box provides a summary of the OCRed text and the as￾sociated question-answer pair. Words in red indicate OCR errors. key insights from lengthy administrative documents, en￾abling quicker decision-making and improving operationa… view at source ↗
Figure 2
Figure 2. The overall pipeline. During pre-training, OCRed text and LLM-generated question-answer pairs are combined as input to further train the pre-trained BART language model, adapting it to domain-specific knowledge. In the fine-tuning phase, selected documents, along with their question-answer pairs and LLM-generated gold summaries, are used to fine-tune the pre-trained DocSum model. Additionally, LLM prompts include OC… view at source ↗
Figure 3
Figure 3. Different input formats according to different prompts. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Analysis of document characteristics from the IIT-CDIP [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Comparison of generated summaries: (a) Example with lower BERTScore due to noisy input, and (b) Example with higher [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 13 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Text summarization us- ing large language models: a comparative study of mpt- 7b-instruct, falcon-7b-instruct, and openai chat-gpt models

    Lochan Basyal and Mihir Sanghvi. Text summarization us- ing large language models: a comparative study of mpt- 7b-instruct, falcon-7b-instruct, and openai chat-gpt models. arXiv preprint arXiv:2310.10449, 2023. 2

    arXiv 2023

  3. [3]

    Post-correction of his- torical text transcripts with large language models: An ex- ploratory study

    Emanuela Boros, Maud Ehrmann, Matteo Romanello, Sven Najem-Meyer, and Fr´ed´eric Kaplan. Post-correction of his- torical text transcripts with large language models: An ex- ploratory study. LaTeCH-CLfL 2024, pages 133–159, 2024. 2

    work page 2024

  4. [4]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 2

    arXiv 2005

  5. [5]

    Im- proving factual consistency of text summarization by adver- sarially decoupling comprehension and embellishment abili- ties of llms

    Huawen Feng, Yan Fan, Xiong Liu, Ting-En Lin, Zekun Yao, Yuchuan Wu, Fei Huang, Yongbin Li, and Qianli Ma. Im- proving factual consistency of text summarization by adver- sarially decoupling comprehension and embellishment abili- ties of llms. arXiv preprint arXiv:2310.19347, 2023. 2

    arXiv

  6. [6]

    Abstractive vs

    Nikolaos Giarelis, Charalampos Mastrokostas, and Nikos Karacapilidis. Abstractive vs. extractive summarization: An experimental review. Applied Sciences, 13(13):7620, 2023. 2

    work page 2023

  7. [7]

    News sum- marization and evaluation in the era of gpt-3

    Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News sum- marization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356, 2022. 2

    arXiv 2022

  8. [8]

    Evaluation of deep convolutional nets for document image classification and retrieval

    Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In 2015 13th International Con- ference on Document Analysis and Recognition (ICDAR) , pages 991–995. IEEE, 2015. 2, 5

    work page 2015

Show all 27 references
  1. [9]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. 2, 3

    arXiv

  2. [10]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2. Minneapolis, Minnesota,

  3. [11]

    Building a test col- lection for complex document information processing

    David Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, David Grossman, and Jefferson Heard. Building a test col- lection for complex document information processing. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information...

    2006

  4. [12]

    Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension

    M Lewis. Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019. 1, 2, 3

    1910 arXiv

  5. [13]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74–81, 2004. 7

    2004

  6. [14]

    On learning to summarize with large language models as references

    Yixin Liu, Kejian Shi, Katherine S He, Longtian Ye, Alexan- der R Fabbri, Pengfei Liu, Dragomir Radev, and Arman Co- han. On learning to summarize with large language models as references. arXiv preprint arXiv:2305.14239, 2023. 2, 3

    2023 arXiv

  7. [15]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 7

    2017 arXiv

  8. [16]

    Selfcheckgpt: Zero-resource black-box hallucination detec- tion for generative large language models

    Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detec- tion for generative large language models. arXiv preprint arXiv:2303.08896, 2023. 4

    2023 arXiv

  9. [17]

    Generative in- terpretation: Toward human-like evaluation for educational question-answer pair generation

    Hyeonseok Moon, Jaewook Lee, Sugyeong Eo, Chanjun Park, Jaehyung Seo, and Heui-Seok Lim. Generative in- terpretation: Toward human-like evaluation for educational question-answer pair generation. In Findings of the Asso- ciation for Computational Linguistics: EACL 2024 , pages...

    2024

  10. [18]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 1, 2

    2020

  11. [19]

    Distildoc: Knowledge distillation for visually-rich document applications

    Jordy Van Landeghem, Subhajit Maity, Ayan Banerjee, Matthew Blaschko, Marie-Francine Moens, Josep Llad ´os, and Sanket Biswas. Distildoc: Knowledge distillation for visually-rich document applications. In International Con- ference on Document Analysis and Recognition, pages 195–

  12. [20]

    Assessing the impact of ocr quality on downstream nlp tasks

    Daniel Van Strien, Kaspar Beelen, Mariona Coll Ardanuy, Kasra Hosseini, Barbara McGillivray, and Giovanni Colav- izza. Assessing the impact of ocr quality on downstream nlp tasks. 2020. 1

    2020

  13. [21]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 2

    2017

  14. [22]

    Want to reduce labeling cost? gpt-3 can help

    Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. Want to reduce labeling cost? gpt-3 can help. arXiv preprint arXiv:2108.13487, 2021. 2

    2021 arXiv

  15. [23]

    A system- atic survey of text summarization: From statistical methods to large language models

    Haopeng Zhang, Philip S Yu, and Jiawei Zhang. A system- atic survey of text summarization: From statistical methods to large language models. arXiv preprint arXiv:2406.11289,

    arXiv

  16. [24]

    Pegasus: Pre-training with extracted gap-sentences for ab- stractive summarization

    Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for ab- stractive summarization. In International conference on ma- chine learning, pages 11328–11339. PMLR, 2020. 1, 2

    2020

  17. [25]

    Bertscore: Evaluating text genera- tion with bert

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. Bertscore: Evaluating text genera- tion with bert. arXiv preprint arXiv:1904.09675, 2019. 7

    1904 arXiv

  18. [26]

    Bench- marking large language models for news summarization

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Bench- marking large language models for news summarization. Transactions of the Association for Computational Linguis- tics, 12:39–57, 2024. 2

    2024

  19. [217]

    Springer, 2024. 4, 5

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.