{"id":"edf0757f-42b6-4471-95d0-d70725bfeee7","arxiv_id":"2412.08196","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Pre-training BART on OCR text and augmenting input with LLM-generated question-answer pairs yields small metric gains on administrative document summarization, measured against LLM-written references.","lead":"DocSum is a two-stage recipe that pre-trains BART on OCR'd administrative documents and fine-tunes it using question-answer pairs and summaries generated by Mistral-7B. The paper reports small ROUGE/BERTScore gains over a BART baseline, but the evaluation uses LLM-generated references as ground truth.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All training and test references are generated by Mistral-7B (Section 3.3), so the reported ROUGE/BERTScore gains measure fidelity to that model, not independent summary quality; without human evaluation the central OCR-robustness and QA-boost claims remain unsupported.","rationale":"The reader identified the weakest load-bearing premise exactly: Mistral-7B-Instruct-generated summaries are used as training labels, test references, and confidence-score filters (Section 3.3), so the automatic metrics in Table 5 largely measure how well BART mimics Mistral. The paper's own discussion admits that the model still fails on heavily degraded OCR, yet the conclusion claims pre-training 'significantly enhances' noisy-OCR handling. The effect sizes are small (ROUGE-1 +1.2, BERTScore +0.52) and no significance tests are reported. Adding QA pairs as input is also confounded: the QA pairs are generated by the same LLM that writes the reference summaries, so lexical overlap between the answer and the reference can mechanically inflate ROUGE. The DTC classification result (Table 6) is an independent, human-labeled signal that pre-training helps, but it does not validate the summarization claim. A human evaluation or an external benchmark with human-written references would settle whether the reported gains reflect real quality; until then, the central claim is unsupported. This does not change the reader's REJECT verdict.","tokens_in":12035,"tokens_out":4016,"duration_ms":45094,"concrete_test":"Take a random sample of 100 RVL-CDIP test documents. Have three annotators independently rate (a) DocSum summaries, (b) S1 baseline summaries, (c) the Mistral-7B-generated reference summaries, and (d) Mistral zero-shot summaries, for factual consistency with the source document and overall quality. Then compute inter-annotator agreement, whether human preferences rank S2 above S1 and above Mistral, and the correlation between human scores and the reported ROUGE/BERTScore deltas. If human judges do not prefer S2 over S1, or if Mistral references are rated as hallucinated or not clearly better than S1, the claimed gains are an artifact of training and evaluating against the same LLM.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 5) that domain-adaptive pre-training and QA pairs improve summarization on noisy OCR rests on automatic metrics computed against Mistral-7B-Instruct-generated references. Section 3.3 states that no ground-truth summaries are available, so Mistral generates the training targets (Eq. 2), the test references, and the confidence scores used to filter data. Therefore, the deltas in Table 5 show how closely BART reproduces Mistral's style and content choices, not whether summaries are accurate or useful. If Mistral hallucinates on OCR noise, the model is trained and evaluated to reproduce those hallucinations. The citation to [14] shows that LLM-as-reference training can align with human judgment in some settings, but this paper provides no human validation. The qualitative example in Fig. 5a even shows a clean reference summary for a noisy input while the model output is corrupted, which indicates reference quality is not the only issue; still, no human ratings are supplied. The DTC result in Table 6, which uses human labels, independently supports the pre-training benefit, but it does not cover the summarization or QA-pair claims. A direct comparison to Mistral-7B-Instruct's own zero-shot summaries is also missing. Without an independent reference set or human evaluation, the strongest claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DocSum, a 139M-parameter BART-base model adapted to administrative document summarization. The pipeline has three stages: (i) domain-adaptive pre-training on 100,000 OCR-transcribed IIT-CDIP pages with masked-language modeling, optionally augmented by Mistral-7B-Instruct-generated QA pairs; (ii) fine-tuning on 29,444 RVL-CDIP documents (after confidence filtering) with Mistral-generated 'gold' summaries as targets (Eq. 2); and (iii) evaluation with ROUGE and BERTScore on a held-out portion of the same Mistral-labeled data, plus document classification accuracy. The paper claims in Section 5 that domain-adaptive pre-training significantly improves robustness to noisy OCR and that adding QA pairs further boosts summary accuracy.","tokens_in":12390,"tokens_out":4140,"duration_ms":39760,"significance":"If the claims were established, DocSum would be a practical recipe for distilling an LLM into a small summarizer for OCR-degraded administrative documents, and the use of public datasets (IIT-CDIP, RVL-CDIP) is a strength. The DTC experiment (Table 6) provides one independent, human-labeled validation that pre-training helps. However, the paper's main summarization results rest entirely on references generated by the same LLM used to create training targets, and no human evaluation or teacher-model comparison is provided; in its current form the evidence does not support the central claims.","major_comments":[{"comment":"The summarization evaluation is circular. The gold summaries used as fine-tuning targets are generated by Mistral-7B-Instruct, and the same prompt/process produces the reference summaries for the held-out test portion (Section 4.2). ROUGE and BERTScore in Table 5 therefore measure how closely DocSum reproduces Mistral's lexical and semantic choices, not whether summaries are accurate or useful. Because the model can be trained to mimic the reference generator, the reported gains over S1 may reflect style matching rather than quality. The manuscript needs an independent reference set or human evaluation of summary faithfulness/informativeness, and it should report agreement with human judgments before claiming 'summary accuracy' in Section 5.","section":"Section 3.3 and Eq. (2)"},{"comment":"The confidence-score filtering introduces selection bias into the test set. Only 29,444 of 40,000 documents with confidence >0.9 remain, and the test split is drawn from this filtered set. The evaluation is therefore on a subset that Mistral itself judged easy, which can inflate the absolute scores and mask failures on harder OCR inputs. Please evaluate on an unfiltered or independently constructed test set, or at least report results separately for low-confidence documents.","section":"Section 4.2"},{"comment":"The DTC result is the only evaluation not tied to Mistral-generated references and it supports the pre-training benefit, but it does not cover the summarization or QA-pair claims. Moreover, the 1.31% accuracy increase is reported without variance or significance testing, so it should be treated cautiously. Please provide confidence intervals or multiple seeds.","section":"Section 4.4.2 and Table 6"},{"comment":"A direct baseline of Mistral-7B-Instruct zero-shot summaries is missing. Since Mistral generates the gold summaries, comparing DocSum's ROUGE/BERTScore to Mistral's own outputs on the same test set is needed to show that the distillation is worthwhile and to calibrate how much score gap is due to mimicking. Without it, the claim that the QA-pair input 'boosts summary accuracy' over the teacher's ability is not established.","section":"Section 4.4.1 and Section 5"}],"minor_comments":[{"comment":"The word 'summarzation' is a typo for 'summarization', and the model name is spelled 'Mistal-7B-Instruct' instead of 'Mistral-7B-Instruct'.","section":"Section 1"},{"comment":"The task acronym is introduced as DAS in Section 1, but the classification task is called DTS in the Introduction and DTC in Section 3.2 and Section 4.4.2; please standardize the terminology.","section":"Sections 1, 3.2, and 4.4.2"},{"comment":"Figure 2 is referenced as illustrating the overall pipeline, but the text does not explain all annotation elements in the caption, such as 'context' versus 'key information'; consider clarifying.","section":"Figure 2"},{"comment":"Table 5 reports ROUGE-Lsum, but the metric is not defined in the metrics paragraph of Section 4.3; please define it or explain its relation to ROUGE-L.","section":"Table 5 and Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's central summarization evaluation is circular, but unlike an unfixable flaw, it can be addressed with a human evaluation and additional baselines. I therefore recommend major revision rather than rejection. If the authors cannot provide independent references or human ratings, the summarization claims should be withdrawn or substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: the evaluation in this paper is circular, and that's a serious problem, but the paper is not a waste of time. It's a plausible extension of known ideas to a niche but real task, and it includes one result that does not depend on the circular setup.\n\nWhat's actually new: applying domain-adaptive pre-training and QA-pair-augmented input to abstractive summarization of OCR'd administrative documents using a 139M BART-base model. Using RVL-CDIP for summarization is new, as it was previously a classification benchmark. The method combines known pieces: domain-adaptive pre-training, LLM-generated references, and adding Q&A to the input.\n\nWhat it does well: the writing is clear and the authors are honest about the model's failures on heavily degraded OCR. The DTC result (Table 6) uses human labels and does independently support the claim that pre-training on OCR'd text helps, albeit for classification, not summarization.\n\nThe soft spot: Section 3.3 states that all training targets, test references, and filtering confidence scores are generated by Mistral-7B-Instruct. So the ROUGE/BERTScore gains in Table 5 measure how closely BART reproduces Mistral's style and content choices, not whether the summaries are actually accurate. That's a load-bearing flaw. There is no human evaluation, no comparison to Mistral's own zero-shot summaries, and no significance testing on the small deltas (0.5–1.2 points). Also, the claim to define a novel task is overstated; it's a new application of an existing task.\n\nIs the circularity fatal? For the central claim, yes, as stated. But it's fixable: independent human-written references or human evaluation, plus a direct baseline against the teacher LLM, would make the comparison meaningful. The authors' qualitative example in Fig. 5a actually shows the model failing on noisy input, which undercuts the claim of robustness.\n\nWho should read this: anyone working on document summarization or OCR robustness. It's more of a work-in-progress than a finished contribution. I'd send it to peer review because the task is relevant and the flaws are addressable, but I'd expect major revision before acceptance. A serious referee could help the authors turn this into a solid workshop or short-paper contribution.\n\nRecommendation: engage with it as a reviewer, but reject in current form.","headline":"The evaluation is circular—same LLM makes the training targets, test references, and filters—so the summarization gains in Table 5 are not evidence of absolute quality, but the DTC result and the honest writing give the paper enough substance to send back for major revision.","tokens_in":12855,"tokens_out":3405,"would_cite":false,"duration_ms":34419,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DocSum claims that continuing BART-base's pre-training on OCR-transcribed administrative text, then fine-tuning it on LLM-generated references with question-answer pairs appended, improves abstractive summarization of noisy administrative…","keywords":["abstractive summarization","domain-adaptive pre-training","OCR noise","administrative documents","question-answer pairs","LLM data annotation","BART","RVL-CDIP"],"falsifier":"Take a random sample of the RVL-CDIP test pages, have human annotators write summaries, and compute ROUGE and BERTScore of DocSum against those human references instead of Mistral references. If the scores collapse while Mistral-reference scores stay high, the reported gains measure imitation of Mistral rather than improved summarization; if they track human references, the claim is supported.","tokens_in":11776,"feed_emoji":"📄","tokens_out":6036,"duration_ms":57219,"temperature":0.7,"pith_summary":"The paper tries to establish that a small, deployable summarizer can be built for administrative documents despite three obstacles: OCR noise, domain-specific vocabulary, and the absence of human-labeled summarization data. Its recipe is to have Mistral-7B-Instruct write gold summaries and question-answer pairs, filter out low-confidence outputs, continue pre-training BART-base on 100,000 OCR-transcribed pages, and then fine-tune on the filtered documents. The paper reports that domain pre-training alone lifts ROUGE-1 from 49.52 to 50.72 and that appending question-answer pairs to the input lifts it further to 52.21, with BERTScore reaching 90.35. If true, this shows a path to domain-specific summarization without hiring human annotators, at the inference cost of a 139M-parameter model rather than a large LLM.","feed_headline":"Domain pre-training plus QA pairs lift summary scores on OCR documents","feed_subtitle":"DocSum adapts BART to administrative text using only Mistral-generated references, reaching BERTScore of 90.35.","key_machinery":"The load-bearing mechanism is domain-adaptive pre-training with a token-masking denoising objective: BART-base is further trained to reconstruct token-masked OCR text, letting the model absorb both administrative vocabulary and the characteristic noise of OCR transcription. The second mechanism is the input format: appending a generated question-answer pair to the document before summarization, which the paper says focuses the decoder on key content. The third is the data-creation loop: Mistral-7B-Instruct generates summaries, question-answer pairs, and confidence scores, and outputs scoring below 0.9 are discarded before fine-tuning.","core_discovery":"On its own terms, the paper's central claim is that a 139M-parameter BART-base model, after continued pre-training on OCR-transcribed administrative text and fine-tuning on LLM-generated references, produces more accurate summaries of administrative documents than the same model fine-tuned without that adaptation. The paper further claims that adding a Mistral-generated question-answer pair to the input improves summary accuracy, and that filtering generated references by a self-reported confidence score above 0.9 keeps the training data reliable. The evidence is the RVL-CDIP test set, where every variant of the proposed pipeline outperforms the baseline across ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-Lsum, and BERTScore, and where the same pre-training also improves document classification accuracy from 88.21 to 89.52.","pith_inferences":["Editorial inference: the same LLM-annotation-plus-filtering loop could be applied to other OCR-heavy domains, such as historical newspapers or medical records, wherever a small inference model is preferred; the paper only demonstrates administrative documents.","Editorial inference: because answer-only input scores slightly higher on ROUGE while question-answer input achieves the best BERTScore, the practical trade-off the paper notes suggests a testable design: train on answers, query with questions, and measure whether summary quality tracks the training format.","Editorial inference: the claim of robustness to OCR noise could be probed directly by adding controlled synthetic OCR corruptions at test time and measuring whether DocSum degrades more gracefully than the baseline; the paper only observes behavior on naturally noisy pages."],"forward_implications":["Domain-adaptive pre-training on OCR text improves both summarization and document classification from the same checkpoint.","Appending LLM-generated answers or question-answer pairs to the input improves summary scores over OCR text alone, with question-answer pairs giving the best BERTScore.","A 139M-parameter model can reach these scores, so the approach is deployable on low-resource devices without running a large LLM at inference time.","The confidence filter at 0.9 keeps only 29,444 of the original documents yet leaves enough data for effective fine-tuning.","Heavily degraded OCR remains a hard failure mode even after domain pre-training, so the method does not fully solve noise robustness."],"supporting_citations":[{"why":"Supplies the BART-base architecture that DocSum continues pre-training and fine-tuning.","marker":"[12]"},{"why":"Generates the gold summaries, question-answer pairs, and confidence scores that form the training data.","marker":"[9]"},{"why":"Provides the 100,000-page OCR-transcribed corpus used for domain-adaptive pre-training.","marker":"[11]"},{"why":"Provides the RVL-CDIP pages and document labels used for fine-tuning and evaluation.","marker":"[8]"},{"why":"Establishes the precedent that small summarization models trained on LLM-generated references can perform well.","marker":"[14]"},{"why":"Defines the ROUGE metrics used to score summary quality.","marker":"[13]"},{"why":"Defines BERTScore, the semantic similarity metric used alongside ROUGE.","marker":"[25]"},{"why":"Motivates the confidence-score approach used to detect and filter hallucinated LLM outputs.","marker":"[16]"},{"why":"Provides the question-answer evaluation and confidence-scoring method adapted for data filtering.","marker":"[19]"}],"fun_headline_variants":["Domain-adaptive pretraining lifts admin document summarization","QA pairs boost OCR summary accuracy in DocSum framework","DocSum adapts BART to administrative text for better summaries","Pre-training on OCR text improves abstractive summarization scores","Domain pre-training plus QA pairs hit BERTScore 90.35 on OCR docs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire evaluation rests on assuming that summaries written by Mistral-7B-Instruct are a valid stand-in for human-quality summaries: the same model supplies the gold references the fine-tuned BART is scored against, and the filtering step that keeps data 'reliable' uses the same model's own confidence scores.","fun_headline_variants_meta":{"raw":{"variants":["Domain-adaptive pretraining lifts admin document summarization","QA pairs boost OCR summary accuracy in DocSum framework","DocSum adapts BART to administrative text for better summaries","Pre-training on OCR text improves abstractive summarization scores","Domain pre-training plus QA pairs hit BERTScore 90.35 on OCR docs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000309,"raw_usage":{"total_tokens":1741,"prompt_tokens":897,"completion_tokens":844,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":758}},"tokens_in":513,"tokens_out":844,"duration_ms":7859,"temperature":1.0,"reasoning_tokens":758,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:05:10.940363+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the RVL-CDIP test pages, have human annotators write summaries, and compute ROUGE and BERTScore of DocSum against those human references instead of Mistral references. If the scores collapse while Mistral-reference scores stay high, the reported gains measure imitation of Mistral rather than improved summarization; if they track human references, the claim is supported.","supporting_citations":[{"cited_title":"Building a test col- lection for complex document information processing","cited_arxiv_id":null,"evidence_quote":"Provides the 100,000-page OCR-transcribed corpus used for domain-adaptive pre-training."},{"cited_title":"Evaluation of deep convolutional nets for document image classification and retrieval","cited_arxiv_id":null,"evidence_quote":"Provides the RVL-CDIP pages and document labels used for fine-tuning and evaluation."},{"cited_title":"Rouge: A package for automatic evaluation of summaries","cited_arxiv_id":null,"evidence_quote":"Defines the ROUGE metrics used to score summary quality."},{"cited_title":"Distildoc: Knowledge distillation for visually-rich document applications","cited_arxiv_id":null,"evidence_quote":"Provides the question-answer evaluation and confidence-scoring method adapted for data filtering."}],"review_version":1}