{"id":"34b70281-10b3-4e74-b6b2-5a494055050d","arxiv_id":"2507.05805","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A single model, DREAM, jointly predicts layout elements, coordinates, and transcriptions for document reconstruction, along with a new metric (DSM) and benchmark (DocRec1K).","lead":"DREAM is an end-to-end autoregressive model that takes a single document image and outputs a structured sequence containing element categories, bounding boxes, and transcription content (paragraphs, tables, formulas) in reading order. A generalist would read it as a candidate replacement for the conventional multi-stage pipeline of OCR, layout analysis, and table/formula recognition.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The DSM recurrence in Eq. (4) is not a valid edit-distance recurrence: it adds Cost(i,j) on insertion/deletion transitions and lacks base cases, so the Table 1 DSM scores cannot support the claimed SOTA.","rationale":"The paper's central claim is that a single end-to-end autoregressive model jointly localizes and transcribes document elements at state-of-the-art quality. The primary evidence is the DSM score on the newly introduced DocRec1K benchmark. The DSM recurrence in Eq. (4) is mathematically inconsistent with the edit-distance formulation it claims to adapt: insertion and deletion transitions are not assigned their own costs, the substitution cost is added on every transition, and base cases are unspecified. This makes the metric's numerical values, including the 91.4 versus 85.2 headline comparison, uninterpretable and potentially non-monotonic. The NED results in Table 1 are a partial alternative, but they evaluate only logical transcription after stripping physical information, so they do not substantiate the physical-plus-logical superiority that distinguishes DREAM. The architecture and subtask results are plausible and mostly competitive, and the paper should get credit for those, but they do not rescue the central claim. Given the invalid metric, the lack of released predictions or code, and the unresolved possibility of train/test overlap between the arXiv-derived training data and DocLayNet-derived DocRec1K, the central claim is not currently supported. The reader's weakest_assumption identifies the same load-bearing issue, and the appropriate verdict is REJECT unless the metric is corrected and the numbers are recomputed on released outputs.","tokens_in":20686,"tokens_out":3486,"duration_ms":40808,"concrete_test":"Obtain DocRec1K predictions for DREAM, Nougat_base*, Pix2Struct_base*, and PaddleOCR (the authors should release these), then recompute DSM with the standard recurrence: D(0,j)=j, D(i,0)=i, D(i,j)=min(D(i-1,j)+1, D(i,j-1)+1, D(i-1,j-1)+Cost(i,j)), and normalize by max sequence length as in Eq. (5). If DREAM's DSM advantage over Nougat_base* shrinks materially below the reported 6.2 points, or the ordering changes, the central SOTA claim is not established. As an immediate analytic check, compute DSM under Eq. (4) for a prediction equal to the ground truth plus one extra correct element at the end: a valid metric should penalize this as one insertion, but Eq. (4) charges Cost(i,j) across the entire alignment path.","verdict_should_be":"REJECT","load_bearing_attack":"The headline result (Table 1: DSM 91.4 vs 85.2 for Nougat_base*) rests entirely on the new DSM metric. Eq. (4) defines D(i,j) = Min(D(i-1,j), D(i,j-1), D(i-1,j-1)) + Cost(i,j) with no stated base cases. This is not standard sequence alignment: in Levenshtein edit distance, moving from D(i-1,j) or D(i,j-1) should incur insertion/deletion costs without adding the substitution cost Cost(i,j). Here every transition through a cell pays Cost(i,j), so pure insertions and deletions are penalized as though they were mismatched elements, and D can exceed the sequence length used for normalization in Eq. (5), making DSM non-monotonic and not bounded in [0,1] as claimed. Because both DREAM and the baselines are scored with this flawed recurrence, the reported 6.2-point gap is uninterpretable as a measure of reconstruction quality. The accompanying NED numbers cover only logical text and do not support the physical-plus-logical joint claim, and no code or predicted outputs are released to independently recompute the scores. If Eq. (4) were corrected, the margin and even the ranking in Table 1 could change.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DREAM, an end-to-end autoregressive model for document reconstruction that outputs, for each document element, a category, a bounding box, and a transcribed content string in a single sequence. The authors introduce a task definition, a new evaluation metric called Document Similarity Metric (DSM), and a 1,000-page benchmark DocRec1K built from DocLayNet. DREAM uses a Swin image encoder, a DETR-style feature aggregator with element queries, and a parallel decoder that generates the elements' physical and logical tokens in parallel. The paper reports a DSM of 91.4 on DocRec1K, outperforming a retrained Nougat-base variant (85.2) and a PaddleOCR pipeline (65.3), and also evaluates on document layout analysis, text recognition, table structure recognition, formula recognition, and reading order detection, showing competitive or slightly lower scores than specialized baselines.","tokens_in":20977,"tokens_out":4413,"duration_ms":50565,"significance":"If the headline result were sound, the paper would be a useful step toward a unified document reconstruction framework: it combines physical layout and logical transcription in one autoregressive sequence, proposes a metric that attempts to measure both, and demonstrates a parallel decoding scheme that reduces inference time. The ablation study is informative, and the authors are candid about limitations such as poor performance on natural scenes. However, the central evaluation rests on a newly introduced metric that is not a valid edit-distance dynamic program, and the benchmark is a new dataset with possible overlap with the training distribution. These issues make the reported state-of-the-art claim unverifiable and potentially non-meaningful as stated. The subtask evaluations on established public benchmarks provide partial independent evidence that the architecture is competitive, but they do not rescue the central document-reconstruction claim.","major_comments":[{"comment":"The DSM recurrence is not a valid edit-distance dynamic program. In standard sequence alignment, moving from D(i-1,j) or D(i,j-1) should incur an insertion/deletion cost without charging the substitution cost Cost(i,j). Eq. (4) adds Cost(i,j) to every transition, so pure insertions and deletions are penalized as if they were mismatched elements. Moreover, no base cases are specified, and the normalization in Eq. (5) cannot guarantee a value in [0,1] because the accumulated cost is bounded by the path length in the grid, which can exceed max(K, eK). Since the headline results in Table 1, including the 6.2-point margin over Nougat_base*, are computed with this recurrence, the central SOTA claim is uninterpretable until the metric is corrected or the raw scores are independently recomputed. I request corrected base cases and DP, or release of predicted outputs so reviewers can recompute the scores.","section":"3.2, Eqs. (4)-(5); Table 1"},{"comment":"DocRec1K is derived from DocLayNet, while DREAM is trained on approximately 10 million image/ground-truth pairs generated from arXiv LaTeX sources. DocLayNet is known to include pages of scientific articles, including arXiv-derived content, so there is a real risk of train/evaluation overlap. The paper does not report any deduplication between DocRec1K and the training set, nor does it provide a breakdown of DocRec1K by source. This is load-bearing because DocRec1K is the only direct benchmark for the document-reconstruction claim. I ask for an overlap analysis, a disjoint evaluation set, or per-source results to establish that the comparison is not inflated by training-set leakage.","section":"5.1 and Appendix B"},{"comment":"The end-to-end baselines Pix2Struct_base* and Nougat_base* are retrained by the authors on the same synthetic pipeline and evaluated only through in-house code, with no released checkpoints, predicted outputs, or uncertainty estimates. Since both the benchmark and the metric are introduced in this paper and all scores are point estimates on 1,000 images, the reported margin could be affected by scoring details, post-processing, or small-sample variance. I request release of the baseline predictions and the evaluation code, and the reporting of confidence intervals or significance tests, before the claim of 'unparalleled performance' can be accepted.","section":"5.4, Table 1"}],"minor_comments":[{"comment":"Cost_tran divides by Maxlen(t_i, t_j); if both strings are empty, the denominator is zero. Please specify a convention for empty transcriptions.","section":"3.2, Eq. (2)"},{"comment":"The notation F_v is used both for the patch embeddings in Sec. 4.2 and for the duplicated visual tokens in Sec. 4.4; using a different symbol, such as F_v_dup, would avoid confusion.","section":"4.4 and Figure 3"},{"comment":"The negative-query filtering threshold of 80% is stated without justification or ablation; a sensitivity analysis would clarify its effect on the DSM and NED scores.","section":"4.4"},{"comment":"The sentence 'DREAM is trained on ... approximately 10 million pairs' is immediately repeated with slightly different wording; please remove the duplication.","section":"5.1"},{"comment":"The input resolution, maximum sequence length, and number of queries are fixed, but the paper does not report how these choices interact with the maximum number of elements or the longest transcription; a brief discussion would be helpful.","section":"5.3"},{"comment":"The statement that DREAM 'struggles with natural scene' is vague; giving a quantitative example would make the stated limitation more actionable.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The DSM recurrence problem is severe and directly invalidates the headline numbers; no amount of polishing can fix the central claim without recomputation or release of raw outputs. I would not reject outright because the architecture and the subtask results suggest a plausible model, and the metric can in principle be corrected, but the authors must be required to address the metric, the dataset overlap, and the transparency of baseline evaluation before the paper can be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the honest take: the architecture in this paper is genuinely interesting, but the headline result rests on a metric that does not do what it claims. You should not cite the DSM numbers as evidence for anything.\n\nWhat's new: DREAM combines DETR-style element queries with parallel autoregressive decoding to predict layout boxes, categories, and content in one sequence. That's a reasonable integration, and the ablations suggest the parallel decoder helps with repetition. The task definition and DocRec1K are contributions in principle, and the subtask results on DocBank, PubLayNet, FUNSD, table and formula benchmarks are mostly within 1-2% of specialized models. That part is credible because it uses standard metrics.\n\nThe soft spot is the DSM metric, and it's load-bearing. Eq. (4) defines D(i,j) as Min(D(i-1,j), D(i,j-1), D(i-1,j-1)) + Cost(i,j) with no base cases. In a real edit distance, moving into a cell from the left or above should pay an insertion/deletion cost, not the substitution cost of the current cell. Here every transition pays Cost(i,j), and Cost includes category/box/transcription mismatches. That means aligning by pure insertion or deletion is punished as if it were a mismatch; D is not a proper distance, the normalization in Eq. (5) is not tied to a meaningful bound, and the claimed [0,1] range is not guaranteed. The 91.4 vs 85.2 DSM gap is uninterpretable. The NED numbers cover only logical text and do not rescue the joint physical-plus-logical claim.\n\nTwo more issues: DocRec1K is built from DocLayNet, which contains arXiv-derived pages, and the training set is arXiv LaTeX; the paper does not check for overlap. And no code or predicted outputs are released, so the scores cannot be independently recomputed. The retrained baselines on the same synthetic pipeline are fair in principle, but they sit on top of a broken metric.\n\nThe math, the data, and the citation pattern are otherwise fine for the subtask experiments. Overall, the architecture may work, but the paper's evidence for its central claim does not. I would take this seriously if the DSM were corrected and the dataset overlap checked; as submitted, the headline is not supported.\n\nRecommendation: engage with the paper for the architecture and subtask results, but treat the reconstruction benchmark as meaningless until the metric is fixed. It deserves a serious referee because the core idea is worth the time, but expect a heavy revision.","headline":"Interesting architecture and credible subtask results, but the DSM metric in Eq. (4) is not a valid edit distance, so the headline reconstruction scores are uninterpretable.","tokens_in":21509,"tokens_out":2727,"would_cite":false,"duration_ms":27437,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single end-to-end autoregressive model can turn a document image into one token sequence carrying each element's category, bounding box, and transcription, and reports top scores on a new DocRec1K benchmark.","keywords":["document reconstruction","end-to-end autoregressive model","document layout analysis","text recognition","table structure recognition","formula recognition","reading order detection","parallel decoding"],"falsifier":"Recompute DSM using a standard edit-distance recurrence where insertion and deletion transitions add a fixed cost without adding the substitution cost $\\text{Cost}(i,j)$; if Table 1's rankings or margins change materially, the claimed superiority is an artifact of Eq. (4) rather than of reconstruction quality.","tokens_in":20492,"feed_emoji":"📄","tokens_out":7802,"duration_ms":79972,"temperature":0.7,"pith_summary":"DREAM reframes document reconstruction as one sequence-to-sequence problem: a page image goes in, and a single token stream comes out in which each element carries its type, its bounding-box coordinates, and its content (paragraph text lines with coordinates, tables as HTML with cell boxes, formulas in LaTeX). The authors argue this end-to-end design avoids the error propagation that hurts multi-stage pipelines and preserves layout information that earlier end-to-end generators drop. To measure the task they introduce a Document Similarity Metric and the DocRec1K dataset, and report that DREAM scores 91.4 DSM versus 85.2 for a retrained Nougat base and 65.3 for a PaddleOCR pipeline. They also report competitive results on layout analysis, text recognition, table structure recognition, formula recognition, and reading order detection.","feed_headline":"DREAM beats multi-stage pipelines at document reconstruction","feed_subtitle":"On a new 1,000-page benchmark it scores 91.4 versus 85.2 for a retrained Nougat baseline.","key_machinery":"The load-bearing design is the element-parallel decoding scheme: N element queries, matched to ground-truth elements by DETR-style bipartite matching, each generate a complete element record (category, discretized coordinates, then text tokens) in parallel rather than as one long serial markup string; a confidence threshold filters out negative queries. This shortens the effective autoregressive sequence and, the authors argue, prevents repetitive degeneration while cutting inference time. The other central object is the reconstruction format itself, the per-element tuple with <Sep> separators, which lets one sequence carry both physical layout and logical content.","core_discovery":"The central claim is that document reconstruction should be defined as transforming an image into a sequence Y={y_k} where each element y_k=(c,b,t,<Sep>) includes the element category, a four-coordinate bounding box, and a transcription in a format suited to the element type. DREAM implements this with an image encoder, a feature aggregator that lets N learnable element queries cross-attend to image features, and a parallel decoder that autoregressively generates the category and coordinates for all retained elements and then each element's text. The model is trained with a document reconstruction loss combining element discrimination via bipartite matching, transcription cross-entropy, and a sequence-level cosine loss intended to enforce reading order. The paper introduces DSM to score reconstruction quality and DocRec1K, a 1,000-page benchmark derived from DocLayNet, and claims DREAM outperforms the compared multi-stage and end-to-end baselines on that benchmark while remaining competitive across five document-understanding subtasks.","pith_inferences":["An editor's check: Eq. (4) adds the element substitution cost to every dynamic-programming transition, including pure insertion and deletion moves; a standard edit-distance recurrence would not do that, so the DSM numbers in Table 1, and the claimed 6.2-point margin over Nougat*, should be recomputed with a corrected recurrence before being taken at face value.","Because DREAM is trained on LaTeX-rendered pages, its strong results likely transfer well to clean digital documents but less to photographed or handwritten pages; the paper itself notes difficulty with natural scenes, so a testable extension is to evaluate on a mixed-domain reconstruction benchmark.","The element-query design suggests each output element is grounded in a specific image region; a natural next probe is whether the predicted bounding boxes are accurate enough for layout-aware editing, retrieval, or question answering, not just for reconstruction scoring."],"forward_implications":["A single DREAM-style model can replace a pipeline of separate detection, OCR, table, and formula models, removing stage-to-stage error propagation.","Document images become structured inputs for large language models: the output sequence contains both where each element is and what it says, so downstream systems get layout-aware content without extra parsing.","The new DSM and DocRec1K give the field a common yardstick for reconstruction quality, making future end-to-end models directly comparable.","Parallel decoding makes end-to-end reconstruction practical at 3.2 seconds per image with 270M parameters, versus 14.1 seconds for the adapted Nougat base."],"supporting_citations":[{"why":"Supplies the end-to-end baseline (retrained Nougat base) and the LaTeX/HTML pipeline used to build training ground truth.","marker":"[2]"},{"why":"Provides the query-based object detection formulation (bipartite matching, set prediction) that the feature aggregator and element discrimination loss adapt to document elements.","marker":"[3]"},{"why":"Provides the multi-stage PaddleOCR baseline and the SLANet/CAN tools used to annotate table cells and formulas in DocRec1K.","marker":"[10]"},{"why":"Supplies Pix2Struct, the other end-to-end baseline retrained for document reconstruction.","marker":"[25]"},{"why":"Provides the KOSMOS-2.5 end-to-end counterpart, the NED/NTED evaluation style, and the text-recognition comparison protocol.","marker":"[39]"},{"why":"DocLayNet is the source dataset from which DocRec1K is built, so DocRec1K inherits its layout annotations.","marker":"[45]"},{"why":"Supplies the reading order detection benchmark (ReadingBank) and the LayoutReader baseline that DREAM is compared against.","marker":"[51]"},{"why":"Provides the IM2LATEX-100K benchmark and IM2TEX baseline for formula recognition.","marker":"[8]"}],"fun_headline_variants":["End-to-end DREAM model sets new bar for document reconstruction","DREAM unifies layout and text for document reconstruction","DREAM model wins on document reconstruction benchmark","New autoregressive DREAM redefines document reconstruction","DREAM: one model for layout, text, and reading order"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that DSM, the paper's new similarity metric, correctly measures reconstruction quality; all headline comparisons use its numbers, so if the metric is miscalibrated the reported margins are not meaningful.","fun_headline_variants_meta":{"raw":{"variants":["End-to-end DREAM model sets new bar for document reconstruction","DREAM unifies layout and text for document reconstruction","DREAM model wins on document reconstruction benchmark","New autoregressive DREAM redefines document reconstruction","DREAM: one model for layout, text, and reading order"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1380,"prompt_tokens":1018,"completion_tokens":362,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":283}},"tokens_in":634,"tokens_out":362,"duration_ms":4124,"temperature":1.0,"reasoning_tokens":283,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:17:32.952163+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute DSM using a standard edit-distance recurrence where insertion and deletion transitions add a fixed cost without adding the substitution cost $\\text{Cost}(i,j)$; if Table 1's rankings or margins change materially, the claimed superiority is an artifact of Eq. (4) rather than of reconstruction quality.","supporting_citations":[{"cited_title":"End-to- end object detection with transformers","cited_arxiv_id":null,"evidence_quote":"Provides the query-based object detection formulation (bipartite matching, set prediction) that the feature aggregator and element discrimination loss adapt to document elements."},{"cited_title":"Pix2struct: Screenshot parsing as pretraining for visual lan- guage understanding","cited_arxiv_id":null,"evidence_quote":"Supplies Pix2Struct, the other end-to-end baseline retrained for document reconstruction."},{"cited_title":"Doclaynet: A large human- annotated dataset for document-layout segmentation","cited_arxiv_id":null,"evidence_quote":"DocLayNet is the source dataset from which DocRec1K is built, so DocRec1K inherits its layout annotations."},{"cited_title":"Image-to-markup generation with coarse-to- fine attention","cited_arxiv_id":null,"evidence_quote":"Provides the IM2LATEX-100K benchmark and IM2TEX baseline for formula recognition."}],"review_version":1}