{"id":"c9d36c35-50a8-4f40-91e5-cc35ed29ef28","arxiv_id":"2506.16407","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Line-level combined bounding-box, pixel, and text perturbations from a unified budgeted attack framework most severely degrade OCR-based visual document understanding models.","lead":"This paper builds and evaluates a unified framework of multi-modal adversarial attacks that shift OCR bounding boxes, alter pixels, and modify text in document images. Its main finding is that line-level, combined perturbations cause the largest accuracy drops in several document-understanding models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The attack is optimized in continuous token-embedding space, but the paper never constructs a document whose OCR output induces that embedding; the claimed budgeted, realistic multi-modal attack may not be physically realizable.","rationale":"The reader's weakest assumption is precisely the soft spot. Section 3.3 optimizes in continuous embedding space, and Section 3.6 feeds perturbed embeddings to the frozen model; no OCR document whose output induces e~ is ever constructed or verified. Without that, the 'plausible' budget (IoU, edit rate) is enforced on g_theta-predicted boxes rather than on real OCR output, so Tables 3-7 may measure internal-embedding sensitivity rather than robustness of OCR-based VDU systems. This is the most load-bearing concern because it attacks the central claim that the proposed attack is a realistic, budgeted, multi-modal attack on OCR-based pipelines. Supporting issues reinforce, but do not replace, this concern: Table 3 has missing PGD entries and no DocVQA PGD column; Table 2 shows no usable predictor for GeoLayoutLM; and Section 4.3.1 claims PGD trends on models for which no direct PGD results are reported. The proposed rendering-plus-OCR check is decisive: if it lands, the paper must either realize the attack or reframe its claims as feature-space sensitivity; if it passes, the central claim is materially supported. The reader's CONDITIONAL verdict remains appropriate: the manuscript has a useful framework and some plausible results, but the core realizability requirement is unproven and must be demonstrated before the findings can be accepted.","tokens_in":9656,"tokens_out":9636,"duration_ms":102785,"concrete_test":"On FUNSD, take the PGD output e~ from Section 3.3 for scenario S6 at IoU>=0.6; compute \\tilde{B}=g_theta(e~), apply the text and pixel edits exactly as specified in Section 3.4, render the modified document images from the source scans, run an off-the-shelf OCR (e.g., Tesseract or Google Vision), feed the resulting OCR text, boxes, and image to LayoutLMv3, and recompute F1 together with the real IoU of the rendered boxes against ground truth. Compare with the 29.18% F1 drop claimed in Table 3. If the reproduced drop is substantially smaller or the real IoU falls below 0.6, the budget/realism claim fails; if the drop and IoU match, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing step is in Section 3.3: PGD perturbs continuous token embeddings e to e~, with \\tilde{B}=g_theta(e~) as the predicted bounding boxes, and the IoU budget is enforced only on these predictions. To evaluate L_task, the perturbed embeddings are fed into the frozen VDU model (Section 3.6). Nowhere does the paper map e~ back to a discrete OCR string, concrete bbox coordinates, and a rendered document image whose OCR would reproduce e~. The text and pixel modules in Section 3.4 are applied afterwards, but the layout channel is never realized as an actual document. Consequently, the F1 drops reported in Tables 3-7 (e.g., 29.18% for S6 on FUNSD) measure sensitivity to internal embeddings, not robustness of OCR-based VDU systems to a budgeted, plausible input perturbation. Section 3.1 promises spatially consistent and text-edit-bounded perturbations, but the only quantities checked against the budget are g_theta-predicted boxes. Since Table 2 shows g_theta is imperfect (e.g., LayoutLMv3 line mIoU is 89.34 on FUNSD), even the predicted-IoU constraint may not hold for actually realized boxes. Unless the embedding-space PGD output is demonstrated to correspond to a realizable OCR or layout edit, the central claim is not verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a unified multi-modal adversarial attack framework for OCR-based Visual Document Understanding (VDU) models, combining bounding-box, pixel, and text perturbations under explicit budgets. The method trains a differentiable bounding-box predictor to enable PGD attacks on token embeddings, defines six attack scenarios at word and line granularity, and reports F1/ANLS drops across FUNSD, CORD, SROIE, and DocVQA on LayoutLMv2, LayoutLMv3, ERNIE, GeoLayoutLM, LayTextLLM, LLaMA, and GPT-4.1 mini. The central claims are that line-level and compound (BBox+Pixel+Text) attacks cause the largest degradation, that PGD outperforms random shift, and that Unicode text attacks are more damaging than random edits.","tokens_in":9928,"tokens_out":6102,"duration_ms":60494,"significance":"If the attack framework were fully realized and the results complete, this would be a valuable robustness benchmark for OCR-based VDU systems, which are widely deployed in document-processing pipelines. The paper's strengths include the breadth of scenarios (six), the coverage of multiple architectures and datasets, the inclusion of a learned differentiable bbox predictor for gradient-based layout attacks, and a transparent limitations section that acknowledges white-box scope and the exclusion of OCR-free models. The ablation on IoU budgets and the comparison of Unicode versus random text edits are informative. However, the central attack is never demonstrated to correspond to a realizable document perturbation, and the key results table is incomplete, both of which currently block acceptance.","major_comments":[{"comment":"The attack is optimized in continuous token-embedding space, and the layout budget is enforced only on boxes predicted by g_theta(e~). No step in the pipeline constructs a real document image whose OCR output would reproduce e~, nor a discrete OCR string and concrete bounding-box edit that realizes the perturbation. The pixel and text modules in §3.4 are applied after the fact, but the layout channel is never realized as an actual document. Consequently, the reported F1/ANLS drops measure sensitivity of the frozen model to internal embedding perturbations, not robustness of OCR-based VDU systems to the claimed budgeted, realistic input attacks. This is the load-bearing missing support for the abstract's central claim of 'up to 29.18% F1 drop' under a plausible perturbation. Please either render the perturbed boxes onto the document and re-run OCR, or otherwise demonstrate that the embedding-space perturbation can be realized as a document whose OCR output induces e~.","section":"§3.3, §3.6, Tables 3–7"},{"comment":"The key results table contains the note 'PGD values will be filled in separately if available,' and the DocVQA columns lack PGD results entirely. Yet §4.3.1 claims 'PGD-based attacks consistently yield greater performance degradation than Random Shift' across all investigated datasets, and §4.3.3 uses these numbers for the KIE-vs-VQA task comparison. Without complete PGD entries, the headline comparison is not supported as presented. Please complete the table or explicitly report which PGD values could not be obtained and why.","section":"Table 3"},{"comment":"The BBox predictor g_theta is trained on the same four datasets used for evaluation, but the paper never states the train/test split between documents used to fit g_theta and documents used to generate and evaluate attacks. If the predictor is trained on the test split, the mIoU values in Table 2 and the resulting attack effectiveness are inflated, and the reported transferability results are not trustworthy. Please specify the split and, if necessary, retrain the predictor on a separate held-out set before generating attacks.","section":"§4.2"}],"minor_comments":[{"comment":"The caption says the table shows 'the number of document images and bounding boxes,' but Table 1 lists only bounding-box counts; no document-image counts are given.","section":"Table 1 caption"},{"comment":"The objective L_adv = L_task - λ_box (1 - IoU(hat B, tilde B)) uses the predicted clean boxes hat B rather than the original ground-truth boxes B. As written, the budget is relative to the predictor's output, not to the actual clean boxes, which is inconsistent with the threat-model definition in §3.1.","section":"§3.3, Eq. (1)"},{"comment":"The text says the PGD update 'projects the resulting boxes back into the feasible set {IoU≥τ}', but no projection operator is specified for embedding space. It is unclear how one projects a token embedding so that the predicted boxes satisfy the IoU constraint.","section":"§3.3"},{"comment":"GeoLayoutLM rows in Table 2 are all dashes, yet Table 5 reports GeoLayoutLM PGD transfer results. Please clarify whether these are transfer attacks generated with the LayoutLMv3 predictor and whether the asterisk '*Word level' applies to all GeoLayoutLM entries.","section":"Table 2 and Table 5"},{"comment":"All results are averaged over 5 random seeds, but no standard deviations or confidence intervals are reported. Given that some reported differences are small (e.g., Table 7, PGD at IoU 0.75 vs 0.9: 6.60 vs 6.50), the claimed 'consistently' and 'more effective' trends need variance estimates to be statistically assessed.","section":"§4.3"},{"comment":"The Unicode diacritic attack is attributed to (Boucher et al., 2023), but the reference entry lacks a year and venue, and the genetic-optimization details are not described in §3.4.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is not ready for publication in its current form: the central attack is not realized as a document perturbation, and the main results table is incomplete. The realizability gap is the principal blocker. If the authors can demonstrate that the embedding-space PGD perturbation corresponds to a realizable OCR or layout edit and complete the PGD results, the work could become a useful contribution. I would also ask the editor to check the completeness of the table and the bbox-predictor training split during any revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a useful paper to know about, but the central attack claim is not actually verified. The authors introduce a unified six-scenario benchmark for attacking OCR-based VDU models across layout, pixel, and text, with budgets on IoU, edit rate, and image transforms. That framing is genuinely new relative to Do-GOOD, RoDLA, and text-in-image attacks. The line-level vs word-level comparison and the transfer experiments are a sensible way to organize robustness results. The bbox predictor itself is a legitimate trick for getting gradients through discrete layout, and the empirical finding that PGD beats random shift under tight IoU budgets is plausible and useful.\n\nThe soft spots are not minor. The load-bearing step in Section 3.3 is PGD on continuous token embeddings, with boxes recovered by a learned predictor g_theta. The budget is enforced only on g_theta's predictions. The paper never constructs a document whose OCR output would induce the adversarial embedding, nor verifies that a real OCR pipeline could produce it. So the attack measures sensitivity of the model to internal embeddings, not robustness of OCR-based VDU systems to a realistic, budget-constrained document perturbation. The pixel and text modules modify the document, but the layout channel never gets realized as an actual rendered box shift. Given the paper's own Table 2 shows g_theta is imperfect, the claimed IoU guarantee may not hold for realized boxes. That is a load-bearing gap.\n\nOther issues: Table 3 has \"PGD values will be filled in separately if available\" for several cells and no PGD column for DocVQA; no error bars; bbox predictor train/test split is unspecified; claims about LayoutLMv2, ERNIE, and GeoLayoutLM are broader than the reported data, since most PGD runs use LayoutLMv3 only. These are fixable, but they make the current version incomplete.\n\nI think the paper deserves a serious referee. The benchmark idea is worth building on, and the realizability gap can be closed by an explicit construction: given an adversarial embedding, find the nearest text and bbox input that reproduces it, or restrict PGD to the space of actual OCR outputs. If the authors do that, the paper becomes a solid contribution. As it stands, the reader's conditional verdict is right. I would not cite it yet.\n\nRecommendation: send to peer review, major revision.","headline":"Useful benchmark idea, but the flagship PGD layout attack is never shown to be realizable as a document; worth reviewing but needs major revision.","tokens_in":10465,"tokens_out":2638,"would_cite":false,"duration_ms":28105,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that OCR-based visual document understanding models are far less robust than clean benchmarks suggest: budgeted adversarial shifts to bounding boxes, pixels, and text can cut F1 or ANLS by as much as 29.18%.","keywords":["visual document understanding","adversarial attacks","OCR robustness","bounding box perturbation","multi-modal attack","layout perturbation","projected gradient descent","document AI"],"falsifier":"Take one of the PGD line-level adversarial embeddings from FUNSD, render or search for a physical document image whose OCR output produces that embedding, feed the image through the full OCR-to-LayoutLMv3 pipeline, and measure the F1 drop; if the drop largely disappears, the vulnerability claim does not survive the image domain.","tokens_in":9426,"feed_emoji":"📄","tokens_out":5038,"duration_ms":45480,"temperature":0.7,"pith_summary":"The paper tries to establish that OCR-based visual document understanding models are substantially less trustworthy than their clean benchmark scores imply. It introduces a unified attack framework with six scenarios that perturb layout (bounding boxes), pixels, and text under explicit budgets, such as an IoU bound of at least 0.6, and evaluates the attacks on four datasets and six model families. The main finding is that line-level attacks and compound perturbations across all three channels cause the most severe degradation, and that gradient-based (PGD) bounding-box perturbations beat random shifts in every tested model. A sympathetic reader should care because OCR-based pipelines are the ones used in finance, tax, and compliance, so these results suggest an attacker who can alter the document image before OCR can collapse system accuracy despite high scores on ordinary tests.","feed_headline":"Multi-modal attacks cut OCR document F1 by 29%","feed_subtitle":"Line-level shifts to boxes, pixels, and text expose hidden fragility in document AI across four benchmarks.","key_machinery":"The central mechanism is a learned bounding-box predictor $g_\\theta$ that maps token embeddings to box parameters (center coordinates plus log width and log height), trained with SmoothL1 and GIoU losses. This predictor makes layout embeddings differentiable, allowing Projected Gradient Descent on the token embeddings under an mIoU-budget loss with projection onto the feasible set $\\text{IoU} \\ge \\tau$. Text and pixel modules bound character edits and use RoDLA-style document transforms (blur, noise, occlusion) to keep perturbations visually coherent.","core_discovery":"On the paper's own terms, the discovery is that a budgeted, multi-modal attack can reveal severe vulnerabilities in OCR-based VDU models: up to a 29.18% F1 drop on FUNSD under the compound BBox + Pixel + Text scenario. Line-level attacks consistently outperform word-level attacks, PGD-based layout perturbations outperform random shift baselines even under tight IoU budgets, Unicode diacritic text modifications cause larger degradation than random character replacement, and PGD-generated examples transfer across model architectures, including to text+layout models that have no image input.","pith_inferences":["The attack optimizes embeddings in model token space via the box predictor but never renders a real document image whose OCR would produce those embeddings; whether the reported drops survive a full image-to-OCR-to-model pipeline is an open question the paper does not answer.","The IoU budget measures geometric overlap, not semantic plausibility, so an even stronger real-world attack might exist that keeps boxes highly overlapping while moving them to layout positions that are still plausible for the document type.","If line-level attacks are this effective, OCR systems that pre-aggregate word boxes into lines are converting a robustness problem into a general layout-embedding problem; the paper's results imply that defense should happen at the line grouping stage, not only at the classifier.","The transfer results to text+layout models suggest the vulnerability lives largely in the layout embedding itself rather than the visual features, which points to alignment or preprocessing defenses on box tokens as the most direct next step."],"forward_implications":["Production OCR-based document systems should treat clean F1 scores as an upper bound, since plausibility-bounded layout, pixel, and text edits can induce double-digit F1 drops.","Because line-level attacks are consistently stronger than word-level attacks, OCR engines that merge word boxes into lines expand the attack surface and should be hardened first.","PGD-generated layout perturbations transfer to architectures with no visual input, meaning a single crafted document can hurt multiple deployed models without per-model optimization.","Stricter IoU budgets reduce but do not eliminate attack strength: on FUNSD, PGD still produced a 6.5% F1 drop at IoU 0.9 while random shifts fell to 0.54%.","Unicode diacritic text edits are more damaging than random character replacement, so text-based defenses must target visually confusable glyphs, not just typo-level noise."],"supporting_citations":[{"why":"LayoutLMv3 is the primary attacked model and supplies the most accurate bounding-box predictor used to generate PGD adversarial examples.","marker":"(Huang et al., 2022)"},{"why":"LayoutLMv2 is the transfer target that shows large F1 drops under PGD attacks, establishing cross-model transferability.","marker":"(Xu et al., 2022)"},{"why":"ERNIE-Layout is the tested model that remains most robust, providing the comparison that isolates model-specific vulnerability.","marker":"(Zhang et al., 2019)"},{"why":"GeoLayoutLM is the third transfer target whose word-level PGD attacks cause substantial F1 drops during transfer tests.","marker":"(Luo et al., 2023)"},{"why":"LayTextLLM represents the text+layout model family that suffers non-trivial degradation even without visual input, supporting the transferability claim.","marker":"(Lu et al., 2024b)"},{"why":"Supplies the Unicode-combining genetic text attack used in the text module and compared against random character replacement.","marker":"(Boucher et al., 2023)"},{"why":"RoDLA provides the set of twelve document-centric image transformations used for the pixel module in the compound attack scenarios.","marker":"(Chen et al., 2024a)"},{"why":"FUNSD is the primary evaluation benchmark where the largest F1 drops are reported, grounding the empirical claims.","marker":"(Jaume et al., 2019)"}],"fun_headline_variants":["Line-level multi-modal attacks crash OCR F1 by 29%","Unified multi-modal attack drops OCR F1 by 29% on FUNSD","Budgeted box, pixel, text attacks cripple document understanding","PGD box attacks beat random shifts across four OCR benchmarks","Adversarial text+layout attacks transfer across model families"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack is optimized in the model's token-embedding space using a trained box predictor, but the paper never constructs or verifies a real document image whose OCR output would induce that exact embedding, so the reported F1 drops could be an artifact of the embedding space rather than a realizable document perturbation.","fun_headline_variants_meta":{"raw":{"variants":["Line-level multi-modal attacks crash OCR F1 by 29%","Unified multi-modal attack drops OCR F1 by 29% on FUNSD","Budgeted box, pixel, text attacks cripple document understanding","PGD box attacks beat random shifts across four OCR benchmarks","Adversarial text+layout attacks transfer across model families"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000491,"raw_usage":{"total_tokens":2358,"prompt_tokens":832,"completion_tokens":1526,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":1437}},"tokens_in":448,"tokens_out":1526,"duration_ms":11840,"temperature":1.0,"reasoning_tokens":1437,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:27:29.726249+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the PGD line-level adversarial embeddings from FUNSD, render or search for a physical document image whose OCR output produces that embedding, feed the image through the full OCR-to-LayoutLMv3 pipeline, and measure the F1 drop; if the drop largely disappears, the vulnerability claim does not survive the image domain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LayoutLMv3 is the primary attacked model and supplies the most accurate bounding-box predictor used to generate PGD adversarial examples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GeoLayoutLM is the third transfer target whose word-level PGD attacks cause substantial F1 drops during transfer tests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Unicode-combining genetic text attack used in the text module and compared against random character replacement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FUNSD is the primary evaluation benchmark where the largest F1 drops are reported, grounding the empirical claims."}],"review_version":1}