{"id":"09852443-fe3b-4342-b660-8643422ab640","arxiv_id":"2412.17092","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"SAIL improves training-free document information extraction by choosing per-sample examples using entity-level text and layout similarities, outperforming prior ICL methods in F1.","lead":"This paper introduces SAIL, a method that selects tailored examples for each document to help large language models extract structured information from scanned forms and receipts without any training. By matching documents on text and layout separately, it claims to match or beat models that were explicitly trained for the task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never names the OCR system that produces T and B, so the comparisons against ICL-D3IE and borrowed full-training numbers may not be input-controlled; if SAIL uses cleaner ground-truth text/boxes, the claimed F1 gains could overstate the selection strategy.","rationale":"The reader's weakest assumption identifies exactly this: the OCR system is unspecified, so the quality of T and B is unknown, and if inputs differ across methods the central comparison is not controlled. I agree because the method's architecture makes T and B the sole interface between the document and the LLM; any difference in that interface can dominate the F1 deltas. The paper does include strong internal support for the selection mechanism, notably the fixed-vs-adaptive ablation in Table 3, the similarity ablations in Table 5, and the Wilcoxon tests in Table A8, so I am not calling the central claim false. However, those controls all operate within SAIL's own pipeline; they do not reconcile the input conditions with the external baseline and full-training numbers. I considered alternative concerns such as the suspicious P=F1=R equality in Table 2, but that is actually expected when every detected entity is assigned exactly one label, so it is not a red flag. I also considered the lack of matched demonstration counts against ICL-D3IE, but the fixed-example ablation in Table 3 largely addresses the role of adaptivity. The missing OCR specification remains the least secured premise because it can invalidate both halves of the central claim: beating ICL-D3IE and approaching full-training. The verdict stays CONDITIONAL pending this disclosure and a matched-input check.","tokens_in":14247,"tokens_out":11804,"duration_ms":112685,"concrete_test":"On one dataset (e.g., CORD), rerun SAIL and ICL-D3IE under two matched input conditions: (i) the dataset's official annotations for T and B, and (ii) the output of a standard public OCR (e.g., PaddleOCR or Tesseract) aligned to the same entity boxes. If the F1 gap between SAIL and ICL-D3IE shifts by more than a few points across conditions, the reported advantage is not attributable to the selection strategy alone. The authors should also state the OCR system and confirm that all compared methods received identical T and B.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SAIL's sample-centric selection (layout, entity-level, document-level) enables it to outperform ICL-D3IE and approach full-training methods. The method, as formalized in Section 3.1, consumes entity texts T and boxes B 'recognized from I by an OCR system' (Eq. 1); every later component, including the layout-image MSE in Eq. 4 and the prompt construction in Eqs. 5-9, depends on this input. Yet the paper does not state which OCR engine produced T and B, whether the dataset's ground-truth annotations were used, or whether the ICL-D3IE baseline and the borrowed full-training results were evaluated on identical T and B. If SAIL was given the datasets' clean annotated text/boxes while the baselines used a noisier external OCR, the reported gaps, e.g., 95.80 vs 87.13 on CORD with GPT-3.5, could reflect input quality rather than the selection or prompt design. This is the load-bearing premise because the entire comparison is meant to isolate the selection strategy, but the input representation is left uncontrolled and unreported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAIL, a training-free in-context learning method for document information extraction (DIE). SAIL constructs a sample-specific prompt for each test document by selecting three types of demonstrations: document-level text-similar examples, entity-level text-similar examples, and layout-similar examples where layout similarity is computed as the inverse MSE between binary layout images rendered from OCR bounding boxes. The prompt template combines these demonstrations with candidate labels, layout analysis, and the test question in OCR text-plus-box form. Experiments on FUNSD, CORD, and SROIE with ChatGLM3, GPT-3.5, and GPT-4 (gpt-4o) report consistent F1 gains over the ICL-D3IE baseline and, for GPT-4, F1 scores close to full-training methods. The authors include ablations of each component, example order, layout-similarity metric, resize method, and entity representation, plus a Wilcoxon significance test against ICL-D3IE.","tokens_in":14503,"tokens_out":5713,"duration_ms":47130,"significance":"If the reported results hold, SAIL would be a useful contribution: it is a simple, training-free, sample-adaptive ICL method with public code, and it shows consistent gains over the established ICL-D3IE baseline across three benchmarks and three backbone models, together with a thorough ablation study. The Wilcoxon signed-rank test (Appendix C) is a welcome addition, and the synthetic-data experiments (Appendix B) attempt to decompose text versus layout contributions. However, the validity of the empirical claims currently rests on several load-bearing evaluation choices: the OCR inputs are not specified, hyperparameters are selected using test sets, the Table 2 precision/recall values for SAIL are implausible, and the multimodal comparison is not input-controlled. These issues need to be resolved before the central claim can be taken at face value.","major_comments":[{"comment":"The OCR system that produces T and B is never identified. The manuscript states only that T and B are \"recognized from I by an OCR system\" (Eq. 1), and all subsequent similarity computations (Eqs. 2–4) and prompt construction (Eqs. 5–9) depend on these inputs. The comparisons in Table 1 against ICL-D3IE and full-training methods are only meaningful if all methods consume the same T and B. If SAIL uses ground-truth or higher-quality OCR while baselines use noisier OCR, the reported F1 gaps could be input artifacts rather than effects of example selection. Please specify the OCR engine for each dataset, state whether T and B come from ground-truth annotations, and re-run or re-derive baseline comparisons under identical inputs, or otherwise quantify sensitivity of the results to OCR quality.","section":"§3.1, Eq. (1) and §4.1"},{"comment":"For CORD and FUNSD, the SAIL row reports precision = recall = F1 (96.41/96.41/96.41 and 84.67/84.67/84.67). This exact equality is implausible for entity-level evaluation and suggests a reporting error. Since Table 2 is the basis for the claim that SAIL significantly surpasses multimodal LLMs, the precision/recall values need to be corrected or, if they are genuinely identical, justified with the evaluation formula used.","section":"Table 2, §4.3"},{"comment":"Key design choices are selected using the test sets: the layout similarity metric (Table A3), resize method (Table A4), number of document-level examples (Figure A1), representation of entity examples (Table A6), and number of entity examples (Table A7) are all chosen by comparing F1 on CORD and/or FUNSD test sets. The main paper does not mention a validation split. This makes the reported numbers test-set-tuned, so the headline F1 scores (e.g., 95.80 on CORD with GPT-3.5) may overstate generalization. Please move hyperparameter selection to a validation split or report the selection procedure explicitly and, if feasible, the corresponding validation performance.","section":"Appendix B, Tables A3–A7 and Figure A1"},{"comment":"The multimodal comparison is not input-controlled: SAIL is given OCR-derived text and boxes, whereas GPT-4o and LLaVA-1.5 receive only document images. The comparison conflates the selection/prompt design with the input representation. A controlled comparison (e.g., giving GPT-4o the same OCR text and boxes, or giving SAIL the image) is needed to claim that the method, rather than the input format, is responsible for the gap. At minimum, the paper should discuss this confound explicitly.","section":"§4.3, Table 2"},{"comment":"The paper reports single runs without error bars, and it invokes \"inherent randomness of LLM generation\" to explain a counterintuitive ablation result (Section 4.4, FUNSD #0 vs. #1). This acknowledges run-to-run variability. To support the claim that small differences (e.g., Table 3 adaptive examples, Table 4 example order) are meaningful, the authors should report means and standard deviations over multiple runs, or at least fix and report seeds for the local models, and provide confidence intervals for the main comparisons.","section":"§4.4 and Appendix B"}],"minor_comments":[{"comment":"The sentence \"In the case of GPT-4o, we only provide text prompts as input, while also testing its multimodal capabilities by providing document images and clear task instructions\" is ambiguous: it is unclear whether the GPT-4o rows in Table 2 are the text-only or image-based condition. Please clarify which setting produced the reported numbers.","section":"§4.1"},{"comment":"The caption contains a typo: \"Grean\" should be \"Green\".","section":"Figure A3"},{"comment":"The phrase \"pulling into a one-dimensional vector\" (Effect of the Layout Similarity Comparison Method) would be clearer as \"flattening into a one-dimensional vector\".","section":"Appendix B"},{"comment":"The paper alternates between \"GPT-4\", \"gpt-4o\", and \"GPT-4 (gpt-4o API version)\". Please standardize the naming so readers know which model is being reported in each table.","section":"§4.1 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"The most serious issues are the unspecified OCR inputs and the test-set-based hyperparameter selection, because both directly affect the validity of the headline comparisons. The identical precision/recall/F1 values in Table 2 for SAIL on CORD and FUNSD also suggest a data-integrity problem that should be clarified. These issues are fixable within the scope of a revision, so I am recommending major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: SAIL is a sensible, well-engineered extension of ICL-D3IE that replaces fixed document-level examples with per-sample selection using layout similarity and entity-level text similarity, wrapped in a unified prompt template. It consistently beats the previous training-free baseline across three datasets and three LLMs, and the ablations support the design. The evaluation has enough loose ends—unnamed OCR, suspicious precision/recall columns, test-set hyperparameter selection—that I would not take the headline F1 numbers at face value, but the core idea is sound and worth refereeing.\n\nWhat's new: the combination of entity-level text similarity and layout-image MSE selection, plus the unified prompt template that transfers across models better than ICL-D3IE's handcrafted prompts. The paper ships code, and the ablations (adaptive vs fixed, layout analysis, example order, synthetic data tests) are more thorough than most ICL papers. The Wilcoxon signed-rank test is a nice touch, though it only tests one run's per-sample scores.\n\nThe soft spots are real but mostly fixable. First, the OCR pipeline is never identified. Section 3.1 says T and B are recognized by an OCR system, but the experiments never say which OCR was used, whether the datasets' ground-truth annotations were used, or whether the ICL-D3IE baseline and the borrowed full-training numbers were based on the same T and B. If SAIL used clean ground-truth boxes while the baselines used noisier OCR, the gaps in Table 1 could partly reflect input quality rather than the selection strategy. This needs a clear statement. Second, Table 2 lists identical precision, recall, and F1 for SAIL on CORD and FUNSD—implausible enough that a careful reader assumes a copy-paste error. Third, several hyperparameters (number of examples, layout metric, resize method, entity representation) were chosen by comparing F1 on the CORD and FUNSD test sets, so the reported numbers are optimistic; the appendix tables should be framed as ablations on test sets rather than as model selection. None of these are disqualifying; they just cap how much the current numbers can be trusted.\n\nWho this is for: people working on document understanding with LLMs who want a cheap, training-free adaptation method; they'll find the prompt template and selection strategy directly usable. The paper deserves a serious referee. I'd send it to a document-AI venue or an EMNLP/ACL short-paper track and ask for clarity on the OCR setup, a fix for Table 2, and either variance across runs or evidence that the selection choices aren't overfit. With those changes, it's a solid paper.","headline":"SAIL is a solid, well-ablated extension of ICL-D3IE, but the unreported OCR setup and test-set-tuned hyperparameters mean the headline F1 numbers shouldn't be taken at face value.","tokens_in":15033,"tokens_out":3618,"would_cite":true,"duration_ms":28936,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No-training document extraction nearly matches fine-tuned models","keywords":["document information extraction","in-context learning","large language models","visually rich documents","layout similarity","entity-level text similarity","sample-centric prompting","training-free extraction"],"falsifier":"One concrete check is to run SAIL on a benchmark while jittering the OCR bounding boxes (for example, shifting each box by a few pixels or removing boxes below a size threshold). If F1 stays essentially unchanged, layout similarity is not doing the claimed work; if it drops sharply, the method is hostage to OCR box quality. Alternatively, replace the layout-similar examples with random documents while keeping the same prompt template; if F1 does not fall, the layout retrieval is superfluous.","tokens_in":14045,"feed_emoji":"🧾","tokens_out":3967,"duration_ms":33338,"temperature":0.7,"pith_summary":"The paper tries to establish that a training-free, in-context learning approach can extract structured information from document images almost as accurately as fully supervised models. It claims that the key is to build a custom prompt for each test document, pulling in examples that resemble it in three ways: same document-level text, same entity-level text, and same spatial layout. On three benchmarks the method reaches F1 scores of 96.41 on CORD and 98.18 on SROIE with GPT-4, beating the previous in-context baseline and approaching fully trained LayoutLMv3. If correct, this means strong document extraction is possible without task-specific training, using only a handful of hand-picked examples.","feed_headline":"No-training document extraction nearly matches fine-tuned models","feed_subtitle":"Retrieving layout- and entity-similar examples per document lifts GPT-4 to 98.18 F1 on receipts.","key_machinery":"The load-bearing mechanism is a three-way similarity retrieval feeding a single prompt template. Document-level text similarity uses Sentence-BERT embeddings of concatenated OCR text; entity-level similarity embeds individual text blocks (excluding pure numbers) and retrieves nearest entities; layout similarity renders the OCR bounding boxes as a binary image, crops to the content area, resizes, and compares with mean squared error, taking the inverse as the similarity score. These three selections populate a template with candidate label descriptions, entity demonstrations, a layout-analysis step, and document demonstrations. The boxes supplied to the LLM are cropped to the content region, and the layout analysis step asks the LLM to state where each label sits, which the paper claims helps the model transfer layout knowledge to the test document.","core_discovery":"The central claim is that sample-centric selection of in-context examples unlocks large language models for document information extraction. The authors argue that previous ICL methods fail because they use fixed examples picked by document-level text similarity alone; SAIL instead retrieves, for every test document, layout-similar documents, entity-level text-similar entities, and document-level text-similar documents, then packs them into a unified prompt template. With this recipe, GPT-4 (specifically GPT-4o) reaches 96.41 F1 on CORD and 98.18 on SROIE, close to the fully trained LayoutLMv3 (96.56 and 96.89 on the same sets), and the method outperforms the ICL-D3IE baseline across GPT-3.5, GPT-4, and ChatGLM3. The paper's claim is that the three-way retrieval plus the unified template is what produces this result, not the particular LLM.","pith_inferences":["The layout-similarity step depends on clean OCR boxes; if a deployed OCR produces noisy or incomplete boxes, the binary layout images may mislead retrieval, so the method's gains may shrink outside benchmark OCR conditions.","The approach could extend to table extraction or form understanding in other languages, since the prompt template and retrieval are language-agnostic as long as the embedding model supports the language.","A testable extension is to combine SAIL's retrieval with multimodal LLMs that take images directly, which might close the remaining gap to full training without OCR at all."],"forward_implications":["Document information extraction no longer requires task-specific fine-tuning; few-shot prompting with retrieved examples can rival fully trained extractors.","Because the method works across GPT-3.5, GPT-4, and ChatGLM3, the main lever for DIE performance may be in-context prompt construction rather than model scale.","The unified template transfers across datasets and label sets with only the candidate-label description changed, suggesting a single prompt recipe for many extraction tasks.","Ablations show each similarity type helps on different datasets (entity-level for long forms, layout for receipts), so future ICL systems should mix retrieval signals rather than rely on one."],"supporting_citations":[{"why":"Defines the ICL-D3IE baseline that SAIL compares against and improves upon across all models and datasets.","marker":"(He et al. 2023)"},{"why":"Provides Sentence-BERT embeddings used for both document-level and entity-level text similarity retrieval.","marker":"(Reimers and Gurevych 2019)"},{"why":"Reports LayoutLMv3 full-training results that serve as the target that SAIL approaches.","marker":"(Huang et al. 2022)"},{"why":"Establishes in-context learning as the paradigm that SAIL builds on for training-free extraction.","marker":"(Brown et al. 2020)"},{"why":"Motivates the importance of selecting good in-context examples, which SAIL operationalizes with three similarity signals.","marker":"(Liu et al. 2022)"},{"why":"Provides the CORD receipt benchmark with its 30 hierarchical labels used in evaluation.","marker":"(Park et al. 2019)"},{"why":"Provides the FUNSD form-understanding benchmark used in evaluation.","marker":"(Jaume, Ekenel, and Thiran 2019)"},{"why":"Provides the SROIE receipt benchmark used in evaluation.","marker":"(Huang et al. 2019)"}],"fun_headline_variants":["SAIL: per-document example retrieval boosts LLM extraction to 98 F1","Sample-centric ICL: similar layouts and entities per sample beat fixed prompts","Training-free document extraction closes gap to full fine-tuning","Retrieve per-document layout and text examples to push LLM extraction higher"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the OCR-derived text and bounding boxes are accurate and that comparing binary images of those boxes by mean squared error captures the layout information that actually determines the entity labels; the paper never names its OCR system.","fun_headline_variants_meta":{"raw":{"variants":["SAIL: per-document example retrieval boosts LLM extraction to 98 F1","Sample-centric ICL: similar layouts and entities per sample beat fixed prompts","Training-free document extraction closes gap to full fine-tuning","Retrieve per-document layout and text examples to push LLM extraction higher"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1675,"prompt_tokens":968,"completion_tokens":707,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":628}},"tokens_in":584,"tokens_out":707,"duration_ms":5838,"temperature":1.0,"reasoning_tokens":628,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:47:58.662966+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete check is to run SAIL on a benchmark while jittering the OCR bounding boxes (for example, shifting each box by a few pixels or removing boxes below a size threshold). If F1 stays essentially unchanged, layout similarity is not doing the claimed work; if it drops sharply, the method is hostage to OCR box quality. Alternatively, replace the layout-similar examples with random documents while keeping the same prompt template; if F1 does not fall, the layout retrieval is superfluous.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Sentence-BERT embeddings used for both document-level and entity-level text similarity retrieval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports LayoutLMv3 full-training results that serve as the target that SAIL approaches."},{"cited_title":"D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al","cited_arxiv_id":null,"evidence_quote":"Establishes in-context learning as the paradigm that SAIL builds on for training-free extraction."},{"cited_title":"B.; Carin, L.; and Chen, W","cited_arxiv_id":null,"evidence_quote":"Motivates the importance of selecting good in-context examples, which SAIL operationalizes with three similarity signals."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CORD receipt benchmark with its 30 hierarchical labels used in evaluation."},{"cited_title":"K.; and Thiran, J.-P","cited_arxiv_id":null,"evidence_quote":"Provides the FUNSD form-understanding benchmark used in evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SROIE receipt benchmark used in evaluation."}],"review_version":1}