{"id":"2ca5723b-718f-4c14-b34c-293b62f62e24","arxiv_id":"2412.02886","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PatchFinder uses VLM token confidence to select patch size and the most confident patch, achieving 94% field-extraction accuracy on 190 noisy scanned well documents.","lead":"PatchFinder splits scanned documents into overlapping patches and uses a vision-language model's own confidence scores to pick the most reliable patch for each answer, reporting 94% accuracy on historical well records. The method runs on a laptop and beats ChatGPT-4o by 18.5 percentage points, which could make digitization of noisy archives cheaper and more accessible.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The confidence-selection mechanism is validated only on centered crops, not on the grid patches PatchFinder actually uses; the 94% claim needs a control selecting non-max-confidence patches.","rationale":"The reader's weakest assumption identified PC reliability as the key risk, and I agree that is the central load-bearing point. I sharpen it: the correlation is demonstrated only under centered crops, whereas the deployed algorithm uses uncentered grid patches, so the evidence does not directly cover the operating regime. The reader also noted small dev set and missing error bars, which compound the issue. However, I do not think this rises to rejection; the method is plausible and the paper includes a quantitative result that, if confirmed under the proposed control, would support the claim. The proposed concrete test—comparing max-PC selection against random or fixed-patch selection—would directly establish whether the confidence mechanism is responsible for the gain, which is the minimal check required to accept the central claim. The verdict remains CONDITIONAL, hence UNCHANGED from the reader's assessment.","tokens_in":11879,"tokens_out":3341,"duration_ms":34904,"concrete_test":"Re-run Algorithm 1 on the 170-document test set (or at least on Old Colorado, n=77, and New Mexico, n=13) with four patch-selection rules: (a) max PC as in the paper; (b) min PC; (c) a random patch; (d) a fixed grid position (e.g., the center patch). If (a) is not statistically significantly better than (c) or (d) under a paired test (e.g., McNemar's test), then the Patch Confidence score is not load-bearing for the reported 94% accuracy. Also report 95% binomial confidence intervals for each category, especially New Mexico (n=13), and state the exact matching rule used for accuracy (exact string match? tolerance on numeric values?).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that PatchFinder's Patch Confidence (PC) score selects the patch that yields the correct answer, and that this selection is what lifts accuracy to 94%. The only direct evidence for the PC-accuracy correlation is Fig. 4 (Sec. 4.2), where the authors 'systematically varied the patch size, ensuring the target information remained centered.' But Algorithm 1 (Sec. 3.2) does not center the target: it divides the document into a fixed grid of overlapping patches, so the target field can be split, near a boundary, or absent from most patches. The paper never validates that PC identifies the correct patch under this grid sampling. If a patch without the target produces a confidently hallucinated value (e.g., a different well's latitude or a similar-looking number elsewhere on the form), PC will select the wrong answer. This is not a hypothetical failure mode: the development set used for Fig. 4 has only 20 documents (10 Colorado, 10 Pennsylvania), no error bars, and no per-category breakdown, so it cannot establish the correlation for Old Colorado, New Mexico, or the noisy financial statements. Because the entire method's improvement over vanilla Phi-3v is attributed to confidence-based patch selection, the absence of a control that replaces PC selection with a non-confidence-based choice is the single most load-bearing gap. The 90.1% unoptimized result (fixed 16.7% patch size) shows that patching alone helps, but it does not show that the PC-based selection step contributes beyond that.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PatchFinder, a method to improve vision-language-model (VLM) information extraction from noisy scanned documents. PatchFinder defines a Patch Confidence (PC) score as the mean log-likelihood of the maximum-softmax-probability tokens in the VLM's generated sequence, uses this score to choose a patch size on a development set, partitions documents into overlapping patches, and returns the prediction from the highest-confidence patch. Experiments on 190 historical well records report 94% accuracy for optimized PatchFinder with Phi-3v, outperforming ChatGPT-4o (75.4%) and vanilla Phi-3v (61.8%); additional experiments on 30 noisy financial statements and on CORD/FUNSD are reported. The central methodological claim is that PC reliably selects the patch containing the correct answer, and that patch-size optimization further improves accuracy.","tokens_in":12136,"tokens_out":3528,"duration_ms":33635,"significance":"If the central claim holds, the paper offers a simple, fine-tuning-free technique that could make small open-source VLMs competitive with much larger commercial models on noisy scanned documents, with direct relevance to the CATALOG orphan-well initiative and to document-processing practice more broadly. The algorithm is clearly specified (Algorithm 1), and the experiments span multiple datasets, including a known benchmark (CORD) and a synthetic noise condition. The paper also honestly acknowledges limitations around noise dependence and computational overhead. However, the headline accuracy and the confidence-selection mechanism rest on several underspecified and under-validated points, detailed in the major comments.","major_comments":[{"comment":"The dataset split is internally inconsistent. The text states that 20 documents (10 Colorado, 10 Pennsylvania) are used for development and 170 for testing, but Table 2 reports results for Pennsylvania (93.3%) despite Pennsylvania having only 10 documents total, and the overall 94% figure appears to include the development documents. Please clarify whether all reported accuracies are on the 170-document test set only, or on the full 190 documents, and if Pennsylvania contributes 0 test documents, explain where the 93.3% comes from. This affects every headline number in the abstract and Section 4.3.","section":"§4.1 and Table 2"},{"comment":"The correlation between Patch Confidence and extraction accuracy is demonstrated only for centered crops, where the target information is kept in the middle of the patch, while Algorithm 1 applies a fixed grid of overlapping patches with no centering. Under grid sampling, the target can be split across patch boundaries, near the edge, or absent from most patches, so the centering experiment does not validate the confidence-selection mechanism actually used. Please add a control that evaluates PatchFinder under grid sampling with a non-confidence-based selection rule (e.g., center patch, random patch, or a fixed patch) and compare it against max-PC selection; alternatively, provide per-patch accuracy and PC statistics on the grid to show that PC identifies the correct patch.","section":"§4.2, Fig. 4, and Algorithm 1"},{"comment":"The output filtering step discards patches whose response is non-numerical, but the paper never reports how often this filter triggers, how many documents have all patches filtered, or what the accuracy is without filtering. If the filter removes a large fraction of incorrect candidates, the reported 94% could be an artifact of filtering rather than of Patch Confidence. Please report the discard rate per dataset and the accuracy with and without the filtering rule.","section":"§3.3, 'Refinement with Output Filtering'"},{"comment":"The accuracy metric is not defined precisely. The paper says 'we use accuracy over F1' but does not specify the matching criterion for latitude, longitude, and TVD: exact string match after normalization? tolerance in decimal degrees or feet? How are the different coordinate formats (DMS vs decimal) handled? Without this definition, the reported accuracies are not reproducible and the comparison to ChatGPT-4o is not auditable. Please state the exact correctness rule and, ideally, provide the per-field accuracy.","section":"§4.1, 'evaluation' paragraph"},{"comment":"The strong OCR+Phi-3-mini baseline from the preliminary experiments (Table 1) is not evaluated on the full 190-document dataset. Since the paper motivates the work by arguing that the two-step OCR+LLM approach is costly and error-prone, and since OCR+Phi-3-mini was the best baseline in the preliminary results, omitting this baseline from the main comparison weakens the central claim of outperforming existing approaches. Please add the OCR+Phi-3-mini result (or a similar OCR+LLM baseline) on the same test split.","section":"Table 2 and §2.3"},{"comment":"The per-category samples are very small (Pennsylvania 10 documents, New Mexico 13) and no error bars, confidence intervals, or significance tests are reported. For example, a single misclassification in Pennsylvania changes the reported accuracy by 10 percentage points, so the 93.3% vs 66.7% difference is not shown to be meaningful. Please report exact counts (X/Y per category) and provide confidence intervals or a statistical test.","section":"Table 2 and §4.3"}],"minor_comments":[{"comment":"The abstract and Section 2 use 'Phi-3v', while Section 3.3 says 'Phi-3.5-Vision model' and Figure 1 says 'Phi-3-mini-128K-instruct'. Please standardize the model name, since these may refer to different model versions.","section":"Abstract and §3.3"},{"comment":"The caption says '10 documents from Colorado and Pennsylvania datasets' but the table reports separate percentages for each state; please clarify how many documents per state were used in this preliminary benchmark.","section":"Table 1 caption"},{"comment":"The definition of PC as (1/l) Σ log p(...) is the mean per-token log-likelihood, not the log-likelihood of the full sequence. The surrounding text says 'average log-likelihood of the maximum-likelihood sequence,' which is ambiguous; please rephrase to make clear that the average is over token positions.","section":"Eq. (2)"},{"comment":"Patch Size Optimization is described as using ground-truth sequences to determine adequate coverage, but Figure 2 and the surrounding text describe a confidence-based curve that does not reference ground truth. Please clarify the role of ground-truth labels in this step and whether the PC curve is computed on documents with known answers.","section":"§3.2 and Fig. 2"},{"comment":"Figure 4 would benefit from error bars or shading and a statement of how many documents and patches underlie each point; the current figure is consistent with the centered-crop protocol but the reader cannot assess variance.","section":"Fig. 4"},{"comment":"There are several typos and inconsistent capitalizations, e.g., 'dicussing' at the start of Section 3, and 'PatchFinder' vs 'Patchfinder' (especially in Table 3 and Section 5). A careful proofread is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The inconsistency between the stated 20/170 split and the Pennsylvania results in Table 2 is the most serious issue; if the headline results include development data, the comparison to ChatGPT-4o is not a clean held-out evaluation and the abstract's '190 documents' claim is misleading. I would also push for the missing OCR+LLM baseline on the full dataset and for the non-confidence-based control, as these are directly load-bearing for the paper's central claims. The paper is a reasonable application-style contribution, and the issues are fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a useful engineering paper with a plausible method and an honest but incomplete evaluation. The idea is simple: use the mean log maximum softmax probability over the decoded tokens as a Patch Confidence score, pick the patch size that maximizes confidence on a small development set, then report the prediction from the most confident patch. Applied to 190 scanned well records with Phi-3v, they report 94% accuracy, beating ChatGPT-4o by 18.5 points. That is a striking result if it holds up.\n\nWhat is genuinely new is the specific combination: using MSP-based confidence not just to flag low-confidence outputs but to select both patch size and the winning patch for VLM extraction. It is not a deep theoretical contribution — it is a straightforward extension of well-established uncertainty estimation to the VLM inference setting — but the application to historical well records is practically motivated and the method is described clearly enough to reproduce. The PC score is well specified, the development/test split is standard, and they go beyond the main dataset by testing on noisy financial statements and on CORD/FUNSD. The limitations section is candid about the method working best on small numeric fields in noisy documents.\n\nThe soft spots are mostly in the evidence, not the method. First, the accuracy metric is never defined. 'Accuracy' for latitude/longitude/TVD could mean exact string match, or a tolerance-based match for coordinates; without this, the headline number is not interpretable. Second, the per-category samples are tiny — 10 for Pennsylvania, 13 for New Mexico — and there are no error bars. A 93.3% on 10 documents is 9 or 10 correct answers; the difference from 66.7% is a few documents. Third, the output filtering rule discards non-numerical responses, but the discard rate is never reported, so the 94% could be inflated if many patches are thrown away. Fourth, the strong OCR + Phi-3-mini baseline from Table 1 is missing from the main comparison in Table 2; that is a notable omission.\n\nThe stress-test concern about centered crops versus grid patches is valid. The PC-accuracy correlation in Fig. 4 is demonstrated on crops where the target is centered, but Algorithm 1 uses a fixed grid where the target can be split or near a boundary. The paper never shows that PC selects the correct patch under grid sampling, and there is no control such as random patch selection or majority voting to isolate the contribution of confidence-based selection. The 90.1% unoptimized result shows patching helps; it does not show that the confidence step is what lifts accuracy. This is not fatal — the method still works on aggregate — but it is the load-bearing gap in the causal story.\n\nOverall, this is a solid engineering contribution that deserves a serious referee, but the revision needs to define the metric, report error bars and discard rates, add the missing baseline, and include a non-confidence-based patch selection control. I would bring it to a reading group as an example of applied uncertainty in VLMs, but I would not cite it in my own work unless I were working on document extraction.","headline":"Useful engineering paper with a plausible method, but evidence for the headline 94% has several fixable gaps: undefined metric, tiny samples, missing baseline, and confidence selection validated only on centered crops.","tokens_in":12690,"tokens_out":3237,"would_cite":false,"duration_ms":29084,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PatchFinder claims that a small vision-language model can reach 94% extraction accuracy on noisy scanned well documents by using its own token-confidence scores to choose the best image patch, outperforming ChatGPT-4o by 18.5 percentage…","keywords":["PatchFinder","vision-language models","information extraction","scanned documents","model uncertainty","Maximum Softmax Probability","well records","document understanding"],"falsifier":"A held-out experiment in which correct and incorrect patches have overlapping Patch Confidence distributions would falsify the central claim: for example, if on the 170 test well records the argmax-confidence patch is no more accurate than a randomly chosen patch, the accuracy–confidence correlation breaks down. Concretely, one could take a document where the model hallucinates a plausible number with high average token probability and show that PatchFinder selects that wrong number over the correct one.","tokens_in":11663,"feed_emoji":"📄","tokens_out":5280,"duration_ms":50936,"temperature":0.7,"pith_summary":"This paper tries to show that a cheap, fine-tuning-free method can extract structured fields from noisy scanned documents as well as or better than much larger commercial models. The proposed algorithm, PatchFinder, crops a scanned page into overlapping patches, asks a small vision-language model to read each patch, and uses a confidence score—the average log-likelihood of the model's most-likely tokens—to pick both the patch size and the single most reliable patch. On 190 historical well records, PatchFinder with the 4.2-billion-parameter Phi-3v reaches 94% accuracy on latitude, longitude, and true vertical depth, outperforming ChatGPT-4o by 18.5 percentage points, and it also beats both baselines on noisy financial statements. The practical payoff would be that organizations with limited compute can automate large-scale digitization of old records without OCR pipelines or fine-tuning.","feed_headline":"Confidence score guides a small VLM to 94% accuracy","feed_subtitle":"PatchFinder reads noisy scanned well records and beats ChatGPT-4o by 18.5 points without fine-tuning.","key_machinery":"The central object is the Patch Confidence score, defined as $PC(m) = \\frac{1}{l}\\sum_{i=1}^{l} \\log p_\\theta(x_i^{k^*}|x_{1\\cdots i-1}, z^{(m)})$, where $k^*$ is the token with maximum softmax probability and $z^{(m)}$ is the visual-plus-text context of patch $m$. In plainer terms, it is the mean log-likelihood of the most-likely decoded tokens, i.e., an average of Maximum Softmax Probabilities over the generated answer. PatchFinder uses this score in two stages: Patch Size Optimization searches the development data for the patch size that yields high, low-variance confidence (around 22–25% of the image), and Confidence-Based Prediction computes $PC(m)$ for every overlapping patch and returns the prediction of $m^* = \\arg\\max_m PC(m)$. The mechanism works by cropping the image into a size the model can attend to well, reducing noise while preserving the target field.","core_discovery":"The paper's central claim is that a vision-language model's own confidence, measured as the average log-likelihood of its maximum-likelihood output tokens, can be used to locate the most reliable region of a noisy scanned document. PatchFinder first uses this Patch Confidence score to select a patch size that keeps the target field inside the model's positional-encoding comfort zone, then evaluates every overlapping patch at that size and returns the prediction of the most confident one. On the authors' 190-document well-record dataset, this lifts Phi-3v's accuracy from 61.8% to 94.0%, surpassing ChatGPT-4o's 75.4%; the same method reaches 53.3% on 30 synthetically noisy financial statements, where ChatGPT-4o scores 36.7% and vanilla Phi-3v scores 0%. The paper presents this as evidence that patching and maximum-softmax-probability-based confidence can improve VLM extraction without fine-tuning.","pith_inferences":["If Patch Confidence reliably tracks answer correctness, the same score could serve as a rejection rule: documents whose best patch confidence falls below a threshold could be routed to human review, which would be directly useful for the orphan-well cataloging use case.","The cropping mechanism implies a general chunking strategy for small-context VLMs: treating image crops like text chunks could let models with short positional-encoding windows handle high-resolution or large-format documents, a consequence the paper states but does not develop.","A natural extension the paper leaves implicit is multi-scale aggregation: instead of choosing one patch size, combine predictions across several sizes weighted by Patch Confidence, which could smooth the sensitivity to the 20–25% optimum.","The confidence–accuracy correlation is established on 20 development documents without error bars; a direct test would be to report per-category calibration of Patch Confidence against correctness, especially for Old Colorado and New Mexico."],"forward_implications":["On the 190 well records, optimized PatchFinder raises accuracy from 61.8% (vanilla Phi-3v) and 75.4% (ChatGPT-4o) to 94.0%.","The largest gains appear on the hardest document categories: Pennsylvania goes from 66.7% to 93.3%, and New Mexico from 57.6% (ChatGPT-4o) to 93.9%.","On 30 synthetically noisy financial statements, PatchFinder reaches 53.3% versus 36.7% for ChatGPT-4o and 0% for vanilla Phi-3v, indicating the method helps most in high-noise settings.","On the CORD receipt dataset, PatchFinder's zero-shot accuracy is 77.2–79.0% versus 46.8% for Donut, showing transfer beyond well records.","The method requires no fine-tuning and runs on a single laptop with 64 GB of RAM, making it accessible to organizations without large compute budgets."],"supporting_citations":[{"why":"Supplies the Phi-3/Phi-3v 4.2B vision-language model that PatchFinder builds on.","marker":"[21]"},{"why":"Provides the Maximum Softmax Probability baseline that the Patch Confidence score is derived from.","marker":"[25]"},{"why":"Defines the well-record extraction task and the two-step OCR-plus-LLM approach that PatchFinder is compared against.","marker":"[6]"},{"why":"ChatGPT-4o is the commercial VLM baseline that PatchFinder claims to outperform by 18.5 percentage points.","marker":"[29]"},{"why":"Donut is the OCR-free document extraction baseline used in the preliminary and CORD/FUNSD comparisons.","marker":"[4]"},{"why":"Google Cloud Vision OCR forms the OCR leg of the two-step baseline pipeline.","marker":"[24]"},{"why":"CLIP's ViT-L/14 image encoder produces the visual tokens inside the Phi-3v architecture.","marker":"[9]"},{"why":"Provides the historical financial statements used to test generalization to a different document type.","marker":"[30]"}],"fun_headline_variants":["Small VLM's own confidence cracks noisy scans: 94% hits","Confidence-guided patching lifts VLM to 94% accuracy","PatchFinder: Use model uncertainty to read messy scans","No fine-tuning needed: Confidence-based patching hits 94%","VLM confidence spots the right patch, beats GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that a higher Patch Confidence score means a more accurate answer; if the model is confidently wrong on a patch, PatchFinder will happily return that wrong answer.","fun_headline_variants_meta":{"raw":{"variants":["Small VLM's own confidence cracks noisy scans: 94% hits","Confidence-guided patching lifts VLM to 94% accuracy","PatchFinder: Use model uncertainty to read messy scans","No fine-tuning needed: Confidence-based patching hits 94%","VLM confidence spots the right patch, beats GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1356,"prompt_tokens":987,"completion_tokens":369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":280}},"tokens_in":603,"tokens_out":369,"duration_ms":4086,"temperature":1.0,"reasoning_tokens":280,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:58:57.850757+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A held-out experiment in which correct and incorrect patches have overlapping Patch Confidence distributions would falsify the central claim: for example, if on the 170 test well records the argmax-confidence patch is no more accurate than a randomly chosen patch, the accuracy–confidence correlation breaks down. Concretely, one could take a document where the model hallucinates a plausible number with high average token probability and show that PatchFinder selects that wrong number over the correct one.","supporting_citations":[{"cited_title":"Information extraction from historical well records using a large language model,","cited_arxiv_id":null,"evidence_quote":"Defines the well-record extraction task and the two-step OCR-plus-LLM approach that PatchFinder is compared against."},{"cited_title":"Chatgpt-4o,","cited_arxiv_id":null,"evidence_quote":"ChatGPT-4o is the commercial VLM baseline that PatchFinder claims to outperform by 18.5 percentage points."},{"cited_title":"Cloud vision api,","cited_arxiv_id":null,"evidence_quote":"Google Cloud Vision OCR forms the OCR leg of the two-step baseline pipeline."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"CLIP's ViT-L/14 image encoder produces the visual tokens inside the Phi-3v architecture."},{"cited_title":"Corporate reports online,","cited_arxiv_id":null,"evidence_quote":"Provides the historical financial statements used to test generalization to a different document type."}],"review_version":1}