{"id":"2b245d8c-0ded-4d1e-8f7b-81c25bcd1225","arxiv_id":"2412.20088","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A VLM-based pipeline with localize, comprehend, and match modules collects artifact catalogs at AP up to 35.4%, though evaluation circularity weakens the claim.","lead":"This paper builds a three-step pipeline using large vision-language models to automatically extract pottery images and their text descriptions from printed archaeological catalogs. The authors report that their method outperforms using off-the-shelf VLMs alone, but the evaluation relies on a ground truth cleaned from the method's own output.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation is circular: the 'cleaned dataset' used as ground truth in §4.2 was generated by the proposed method itself (§4.1), so Table 1's AP gains do not independently validate the claimed reliable solution.","rationale":"The reader's weakest assumption is the right one and is confirmed by the paper's own text: §4.1 constructs the dataset with the proposed method, and §4.2 uses that dataset as ground truth. This makes the evaluation circular in exactly the way that matters for the central claim. I find no independent support that would offset the issue: no code/data release, no independent annotation study, and no external benchmark. The pipeline's components are standard (open-set detection, VLM prompting, bipartite matching) and the comparison against Qwen-VL-only is reasonable, but those facts do not establish real-world reliability. I therefore keep the reader's REJECT verdict, with moderate confidence; a revision that adds an independently annotated evaluation set and quantifies the manual corrections could change this assessment.","tokens_in":7617,"tokens_out":2937,"duration_ms":31507,"concrete_test":"Select a random held-out sample of 30 of the 302 catalog pages. Have two archaeologists with no access to the pipeline outputs independently annotate every pottery image, its excavation unit, class label, and bounding box following a written protocol; compute inter-annotator agreement. Recompute Table 1 AP@0.9 for Ours (Claude 3.5 Sonnet) and Qwen-VL-only against this independent ground truth. Also log the number and nature of the manual corrections used to create the original cleaned dataset. If the proposed method's AP drops materially, or the baseline's relative gap shrinks, the self-generated ground truth was inflating the reported improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the three-module pipeline significantly improves accuracy and provides a reliable solution—rests on the comparison experiments in §4.2, which state: \"We use the cleaned dataset as ground truth.\" But §4.1 says the dataset was obtained \"using the proposed method\" with \"minimal manual correction.\" The reference boxes, class labels, and pairings used to compute AP at IoU 90% are therefore outputs of the same detection and comprehension modules being evaluated. If the manual corrections were light, the metric largely measures self-consistency; if they were extensive, the paper's characterization is misleading and the task difficulty is hidden. Either way, the 33.8-point improvement over Qwen-VL-only is not evidence of external accuracy. No released code, data, or annotation protocol is provided, so there is no way to check whether ground-truth construction was biased by model outputs. This is a load-bearing flaw: without an independent gold standard, the paper's headline effectiveness claim is unsupported, even though the pipeline itself is internally coherent and the comparison against a no-pipeline baseline is a reasonable experimental design.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-module pipeline for automated archaeological catalog collection from scanned pottery catalog PDFs: a document localization module based on an open-set object detection model, a block comprehension module that uses a vision-language model (VLM) to convert image and text blocks into structured attribute dictionaries, and a block matching module that aligns image and text blocks through foreign-key overlap followed by distance-based bipartite matching. The authors report experiments on 302 pages from the Dabagou and Miaozigou pottery catalogs, claiming an AP of 35.4% with Claude 3.5 Sonnet, a 33.8-percentage-point improvement over Qwen-VL alone, and conclude that the approach provides a reliable solution for automated archaeological catalog collection.","tokens_in":8033,"tokens_out":6840,"duration_ms":71057,"significance":"If the claimed accuracy were established, the pipeline would be a practically useful contribution to cultural-heritage digitization and to VLM-based document understanding, since it couples open-set detection with structured attribute extraction and a principled matching rule. The three-module decomposition is sensible, and the inclusion of a no-pipeline baseline is a reasonable experimental design. However, the significance is currently capped by the circularity of the evaluation, the incomplete baseline table, and the absence of released data or code; these issues must be resolved before the effectiveness and reliability claims can be accepted.","major_comments":[{"comment":"The evaluation is circular: Section 4.1 states that the dataset was collected \"using the proposed method\" with \"minimal manual correction,\" and Section 4.2 states \"We use the cleaned dataset as ground truth to calculate metrics.\" The ground-truth boxes, class labels, and image-text pairings used to compute AP are therefore derived from the same detection, comprehension, and matching pipeline being evaluated. The paper does not quantify the manual corrections (e.g., how many boxes were added, deleted, or moved; how many class labels were corrected; how many pairings were fixed) and does not describe an annotation protocol that would prevent the model's outputs from biasing the corrections. This is load-bearing because the abstract and Section 5 base the \"reliable solution\" and \"significantly improves accuracy\" claims on the AP values in Table 1. The authors should construct an independent gold standard, for example by having annotators label pages without seeing the method's outputs, and report the comparison on that gold standard together with correction statistics.","section":"Sections 4.1 and 4.2"},{"comment":"The baseline comparison is incomplete: \"GPT-4o only\" and \"Claude 3.5 Sonnet only\" are listed as \"-\" without any score or explicit failure criterion, and the statement that these models \"lack object detection capabilities\" is an assertion rather than a measured result. The headline 33.8-point improvement is computed only against Qwen-VL-only, so the reader cannot verify that the pipeline outperforms all direct VLM baselines. In addition, all AP values appear to come from a single evaluation without error bars, confidence intervals, or significance tests; given the strongly long-tailed class distribution shown in Figure 4(a), the reported differences may be driven by a few frequent classes. Please report failure rates for the incomplete baselines, per-class or at least page-level bootstrap intervals, and the exact detection recall before matching.","section":"Table 1 and Section 4.2"},{"comment":"The detection model used inside the proposed pipeline is never identified: Eq. (1) refers only to \"open-set object detection model,\" while the baseline explicitly uses Grounding-DINO. The reader cannot tell whether the proposed method also uses Grounding-DINO or a different detector, which makes the method irreproducible and leaves open the possibility that the Table 1 gain comes from detector choice rather than from the proposed pipeline. Please specify the detection model, the exact prompts P_img, P_text, and P_c, the VLM versions and decoding settings, and the concrete foreign-key attribute set K_foreign used in Eq. (4).","section":"Section 3.2, Eq. (1); Section 3.4, Eq. (4); Section 4.2"},{"comment":"Two thresholds that materially affect the reported numbers are asserted without justification or sensitivity analysis: the foreign-key matching threshold M = 1 in Eq. (6) and the AP IoU threshold of 0.9 in Section 4.2. A single OCR or comprehension error in a foreign key makes M < 1 and rejects a correct pair, and an IoU threshold of 0.9 is unusually strict for detection evaluation. Please report AP at standard IoU thresholds (e.g., 0.5 and 0.75), vary the matching threshold (e.g., 0.75, 0.9, 1.0), and justify the chosen operating points with data.","section":"Section 4.2 and Eqs. (5)-(9)"}],"minor_comments":[{"comment":"The manuscript contains two objects labeled \"Table 1\": one for artifact-per-unit statistics in Section 4.1 and one for the performance comparison in Section 4.2. Renumber the tables to avoid ambiguity.","section":"Sections 4.1 and 4.2"},{"comment":"The axis labels in Figure 4 are rendered as garbled Unicode escape sequences (e.g., \"/uni00000026/uni00000044/...\") rather than readable category names; replace them with the actual class and unit labels.","section":"Figure 4"},{"comment":"The ACM reference block contains placeholder text (\"Make sure to enter the correct conference title from your rights confirmation email\") and a 2018 copyright date for a 2025 submission; these must be corrected before publication.","section":"ACM reference block"},{"comment":"The notation is inconsistent: Eq. (8) writes M* in the minimization formula while the surrounding text refers to M*_k. Make the notation uniform.","section":"Eq. (8)"},{"comment":"The foreign-key attribute set K_foreign is used in Eq. (4) but never explicitly defined; give a concrete example for the pottery catalogs (e.g., which attribute keys are treated as foreign keys).","section":"Section 3.4"},{"comment":"The term \"data pair\" is not formally defined; clarify whether it denotes one image-text correspondence, one artifact record, or one page-level pair.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be an early short-conference version with placeholder ACM metadata and garbled figure text. The main risk is not the architectural idea but the evaluation: as written, the ground truth is derived from the method being evaluated. I would ask for an independent annotation study before considering publication; if that is not feasible, the effectiveness claims should be substantially weakened. The two Table 1 references and the missing detector specification also suggest that the manuscript needs careful editorial and technical revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper assembles three well-known pieces (open-set detection, VLM comprehension, foreign-key plus bipartite matching) into a pipeline for extracting pottery catalog entries from PDF pages. That assembly is reasonable and the problem is real. But the evaluation is circular, so the headline numbers — 35.4% AP with Claude 3.5 Sonnet, a 33.8-point jump over Qwen-VL alone — tell you more about the cleaning process than about the method.\n\nThe design is clean and the matching stage is a sensible answer to a genuine many-to-many problem in catalogs. The authors correctly identify that off-the-shelf VLMs can't ground artifact images reliably, and they show a non-trivial baseline comparison. The use of AP at IoU 0.9 is strict and appropriate.\n\nThe load-bearing problem is in Sections 4.1–4.2. The ground truth is the 'cleaned dataset' produced by the proposed method with 'minimal manual correction.' That makes the AP a self-consistency score. We don't know how much the human corrected, what the protocol was, or whether the corrections were guided by the method's failures. Without an independent annotation set, the improvement over baselines is not evidence of external accuracy. The paper also leaves GPT-4o and Claude-only baselines as '-' without explaining what that means, provides no error bars, no released code/data, and only two catalogs. Minor: Figure 4's text appears garbled in the PDF, but that's production not substance.\n\nIf the evaluation were fixed — e.g., a separate human annotation produced without seeing the method's output, plus a description of the correction load — the result could be useful to the digital heritage community. As it stands, the abstract's claim of a 'reliable solution' is not supported.\n\nI would not send this to peer review in its current form; the circularity is load-bearing. I'd desk reject with an invitation to resubmit after a proper independent evaluation. For a reading group, it's a useful case study in how evaluation design can invalidate an otherwise coherent pipeline.","headline":"The pipeline is a sensible engineering combination, but the evaluation's ground truth is the method's own corrected output, so the reported AP numbers don't support the reliability claim.","tokens_in":8356,"tokens_out":2724,"would_cite":false,"duration_ms":26529,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-module pipeline—document localization, block comprehension, block matching—lets large vision-language models extract pottery catalog entries, reaching 35.4% average precision, 33.8 points above the best direct baseline.","keywords":["archaeological catalog","vision-language models","open-set object detection","document localization","block matching","modal matching","pottery catalog","cultural heritage digitization"],"falsifier":"Take a random sample of the 302 catalog pages, have two annotators who have never seen the pipeline's outputs transcribe all image–caption pairs by hand, and recompute the pipeline's AP against that independent ground truth; if the number approaches the direct-baseline level (1.6% AP) or the human annotations diverge substantially from the paper's cleaned data, the central reliability claim is false.","tokens_in":7451,"feed_emoji":"🏺","tokens_out":8286,"duration_ms":72052,"temperature":0.7,"pith_summary":"Archaeological catalogs—artifact images paired with excavation unit, class, and measurements—are scattered across thousands of PDFs, and existing vision-language models either cannot detect small artifact images or cannot link them to their text rows. This paper claims that a three-module pipeline solves both problems: an open-set detector isolates image and text blocks, a vision-language model converts each block into structured attributes, and a matcher aligns images to text by shared catalog numbers, using spatial distance to break ties. On 302 pages from two Neolithic pottery catalogs, the pipeline with the strongest tested model reaches 35.4% average precision at 90% IoU, a 33.8-point improvement over the best direct VLM baseline. If the result holds, automated collection of archaeological catalogs becomes practical, turning scattered legacy publications into structured data for classification, dating, and evolution studies.","feed_headline":"Pottery catalog AI jumps from 1.6% to 35.4% average precision","feed_subtitle":"Detection, comprehension and matching turn scattered archaeology PDFs into structured records.","key_machinery":"The load-bearing object is the matching module operating on structured attribute blocks. Each detected block is converted by a VLM into a dictionary of attributes; foreign-key matching uses the Jaccard overlap of the key sets, accepting a pair only when the overlap is exactly 1, and the distance-bipartite stage resolves the resulting many-to-many groups by solving a minimum-weight assignment on Euclidean distances between bounding-box centers. The design converts modal alignment from an open-ended grounding problem into two closed-form matching problems, which is what lets a general-purpose VLM succeed without specialized detection heads or fine-tuning.","core_discovery":"On its own terms, the paper claims that direct VLM prompting fails at catalog collection because a single model cannot reliably localize small artifact images and bind them to their caption rows, while splitting the task into three modular steps succeeds. An open-set detector segments a page into image and text blocks; a VLM converts each block into a structured attribute dictionary (index, item code, class); and a matcher aligns the two modalities by accepting pairs whose foreign-key sets overlap completely, then resolves many-to-many groups with minimum-weight bipartite matching on the Euclidean distance between block centers. With Claude 3.5 Sonnet driving comprehension, the full pipeline reaches 35.4% AP on the cleaned ground truth, versus 1.6% AP for Qwen-VL used alone and 0% AP for a detector-plus-VLM combination without the matching module; GPT-4o inside the pipeline reaches 32.8% AP. These margins are the paper's evidence that all three modules are necessary and that the pipeline transfers across different vision-language models.","pith_inferences":["The reported 35.4% AP is best read as an upper bound because the ground truth came from the method itself with manual correction; a fresh independent annotation of a page subset would give the realistic number.","The method's matching rules depend on the regular layout of pottery catalogs, where an index and item code sit beside each image; irregular archaeological documents would likely need learned matching rather than fixed foreign keys.","A straightforward stress test is to rotate pages, merge columns, or erase item codes and measure how much of the 33.8-point gain survives, which would separate the contribution of layout regularity from the VLM's semantic comprehension."],"forward_implications":["Automated collection of archaeological catalog pages is feasible with current off-the-shelf VLMs when localization and matching are handled by dedicated modules, rather than by the VLM alone.","The pipeline is model-agnostic but backend-sensitive: swapping the comprehension VLM moves AP between 27.4% and 35.4%.","The 90% IoU threshold means the gains are in precise pixel-level localization of artifact images, not just coarse page-level understanding.","The resulting 2,301-pair dataset with 44 shape classes and 310 excavation units provides a structured resource for downstream archaeological tasks such as classification and dating."],"supporting_citations":[{"why":"Supplies Claude 3.5 Sonnet, the comprehension engine that achieves the paper's best 35.4% AP result.","marker":"[1]"},{"why":"Supplies Qwen-VL, the baseline that reaches only 1.6% AP alone and 27.4% inside the pipeline.","marker":"[2]"},{"why":"Grounding-DINO, the open-set detector used in the document localization module and in the strongest baseline variant.","marker":"[7]"},{"why":"Supplies GPT-4o, which scores 32.8% AP inside the pipeline but cannot complete the task alone.","marker":"[8]"},{"why":"Describes PDFChatAnnotator, the prior catalog annotation tool whose simple binding rules motivate the matching module.","marker":"[12]"},{"why":"LMDX, the VLM document-extraction method that performs modal binding within a single modality and provides the baseline context.","marker":"[10]"}],"fun_headline_variants":["Archaeology AI: 1.6% to 35.4% catalog precision leap","VLM pipeline triples pottery catalog accuracy","From 1.6% to 35.4%: Modular AI reads archaeology PDFs","Split tasks boost pottery catalog AI 20-fold"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the cleaned dataset used as ground truth is an independent gold standard, even though the dataset was assembled by the proposed method with minimal manual correction, so any systematic failures that the corrections did not remove would inflate the measured AP.","fun_headline_variants_meta":{"raw":{"variants":["Archaeology AI: 1.6% to 35.4% catalog precision leap","VLM pipeline triples pottery catalog accuracy","From 1.6% to 35.4%: Modular AI reads archaeology PDFs","Split tasks boost pottery catalog AI 20-fold"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000333,"raw_usage":{"total_tokens":1817,"prompt_tokens":882,"completion_tokens":935,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":858}},"tokens_in":498,"tokens_out":935,"duration_ms":7751,"temperature":1.0,"reasoning_tokens":858,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:33:57.672609+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 302 catalog pages, have two annotators who have never seen the pipeline's outputs transcribe all image–caption pairs by hand, and recompute the pipeline's AP against that independent ground truth; if the number approaches the direct-baseline level (1.6% AP) or the human annotations diverge substantially from the paper's cleaned data, the central reliability claim is false.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes PDFChatAnnotator, the prior catalog annotation tool whose simple binding rules motivate the matching module."}],"review_version":1}