{"id":"c54ad78b-3aab-42c5-968c-bfc3e054ae02","arxiv_id":"2412.11574","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PyPotteryLens detects, segments, orients, and labels pottery drawings from archaeological PDFs using YOLO and EfficientNetV2, reporting above 96% precision and up to 20x faster processing.","lead":"PyPotteryLens is an open-source tool that uses image-recognition models to find and extract pottery drawings from archaeological PDFs, then sorts and labels each drawing automatically. It reports high accuracy and large time savings for archaeologists, and it could unlock decades of legacy illustrations for digital research.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Classification validation split may leak augmented variants of the same original vessel, inflating the reported >97% classification metrics.","rationale":"The reader's conditional verdict is well founded. I share the concern about the two OOD contexts being reported only as boxplot points without numeric values and with unknown annotation reliability. But the most load-bearing risk to the central claim is in the classification pipeline (§3.2.2). The authors create 8 augmented variants of each of 4,563 vessel instances, producing 18,252 examples, and then say '80% of the images are used for training and the remaining 20% for validation.' If that split is at the level of augmented images, the validation set contains flipped/rotated copies of vessels already seen in training. The near-100% accuracy in Table 3 and Figure 9 would then be an upper bound on memorization, not an estimate of performance on unseen drawings. This directly threatens the abstract's 'over 97% precision and recall in ... classification tasks'. The manuscript does not state that the split was grouped by base instance or by publication, and the phrase '80% of the images' weighs against that interpretation. The OOD generalization concern raised by the reader is real and should be fixed by reporting numeric per-context metrics, but it is about the strength of a qualitative claim rather than the validity of the central quantitative claim. I therefore keep the CONDITIONAL verdict: the framework is plausible and likely useful, but the classification numbers must be re-validated with a leakage-free split and the OOD metrics must be reported numerically before the central claim can be accepted as stated.","tokens_in":14273,"tokens_out":5755,"duration_ms":52484,"concrete_test":"Inspect the repository's dataset split code (or request the authors' split indices) to determine whether the 80/20 split was applied to the 18,252 augmented images or to the 4,563 base instances. Then recompute Table 3 with a split grouped by base instance (and ideally by publication) using the same training configuration. If per-class precision/recall remains above the claimed thresholds, the concern is resolved; if it drops, the classification claim must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.2.2 the classifier is trained on 4,563 base instances that are manually augmented into 8 orientation/flip configurations, giving 18,252 'examples'; the text then says 80% of the images are used for training and 20% for validation. If the 80/20 split was made over the 18,252 augmented images rather than over the 4,563 original vessels, then the same physical drawing appears in both training and validation in different orientations. Validation accuracy then partly measures the model's ability to recognize already-seen vessels, not its ability to classify new pottery. The paper never states that the split was stratified by base instance or publication, and the phrasing '80% of the images' suggests the opposite. Since the abstract's central claim explicitly includes 'over 97% precision and recall in ... classification tasks', Table 3's near-99% numbers may be optimistically biased and the classification portion of the central claim is not yet reliable. This is a correctness risk distinct from the (valid) concern that OOD detection metrics are only shown as boxplots.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PyPotteryLens, an open-source deep-learning framework for automatically detecting, segmenting, classifying, and digitising archaeological pottery drawings from legacy publications. The system combines YOLOv8/YOLOv11 instance segmentation with a multi-head EfficientNetV2 classifier and a Gradio-based user interface, plus a self-annotation module. The authors report detection/segmentation precision and recall near 97% on a validation set, classification precision and recall above 98%, a 4.7× to 20× reduction in processing time compared to manual recording in a single case study, and qualitative evidence of generalisation to two publications outside the training set.","tokens_in":14356,"tokens_out":4183,"duration_ms":37396,"significance":"If the reported performance holds up, this is a practically useful contribution to digital heritage and computational archaeology: it addresses a real bottleneck (legacy pottery drawings locked in printed publications), provides an open-source and modular tool with a UI aimed at non-programmers, and includes a self-annotation loop that can generate further training data. The time-saving and standardised-output aspects are directly relevant to archaeology workflows. The paper also ships code, documentation, and models on GitHub/HuggingFace, which supports reproducibility. The two external-context tests, while limited, are a commendable step beyond simple within-corpus validation.","major_comments":[{"comment":"The classifier training/validation split is ambiguous and potentially leaky. The text states that 4,563 base instances are manually augmented to 8 configurations each, giving 18,252 examples, and then \"80% of the images are used for training and the remaining 20% for validation.\" If the 80/20 split was applied over the 18,252 augmented images (as the phrasing suggests), the same original vessel will appear in both training and validation in different orientations, and the near-99% classification metrics in Table 3 would partly reflect recognition of already-seen vessels rather than classification of unseen pottery. Please clarify whether the split was stratified by base instance (or by publication) and report the number of distinct base vessels in each fold. This is load-bearing for the abstract's classification accuracy claim.","section":"§3.2.2, Table 3"},{"comment":"The generalisation claim is not supported by the reported numbers. Section 4.1 states that the model's performance on the two extra-training contexts (Morel 1981; Dyrdahl and Montalvo 2022) \"aligned closely with training results,\" but these contexts appear only as individual points on boxplots (Figure 7) with no numeric mAP, precision, or recall values. Without quantitative results for these two external publications, the assertion of \"robust generalisation capabilities\" in the Abstract is unsubstantiated. Please provide the actual metric values for each external context, or explicitly mark them as qualitative observations.","section":"§4.1, Figure 7"},{"comment":"The time-saving claim is inconsistent with the reported overall speed-up. The Abstract states that the framework reduces processing time \"by up to 5× to 20×,\" but Section 4.3.1 reports that the overall speed-up for the Ponte Nuovo case study was 4.7×, with 20× achieved only after excluding the Tabular information input phase. The lower bound of the claimed range (5×) is not met by the overall measured value. Please reconcile these numbers and present the speed-up as a range of per-step speed-ups, with the overall 4.7× clearly distinguished from the conditional 20× figure.","section":"§4.3.1, Abstract"}],"minor_comments":[{"comment":"The statement \"YOLOv8 has a higher Precision than YOLOv11 (≈ 10%)\" is numerically wrong: Table 2 shows box precision 0.972 vs. 0.966, a difference of about 0.6 percentage points, not 10%. Please correct this to avoid misleading readers.","section":"§4.1"},{"comment":"The boxplots in Figure 7 are described as showing \"model's validation performance metrics,\" but the figure caption does not specify what the boxes represent (e.g., per-page metrics, per-publication metrics, or per-annotation metrics). Adding axis labels and a clear caption would improve interpretability.","section":"§4.1, Figure 7"},{"comment":"The section heading reads \"Harware and Software used\"; this should be \"Hardware and Software used.\"","section":"§3.4"},{"comment":"The self-annotation module is described as enabling \"self-training,\" but it is actually a manual-correction-plus-export workflow that produces new training data for later fine-tuning. Using a term like \"human-in-the-loop annotation\" or \"iterative re-training\" would be more accurate and avoid confusion with self-supervised learning.","section":"§3.2.3"},{"comment":"The VAE case study is presented as a demonstration, but the architecture description is sparse (only the loss and β value are given). Since the paper does not claim this as a central contribution, a brief sentence pointing to a repository or prior publication would suffice.","section":"§4.3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially a systems/application paper. The two load-bearing issues—the classifier split ambiguity and the lack of numeric out-of-distribution results—are both fixable within the manuscript's scope. The time-saving inconsistency is also a straightforward correction. I do not see a fundamental methodological flaw that would require rejection, but the current level of reporting detail is not yet sufficient for the strong claims made in the abstract. The author's single-user manual-comparison design is a limitation worth noting in the revised text, but it is not disqualifying for a software-demonstration paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper delivers a working, open-source pipeline for a real bottleneck: turning legacy pottery drawings in PDFs into segmented, classified, reusable digital records. What's new is the integration itself—YOLO instance segmentation plus a multi-head EfficientNetV2 classifier, a Gradio UI, and a self-annotation loop—wrapped for archaeologists without programming skills. The code and models are on GitHub/HuggingFace, which is more than most archaeology papers ship. The detection results are plausible: YOLOv8/V11 box mAP50 near 0.99 and segmentation mAP50-95 around 0.89 on a hold-out set are consistent with what these models do on a single homogeneous class. That part of the paper is credible.\n\nThe soft spots are real, and the biggest one is the classification validation split. Section 3.2.2 augments each of 4,563 base vessels into 8 orientation variants, giving 18,252 images, then says 80% of the images are used for training and 20% for validation. If that split was made over the augmented images rather than stratified by base vessel—and the wording suggests it was—then the same physical drawing appears in both training and validation. The near-99% precision/recall for Type, Position, and Rotation then partly measure recognition of already-seen vessels, not classification of new pottery. This is a data leak, and it directly inflates the abstract's 'over 97% precision and recall in classification.' The authors need to re-run with a split at the vessel level and report whether the numbers hold. If they don't, the classification claim may fall to something more modest.\n\nOther issues are smaller but still worth fixing. The abstract's 'over 97%' is not met by every number in Table 2 (YOLOv11 precision is 0.966). The text claims YOLOv8 precision is '≈10%' higher than YOLOv11 when the table shows about 0.6 percentage points—sloppy. The 'robust generalisation' evidence consists of two extra-training contexts (Morel, Ecuador) shown only as boxplot points with no numeric metrics; we can't see how well they actually did. The 5–20x time saving comes from one author's self-comparison with no protocol, so treat it as suggestive. There are no confidence intervals or repeated runs anywhere.\n\nMy verdict: the detection core is sound and the framework is a real contribution to computational archaeology, but the classification results may be optimistically biased by the split issue, and the reporting overstates the evidence. This deserves a serious referee—it should not be desk-rejected. Send it to review with a strong request for vessel-level splits, numeric OOD metrics, corrected claims, and a more rigorous timing study. If the classification numbers survive re-evaluation, this becomes a very useful applied paper.","headline":"A useful open-source tool for pottery drawing digitisation, with solid detection results but a likely data leak in the classification validation split that inflates the headline numbers.","tokens_in":14978,"tokens_out":2054,"would_cite":true,"duration_ms":20378,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An open-source deep-learning framework digitizes archaeological pottery drawings at over 97% precision and recall.","keywords":["deep learning","archaeological pottery","instance segmentation","YOLO","EfficientNetV2","legacy data","digital heritage","open-source software"],"falsifier":"Run the released models on a complete previously unpublished pottery plate from a region and artistic tradition well outside the training corpus, with ground-truth masks drawn by an independent archaeologist, and compute per-page precision and recall; the generalization claim is falsified if either metric falls clearly below the claimed 97% on an ordinary page, or if the model systematically attaches vessel contours to scale bars, shading, or other non-vessel elements. A minimal version of the same test is to publish numeric mAP, precision, and recall for the two extra-training contexts (Morel 1981; Dyrdahl and Montalvo 2022) rather than boxplot positions.","tokens_in":13930,"feed_emoji":"🏺","tokens_out":16826,"duration_ms":128973,"temperature":0.7,"pith_summary":"The paper tries to establish that the laborious conversion of published pottery drawings into digital records can be automated without losing archaeological information. It introduces PyPotteryLens, an open-source pipeline that finds each vessel drawing on a scanned plate, cuts it out as a clean segmentation mask, classifies it as complete or fragmentary, standardizes its orientation, and exports the result as an image plus tabular metadata. The reported performance is precision and recall above 97% for detection and classification, with processing time reduced by 5× to 20× compared with manual recording, and the author argues that the approach generalizes to publications outside the training set. A reader should care because hundreds of thousands of legacy pottery illustrations are currently locked in print, and making them machine-readable would unlock larger comparative studies and new uses of deep learning in archaeology.","feed_headline":"97% precision: software digitizes old pottery plates","feed_subtitle":"Open-source tool digitizes scanned pottery plates at 97%+ precision and cuts documentation time by up to 20×.","key_machinery":"The load-bearing machinery is YOLO instance segmentation, a single-shot network that predicts both boxes and per-pixel masks, fine-tuned on 4,097 manually annotated pottery instances from 13 Italian publications, paired with a custom multi-head EfficientNetV2 classifier, a shared convolutional backbone with three specialized classification heads, trained on 18,252 augmented images that cover all completeness and orientation combinations. The YOLO model makes clean contour extraction possible, and the multi-head classifier standardizes orientation and completeness. Around these models, a browser-based graphical interface keeps a human in the loop for correction and validation, and a self-annotation module writes corrected masks back into YOLO training format, closing a feedback loop that is meant to let the detector improve on each user's own publication styles.","core_discovery":"The central claim is that a single-class, segmentation-first deep-learning pipeline can be a practical digitization tool for archaeological pottery, not just a research prototype. Rather than returning bounding boxes, the YOLO model traces the outline of each pottery drawing, so decorations and non-vessel elements are excluded from the extracted record; a multi-head EfficientNetV2 classifier then assigns each extracted vessel a completeness label (ENT/FRAG) and two orientation labels (TOP/BOTTOM, LEFT/RIGHT), so every record is presented consistently. The paper reports mAP50 near 0.99 for detection, segmentation precision and recall above 0.96, classification precision and recall above 0.98, and a speed-up of 4.7× on the full Ponte Nuovo workflow, or about 20× when tabular data entry is excluded. The Osteria dell'Osa demonstration is offered as evidence that the standardized output is clean enough to train an unsupervised variational autoencoder that clusters over 2,300 drawings in about half an hour.","pith_inferences":["A natural extension the paper leaves implicit is that a third out-of-corpus evaluation on a non-Western or non-Mediterranean publication tradition would be the strongest stress test, since the two current extra-training contexts are limited in number and are reported without per-context numeric precision or recall.","The self-annotation feedback loop could amplify an annotator's systematic blind spots: a model fine-tuned on user-corrected masks learns what the user chose to correct, not necessarily what is true.","Because the pipeline already outputs clean object masks rather than boxes, a natural next step not taken in the paper is to compute morphological vessel measurements directly from the masks, turning the digitization tool into a quantitative metrology tool.","The reported speed-up is measured against one experienced manual recorder, so the practical gain will vary with the user's manual proficiency and the structure of the source publication."],"forward_implications":["Legacy pottery plates can be bulk-processed into per-vessel image files with associated metadata, turning print-only corpora into queryable digital archives.","The archaeologist's role shifts from tracing and cutting out drawings to verifying and correcting machine output, with the paper reporting the largest time gains in report generation, post-processing, and card creation.","Extracted masks can be recycled as training data through the self-annotation module, allowing the detector to be fine-tuned on a user's own materials and publication styles.","Standardized, clean vessel images are directly usable as training input for other machine-learning analyses; the paper demonstrates this with an unsupervised VAE that clusters over 2,300 pottery drawings in about half an hour.","The modular design means the same pipeline of document processing, segmentation, classification, and export can be retrained for other artifact classes such as lithics or metal objects."],"supporting_citations":[{"why":"Introduces the YOLO single-shot detection architecture that the paper fine-tunes for pottery instance segmentation.","marker":"Redmon et al. (2016)"},{"why":"Provides the YOLOv8 and YOLOv11 implementation and training infrastructure used in the experiments.","marker":"Jocher, Qiu, and Chaurasia (2023)"},{"why":"Defines EfficientNetV2, the backbone of the multi-head classification model for completeness and orientation.","marker":"Tan and Le (2021)"},{"why":"Presents the PDF-analysis object-detection approach for archaeological catalogues that PyPotteryLens positions itself against by adding instance segmentation.","marker":"Klein et al. (2024)"},{"why":"Serves as one of the two out-of-training test corpora used to support the generalization claim.","marker":"Morel (1981)"},{"why":"Serves as the second out-of-training test corpus, from a pre-Columbian Ecuadorian context.","marker":"Dyrdahl and Montalvo (2022)"},{"why":"Surveys self-training, the basis of the self-annotation module that recycles corrected masks into new training data.","marker":"Amini et al. (2024)"},{"why":"Provides the comparative benchmark used to interpret mAP values, including the claim that segmentation mAP above 50% is very good.","marker":"Jegham et al. (2024)"},{"why":"Introduces the variational autoencoder used in the Osteria dell'Osa unsupervised-analysis demonstration.","marker":"Kingma and Welling (2013)"},{"why":"Introduces the Gradio web-interface framework that makes the models usable without programming skills.","marker":"Abid et al. (2019)"}],"fun_headline_variants":["Open-source AI digitizes pottery plates at 97% precision","AI tool cuts pottery documentation time by up to 20x","Deep learning framework automates pottery drawing digitization","PyPotteryLens: open-source deep learning for pottery plates","Pottery digitization: 97% precision, 20x faster with open-source"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the two publications not used in training are representative enough of diverse archaeological contexts and were labelled as reliably as the training data; the paper does not report numeric per-context metrics for them, and it concedes that unusual publication styles can make the model learn style rather than pottery.","fun_headline_variants_meta":{"raw":{"variants":["Open-source AI digitizes pottery plates at 97% precision","AI tool cuts pottery documentation time by up to 20x","Deep learning framework automates pottery drawing digitization","PyPotteryLens: open-source deep learning for pottery plates","Pottery digitization: 97% precision, 20x faster with open-source"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1509,"prompt_tokens":969,"completion_tokens":540,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":451}},"tokens_in":585,"tokens_out":540,"duration_ms":5207,"temperature":1.0,"reasoning_tokens":451,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:47:57.866309+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released models on a complete previously unpublished pottery plate from a region and artistic tradition well outside the training corpus, with ground-truth masks drawn by an independent archaeologist, and compute per-page precision and recall; the generalization claim is falsified if either metric falls clearly below the claimed 97% on an ordinary page, or if the model systematically attaches vessel contours to scale bars, shading, or other non-vessel elements. A minimal version of the same test is to publish numeric mAP, precision, and recall for the two extra-training contexts (Morel 1981; Dyrdahl and Montalvo 2022) rather than boxplot positions.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the YOLOv8 and YOLOv11 implementation and training infrastructure used in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as one of the two out-of-training test corpora used to support the generalization claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the second out-of-training test corpus, from a pre-Columbian Ecuadorian context."}],"review_version":1}