{"id":"c406b676-7716-413b-ae05-c88e9522144b","arxiv_id":"2505.03214","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new document annotation platform with a human-in-the-spiral workflow claims at least a 41% reduction in annotation time and improved extraction models across iterations.","lead":"DocSpiral is a web platform for annotating scanned documents that uses AI to propose labels and humans to correct them, with corrections fed back to improve the AI. The authors report that this loop cuts annotation time by at least 41% and improves extraction models with each round.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's mAP gains are unexplained because no held-out test set is specified; the spiral-improvement claim rests on this protocol detail.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the quantitative evidence for the central claim depends on an evaluation protocol that is not specified. The paper's strongest claim bundles two quantities, time reduction and mAP gains. The time reduction is plausible given pre-populated model outputs, but it comes from a single small study with no controls. The mAP gains are the only evidence that the iterative loop actually improves models, and without train/test separation they could simply reflect the model being evaluated on pages it has already seen. This is not an accusation of fraud; it is an ordinary correctness risk that the authors can resolve by releasing the evaluation protocol, the data split, or the code. The paper has genuine independent support in the form of a live demo, a freely accessible deployment, and a detailed system description, which makes the central platform claim credible enough to warrant conditional acceptance. The concern is addressable and does not by itself invalidate the system design, so the correct verdict remains conditional, unchanged from the reader's assessment.","tokens_in":66,"tokens_out":2859,"duration_ms":37359,"concrete_test":"Ask the authors to provide the exact evaluation protocol for Table 2: list the page IDs used for training and for mAP evaluation at each iteration and demonstrate that they are disjoint. Then independently recompute the final mAP using a fixed held-out set of pages never used for annotation or training, and repeat the experiment with a control that trains Faster-RCNN on the same number of pages drawn from the same document pool without the spiral's human-verified corrections; if the held-out gains are within noise of the control, the spiral-improvement claim is not supported. For the time claim, run a five-annotator study with counterbalanced order and report per-page times with standard deviations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim of \"consistent performance gains across three iterations during model training\" rests entirely on Table 2, a four-row Faster-RCNN mAP progression (0.053 -> 0.12 -> 0.21 -> 0.33). However, Section 4 gives no information about how the test set for these mAP numbers was constructed. Each iteration \"adds 100 new pages of data\" to training, but if the evaluation set is the same pool of annotated pages (or overlaps with the newly added data), the mAP increase is a trivial consequence of the model seeing the evaluation pages during training. The mAP numbers are also reported without confidence intervals, test-set size, or a comparison to a control condition that trains on the same number of randomly selected pages. This is not an internal inconsistency, but it is a correctness risk: the \"upward spiral\" may be a data-leakage artifact rather than a genuine property of the human-in-the-spiral workflow. The annotation-time reduction (28.4s vs 16.7s per page, 41%, with 75% on low-quality scans) is similarly under-specified: a single 90-page study, no annotator count, no counterbalancing for learning effects, no error bars. Both numbers support the central claim, and neither has a specified protocol; the mAP protocol is the more load-bearing because it is the only evidence that the loop improves models, not just that pre-filled predictions save typing time.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DocSpiral, a web-based annotation platform that integrates document format normalization (Anything2PDF), layout detection, OCR, table/figure/formula annotation, and model-development API endpoints into a single human-in-the-spiral workflow. The central idea is that human corrections on baseline model outputs generate training data that improve progressive models, which in turn reduce future manual effort. The authors report two quantitative results: a 41% overall annotation-time reduction (from 28.4s to 16.7s per page) in a 90-page experiment, and a Faster-RCNN layout-detection mAP progression from 0.053 to 0.33 across three training iterations, each adding 100 pages. The paper also provides a feature comparison table and a system-architecture description. The main contributions claimed are the comprehensive annotation interface, the assisted spiral improvement framework, and the open, deployable system.","tokens_in":8704,"tokens_out":2348,"duration_ms":24640,"significance":"If the human-in-the-spiral claims are substantiated, DocSpiral addresses a real pain point: curating training data for domain-specific, image-heavy document collections where existing tools stop at isolated annotation tasks. The platform design is constructive, and the paper's contribution as a systems paper is valuable: it unifies layout, OCR, table, figure, and formula annotation; supports custom schemas; and exposes RESTful APIs for model development, all in an openly accessible deployment. The software appears usable, and the freemium access lowers the barrier for researchers in geoscience and healthcare. However, the quantitative evidence is presently insufficient to support the headline claims, especially the mAP improvement, which is the only direct support for the 'spiral' feedback loop. The evaluation needs a clearly defined test protocol, controls, and uncertainty quantification before the contributions can be deemed valid.","major_comments":[{"comment":"The claim of 'consistent performance gains across three iterations' is not supported without specifying the evaluation protocol. The manuscript states that each iteration adds 100 new pages of data to training, but it does not state how the 90-page experiment relates to the mAP evaluation, how the test set was constructed, or whether the test pages overlap with the incrementally added training pages. If the mAP is measured on pages that were used for training in a later iteration, the improvement from 0.053 to 0.33 is a data-leakage artifact rather than evidence of an upward spiral. Please provide the full protocol: the split between training and test pages at each iteration, the independence of the test set from all training data, the number of evaluation pages, and confidence intervals or error bars. In addition, a control condition that trains on the same number of randomly selected pages (rather than pages selected through the human-in-the-spiral loop) is needed to attribute the gains to the workflow rather than to mere additional data.","section":"Section 4, Table 2"},{"comment":"The annotation-time comparison (28.4s vs. 16.7s per page, yielding 'at least 41%' reduction, and 75% for low-quality scans) is a single point estimate with no reported variance, number of annotators, or experimental design details. With only 90 pages, individual annotator skill and learning effects can easily shift the measured ratio. The manuscript does not describe whether the manual and assisted conditions were performed by the same annotators, whether tasks were counterbalanced, or whether the reported times include model-inference latency. Please report the number of annotators, the distribution of per-page times, a statistical significance test, and a clear definition of what constitutes 'assisted' versus 'manual' annotation, including how pre-filled predictions are scored.","section":"Section 4, timing experiment"},{"comment":"The 'human-in-the-spiral' claim is only partially tested by the mAP progression. The argument that the loop reduces manual intervention requires measuring the human correction effort per iteration (e.g., number of edits per page, time per page) as the progressive models improve. Table 2 shows model quality improving, but it does not show that the human workload decreases across iterations, which is the second half of the spiral. Without such a measurement, the abstract's statement that the platform makes models 'progressively require less manual intervention' is an extrapolation rather than an empirical result. Please add per-iteration measurements of annotation effort, or explicitly restrict the claim to model-performance gains.","section":"Section 4, spiral mechanism"},{"comment":"The Faster-RCNN training setup is under-specified. The manuscript does not state the backbone, hyperparameters, or how the 100 pages per iteration were selected and annotated. It also does not provide the mAP definition (e.g., COCO-style AP@IoU=0.5 or AP@[0.5:0.95]) or the number of classes and class distribution in the evaluation. Because the default layout classes in the platform are few (content, title, figure, table, formula, footnote), a small test set with imbalanced classes could make the reported mAP values highly variable. Please provide the exact evaluation metric, the test-set construction, and the per-class breakdown, and consider reporting multiple random seeds or trial repetitions.","section":"Section 4, Table 2 and Section 3.3"}],"minor_comments":[{"comment":"The claim 'at least 41% time reduction' is inconsistent with the single experimental point; 'at least' implies a lower bound across conditions, but the manuscript only reports one average. Please rephrase to 'a 41% reduction in our experiment' unless more data are provided.","section":"Abstract and Section 1"},{"comment":"The affiliation string contains a typo: 'Melborune' should be 'Melbourne'.","section":"Author affiliation"},{"comment":"The word 'domain-specfic' is misspelled; should be 'domain-specific'.","section":"Section 1"},{"comment":"The phrase 'you can speficy your own baseline model' contains a typo: 'speficy' should be 'specify'.","section":"Section 3.3"},{"comment":"The caption reads 'Open evaluation eetrics'; it should be 'Open evaluation metrics'.","section":"Figure 5 caption"},{"comment":"The table claims DocSpiral supports 'Layout Ann.' etc., but the binary checkmarks do not indicate the depth or quality of support. A brief note in the caption on how the capabilities were verified would strengthen the comparison.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a systems/software contribution, and the platform itself seems useful and timely. However, the evaluation section is thin for a journal submission: two experiments, no uncertainty quantification, no control, and no independent replication. The mAP protocol gap is serious enough that the central contribution cannot be accepted as stated. The authors should be encouraged to add a proper evaluation protocol, including a held-out test set, a random-data control, per-iteration workload measurements, and error bars. If that is done, the work could become a solid systems paper; as it stands, the evidence supports a platform demo but not the claimed spiral effect. I have no concerns about citation practices or novelty disclosure beyond the somewhat strong 'first' claim, which should be softened given the existence of other human-in-the-loop annotation tools."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead DocSpiral last night. The platform part is genuinely solid: an integrated annotation pipeline covering layout, OCR, tables, figures, and formulas, with a dynamic form schema and an iterative human-in-the-spiral retraining loop. That combination is not, as far as I can tell from the cited tools, present in any single prior system. The live demo and the API design are concrete and useful. If I were setting up a document-annotation effort for a scanned-report project, I would seriously look at this.\n\nThe evaluation, though, is much thinner than the claims. The 41% time reduction is one 90-page study with no annotator count, no error bars, no counterbalancing for learning effects. The 75% on low-quality scans is a nice datapoint but same caveats. The Faster-RCNN mAP table (0.053 to 0.33) is the only evidence for the upward-spiral claim, and it is genuinely under-specified: no stated held-out test set, no confidence intervals, no control condition training on random pages. The stress-test note is right: the mAP gains could be a data-leakage artifact if the added training pages overlap with the evaluation pool. The paper needs to say explicitly how test pages were chosen and separated. This is the load-bearing detail, not a nitpick.\n\nThat said, the shortcomings are fixable and the central idea is not broken. The platform is designed thoughtfully, and the demo gives me reason to believe the workflow works even if the numbers are preliminary. The authors also make a fair point that they plan to open-source once the system stabilizes; that softens the reproducibility concern, though it does not remove it.\n\nSo my take: this is a systems/demo paper. It should not be held to the standard of an experimental ML paper, but it should stop overstating. Drop 'at least 41%' to 'a single pilot study suggests a 41% reduction,' and add a paragraph on evaluation protocol. That is a normal revision, not a rejection.\n\nWho is this for? Anyone building annotation tooling for image-heavy document workflows, especially in geoscience or healthcare. It deserves a serious referee: the demo works, the comparisons are useful, and the spiral idea is worth a look. Send it to review, but ask for the evaluation details before you publish.\n\nBest,\n[you]","headline":"DocSpiral is a real system contribution with a plausible 'first' claim for its integrated workflow, but its headline numbers are thin and need a protocol before they are trustable.","tokens_in":9212,"tokens_out":1352,"would_cite":false,"duration_ms":15416,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Human-in-the-spiral annotation cuts document labeling time by 41%.","keywords":["document annotation","human-in-the-loop","layout detection","OCR verification","iterative model training","scanned document processing","structured data extraction","assistive annotation"],"falsifier":"Reproduce the Faster-RCNN training with a strict held-out test set that is never seen during the three rounds; if mAP does not rise as in Table 2, or declines when test pages are re-randomized, the spiral's performance gain is an artifact. Similarly, run a counterbalanced annotation study with fresh annotators assigned to manual and assisted conditions; if the assisted condition does not show a significant per-page time reduction, the 41% claim is refuted.","tokens_in":8273,"feed_emoji":"🔄","tokens_out":5572,"duration_ms":46208,"temperature":0.7,"pith_summary":"The paper introduces DocSpiral, a web-based annotation platform for image-based documents such as scanned reports. Its central claim is that a \"human-in-the-spiral\" workflow, where humans verify and correct the outputs of baseline models and those corrections train progressively better models, reduces manual annotation time by at least 41% and improves model performance with each cycle. The motivation is that domain-specific documents like geological reports and medical records are often scanned images, and extracting structured data from them currently demands heavy human labeling. If the claim holds, DocSpiral offers a practical route to cheaper, faster creation of training data for document-understanding models in fields where such data are scarce.","feed_headline":"Human-in-the-spiral annotation cuts document labeling time by 41%.","feed_subtitle":"An iterative human-model loop improves document extraction models while cutting annotation effort.","key_machinery":"The central mechanism is the human-in-the-spiral loop: an iterative cycle in which uploaded documents are converted to PDF and processed by baseline models, human annotators review and correct the outputs through an interactive web interface with dynamic form generation, the verified annotations become training data, and retrained progressive models are fed back into the pipeline to reduce the human effort needed in the next round. This loop is what differentiates DocSpiral from one-shot annotation tools; the platform's other components, such as format normalization, customizable layout schemas, a metrics dashboard, and API endpoints, support the loop rather than replace it.","core_discovery":"DocSpiral's central discovery is that a closed loop between human annotation and model training makes document annotation self-improving. Starting from baseline models for layout detection, OCR, and table, formula, and figure processing, the platform lets experts review, correct, and augment model outputs. The corrected data are then used to train or fine-tune progressive models, which in turn produce better initial outputs for the next round of human review. In a 90-page annotation experiment, this assistance cut per-page processing time from 28.4 seconds to 16.7 seconds, a 41% reduction, and up to 75% for low-quality scans. Separate training runs with a Faster-RCNN detector over three iterations, adding 100 pages each round, raised layout-detection mAP from 0.053 to 0.33, demonstrating the upward performance spiral.","pith_inferences":["If the 41% figure generalizes, the cost model for creating domain-specific document datasets would shift from raw labeling effort to expert review capacity, making small expert teams viable for large collections.","The spiral design implies a concrete testable hypothesis: the marginal time saving per page should increase with each iteration and approach a floor set by the ceiling of baseline-model accuracy; tracking per-round review time would verify this.","A potential hidden risk is feedback-loop bias: if progressive models are trained on the same pages whose corrections seed them, the reported mAP gains could reflect memorization rather than true generalization, making the paper's omitted test-set protocol the first thing to check.","The architecture could generalize beyond documents: any image-to-structured-output task with a measurable model, such as medical imaging annotations, could adopt the same human-in-the-spiral loop."],"forward_implications":["Domain teams can bootstrap document extraction from scanned collections with limited initial labels, since baseline models provide a first pass and only corrections require human effort.","Each deployment round compounds: more annotated data yields better progressive models, which lower the cost of annotating the next batch.","The same platform can serve rule-based fixes, deep-learning fine-tuning, and LLM alignment, because the annotation schema and API endpoints are model-agnostic.","Open, freely accessible hosting lowers the entry barrier for geoscience, healthcare, and other document-intensive fields to adopt LLM-based extraction.","Dynamic form generation lets a single tool handle diverse output formats, including LaTeX, HTML, JSON, and descriptive text, for tables, formulas, and figures."],"supporting_citations":[{"why":"Provides the Faster-RCNN detector whose three-round training demonstrates the spiral's performance gains.","marker":"Ren et al., 2016"},{"why":"Supplies the DocLayout-YOLO baseline that seeds layout detection for human correction.","marker":"Zhao et al., 2024d"},{"why":"Supplies PaddleOCR, the baseline OCR model whose outputs annotators verify and correct.","marker":"PFCCLab, 2020"},{"why":"Label Studio is the primary open comparison point for annotation interfaces the paper argues are insufficient.","marker":"Tkachenko et al., 2020"},{"why":"PAWLS represents the PDF-specialized annotation baseline that DocSpiral extends.","marker":"Neumann et al., 2021"},{"why":"Supplies the human-in-the-loop background that motivates shifting from offline labeling to iterative assistance.","marker":"Wu et al., 2022"}],"fun_headline_variants":["DocSpiral cuts document annotation time by 41%","Human-in-the-spiral loop makes annotation 41% faster","DocSpiral boosts layout detection mAP 6x","DocSpiral cuts annotation time 75% on poor scans","Spiral human-AI loop cuts annotation time 41%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 41% time savings and steady mAP gains depend on a controlled evaluation where the mAP reflects a held-out test set not contaminated by the incrementally added training pages, and where the manual-versus-assisted time comparison isolates annotator skill and learning effects; the paper does not spell out this protocol.","fun_headline_variants_meta":{"raw":{"variants":["DocSpiral cuts document annotation time by 41%","Human-in-the-spiral loop makes annotation 41% faster","DocSpiral boosts layout detection mAP 6x","DocSpiral cuts annotation time 75% on poor scans","Spiral human-AI loop cuts annotation time 41%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001343,"raw_usage":{"total_tokens":5452,"prompt_tokens":933,"completion_tokens":4519,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":4435}},"tokens_in":549,"tokens_out":4519,"duration_ms":28984,"temperature":1.0,"reasoning_tokens":4435,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:56:07.592688+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the Faster-RCNN training with a strict held-out test set that is never seen during the three rounds; if mAP does not rise as in Table 2, or declines when test pages are re-randomized, the spiral's performance gain is an artifact. Similarly, run a counterbalanced annotation study with fresh annotators assigned to manual and assisted conditions; if the assisted condition does not show a significant per-page time reduction, the 41% claim is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies PaddleOCR, the baseline OCR model whose outputs annotators verify and correct."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Label Studio is the primary open comparison point for annotation interfaces the paper argues are insufficient."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PAWLS represents the PDF-specialized annotation baseline that DocSpiral extends."}],"review_version":1}