{"id":"e6c49282-9171-404b-911c-35316378fd94","arxiv_id":"2507.10053","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CoSMo, a lightweight multimodal Transformer, segments comic book page streams into semantic sections and outperforms larger zero-shot vision-language models on a new 20,800-page dataset.","lead":"This paper introduces CoSMo, a Transformer that automatically divides scanned comic books into stories, advertisements, and cover pages, and releases a 20,800-page annotated dataset. It shows a vision-only model is nearly as accurate as one that also reads text, with multimodal input helping only on ambiguous pages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported SOTA numbers are not yet verifiable: Section 6.4 gives training details but no train/validation/test split, so CoSMo's F1-Macro 98.10 / PQ 95.08 could reflect in-sample memorization rather than generalization to new comic books.","rationale":"I read the paper's central contribution as the first comic PSS method with SOTA numbers. The reader's weakest assumption—manual label quality—is plausible but not the place the argument is most fragile: labels are at least described as manually annotated and quality-checked, and the paper's qualitative examples show genuine annotation. The largest unstated condition for the headline claim is that the metrics come from a held-out set. Section 6.4 never defines one; Section 5 never says how 430 books are partitioned. Because the model operates on full page streams, a training/evaluation overlap would be especially damaging: the transformer could memorize per-book transitions, and 'First-Page' boundaries are precisely the kind of infrequent, context-dependent event that memorization would inflate. The same issue affects every baseline comparison, so the 'consistently outperforms' claim is globally at risk. A concrete book-level cross-validation with per-fold variance would settle this cleanly. If results survive such an evaluation, I see no other objection that blocks the central claim; model size, zero-shot comparisons, and dataset construction are acceptable. Hence I keep the conditional verdict and ask for the split and reproducibility package.","tokens_in":9446,"tokens_out":4087,"duration_ms":51099,"concrete_test":"Obtain the public GitHub repository, locate or reconstruct the exact split used for Tables 1–3. If no split file exists, run a book-level 5-fold cross-validation (stratify 430 books, keep books intact across folds), retrain CoSMo Multimodal and Vision-Only with the Section 6.4 settings, and report mean ± std of F1-Macro, Accuracy, PQ, and MnDD per fold. If the cross-validated F1-Macro for Multimodal drops below 97 or the 95% interval overlaps Vision-Only, the superiority claim needs qualification. Also report the corresponding split for baselines to ensure fair comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim—CoSMo consistently outperforms baselines and larger VLMs—depends on the reported metrics being measured on a held-out set. The paper never states one. Section 5 describes the 430-book/20,800-page dataset, and Section 6.4 describes loss, learning rate, and early stopping, but no train/validation/test split, no per-book grouping, no number of books per split, and no random seed or repeated runs. Tables 1–3 report single numbers without provenance. This matters especially because CoSMo is a Transformer over entire book streams: if any book used for training also appears in evaluation, the model has seen all page-level embeddings and positional context for that book, and the sequential attention can memorize transitions. The 'First-Page' derived label makes this worse, since story boundaries are exactly the kind of local structure that can be memorized from training books. Because all comparisons (XGBoost, linear probe, zero-shot Qwen, CoSMo variants) are reported on the same unspecified evaluation set, every headline number inherits this risk. The concern is not that the authors are hiding results; it is that, as written, the protocol is incomplete and the central claim is unverifiable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CoSMo, a transformer-encoder architecture for page stream segmentation (PSS) in comic books, with vision-only and multimodal variants. The authors formalize the task, curate a 430-book/20,800-page dataset labeled into Cover, Advertisement, Text Story, Story, and the derived First-Page classes, and report experiments in which CoSMo outperforms XGBoost, linear probes, zero-shot CLIP/SigLIP, and zero-shot Qwen2.5-VL-32B across F1-Macro, Panoptic Quality (PQ), and the stream-level MnDD metric. The central claim is that a lightweight, task-specific sequential model beats larger general-purpose vision-language models on this task.","tokens_in":9726,"tokens_out":4681,"duration_ms":52936,"significance":"If the reported results withstand scrutiny, CoSMo would be a strong new baseline for comic PSS, and the released dataset would be a useful public resource for a previously absent domain in the PSS literature. The paper's strengths include a clear task formalization, a modular architecture description, systematic ablations over visual backbones and fusion strategies, and the stated intention to release code and data. The finding that visual features dominate while text helps resolve ambiguous cases is plausible and actionable. The main weakness is that the evaluation protocol is incompletely specified, so the generalization claims are currently unverifiable.","major_comments":[{"comment":"The paper never specifies the train/validation/test split: it does not state the number of books per split, whether splits are performed at book level, or whether a book used for training can also appear in evaluation. Because CoSMo encodes the full book stream with positional encoding and attention over the entire sequence, any overlap between training and test books would permit memorization of page-transition structure, including the derived First-Page boundaries. All headline numbers (e.g., F1-Macro 98.10, PQ 95.08, MnDD 0.437 in Table 2) therefore lack provenance as measures of generalization. Please state the exact split, guarantee book-level separation, and report results over multiple seeds with mean and standard deviation.","section":"Section 6.4; Tables 1-3"},{"comment":"First-Page is described as a derived label marking the first page in a narrative block, but the derivation protocol is not given and no annotation-quality metrics are reported. The paper mentions 'manual annotation' and 'quality checks' without inter-annotator agreement, label error rate, or adjudication details. Since First-Page is consistently the hardest class and stream-level metrics depend on segment boundaries, systematic ambiguity or noise in this derived label propagates into every comparison in Tables 2 and 3. Please provide the annotation protocol and a quantitative assessment of label reliability.","section":"Section 5; First-Page label"},{"comment":"All experimental results are single numbers without variance. Given the class imbalance (Section 5) and the modest differences between some variants, such as multimodal versus vision-only CoSMo (F1-Macro 98.10 vs 97.30; PQ 95.08 vs 94.50), error bars or repeated-run statistics are needed to support the claims of consistent improvement and of the multimodal variant being 'best overall.' Report mean and standard deviation over at least three seeds or a book-level bootstrap.","section":"Section 6.4 and Tables 1-3"}],"minor_comments":[{"comment":"The abstract and the introductory summary say CoSMo outperforms 'significantly larger general-purpose vision-language models' without consistently adding that the LLM comparisons are zero-shot. Section 6.1 is explicit about zero-shot evaluation, so the high-level statements should carry the same qualifier to avoid overstatement.","section":"Abstract and Section 1"},{"comment":"The two Multimodal rows in Table 2 are not labeled by fusion strategy, even though Section 6.2 describes two distinct strategies, fused and multitoken. Add explicit row labels so the reader can map the table to the described variants.","section":"Table 2; Section 6.2"},{"comment":"The MnDD formula as printed, MnDD = N - sum_{i,j} max_i |Gi ∩ Pj|, is not well-formed because the maximization index appears over the same index used in the summation. Please align the notation with reference [8].","section":"Equation (4)"},{"comment":"The training protocol lists only the learning rate and early stopping. Please also report the optimizer, batch size, number of epochs, and the exact inverse-frequency weighting used in the cost-sensitive cross-entropy loss.","section":"Section 6.4"},{"comment":"The manuscript uses 'lightweight' to describe CoSMo without giving parameter counts. Add parameter counts for the vision-only and multimodal variants to support the comparison with Qwen2.5-VL-32B.","section":"Section 4.1; Section 7.1"},{"comment":"The single-page CoSMo evaluation is reported without describing how a sequence-trained model is applied to an isolated page. Specify whether the page is fed as a one-page sequence, whether context is masked, or whether a separate inference procedure is used.","section":"Section 7.1"},{"comment":"The introduction states that CoSMo has 'an impressive error rate of approximately 1%,' but Table 2 reports accuracy of 98.65% for the best model, corresponding to a 1.35% error rate. Please reconcile the wording with the reported numbers.","section":"Section 1 and Table 2"}],"recommendation":"major_revision","confidential_remarks":"The missing train/validation/test split is the main editor-level concern. I would not recommend rejection: the architecture and dataset are useful, and the missing protocol details are fixable within the manuscript's scope. The paper's related-work framing relies heavily on the authors' own LoCU framework and resources; this is not a correctness problem, but the novelty claim would be strengthened by an external evaluation or an independent baseline comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the first credible formalization of page stream segmentation for comic books, with a genuinely new 430-book/20,800-page dataset and sensible stream-level metrics. But the evaluation never states a train/validation/test split, so the reported SOTA numbers—F1-Macro 98.10, PQ 95.08—could be in-sample. That's not a minor omission; it is the difference between a benchmark you can trust and a claim you can't check.\n\nWhat's actually good: the task framing with Cover, Advertisement, Text Story, Story, and derived First-Page labels is right and fills a real gap. The dataset, manually curated from public archives and aligned to comics.org, looks like a useful resource even if label quality is only asserted. The architecture is standard—frozen SigLIP features plus a four-layer transformer—but the paper doesn't oversell it. The ablations across backbones and modalities are informative, and the finding that vision dominates while text helps on ambiguous First-Page cases is a fair, non-obvious empirical point. The GitHub release is the right move.\n\nThe soft spots are real. Most importantly, Section 6.4 gives learning rate, loss, and early stopping but no split, no seeds, no per-book grouping, no repeated runs. Tables 1–3 report single numbers. Since CoSMo is a transformer over entire book streams, any book appearing in both training and evaluation allows the model to memorize story transitions—exactly the local structure First-Page labels capture. All baselines are evaluated on the same unspecified set, so every comparison inherits this risk. The stress-test note lands.\n\nSmaller issues: no inter-annotator agreement or error rate for the manual labels; the First-Page definition is described casually; the LLM baselines are zero-shot only, which is fine as a comparison but shouldn't be phrased as consistent outperformance. The detection-feature experiment uses a manga-trained detector, which the authors acknowledge may hurt.\n\nNet: the task and dataset are genuinely useful and the paper deserves a careful referee. But I would not accept the results as stated. The authors need to report the split, ideally release a commit-pinned data split and seeds, and rerun the numbers. With that fixed, this becomes a straightforward acceptance.\n\nFor you: worth watching, and I'd cite the dataset if the split gets resolved. Bring to reading group if you want a concrete case of why evaluation protocol matters.","headline":"Useful new task and dataset for comic-book page stream segmentation, but the missing train/test split makes the headline numbers unverifiable as written.","tokens_in":10200,"tokens_out":2226,"would_cite":true,"duration_ms":25504,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A compact Transformer that splits comic books into stories, ads, and covers reaches 98.10 F1-Macro on a new 20,800-page benchmark.","keywords":["Page Stream Segmentation","comic book understanding","multimodal Transformer","sequence labeling","document layout analysis","vision-language model","metadata enrichment","panoptic quality"],"falsifier":"Have several independent annotators re-label a random sample of about 50 books with the same protocol, measure agreement specifically on First-Page boundaries, and retrain and evaluate CoSMo on majority-vote labels. If F1-Macro or Panoptic Quality falls materially, or if annotators disagree on a large share of story starts, the reported state of the art is partly an artifact of one annotation protocol.","tokens_in":9301,"feed_emoji":"📚","tokens_out":9134,"duration_ms":96701,"temperature":0.7,"pith_summary":"The paper asks whether a small, task-specific Transformer can automatically divide scanned comic books into their semantic sections—stories, advertisements, text stories, and covers—well enough to replace manual metadata annotation. It answers yes: CoSMo, trained on a newly curated 20,800-page dataset of 430 books, reaches 98.10 F1-Macro and 95.08 Panoptic Quality in its multimodal form, and 97.30 F1-Macro in a cheaper vision-only form. The authors report that both variants beat much larger zero-shot vision-language models and traditional layout-feature baselines. The practical point is that the first stage of comic-book content understanding, knowing where one story ends and the next begins, can be automated at scale.","feed_headline":"Small model beats 32-billion-parameter AI at comic segmentation","feed_subtitle":"CoSMo splits scanned comic books into stories, ads and covers at about 98 percent F1, easing manual metadata tagging.","key_machinery":"The load-bearing mechanism is a dual-token Transformer encoder: each page is represented by a frozen pretrained visual embedding and, in the multimodal variant, a text embedding of structured OCR, and the two tokens are interleaved so the encoder can attend over the whole page sequence. Four Transformer layers with absolute positional encoding model narrative flow, and only one token per page feeds a three-layer classification head. A cost-sensitive cross-entropy loss reweights classes by inverse frequency. The same architecture with only visual tokens forms the lightweight vision-only variant; the design point is that decisions come from cross-page context rather than page-local features alone.","core_discovery":"The central discovery is that comic-book page-stream segmentation can be recast as multiclass sequence labeling over a page stream, with each page assigned one of Cover, Advertisement, Text Story, Story, or First-Page, the last being a derived label for the first page of a narrative block. A compact encoder-only Transformer that attends across the whole book solves this task to near-ceiling accuracy on the curated dataset: the multimodal variant reaches 98.10 F1-Macro, 95.08 Panoptic Quality, and a minimum-drags metric of 0.437, while the vision-only variant is nearly as strong. The authors interpret this as evidence that visual layout and style carry most of the segmentation signal, with text helping specifically on ambiguous pages such as ads that resemble stories or first pages with weak title cues. They also position the work as the first method that ingests an entire volume, in contrast to prior single-page or panel-sequence comic understanding work.","pith_inferences":["The dataset is drawn from one public-domain Golden Age archive; whether the roughly 98 percent F1 transfers to modern comics, manga, or European albums is untested and would be a natural domain-shift study.","A minimum-drags value below one suggests a human annotator would need fewer than one drag-and-drop correction per book on average, but how sensitive that metric is to label noise is not examined.","A concrete extension would be fine-tuning on a small sample of modern books to measure transfer; the paper's own failure cases already point to atypical first pages and ad-like mixed content as the error clusters to attack.","Since First-Page is a derived label, publishing the exact derivation protocol is what would make the benchmark reproducible; a different story-boundary definition would change labels and headline numbers."],"forward_implications":["If the reported accuracy holds, archives can auto-generate story-boundary metadata for backlogs of scanned comic books at roughly one percent error instead of tagging pages by hand.","A task-specific model with a small encoder outperforming a much larger zero-shot vision-language model suggests that general-purpose models are not the cheapest route for structured document segmentation.","The strong vision-only results imply that OCR is not required for most pages, so segmentation can run cheaply where text extraction is costly or impossible.","Reliable page-stream segmentation unlocks downstream analysis, character tracking, story indexing, and metadata enrichment, on story blocks rather than on whole issues.","First-Page detection is the hardest class and the main source of residual error; further gains depend on better long-range context modeling for story-boundary cues."],"supporting_citations":[{"why":"Supplies the unified page-stream-segmentation evaluation protocol and the document-level and stream-level metrics used in the experiments.","marker":"[5]"},{"why":"Defines the minimum-drag-and-drop stream metric and document-level F1 that measure how useful the predictions are in real annotation workflows.","marker":"[8]"},{"why":"Defines panoptic segmentation quality, which the paper adapts to evaluate page-segment recognition and boundary quality together.","marker":"[7]"},{"why":"Provides the large zero-shot vision-language baseline that CoSMo must beat and supplies the OCR engine used to extract page text features.","marker":"[1]"},{"why":"Provides the text embedding model that converts the OCR output into the page-level textual representation used by the multimodal variant.","marker":"[17]"},{"why":"Earlier page-stream-segmentation formulation for document analysis that grounds the task formalization this paper extends to comics.","marker":"[3]"}],"fun_headline_variants":["Tiny transformer reads entire comic book, nails page roles","CoSMo: Small model, 98% F1 on comic page segmentation","Comic segmentation: compact transformer beats giant VLMs","New model parses full comic volumes with 98% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results rest on the assumption that the manually curated, archive-aligned labels, especially the derived First-Page class, are accurate and consistent across all 20,800 pages; quality checks are reported, but no inter-annotator agreement or annotation-error rate is given, so systematic label noise would propagate into every headline number.","fun_headline_variants_meta":{"raw":{"variants":["Tiny transformer reads entire comic book, nails page roles","CoSMo: Small model, 98% F1 on comic page segmentation","Comic segmentation: compact transformer beats giant VLMs","New model parses full comic volumes with 98% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2706,"prompt_tokens":875,"completion_tokens":1831,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":1759}},"tokens_in":491,"tokens_out":1831,"duration_ms":13258,"temperature":1.0,"reasoning_tokens":1759,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:40:12.992820+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have several independent annotators re-label a random sample of about 50 books with the same protocol, measure agreement specifically on First-Page boundaries, and retrain and evaluate CoSMo on majority-vote labels. If F1-Macro or Panoptic Quality falls materially, or if annotators disagree on a large share of story starts, the reported state of the art is partly an artifact of one annotation protocol.","supporting_citations":[{"cited_title":"Openpss: An open page stream segmentation benchmark","cited_arxiv_id":null,"evidence_quote":"Supplies the unified page-stream-segmentation evaluation protocol and the document-level and stream-level metrics used in the experiments."},{"cited_title":"Tab this folder of documents: Page stream seg- mentation of business documents","cited_arxiv_id":null,"evidence_quote":"Defines the minimum-drag-and-drop stream metric and document-level F1 that measure how useful the predictions are in real annotation workflows."},{"cited_title":"Panoptic segmentation","cited_arxiv_id":null,"evidence_quote":"Defines panoptic segmentation quality, which the paper adapts to evaluate page-segment recognition and boundary quality together."},{"cited_title":"Qwen3 em- bedding: Advancing text embedding and reranking through foundation models, 2025","cited_arxiv_id":null,"evidence_quote":"Provides the text embedding model that converts the OCR output into the page-level textual representation used by the multimodal variant."},{"cited_title":"Document classification and page stream segmentation for digital mailroom applications","cited_arxiv_id":null,"evidence_quote":"Earlier page-stream-segmentation formulation for document analysis that grounds the task formalization this paper extends to comics."}],"review_version":1}