{"id":"4fe5933b-307e-49e5-8491-81007da1ad10","arxiv_id":"2506.20255","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Early fusion of image patches and pen-stroke tokens in a transformer reaches 96.4% accuracy on isolated IAMOn-DB characters, slightly above the prior dual-input system.","lead":"This paper introduces HATCharClassifier, a transformer that recognizes handwritten characters by jointly using a scanned image of the glyph and the pen trajectory that produced it. The authors report top accuracy on three handwriting datasets, with the fused model beating prior systems by about one percentage point.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 6 reports Dual→Dual accuracy as 92.4% while Table 2 reports HATD as 96.4% on IAMOn-DB; without a stated protocol difference, the central SOTA claim is internally contradicted.","rationale":"I read the strongest claim as the early-fusion design beating late fusion on isolated character recognition, supported primarily by Table 2's dual-input numbers. The reader's weakest assumption was baseline comparability; that is a real issue, but the paper's own tables contain a sharp internal inconsistency that is even more decisive. Table 2 gives HATD 96.4% on IAMOn-DB, while Table 6's Dual→Dual row gives 92.4% for the same described task (full dual input after dual training). If the 92.4% number is correct, the headline SOTA claim fails by several points. The prose in Section 5 also contradicts Table 2 by claiming image-only gains and a positive mean improvement that are not present in the table. The lack of code, the undefined ISI-Air test split, and the absence of error bars compound the problem. Table 5's comparison of early vs late fusion also depends on the same unverified 96.4% figure. I am not objecting to the architecture itself; early fusion is a plausible idea. The issue is that the central quantitative evidence is internally incoherent as written. Thus the verdict should be unverified rather than conditional: the authors must reconcile Table 6 and Table 2 and provide reproducible artifacts before the claim can be assessed. Once that is done, the paper may be salvageable, but as submitted the SOTA claim is not supportable from the reported evidence.","tokens_in":10397,"tokens_out":5242,"duration_ms":58737,"concrete_test":"Obtain the released code and reproduce Table 2's HATD row on the Table 1 IAMOn-DB test split under the same protocol used for Table 6, then evaluate the same checkpoint with full dual input. If the reproduced accuracy is approximately 92.4% rather than 96.4%, the claimed SOTA margin of 1.1 points over OLHTR reverses. If the two tables intentionally use different protocols, the authors must report both configurations and re-derive the headline number accordingly.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The paper's central claim is that early fusion of image and stroke streams achieves state-of-the-art accuracy, with HATD at 96.4% versus OLHTR's 95.3% on IAMOn-DB (Table 2). But Table 6, described as evaluating the dual-trained model on full input, reports Dual→Dual accuracy of 92.4%, a 4.0-point gap. No difference in split, backbone, training schedule, checkpoint, or evaluation protocol is stated to reconcile these numbers. Table 6's own delta values (−4.3, −6.7) are consistent with a 92.4% baseline, so the inconsistency is specifically with Table 2. A second internal reporting error appears in the Section 5 text: it claims a mean 1.5% improvement across all modes and a 3.8% image-only gain, but Table 2 shows HATI at 91.5% versus HTR-VT at 95.3% (a 3.8% loss) and a mean change of about −1.3 accuracy points across the three modes. Since no code or data artifacts are provided, the reader cannot determine which number is correct. If 92.4% is the true full-input accuracy, the claimed margin over OLHTR reverses, and the case for early fusion over late fusion loses its main quantitative support. This internal contradiction is more load-bearing than the baseline-comparability issue because it cannot be resolved by external comparison alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HATCharClassifier, a transformer-based isolated handwritten-character recognizer that fuses offline image patches with online pen-stroke sequences in a shared latent space using learnable latent queries and cross-modal attention before classification. The model is evaluated in image-only, stroke-only, and dual-input modes on IAMOn-DB, VNOn-DB, and ISI-Air, reporting state-of-the-art accuracy with claimed gains of up to 1% over prior work. The paper also provides ablations over image backbones, a convergence comparison, a fusion-level comparison, and a modality-dropout robustness study.","tokens_in":10684,"tokens_out":5040,"duration_ms":49369,"significance":"If its empirical claims hold, the paper provides a clean demonstration that early fusion of stroke and image tokens via cross-modal querying can beat late-fusion baselines for isolated character recognition, and the reported convergence speed is practically attractive. The architecture is specified in enough detail to be reimplemented, and the ablation and robustness tables are useful. However, the central quantitative claims are currently compromised by internal numerical contradictions and by unanswered questions about baseline comparability, so the significance is conditional on resolution of those issues.","major_comments":[{"comment":"Table 6 reports Dual→Dual accuracy of 92.4% for the dual-trained HAT model on IAMOn-DB, while Table 2 reports HATD accuracy of 96.4% on the same dataset; no difference in split, checkpoint, training schedule, or evaluation protocol is stated to reconcile the 4.0-point gap. The Table 6 deltas (−4.3 and −6.7) are consistent with a 92.4% baseline, so the contradiction is specifically with Table 2. This is load-bearing because the claimed state-of-the-art margin over OLHTR (95.3%) reverses if 92.4% is the correct full-input accuracy; the authors must state which number is correct and why.","section":"§5, Table 6 and Table 2"},{"comment":"The results paragraph claims a 'mean 1.5% improvement in accuracy across all modes' and a '3.8% gain' for image-only mode on IAMOn-DB. Table 2 shows HATI at 91.5% versus HTR-VT at 95.3% (a 3.8-point loss), HATS at 89.5% versus LSTM at 90.7%, and HATD at 96.4% versus OLHTR at 95.3%; the mean change across the three IAMOn-DB modes is about −1.3 points, not +1.5. The sentence must be corrected to match the table.","section":"§5, text after Table 2"},{"comment":"The comparisons to HTR-VT, LSTM, OLHTR, and CNN-LSTM do not establish that those baselines were trained and evaluated under the same character-level preprocessing, train/validation/test partitions, and character-isolation protocol as HAT. Table 1 reports post-preprocessing character counts for the HAT experiments, but no analogous protocol is given for the cited baselines; HTR-VT [24] is a line-level recognizer in its original form. Without this information the reported state-of-the-art margins, including the VNOn-DB and ISI-Air comparisons, are not verifiable.","section":"§5, Table 2 and §4.1"},{"comment":"Table 1 lists the ISI-Air test set as '-' (no test instances), yet Table 2 reports four accuracy/precision/recall/F1 rows for ISI-Air. A defined test split is required for these numbers to be meaningful or reproducible; the paper should state how the test set was constructed and how the RNN-LSTM baseline [33] was evaluated under the same protocol.","section":"§4.1, Table 1 vs §5, Table 2"}],"minor_comments":[{"comment":"The claims 'exceeding previous bests by up to 1%' and 'state-of-the-art' are not supported for VNOn-DB, which has no dual-input baseline in Table 2; the only datasets with same-mode baselines are IAMOn-DB (+1.1 for HATD vs OLHTR) and ISI-Air (+1.1 for HATD vs RNN-LSTM).","section":"§5, abstract"},{"comment":"'Our code can be found here' provides no URL or repository identifier, so the reproducibility claim cannot be checked.","section":"Abstract"},{"comment":"There are several typos: 'IAM0n-DB' in Section 6, 'IAM-OnDB' in the Figure 6 caption, 'deisgn' in Section 1, and 'Lattent' in Figure 2.","section":"Throughout"},{"comment":"Table 4 reports HAT (Fusion) accuracy as 96.42 while Table 2 reports 96.4; the two-decimal convention should be aligned across tables.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The central problem is the unresolved 96.4 vs 92.4 discrepancy for the same reported model and dataset; if the authors confirm 92.4 as the full-input accuracy, the abstract's SOTA claim and the early-fusion advantage over OLHTR lose their main quantitative support. Because the discrepancy may be a reporting error, major revision is appropriate, but the revision must include protocol-level reconciliation and, ideally, code or data release to permit external checks. Given that baseline comparability is also unverified, the authors should be pressed to provide the exact evaluation setups of the cited baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis one has an interesting core and a load-bearing data problem. The early-fusion design—perceiver-style latent queries over image patches, then stroke tokens querying those latents—is a legitimately new arrangement for online/offline handwriting recognition, and the fusion-level ablation (early vs middle vs late) is exactly the experiment you'd want to see.\n\nBut the numbers do not hang together. Table 2 reports HATD at 96.4% on IAMOn-DB. Table 6, described as the dual-trained model on full input, reports 92.4% on the same dataset. That 4-point gap is not explained by any stated protocol difference. The results text then claims a '3.8% gain' for image-only mode, while Table 2 shows HATI at 91.5% versus HTR-VT at 95.3%—a 3.8-point loss, not a gain. The text also claims a mean 1.5% improvement across modes when the table averages to about -1.3 points. That is not a minor typo; it is the central quantitative evidence for early fusion being better than late fusion.\n\nOther issues: baseline comparability is not established (no evidence that HTR-VT, LSTM, OLHTR, or CNN-LSTM were trained on the same character-level splits and preprocessing), the ISI-Air test split is undefined (Table 1 says '-'), no code or data are provided despite the 'code can be found here' line, and there are no error bars or seeds.\n\nThe architecture and the convergence results are worth a look, and the limitations section is honest about scope. But as written, the empirical claims are not trustworthy. A referee should not have to guess which table is correct.\n\nRecommendation: don't take the SOTA claim at face value; send it to review with a strong request to reconcile the numbers, specify the ISI-Air protocol, and release the code and data. If the 96.4 holds after that, this is a solid contribution. If 92.4 is the real number, the main argument evaporates.","headline":"A genuinely new early-fusion architecture, but the paper's own tables contradict the headline SOTA claim; the numbers need reconciliation before the central finding can be trusted.","tokens_in":11282,"tokens_out":3929,"would_cite":false,"duration_ms":40303,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fusing glyph images with pen trajectories before classification achieves state-of-the-art handwriting recognition.","keywords":["handwritten text recognition","online-offline fusion","early fusion","transformer","cross-modal attention","character recognition","IAMOn-DB","VNOn-DB"],"falsifier":"Run HTR-VT, OLHTR, and CNN-LSTM on HAT's exact character-level train/validation/test splits and preprocessing, and compare on the same test set; if any baseline reaches or exceeds 96.4% on IAMOn-DB or 95.8% on VNOn-DB, the early-fusion advantage claim collapses. Separately, the ISI-Air test set is listed as undefined while Table 2 reports ISI-Air accuracy, so a reader should verify which samples form the ISI-Air test set.","tokens_in":10159,"feed_emoji":"✍️","tokens_out":3766,"duration_ms":34158,"temperature":0.7,"pith_summary":"This paper claims that handwriting recognition improves when the visual glyph and the pen trajectory are fused before any high-level classification, rather than after each modality is processed independently. It introduces HATCharClassifier, a transformer network that projects image patches and stroke points into a shared token space, lets stroke tokens query image tokens through cross-attention, and classifies from an attention-pooled embedding. On character-level benchmarks, the dual-input model reaches the highest reported accuracy on both IAMOn-DB and VNOn-DB, exceeding the previous best by about one percentage point each, and converges in three epochs where comparators need hundreds. The significance is that a simple early-fusion design can capture the correlation between how a character looks and how it is traced, making the system more writer-independent.","feed_headline":"Early fusion of images and pen strokes wins handwriting recognition","feed_subtitle":"A dual-input transformer reaches 96.4% and 95.8% accuracy and converges in three epochs, beating late-fusion rivals.","key_machinery":"The load-bearing object is the HATCharClassifier and its cross-modal querying step. A frozen Swin-B patch encoder turns the glyph image into 49 visual tokens; a lightweight rotary-position-encoded transformer embeds the $(x, y, \\text{pen})$ stroke sequence; learnable latent queries first compress the image tokens via Perceiver-style cross-attention; then the stroke tokens attend to those latent image tokens once, and the result is pooled and passed to a linear classifier. Early fusion happens in that single cross-attention step before any stroke-only or image-only classification layer.","core_discovery":"On the paper's own terms, the central discovery is that early fusion—mixing offline image tokens and online stroke tokens in a shared latent space before task-specific transformer layers—yields state-of-the-art isolated character recognition across English, Vietnamese, and air-written digit datasets. The HATCharClassifier reports 96.4% accuracy on IAMOn-DB versus 95.3% for the late-fusion OLHTR baseline, and 95.8% on VNOn-DB, the first dual-input benchmark on that dataset, versus 95.3% for an image-only CNN-LSTM. The same network also degrades gracefully when one modality is removed at test time, losing only 4–7% accuracy, which the paper reads as evidence that the fused representation is robust to sensor failure.","pith_inferences":["If the early-fusion margin survives a controlled comparison on identical splits and preprocessing, the same latent-query design is a natural candidate for word- and line-level recognition, where temporal and visual cues are richer and harder to align.","The modality-dropout robustness suggests the shared latent space encodes redundant cues; one testable extension is measuring accuracy under partial stroke occlusion or blurred images to see where the redundancy breaks.","The three-epoch convergence hints that the frozen Swin-B image tokens act as a strong prior; a lighter trainable backbone with the same fusion could reveal whether the gain comes from the fusion itself or from the backbone's strength."],"forward_implications":["If correct, early fusion is a better design than late fusion for isolated character recognition, since HAT's dual-input accuracy beats the late-fusion OLHTR baseline on IAMOn-DB.","The model converges in three epochs with 94.4M parameters, suggesting a frozen image backbone plus cross-modal querying is an efficient recipe for multimodal handwriting recognition.","The same architecture handles image-only, stroke-only, and dual-input modes without parameter changes, and degrades gracefully when a modality is missing at test time.","On VNOn-DB, stroke-only input drops to 72.1%, while adding image input restores 95.8%, indicating image context is essential for diacritic-rich scripts but the fusion still helps over image alone.","On ISI-Air, the dual-input model reaches 99.8% accuracy, outperforming the RNN-LSTM baseline by 1.1% and showing the approach transfers to air-writing data."],"supporting_citations":[{"why":"Supplies the IAMOn-DB online handwriting dataset used for the main English benchmark.","marker":"[27]"},{"why":"Supplies the VNOn-DB Vietnamese online handwriting dataset, where the paper reports its first dual-input results.","marker":"[30]"},{"why":"HTR-VT, the image-only transformer baseline whose 95.3% accuracy HAT improves upon on IAMOn-DB.","marker":"[24]"},{"why":"OLHTR, the late-fusion baseline that HAT's early fusion outperforms on IAMOn-DB.","marker":"[37]"},{"why":"CNN-LSTM, the image-only baseline whose 95.3% accuracy HAT surpasses on VNOn-DB.","marker":"[21]"},{"why":"Perceiver-IO, the latent cross-attention construction that HAT adapts for compressing image tokens.","marker":"[19]"},{"why":"Swin-B, the pretrained visual backbone whose features HAT's patch encoder uses and ablates.","marker":"[26]"}],"fun_headline_variants":["Early fusion of pen strokes and images tops handwriting benchmarks","Dual-input transformer achieves new handwriting accuracy records","Fusing online and offline features lifts handwriting recognition","Shared latent space boosts handwriting recognition accuracy","Early fusion of visual and stroke data wins handwriting AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed state-of-the-art margins assume the published baselines were trained and evaluated under the same character-level preprocessing, splits, and protocol as HAT; the paper does not demonstrate that comparability.","fun_headline_variants_meta":{"raw":{"variants":["Early fusion of pen strokes and images tops handwriting benchmarks","Dual-input transformer achieves new handwriting accuracy records","Fusing online and offline features lifts handwriting recognition","Shared latent space boosts handwriting recognition accuracy","Early fusion of visual and stroke data wins handwriting AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000289,"raw_usage":{"total_tokens":1664,"prompt_tokens":885,"completion_tokens":779,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":708}},"tokens_in":501,"tokens_out":779,"duration_ms":8871,"temperature":1.0,"reasoning_tokens":708,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:52:42.637356+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HTR-VT, OLHTR, and CNN-LSTM on HAT's exact character-level train/validation/test splits and preprocessing, and compare on the same test set; if any baseline reaches or exceeds 96.4% on IAMOn-DB or 95.8% on VNOn-DB, the early-fusion advantage claim collapses. Separately, the ISI-Air test set is listed as undefined while Table 2 reports ISI-Air accuracy, so a reader should verify which samples form the ISI-Air test set.","supporting_citations":[{"cited_title":"In: Eighth International Conference on Document Analysis and Recognition (ICDAR’05)","cited_arxiv_id":null,"evidence_quote":"Supplies the IAMOn-DB online handwriting dataset used for the main English benchmark."},{"cited_title":"In: 2018 16th International conference on frontiers in handwriting recognition (ICFHR)","cited_arxiv_id":null,"evidence_quote":"Supplies the VNOn-DB Vietnamese online handwriting dataset, where the paper reports its first dual-input results."},{"cited_title":"Pattern Recognition158, 110967 (2025)","cited_arxiv_id":null,"evidence_quote":"HTR-VT, the image-only transformer baseline whose 95.3% accuracy HAT improves upon on IAMOn-DB."},{"cited_title":"In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","cited_arxiv_id":null,"evidence_quote":"OLHTR, the late-fusion baseline that HAT's early fusion outperforms on IAMOn-DB."},{"cited_title":"SN Computer Science1(1), 7 (2020)","cited_arxiv_id":null,"evidence_quote":"CNN-LSTM, the image-only baseline whose 95.3% accuracy HAT surpasses on VNOn-DB."}],"review_version":1}