REVIEW 4 major objections 4 minor 37 references
A Transformer Based Handwriting Recognition System Jointly Using Online and Offline Features
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Fusing glyph images with pen trajectories before classification achieves state-of-the-art handwriting recognition.
desk verdict A genuinely new early-fusion architecture, but the paper's own tables contradict the headline SOTA claim; the numbers need reconciliation before the central finding can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the HATCharClassifier and its cross-modal querying step. A frozen Swin-B patch encoder turns the glyph image into 49 visual tokens; a lightweight rotary-position-encoded transformer embeds the $(x, y, \text{pen})$ stroke sequence; learnable latent queries first compress the image tokens via Perceiver-style cross-attention; then the stroke tokens attend to those latent image tokens once, and the result is pooled and passed to a linear classifier. Early fusion happens in that single cross-attention step before any stroke-only or image-only classification layer.
What would settle it
Run HTR-VT, OLHTR, and CNN-LSTM on HAT's exact character-level train/validation/test splits and preprocessing, and compare on the same test set; if any baseline reaches or exceeds 96.4% on IAMOn-DB or 95.8% on VNOn-DB, the early-fusion advantage claim collapses. Separately, the ISI-Air test set is listed as undefined while Table 2 reports ISI-Air accuracy, so a reader should verify which samples form the ISI-Air test set.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that early fusion—mixing offline image tokens and online stroke tokens in a shared latent space before task-specific transformer layers—yields state-of-the-art isolated character recognition across English, Vietnamese, and air-written digit datasets. The HATCharClassifier reports 96.4% accuracy on IAMOn-DB versus 95.3% for the late-fusion OLHTR baseline, and 95.8% on VNOn-DB, the first dual-input benchmark on that dataset, versus 95.3% for an image-only CNN-LSTM. The same network also degrades gracefully when one modality is removed at test time, losing only 4–7% accuracy, which the paper reads as evidence that the fused representation is robust to sensor failure.
Load-bearing premise
The claimed state-of-the-art margins assume the published baselines were trained and evaluated under the same character-level preprocessing, splits, and protocol as HAT; the paper does not demonstrate that comparability.
Editorial extensions
If this is right
- If correct, early fusion is a better design than late fusion for isolated character recognition, since HAT's dual-input accuracy beats the late-fusion OLHTR baseline on IAMOn-DB.
- The model converges in three epochs with 94.4M parameters, suggesting a frozen image backbone plus cross-modal querying is an efficient recipe for multimodal handwriting recognition.
- The same architecture handles image-only, stroke-only, and dual-input modes without parameter changes, and degrades gracefully when a modality is missing at test time.
- On VNOn-DB, stroke-only input drops to 72.1%, while adding image input restores 95.8%, indicating image context is essential for diacritic-rich scripts but the fusion still helps over image alone.
- On ISI-Air, the dual-input model reaches 99.8% accuracy, outperforming the RNN-LSTM baseline by 1.1% and showing the approach transfers to air-writing data.
Reading between the lines
- If the early-fusion margin survives a controlled comparison on identical splits and preprocessing, the same latent-query design is a natural candidate for word- and line-level recognition, where temporal and visual cues are richer and harder to align.
- The modality-dropout robustness suggests the shared latent space encodes redundant cues; one testable extension is measuring accuracy under partial stroke occlusion or blurred images to see where the redundancy breaks.
- The three-epoch convergence hints that the frozen Swin-B image tokens act as a strong prior; a lighter trainable backbone with the same fusion could reveal whether the gain comes from the fusion itself or from the backbone's strength.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents HATCharClassifier, a transformer-based isolated handwritten-character recognizer that fuses offline image patches with online pen-stroke sequences in a shared latent space using learnable latent queries and cross-modal attention before classification. The model is evaluated in image-only, stroke-only, and dual-input modes on IAMOn-DB, VNOn-DB, and ISI-Air, reporting state-of-the-art accuracy with claimed gains of up to 1% over prior work. The paper also provides ablations over image backbones, a convergence comparison, a fusion-level comparison, and a modality-dropout robustness study.
Significance. If its empirical claims hold, the paper provides a clean demonstration that early fusion of stroke and image tokens via cross-modal querying can beat late-fusion baselines for isolated character recognition, and the reported convergence speed is practically attractive. The architecture is specified in enough detail to be reimplemented, and the ablation and robustness tables are useful. However, the central quantitative claims are currently compromised by internal numerical contradictions and by unanswered questions about baseline comparability, so the significance is conditional on resolution of those issues.
major comments (4)
- [§5, Table 6 and Table 2] Table 6 reports Dual→Dual accuracy of 92.4% for the dual-trained HAT model on IAMOn-DB, while Table 2 reports HATD accuracy of 96.4% on the same dataset; no difference in split, checkpoint, training schedule, or evaluation protocol is stated to reconcile the 4.0-point gap. The Table 6 deltas (−4.3 and −6.7) are consistent with a 92.4% baseline, so the contradiction is specifically with Table 2. This is load-bearing because the claimed state-of-the-art margin over OLHTR (95.3%) reverses if 92.4% is the correct full-input accuracy; the authors must state which number is correct and why.
- [§5, text after Table 2] The results paragraph claims a 'mean 1.5% improvement in accuracy across all modes' and a '3.8% gain' for image-only mode on IAMOn-DB. Table 2 shows HATI at 91.5% versus HTR-VT at 95.3% (a 3.8-point loss), HATS at 89.5% versus LSTM at 90.7%, and HATD at 96.4% versus OLHTR at 95.3%; the mean change across the three IAMOn-DB modes is about −1.3 points, not +1.5. The sentence must be corrected to match the table.
- [§5, Table 2 and §4.1] The comparisons to HTR-VT, LSTM, OLHTR, and CNN-LSTM do not establish that those baselines were trained and evaluated under the same character-level preprocessing, train/validation/test partitions, and character-isolation protocol as HAT. Table 1 reports post-preprocessing character counts for the HAT experiments, but no analogous protocol is given for the cited baselines; HTR-VT [24] is a line-level recognizer in its original form. Without this information the reported state-of-the-art margins, including the VNOn-DB and ISI-Air comparisons, are not verifiable.
- [§4.1, Table 1 vs §5, Table 2] Table 1 lists the ISI-Air test set as '-' (no test instances), yet Table 2 reports four accuracy/precision/recall/F1 rows for ISI-Air. A defined test split is required for these numbers to be meaningful or reproducible; the paper should state how the test set was constructed and how the RNN-LSTM baseline [33] was evaluated under the same protocol.
minor comments (4)
- [§5, abstract] The claims 'exceeding previous bests by up to 1%' and 'state-of-the-art' are not supported for VNOn-DB, which has no dual-input baseline in Table 2; the only datasets with same-mode baselines are IAMOn-DB (+1.1 for HATD vs OLHTR) and ISI-Air (+1.1 for HATD vs RNN-LSTM).
- [Abstract] 'Our code can be found here' provides no URL or repository identifier, so the reproducibility claim cannot be checked.
- [Throughout] There are several typos: 'IAM0n-DB' in Section 6, 'IAM-OnDB' in the Figure 6 caption, 'deisgn' in Section 1, and 'Lattent' in Figure 2.
- [Table 4] Table 4 reports HAT (Fusion) accuracy as 96.42 while Table 2 reports 96.4; the two-decimal convention should be aligned across tables.
Circularity Check
No derivation-level circularity: central IAM/VN SOTA comparisons use external baselines and are empirical; the only same-institution reference is the ancillary ISI-Air baseline, which is not load-bearing. A Table 2/Table 6 accuracy discrepancy is flagged as a correctness issue, not circularity.
full rationale
The paper's claimed derivation is an architecture (Eqs. 1-10) plus empirical evaluation; there is no step in which a prediction is defined in terms of its input or in which a fitted parameter is renamed as a prediction. The main IAMOn-DB and VNOn-DB claims compare against external published systems (HTR-VT [24], LSTM [13], OLHTR [37], CNN-LSTM [21]), not against values produced by the model itself, so the central SOTA statement is not forced by construction. The only self-referential element is the ISI-Air experiment: the dataset and the RNN-LSTM baseline [33] originate from the same institution as the present authors (ISI / Umapada Pal). That reference is an already-published external baseline rather than a model-derived output, and it supports only the secondary air-writing demonstration, not the paper's central early-fusion claim on IAM/VN, so it does not constitute load-bearing circularity. Separately, the manuscript contains an internal numerical inconsistency: Table 6 reports Dual->Dual accuracy as 92.4% while Table 2 reports HATD as 96.4% on IAMOn-DB, and Section 5's stated 'mean 1.5% improvement' and '3.8% gain' are not supported by Table 2's numbers. This is a reporting/comparability defect that undermines confidence in the exact margins, but it is not a circularity: it does not show that an output equals its input by definition. Under the rubric, the correct finding is no significant circularity, with a minor same-institution citation reducing the score to 2.
Assumptions & free parameters
free parameters (7)
- latent dimension d =
256
- number of latent queries L =
64
- input image resolution =
224x224
- image backbone =
Swin-B 224, trainable (94.48M params)
- label smoothing =
0.1
- dropout rate =
0.1
- weight decay =
0.01
assumptions (5)
- domain assumption The cited baseline accuracies were obtained under the same character-level protocol as HAT.
- domain assumption A pretrained Swin-B backbone provides useful visual features for isolated handwritten characters.
- domain assumption The representation (x, y, pen-state) is sufficient to capture online handwriting information.
- domain assumption Perceiver-style latent cross-attention can mediate useful fusion between modalities.
- domain assumption IAMOn-DB and VNOn-DB ground-truth character segmentations are correct.
Cite this review
Pith. "Pith review of A Transformer Based Handwriting Recognition System Jointly Using Online and Offline Features." pith.science (2026). https://pith.science/paper/I2QL2PH2
@misc{pith2026250620255,
author = {Pith},
title = {Pith review of: A Transformer Based Handwriting Recognition System Jointly Using Online and Offline Features},
year = {2026},
howpublished = {\url{https://pith.science/paper/I2QL2PH2}},
note = {Machine review of arXiv:2506.20255}
}
abstract
We posit that handwriting recognition benefits from complementary cues carried by the rasterized complex glyph and the pen's trajectory, yet most systems exploit only one modality. We introduce an end-to-end network that performs early fusion of offline images and online stroke data within a shared latent space. A patch encoder converts the grayscale crop into fixed-length visual tokens, while a lightweight transformer embeds the $(x, y, \text{pen})$ sequence. Learnable latent queries attend jointly to both token streams, yielding context-enhanced stroke embeddings that are pooled and decoded under a cross-entropy loss objective. Because integration occurs before any high-level classification, temporal cues reinforce each other during representation learning, producing stronger writer independence. Comprehensive experiments on IAMOn-DB and VNOn-DB demonstrate that our approach achieves state-of-the-art accuracy, exceeding previous bests by up to 1\%. Our study also shows adaptation of this pipeline with gesturification on the ISI-Air dataset. Our code can be found here.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[24]
Pattern Recognition158, 110967 (2025)
Li, Y., Chen, D., Tang, T., Shen, X.: Htr-vt: Handwritten text recognition with vision transformer. Pattern Recognition158, 110967 (2025)
work page 2025
-
[33]
SN Computer Science2(1), 20 (2021)
Rahman, A., Roy, P., Pal, U.: Air writing: Recognizing multi-digit numeral string traced in air using rnn-lstm architecture. SN Computer Science2(1), 20 (2021)
work page 2021
-
[1]
AlKendi, W., Gechter, F., Heyberger, L., Guyeux, C.: Advancements and chal- lengesinhandwrittentextrecognition:Acomprehensivesurvey.JournalofImaging 10(1) (2024).https://doi.org/10.3390/jimaging10010018,https://www.mdpi. com/2313-433X/10/1/18
-
[2]
Ansel, J., Yang, E., He, H., Gimelshein, N., Jain, A., Voznesensky, M., Bao, B., Bell, P., Berard, D., Burovski, E., Chauhan, G., Chourdia, A., Constable, W., Desmaison, A., DeVito, Z., Ellison, E., Feng, W., Gong, J., Gschwind, M., Hirsh, B., Huang, S., Kalambarkar, K., Kirsch, L., Lazos, M., Lezcano, M., Liang, Y., Liang, J., Lu, Y., Luk, C.K., Maher, B...
arXiv 2024
-
[3]
In: Proceedings eighth international workshop on frontiers in handwriting recognition
Bahlmann, C., Haasdonk, B., Burkhardt, H.: Online handwriting recognition with support vector machines-a kernel approach. In: Proceedings eighth international workshop on frontiers in handwriting recognition. pp. 49–54. IEEE (2002)
work page 2002
-
[4]
In: Eighth International Conference on Doc- ument Analysis and Recognition (ICDAR’05)
Bai, Z.L., Huo, Q.: A study on the use of 8-directional features for online hand- written chinese character recognition. In: Eighth International Conference on Doc- ument Analysis and Recognition (ICDAR’05). pp. 262–266. IEEE (2005)
work page 2005
-
[5]
Neural computation7(6), 1289–1303 (1995)
Bengio, Y., LeCun, Y., Nohl, C., Burges, C.: Lerec: A nn/hmm hybrid for on-line handwriting recognition. Neural computation7(6), 1289–1303 (1995)
work page 1995
-
[6]
Information Fusion57, 1–14 (2020)
Bhunia, A.K., Mukherjee, S., Sain, A., Bhunia, A.K., Roy, P.P., Pal, U.: Indic handwritten script identification using offline-online multi-modal deep network. Information Fusion57, 1–14 (2020)
work page 2020
Show all 37 references
-
[7]
IEEE Transactions on pattern analysis and machine intelligence11(1), 68–83 (1989)
Bozinovic, R.M., Srihari, S.N.: Off-line cursive script word recognition. IEEE Transactions on pattern analysis and machine intelligence11(1), 68–83 (1989)
1989
-
[8]
IEEE Transactions on Multimedia26, 5769–5781 (2023) 14 A
Chen, Y., Zheng, H., Li, Y., Ouyang, W., Zhu, J.: Online handwritten chinese char- acter recognition based on 1-d convolution and two-streams transformers. IEEE Transactions on Multimedia26, 5769–5781 (2023) 14 A. Lodh, R. Chakraborty et al
2023
-
[9]
arXiv preprint arXiv:2010.11929 (2020)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
2020 arXiv
-
[10]
In: Second InternationalConferenceonDocumentImageAnalysisforLibraries(DIAL’06).pp
Feng, S., Manmatha, R., McCallum, A.: Exploring the use of conditional random field models and hmms for historical handwritten document recognition. In: Second InternationalConferenceonDocumentImageAnalysisforLibraries(DIAL’06).pp. 8–pp. IEEE (2006)
2006
-
[11]
In: Proceedings of the 23rd international conference on Machine learning
Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd international conference on Machine learning. pp. 369– 376 (2006)
2006
-
[12]
IEEE transactions on pattern analysis and machine intelligence31(5), 855–868 (2008)
Graves, A., Liwicki, M., Fernández, S., Bertolami, R., Bunke, H., Schmidhuber, J.: A novel connectionist system for unconstrained handwriting recognition. IEEE transactions on pattern analysis and machine intelligence31(5), 855–868 (2008)
2008
-
[13]
Greff, K., Srivastava, R.K., Koutník, J., Steunebrink, B.R., Schmidhuber, J.: Lstm: Asearchspaceodyssey.IEEEtransactionsonneuralnetworksandlearningsystems 28(10), 2222–2232 (2016)
2016
-
[14]
In: Proceedings of the 12th IAPR International Conference on Pattern Recognition, Vol
Guyon, I., Schomaker, L., Plamondon, R., Liberman, M., Janet, S.: Unipen project of on-line data exchange and recognizer benchmarks. In: Proceedings of the 12th IAPR International Conference on Pattern Recognition, Vol. 3-Conference C: Sig- nal Processing (Cat. No. 94CH3440-5)...
1994
-
[15]
He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
2016
-
[16]
Www.cedar.buffalo.edu/ilt/research.html
-
[17]
arXiv preprint arXiv:1508.01991 (2015)
Huang, Z., Xu, W., Yu, K.: Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015)
2015 arXiv
-
[18]
Pattern Recognition47(3), 969–981 (2014)
Impedovo, D., Pirlo, G.: Zoning methods for handwritten character recognition: A survey. Pattern Recognition47(3), 969–981 (2014)
2014
-
[19]
arXiv preprint arXiv:2107.14795 (2021)
Jaegle,A.,Borgeaud,S.,Alayrac,J.B.,Doersch,C.,Ionescu,C.,Ding,D.,Koppula, S., Zoran, D., Brock, A., Shelhamer, E., et al.: Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795 (2021)
2021 arXiv
-
[20]
In: International Conference on Document Analysis and Recognition
Jungo, M., Wolf, B., Maksai, A., Musat, C., Fischer, A.: Character queries: A transformer-based approach to on-line handwritten character segmentation. In: International Conference on Document Analysis and Recognition. pp. 98–114. Springer (2023)
2023
-
[21]
SN Computer Science1(1), 7 (2020)
Le, A.D., Nguyen, H.T., Nakagawa, M.: An end-to-end recognition system for un- constrained vietnamese handwriting. SN Computer Science1(1), 7 (2020)
2020
-
[22]
nature521(7553), 436–444 (2015)
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature521(7553), 436–444 (2015)
2015
-
[23]
Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z., Wei, F.: Trocr:Transformer-basedopticalcharacterrecognitionwithpre-trainedmodels.In: Proceedings of the AAAI conference on artificial intelligence. vol. 37, pp. 13094– 13102 (2023)
2023
-
[25]
Digital Signal Processing161, 105080 (2025) Joint Utilisation of Online and Offline Features for Handwriting Recognition 15
Li, Z., Zhao, H., Nishizaki, H., Leow, C.S., Shen, X.: Chinese character recognition based on swin transformer-encoder. Digital Signal Processing161, 105080 (2025) Joint Utilisation of Online and Offline Features for Handwriting Recognition 15
2025
-
[26]
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer:Hierarchical visiontransformerusingshiftedwindows.In:Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012–10022 (2021)
2021
-
[27]
In: Eighth International Conference on Document Analysis and Recognition (ICDAR’05)
Liwicki, M., Bunke, H.: Iam-ondb-an on-line english sentence database acquired from handwritten text on a whiteboard. In: Eighth International Conference on Document Analysis and Recognition (ICDAR’05). pp. 956–961. IEEE (2005)
2005
-
[28]
In: International Conference on Learning Representations (2019),https://openreview.net/forum? id=Bkg6RiCqY7
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019),https://openreview.net/forum? id=Bkg6RiCqY7
2019
-
[29]
International journal on document analysis and recogni- tion5, 39–46 (2002)
Marti, U.V., Bunke, H.: The iam-database: an english sentence database for offline handwriting recognition. International journal on document analysis and recogni- tion5, 39–46 (2002)
2002
-
[30]
In: 2018 16th International conference on frontiers in handwriting recognition (ICFHR)
Nguyen, H.T., Nguyen, C.T., Nakagawa, M.: Icfhr 2018–competition on vietnamese online handwritten text recognition using hands-vnondb (vohtr2018). In: 2018 16th International conference on frontiers in handwriting recognition (ICFHR). pp. 494–
2018
-
[31]
IEEE Access11, 94148–94172 (2023).https://doi.org/10.1109/ACCESS
Ott,F.,Rügamer,D.,Heublein,L.,Bischl,B.,Mutschler,C.:Auxiliarycross-modal representation learning with triplet loss functions for online handwriting recogni- tion. IEEE Access11, 94148–94172 (2023).https://doi.org/10.1109/ACCESS. 2023.3310819
2023
-
[32]
IEEE Transactions on pattern analysis and machine intelligence 22(1), 63–84 (2000)
Plamondon, R., Srihari, S.N.: Online and off-line handwriting recognition: a com- prehensive survey. IEEE Transactions on pattern analysis and machine intelligence 22(1), 63–84 (2000)
2000
-
[34]
IEEE transac- tions on pattern analysis and machine intelligence39(11), 2298–2304 (2016)
Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE transac- tions on pattern analysis and machine intelligence39(11), 2298–2304 (2016)
2016
-
[35]
Neurocomputing568, 127063 (2024)
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., Liu, Y.: Roformer: Enhanced trans- former with rotary position embedding. Neurocomputing568, 127063 (2024)
2024
-
[36]
Advances in neural information pro- cessing systems30(2017)
Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30(2017)
2017
-
[37]
In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Xu, Z., Chen, Z., Wu, Y., Li, H., Lv, W., Jin, L., Wang, Q.: A multi-scale bi- modal fusion network for robust and accurate online handwriting recognition. In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6460–6464. IE...
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.