Pith. sign in

REVIEW 4 major objections 5 minor 72 references

HAND: Hierarchical Attention Network for Multi-Scale Handwritten Document Recognition and Layout Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single end-to-end network can transcribe and layout-parse handwritten documents from one line to a three-page spread, the paper claims, with state-of-the-art error reductions on the READ 2016 benchmark.

desk verdict The triple-page benchmark is genuinely new, but the headline HAND+mT5 results are unverified until the authors state whether the mT5 post-processor and the HAND feedback loop saw the READ 2016 test split. read the letter →

arxiv 2412.18981 v1 pith:EU7JKSAB submitted 2024-12-25 cs.CV cs.LG

Mohammed Hamdan , Abderrahmane Rahiche , Mohamed Cheriet This is my paper · ORCID
classification cs.CVcs.LG
keywords handwrittendocumentrecognitionlayoutanalysissegmentation-freehierarchicalattentionmulti-scaleprocessingcurriculumlearningmT5post-processinghistoricaldocuments
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a single end-to-end, segmentation-free neural network can simultaneously transcribe historical handwritten text and recover the document's layout structure, at scales from a single line up to a three-page spread, while using fewer parameters than existing page-level models. On the READ 2016 benchmark of Early Modern German manuscripts, HAND plus a fine-tuned mT5 post-processor is reported to cut character error to 1.65% at line level and 2.36% on single pages, reductions of 59.8% and 31.2% over the DAN baseline, and HAND is claimed to be the first system to process triple-page documents. If these numbers hold, a compact model of around 5.60M parameters can carry out joint transcription and layout analysis on entire historical documents without a separate segmentation stage.

What carries the argument

The central object is the HAND architecture: a fully convolutional encoder with gated depth-wise separable convolutions, octave convolutions, and squeeze-and-excitation blocks that maps a document image into a 2D feature map, plus a six-layer transformer decoder operating on flattened features. The MSAP framework is the mechanism that carries the multi-scale claim: a small complexity network scores each document and modulates feature extraction, query construction, and attention weighting so the same weights process a single line and a three-page column. The decoder's memory-augmented attention concatenates a learnable memory matrix with keys and values to retain global context, while sparse attention restricts computation to relevant local regions and adaptive feature fusion balances hierarchical levels. The post-processing stage is a fine-tuned mT5-Small language model that corrects residual character errors while layout tokens preserve document structure.

What would settle it

Retrain mT5 using only HAND predictions on the official train split and then evaluate on the official test split: if line-level CER exceeds 1.65%, the post-processor had memorized test-set ground truth. Alternatively, inspect the training-data preparation script to confirm that the test split was excluded from both mT5 fine-tuning and the retraining loop described in the appendix.

Watch

Extended reading notes

Core claim

The central claim is that the hierarchical attention network HAND solves handwritten document recognition and layout analysis jointly, across five complexity scales, using a curriculum that progresses from single lines to paragraphs, pages, double pages, and triple pages. The load-bearing novelty is the Multi-Scale Adaptive Processing (MSAP) framework: a complexity-scoring module estimates how hard a document is, then guides two passes of feature extraction and a hierarchical transformer decoder whose attention heads combine memory-augmented and sparse mechanisms with adaptive query generation. On top of this, a domain-adapted mT5 model is fine-tuned on pairs of HAND predictions and ground truth to correct residual errors in historical German script. The paper reports new state-of-the-art results on READ 2016 at every scale, including a line-level CER of 1.65% and a triple-page CER of 2.18%, with layout metrics also improving, all with a 5.60M parameter main model.

Load-bearing premise

The reported CER gains from the mT5 post-processor assume that the READ 2016 test split never fed into mT5's fine-tuning or into the iterative feedback-loop retraining of HAND, even though the paper never states that this separation was enforced.

Editorial extensions

If this is right

  • HAND+mT5 is the first reported system to transcribe triple-page documents end-to-end on READ 2016, reaching a CER of 2.18%.
  • The 5.60M-parameter model is roughly 20% smaller than DAN, Faster-DAN, and DANCER, while also being faster at inference, so joint layout and text recognition does not require a larger model.
  • Ablation results show that the five-level curriculum is essential for scaling: removing it raises triple-page CER from 2.18% to 69.05%.
  • Because HAND is segmentation-free, errors from a separate line-detection step cannot propagate into the transcription stage, which addresses a limitation identified in earlier two-stage pipelines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The mT5 post-processor's benefit shrinks as document size grows (92.27% error reduction at line level versus 15.00% at triple-page), which suggests that the base HAND already captures most linguistic context on long inputs, so further language-model patching will likely yield diminishing returns on even longer documents.
  • If the evaluation split is held out cleanly, the combination of complexity-aware two-pass decoding and language-model post-correction could transfer to other historical collections with few transcriptions, since the method leans heavily on synthetic fonts and curriculum pre-training rather than large annotated page corpora.
  • The MSAP complexity-scoring mechanism could be reused as a document-difficulty predictor for digitization workflows or for active learning, though this goes beyond what the paper evaluates.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HAND, an end-to-end, segmentation-free encoder-decoder architecture for simultaneous handwritten text recognition and layout analysis on historical documents. The architecture combines a convolutional encoder with gated depth-wise separable and octave convolutions, a Multi-Scale Adaptive Processing framework, a hierarchical attention decoder with memory-augmented and sparse attention, curriculum learning across five document scales, and an mT5-based post-processing stage. On READ 2016, the authors report state-of-the-art CER/WER at line, paragraph, single-page, double-page, and triple-page levels, with HAND+mT5 reducing line-level CER by 59.8% relative to DAN and page-level CER by 31.2%; they also report layout metrics and a compact 5.60M-parameter HAND model. The central claim is that HAND alone already beats prior end-to-end page-level models and that HAND+mT5 sets new benchmarks while being the first system to process triple-page documents.

Significance. If the clean-split results hold, the paper makes a solid empirical contribution: it demonstrates joint text recognition and layout analysis at scales (triple-page) not previously handled end-to-end, with a relatively small HAND model and publicly released code and pretrained models. The inclusion of machine-checkable artifacts and detailed algorithmic pseudocode is a strength. However, the headline accuracy gains are carried by the HAND+mT5 pipeline, and the manuscript as written does not rule out leakage of the READ 2016 test split into mT5 fine-tuning or into the iterative HAND retraining loop. That issue, together with the undefined complexity target and inconsistent error-rate tables, places the central empirical claims in need of substantial clarification and likely re-experimentation.

major comments (4)
  1. [Section V and Appendix VIII-D] The load-bearing claim that HAND+mT5 sets new state-of-the-art results depends on the mT5 post-processor and the iterative HAND refinement loop being trained without access to the READ 2016 test split. The paper states that paired (HADN prediction, ground truth) examples were generated on the READ 2016 dataset and that HADN was periodically retrained with corrected outputs from mT5, but it never states that test predictions and test ground truth were excluded from these steps. If any test-derived pairs entered fine-tuning or the feedback loop, the reported 59.8% line-level and 31.2% page-level CER reductions are not generalization measurements. Please specify the exact data split used for mT5 fine-tuning and for each round of HAND retraining, and rerun the evaluation if test data were used in any training step.
  2. [Table III versus Tables VI and VII] Table III reports base error rates of 68.66% for line-level, 89.21% for paragraph-level, and 100% for page-level inputs, with HAND+mT5 error rates of 5.31%, 17.35%, and 42.29% or higher. Tables VI and VII report HAND CER of 2.71% (line), 3.18% (paragraph), 3.41% (single-page), 3.46% (double-page), and 3.52% (triple-page), with HAND+mT5 CER between 1.65% and 2.36%. These two sets of numbers cannot describe the same metric; Table III appears to report a different, undefined error rate. Please define the metric in Table III, compute it consistently, and reconcile the table with the CER values used for the claimed relative reductions.
  3. [Equations (32)-(36)] The total loss in Eq. (32) includes the complexity loss L_c in Eq. (35), but C_target(x) is never defined anywhere in the manuscript; without a defined target complexity score, the MSE term in the complexity loss is vacuous. Similarly, L_layout in Eq. (33) is written as a generic cross-entropy over y_layout and p_layout, but the manuscript never specifies how these quantities are derived from the XML/graph annotations or how the decoder emits layout predictions. Please define C_target and connect L_layout to the concrete output representation and annotations; otherwise the joint loss and the complexity-aware weighting in Eq. (36) are not reproducible.
  4. [Abstract, Section VI-G, Table IV] The abstract and conclusion state that HAND maintains a compact size of 5.60M parameters, and Table IV lists HAND+mT5 as having 5.60M parameters. The system whose results are reported as HAND+mT5 also includes mT5-Small with roughly 300M parameters, so the total parameter count of the reported pipeline is about 305.6M. Please state both parameter counts separately and adjust the compactness claim so that it applies to HAND alone or to the full post-processed pipeline.
minor comments (5)
  1. [Throughout] The acronym HADN appears in multiple places (e.g., Algorithm 4 and Appendix VIII-D) where HAND is intended; please correct these occurrences.
  2. [Section V] The text contains an unresolved reference '(see Appendix??)' after the description of mT5 adaptation; this should be a specific appendix section.
  3. [Table VIII] In the row 'w/o Augmentation', the triple-page WER of 14.62% is inconsistent with the adjacent CER of 89.81%; please verify this entry.
  4. [Section VI-B] Given that the test sets are small (e.g., 15 triple-page documents), reporting confidence intervals or variance across runs would substantially strengthen the comparison.
  5. [References] Reference [9] lists 'M. Coquenet' while [10] lists 'D. Coquenet' for the same author group; please standardize the author names.

Circularity Check

1 steps flagged · score 4.0 of 10

mT5 post-processing and the HAND feedback loop are trained on READ 2016 prediction/ground-truth pairs with no stated exclusion of the test split, so the headline CER reductions are not demonstrably independent predictions.

  1. fitted input called prediction [Section V; Appendix VIII-D.3 'Training Data Preparation'; Appendix VIII-D.6]
    "Our process began with generating initial predictions on the READ 2016 dataset using HADN after 1000 epochs of training. We then created paired examples of (HADN prediction, ground truth) for each structural level, ranging from sentences to triple-column layouts. Lastly, we implemented an iterative refinement process, periodically retraining HADN with corrected outputs from mT5. This created a feedback loop that progressively improved both models, helping to tackle persistent error patterns."

    The fine-tuned mT5 is fitted to (HADN prediction, ground truth) pairs generated from the same READ 2016 benchmark on which the paper reports its headline results, and HAND is then retrained on mT5-corrected outputs in a feedback loop. The paper never states that the READ 2016 test predictions and test ground truth were excluded from mT5 fine-tuning or from the HAND retraining loop; Appendix VIII-D.6 even says the authors 'continuously evaluated' and 'iteratively refined' the model based on CER/WER/LOER/mAPCER evaluations. If any test-derived pair entered either stage, the reported 59.8% line-level and 31.2% single-page CER reductions are not generalization measurements but reflect fitting to the test transcriptions.

full rationale

The HAND architecture itself is a self-contained encoder-decoder design evaluated against external baselines (DAN, Faster-DAN, DANCER); no theoretical derivation in the paper reduces to its own inputs, and no load-bearing self-citation chain or imported uniqueness theorem is present. The only circularity concern is empirical: the mT5 post-processor is fine-tuned on paired (HAND prediction, ground truth) examples from READ 2016, and an iterative feedback loop retrains HAND with mT5-corrected outputs, while the headline CER reductions are reported on READ 2016. If standard train/validation/test discipline was followed for the mT5 stage and the HAND feedback loop, the results would be legitimate; the paper simply never documents that exclusion. Because the central claim depends on this unstated split, the circularity score is raised to 4 rather than 0. The separate 'compact 5.60M parameters' claim omits the 300M-parameter mT5-Small used in the reported system, but that is a reporting inconsistency, not circularity.

Assumptions & free parameters 5 free parameters · 6 assumptions · 3 invented entities

The central empirical claims rest on the integrity of the READ 2016 annotations, on the undefined complexity target in the loss, on an unstated specification for the layout loss, and on the assumption that mT5 post-processing used only training data. The free parameters are mostly hand-set hyperparameters and learned scaling constants with no reported values.

free parameters (5)
  • Positional encoding warmup schedule: alpha0, gamma, E_warmup = 0.1, 0.5, 150
    Hand-set constants in Eq. 26/46 that control how quickly positional encoding is blended into first-pass features; the central performance likely depends on them.
  • Attention balancing weights lambda_mem, lambda_sparse = 0.5, 0.5
    Initialized in Algorithm 3 to balance memory-augmented and sparse attention; no search or sensitivity analysis is reported.
  • mT5 post-processing loss weights alpha, beta, gamma = 0.6, 0.3, 0.1
    Weights for cross-entropy, cosine similarity, and layout loss in Eq. 52; described as empirically determined without a reported search.
  • Complexity scaling parameters gamma_alpha, gamma_beta, theta_alpha, theta_beta, delta_alpha, delta_beta = not reported (learned)
    Parameters in Eq. 28/48 that control complexity-dependent scaling of queries and attention; no values or initialization is given.
  • Complexity network dropout p_d = 0.2
    Dropout in the complexity assessment network, Eq. 43; chosen by hand.
assumptions (6)
  • domain assumption The READ 2016 dataset's transcripts and layout annotations are correct and consistently aligned for the constructed line, paragraph, page, and triple-page samples.
    All recognition and layout losses are supervised by these annotations; any noise or misalignment propagates into every reported metric (Section VI-A, Eq. 33).
  • ad hoc to paper The complexity target C_target used in the complexity loss exists and is well-defined.
    Eq. 35 defines Lc = ||C(x)-C_target(x)||^2, but no definition or source for C_target appears anywhere in the paper; without it the MSAP mechanism is under-specified.
  • ad hoc to paper The layout loss Llayout in Eq. 33 can be computed from the graph/XML representation, and the model outputs layout elements in a form that supports this loss.
    The paper presents a hierarchical graph and XML but never specifies how the decoder output is aligned to layout nodes or how the cross-entropy over layout classes is computed at page scale.
  • domain assumption mT5 post-processing and iterative self-training use only training data, so the test set remains unseen.
    The paper does not state whether test predictions or ground truth were excluded from mT5 training pairs and feedback retraining (Section V, Appendix VIII-D); the validity of the headline CER reductions depends on this.
  • domain assumption Consecutive pages combined into triple-page images are an authentic test of triple-column layout understanding.
    Triple-page samples were created by concatenating consecutive pages of the Ratsprotokolle collection; this artificial construction may not reflect true triple-page documents (Section VI-A, Appendix VIII-E).
  • standard math Standard transformer attention, backpropagation, and sequence-to-sequence training behave as assumed.
    The architecture relies on standard softmax attention and loss gradients without formal verification (Section III-B).
invented entities (3)
  • Complexity score C(x) and its target C_target
    purpose: Scalar in [0,1] that modulates feature selection, query generation, and attention scaling in MSAP; C_target is the supervision target for the complexity loss.
    The complexity score is only defined inside the model; no external, falsifiable prediction is made from it, and C_target is never specified.
  • Learnable memory matrix M in memory-augmented attention
    purpose: Concatenated to keys and values to carry global document context across decoding steps (Eq. 11).
    A learned parameter with no external handle; whether it stores meaningful document context is not tested beyond overall accuracy.
  • Sparse attention mask W
    purpose: Limits cross-attention to critical local regions for long sequences (Eq. 12).
    The construction and learning of W are not specified; no independent evidence of its effect is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HAND: Hierarchical Attention Network for Multi-Scale Handwritten Document Recognition and Layout Analysis." pith.science (2026). https://pith.science/paper/EU7JKSAB

@misc{pith2026241218981,
  author       = {Pith},
  title        = {Pith review of: HAND: Hierarchical Attention Network for Multi-Scale Handwritten Document Recognition and Layout Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EU7JKSAB}},
  note         = {Machine review of arXiv:2412.18981}
}
read the original abstract

Handwritten document recognition (HDR) is one of the most challenging tasks in the field of computer vision, due to the various writing styles and complex layouts inherent in handwritten texts. Traditionally, this problem has been approached as two separate tasks, handwritten text recognition and layout analysis, and struggled to integrate the two processes effectively. This paper introduces HAND (Hierarchical Attention Network for Multi-Scale Document), a novel end-to-end and segmentation-free architecture for simultaneous text recognition and layout analysis tasks. Our model's key components include an advanced convolutional encoder integrating Gated Depth-wise Separable and Octave Convolutions for robust feature extraction, a Multi-Scale Adaptive Processing (MSAP) framework that dynamically adjusts to document complexity and a hierarchical attention decoder with memory-augmented and sparse attention mechanisms. These components enable our model to scale effectively from single-line to triple-column pages while maintaining computational efficiency. Additionally, HAND adopts curriculum learning across five complexity levels. To improve the recognition accuracy of complex ancient manuscripts, we fine-tune and integrate a Domain-Adaptive Pre-trained mT5 model for post-processing refinement. Extensive evaluations on the READ 2016 dataset demonstrate the superior performance of HAND, achieving up to 59.8% reduction in CER for line-level recognition and 31.2% for page-level recognition compared to state-of-the-art methods. The model also maintains a compact size of 5.60M parameters while establishing new benchmarks in both text recognition and layout analysis. Source code and pre-trained models are available at : https://github.com/MHHamdan/HAND.

Figures

Figures reproduced from arXiv: 2412.18981 by the authors.

Figure 1
Figure 1. Hierarchical recognition and organization of the content of a triple-page document image. The left side illustrates [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Document recognition complexity across multiple scales: from line-level to triple-page documents. (a) Line level, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of the HAND Architecture: The HAND [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: An exemple of handwritten text recognition using [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Extended hierarchical structure of a triple-column [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 68 canonical work pages

  1. [1]

    Historical word-spotting in handwritten documents: The challenges,

    A. Fischer, V. Frinken, and H. Bunke, “Historical word-spotting in handwritten documents: The challenges,”International Con- ference on Frontiers in Handwriting Recognition , pp. 106–111, 2012

    work page 2012

  2. [2]

    A transformer-based neural network architecture for handwritten document analysis,

    T. Strauss, G. Leifert, T. Grüning, and R. Labahn, “A transformer-based neural network architecture for handwritten document analysis,” Pattern Recognition Letters, vol. 136, pp. 187–195, 2020

    work page 2020

  3. [3]

    Layout analysis for arabic historical document images using machine learning,

    S. S. Bukhari, T. M. Breuel, and F. Shafait, “Layout analysis for arabic historical document images using machine learning,” International Workshop on Historical Document Imaging and Processing, pp. 130–137, 2012

    work page 2012

  4. [4]

    Acomprehen- sivesurveyofmostlytextualdocumentsegmentationalgorithms since 2008,

    S.Eskenazi,P.Gomez-Krämer,andJ.-M.Ogier,“Acomprehen- sivesurveyofmostlytextualdocumentsegmentationalgorithms since 2008,” Pattern Recognition, vol. 64, pp. 1–14, 2017

    work page 2008

  5. [5]

    A survey on handwritten document understanding technique,

    S. Ahmed, M. I. Malik, M. Liwicki, and A. Dengel, “A survey on handwritten document understanding technique,” Pattern Recognition Letters, vol. 94, pp. 39–57, 2016

    work page 2016

  6. [6]

    Efficient text line segmentation for historical documents,

    C. Clausner, A. Hayes, and A. Antonacopoulos, “Efficient text line segmentation for historical documents,” in International Conference on Document Analysis and Recognition (ICDAR) . IEEE, 2019, pp. 723–728

    work page 2019

  7. [7]

    End-to-end handwritten text recognition and word spotting with deep neural networks,

    C. Wei, E. Boudreau, and R. Singh, “End-to-end handwritten text recognition and word spotting with deep neural networks,” Pattern Recognition Letters, vol. 129, pp. 158–165, 2020

    work page 2020

  8. [8]

    Recognition of historical documents with few labeled samples,

    F. Simistira, M. Seuret, N. Eichenberger, A. Garz, M. Liwicki, and R. Ingold, “Recognition of historical documents with few labeled samples,” in International Conference on Document Analysis and Recognition (ICDAR). IEEE, 2017, pp. 251–255

    work page 2017

Show all 72 references
  1. [9]

    DAN: A segmentation-free document attention network for handwritten document recognition,

    M. Coquenet, C. Chatelain, and T. Paquet, “DAN: A segmentation-free document attention network for handwritten document recognition,”IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 7, pp. 8229–8242, 2023

    2023

  2. [10]

    Faster dan: Multi-target queries with document positional encoding for end-to-end handwritten document recognition,

    D. Coquenet, C. Chatelain, and T. Paquet, “Faster dan: Multi-target queries with document positional encoding for end-to-end handwritten document recognition,”arXiv preprint arXiv:2301.10593, 2023

    2023 arXiv

  3. [11]

    DANCER: A compu- tationally efficient end-to-end model for handwritten document recognition,

    S. Castro, E. Vidal, and F. Casacuberta, “DANCER: A compu- tationally efficient end-to-end model for handwritten document recognition,” in Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR) , 2024

    2024

  4. [12]

    Pay attention to what youread: Non-recurrent handwritten text-line recognition,

    L. Kang, P. Riba, M. Rusi nol, A. Forn’es, and M. Villegas, “Pay attention to what youread: Non-recurrent handwritten text-line recognition,” Pattern Recognition, vol. 129, p. 108766, 2022

    2022

  5. [13]

    ollner, and T. Gr

    C. Wick, J. Z"ollner, and T. Gr"uning, “Transformer for hand- written text recognition using bidirectional post-decoding,” in International Conference on Document Analysis and Recogni- tion. Springer, 2021, pp. 112–126

    2021

  6. [14]

    Transformer-based approach for joint handwriting and named entity recognition in historical documents,

    A. C. Rouhou, M. Dhiaf, Y. Kessentini, and S. B. Salem, “Transformer-based approach for joint handwriting and named entity recognition in historical documents,”Pattern Recognition Letters, vol. 155, pp. 128–134, 2022

    2022

  7. [15]

    Resnest-transformer: Joint at- tentionsegmentation-freeforend-to-endhandwritingparagraph recognition model,

    M. Hamdan and M. Cheriet, “Resnest-transformer: Joint at- tentionsegmentation-freeforend-to-endhandwritingparagraph recognition model,” Array, vol. 19, p. 100300, 2023

    2023

  8. [16]

    On the computational complexity of self-attention,

    F. D. Keles, P. M. Wijewardena, and C. Hegde, “On the computational complexity of self-attention,” in International Conference on Algorithmic Learning Theory . PMLR, 2023, pp. 597–619

    2023

  9. [17]

    A practical survey on faster and lighter transformers,

    Q. Fournier, G. M. Caron, and D. Aloise, “A practical survey on faster and lighter transformers,”ACM Computing Surveys , vol. 55, no. 14s, pp. 1–40, 2023. THIS PAPER IS CURRENTLY UNDER REVIEW AT IEEE TRANSACTIONS. 13

    2023

  10. [18]

    Handwriting recognition with large multidimensional long short-term memory recurrent neural networks,

    P. Voigtlaender and H. Doetsch, Nay, “Handwriting recognition with large multidimensional long short-term memory recurrent neural networks,” in15th International Conference on Frontiers in Handwriting Recognition. IEEE, 2016, pp. 228–233

    2016

  11. [19]

    Jointlinesegmentationandtranscriptionforend-to- end handwritten paragraph recognition,

    T.Bluche,“Jointlinesegmentationandtranscriptionforend-to- end handwritten paragraph recognition,” inAdvances in neural information processing systems, 2016, pp. 838–846

    2016

  12. [20]

    Data augmentation for recognition of handwritten words and lines using a cnn-lstm network,

    C. Wigington, S. Stewart, B. Davis, B. Barrett, B. Price, and S. Cohen, “Data augmentation for recognition of handwritten words and lines using a cnn-lstm network,” in2017 14th IAPR International Conference on Document Analysis and Recogni- tion (ICDAR), vol. 1. IEEE, 2017, pp...

    2017

  13. [21]

    Scan, attend and read: End-to-end handwritten paragraph recognition with mdl- stm attention,

    T. Bluche, J. Louradour, and R. Messina, “Scan, attend and read: End-to-end handwritten paragraph recognition with mdl- stm attention,” 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1, pp. 1050– 1055, 2017

    2017

  14. [22]

    Have convolutions already made recurrence obsolete for uncon- strained handwritten text recognition?

    D. Coquenet, Y. Soullard, C. Chatelain, and T. Paquet, “Have convolutions already made recurrence obsolete for uncon- strained handwritten text recognition?” in2019 International Conference on Document Analysis and Recognition Workshops (ICDAR W), vol. 5. IEEE, 2019, pp. 65–70

    2019

  15. [23]

    uning, and J. Z

    J. Michael, R. Labahn, T. Gr"uning, and J. Z"ollner, “Evaluat- ing sequence-to-sequence models for handwritten text recogni- tion,” in 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, 2019, pp. 1286–1293

    2019

  16. [24]

    Recurrence-free unconstrained handwritten text recognition using gated fully convolutional network,

    D. Coquenet, C. Chatelain, and T. Paquet, “Recurrence-free unconstrained handwritten text recognition using gated fully convolutional network,” in2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR) . IEEE, 2020, pp. 19–24

    2020

  17. [25]

    Origaminet: Weakly-supervised, segmentation-free, one-step, full page text recognition by learn- ing to unfold,

    M. Yousef and T. E. Bishop, “Origaminet: Weakly-supervised, segmentation-free, one-step, full page text recognition by learn- ing to unfold,” in Proceedings of the conference on computer vision and pattern recognition, 2020, pp. 14710–14719

    2020

  18. [26]

    Trocr: Transformer-based optical character recognition with pre-trained models,

    M. Li, T. Lv, L. Cui, Y. Lu, D. Florencio, C. Zhang, Z. Li, and F. Wei, “Trocr: Transformer-based optical character recognition with pre-trained models,”arXiv arXiv:2109.10282, 2021

    2021 arXiv

  19. [27]

    Full page handwriting recognition via image to sequence extraction,

    S. Singh and S. Karayev, “Full page handwriting recognition via image to sequence extraction,” inInternational Conference on Document Analysis and Recognition. Springer,2021,pp.55–69

    2021

  20. [28]

    End-to-end hand- written paragraph text recognition using a vertical attention network,

    D. Coquenet, C. Chatelain, and T. Paquet, “End-to-end hand- written paragraph text recognition using a vertical attention network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 508–524, 2023

    2023

  21. [29]

    Refocus attention span networks for handwriting line recognition,

    M. Hamdan, H. Chaudhary, A. Bali, and M. Cheriet, “Refocus attention span networks for handwriting line recognition,”IJ- DAR, vol. 26, no. 2, pp. 131–147, Jun. 2023

    2023

  22. [30]

    Span: A simple predict and align network for handwritten paragraph recogni- tion,

    D. Coquenet, C. Chatelain, and T. Paquet, “Span: A simple predict and align network for handwritten paragraph recogni- tion,” in International Conference on Document Analysis and Recognition. Springer, 2021, pp. 70–84

    2021

  23. [31]

    Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,

    A. Graves, S. Fern’andez, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” inProceedings of the 23rd conference on Machine learning , 2006, pp. 369–376

    2006

  24. [32]

    Recent progress in deep learning for historical document processing,

    Y.-Y. Tang and C. Suen, “Recent progress in deep learning for historical document processing,”Pattern Recognition, vol. 112, p. 107749, 2021

    2021

  25. [33]

    Document layout analysis: A comprehensive survey,

    G. M. Binmakhashen and S. A. Mahmoud, “Document layout analysis: A comprehensive survey,”ACM Computing Surveys , vol. 52, no. 6, pp. 1–36, 2019

    2019

  26. [34]

    Learning to extract semantic structure from documents using multimodal fully convolutional neural networks,

    X. Yang, E. Yumer, P. Asente, M. Kraley, D. Kifer, and C. L. Giles, “Learning to extract semantic structure from documents using multimodal fully convolutional neural networks,” inPro- ceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4342–4351

    2017

  27. [35]

    dhsegment: A generic deep-learning approach for document segmentation,

    S. A. Oliveira, B. Seguin, and F. Kaplan, “dhsegment: A generic deep-learning approach for document segmentation,” in 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE, 2018, pp. 7–12

    2018

  28. [36]

    Multi-scale gated fully convolutional densenets for se- mantic labeling of historical newspaper images,

    Y. Soullard, P. Tranouez, C. Chatelain, S. Nicolas, and T. Pa- quet, “Multi-scale gated fully convolutional densenets for se- mantic labeling of historical newspaper images,”Pattern Recog- nition Letters, vol. 131, pp. 435–441, 2020

    2020

  29. [37]

    Lay- outlm: Pre-training of text and layout for document image understanding,

    Y. Xu, M. Li, L. Cui, S. Huang, F. Wei, and M. Zhou, “Lay- outlm: Pre-training of text and layout for document image understanding,” in Proceedings of the 26th ACM SIGKDD In- ternational Conference on Knowledge Discovery & Data Mining, 2020, pp. 1192–1200

    2020

  30. [38]

    Visual attention for multi-task visual ques- tion answering,

    C. Soto and S. Yoo, “Visual attention for multi-task visual ques- tion answering,” inProceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 1298–1307

    2019

  31. [39]

    A neural model for text localization, transcription and named entity recognition in full pages,

    M. Carbonell, A. Forn’es, M. Villegas, and J. Llad’os, “A neural model for text localization, transcription and named entity recognition in full pages,”Pattern Recognition Letters, vol. 136, pp. 219–227, 2020

    2020

  32. [40]

    A computationally efficient pipeline approach to full page offline handwritten text recognition,

    J. Chung and T. Delteil, “A computationally efficient pipeline approach to full page offline handwritten text recognition,” in 2019 International Conference on Document Analysis and Recognition Workshops, vol. 5. IEEE, 2019, pp. 35–40

    2019

  33. [41]

    A comprehensive studyofdocument imagelayoutanalysis,

    L. Studer, M. Alberti, V. Pondenkandath, P. Goktepe, T. Kolonko, A. Fischer, and M. Liwicki, “A comprehensive studyofdocument imagelayoutanalysis,”in 2019 International Conference on Document Analysis and Recognition (ICDAR) . IEEE, 2019, pp. 1439–1446

    2019

  34. [42]

    A scalable handwritten text recognition system,

    R. R. Ingle, Y. Fujii, T. Deselaers, J. Baccash, and A. C. Popat, “A scalable handwritten text recognition system,” in2019 In- ternational Conference on Document Analysis and Recognition (ICDAR). IEEE, 2019, pp. 17–24

    2019

  35. [43]

    Boosting handwriting text recognition in small databases with transfer learning,

    J. C. A. Jaramillo, J. J. Murillo-Fuentes, and P. M. Olmos, “Boosting handwriting text recognition in small databases with transfer learning,” in 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR) . IEEE, 2018, pp. 429–434

    2018

  36. [44]

    Intelligent character recognition using fully con- volutional neural networks,

    R. Ptucha, F. P. Such, S. Pillai, F. Brockler, V. Singh, and P. Hutkowski, “Intelligent character recognition using fully con- volutional neural networks,” Pattern recognition, vol. 88, pp. 604–613, 2019

    2019

  37. [45]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 770–778

    2016

  38. [46]

    Fully convolutional networks for semantic segmentation,

    J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2015, pp. 3431–3440

    2015

  39. [47]

    Xception: Deep learning with depthwise separable convolutions,

    F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 1251–1258

    2017

  40. [48]

    Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution,

    Y. Chen, Y. Kalantidis, J. Li, S. Yan, and J. Feng, “Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution,” inProceedings of the IEEE International Conference on Computer Vision (ICCV) , 2019, pp. 3435–3444

    2019

  41. [49]

    Squeeze-and-excitation networks,

    J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 7132–7141

    2018

  42. [50]

    Free- form image inpainting with gated convolution,

    J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, “Free- form image inpainting with gated convolution,” inProceedings of the IEEE International Conference on Computer Vision (ICCV), 2019, pp. 4471–4480

    2019

  43. [51]

    Instancenormaliza- tion: The missing ingredient for fast stylization,

    D.Ulyanov,A.Vedaldi,andV.Lempitsky,“Instancenormaliza- tion: The missing ingredient for fast stylization,”arXiv preprint arXiv:1607.08022, 2016

    2016 arXiv

  44. [52]

    Dropout: A simple way to prevent neural networks from overfitting,

    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Re- search, vol. 15, no. 1, pp. 1929–1958, 2014

    1929

  45. [53]

    Efficient object localization using convolutional networks,

    J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler, “Efficient object localization using convolutional networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 648–656

    2015

  46. [54]

    Attentionisallyouneed,

    A.Vaswani,N.Shazeer,N.Parmar,J.Uszkoreit,L.Jones,A.N. Gomez,Ł.Kaiser,andI.Polosukhin,“Attentionisallyouneed,” in Advances in neural information processing systems, 2017, pp. 5998–6008

    2017

  47. [55]

    End- to-end memory networks,

    S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus, “End- to-end memory networks,” in Advances in neural information processing systems, 2015, pp. 2440–2448

    2015

  48. [56]

    Scheduled sampling for sequence prediction with recurrent neural net- works,

    S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Scheduled sampling for sequence prediction with recurrent neural net- works,” Neural information processing systems, vol. 28, 2015. THIS PAPER IS CURRENTLY UNDER REVIEW AT IEEE TRANSACTIONS. 14

    2015

  49. [57]

    Curriculumdropout,

    P. Morerio, J. Cavazza, R. Volpi, R. Vidal, and V. Murino, “Curriculumdropout,”in Proceedings of the IEEE International Conference on Computer Vision , 2017, pp. 3544–3552

    2017

  50. [58]

    Adabatch: Adaptive batch sizes for training deep neural networks,

    A. Devarakonda, M. Naumov, and M. Garland, “Adabatch: Adaptive batch sizes for training deep neural networks,”arXiv preprint arXiv:1712.02029, 2017

    2017 arXiv

  51. [59]

    Browse Fonts - Google Fonts,

    “Browse Fonts - Google Fonts,” May 2024, [Online; accessed

    2024

  52. [60]

    May. 2024]. [Online]. Available: https://fonts.google.com

    2024

  53. [61]

    Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,

    T. Kudo and J. Richardson, “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” inProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2018, pp. 66–71

    2018

  54. [62]

    Icfhr2016 competition on handwritten text recognition on the read dataset,

    J. A. Sánchez, V. Romero, A. H. Toselli, and E. Vidal, “Icfhr2016 competition on handwritten text recognition on the read dataset,” in2016 15th International Conference on Fron- tiers in Handwriting Recognition. IEEE, 2016, pp. 630–635

    2016

  55. [63]

    Learning string-edit distance,

    E. S. Ristad and P. N. Yianilos, “Learning string-edit distance,” IEEE Transactions on Pattern Analysis and Machine Intelli- gence, vol. 20, pp. 522–532, 2002

    2002

  56. [64]

    End-to-end hand- written paragraph text recognition using a vertical attention network,

    D. Coquenet, C. Chatelain, and T. Paquet, “End-to-end hand- written paragraph text recognition using a vertical attention network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 508–524, 2022. VIII. Appendix A. Complexity Analysis of Multi...

    2022

  57. [65]

    Adaptive Query Generation: In contrast to the se- quential processing in FasterDAN, our pipeline operates through two coordinated passes integrated within the MSAP framework (Algorithm hand training), lines 16-17). Each pass is optimized for different aspects of document under...

  58. [66]

    Dynamic Scale Adaptation: The complexity- dependent scaling employs sophisticated adaptation: α(Cl) =α0 1 +γαCl 1 + exp(δα(Cl−θα)) β(Cl) =β0 1 +γβCl 1 + exp(δβ(Cl−θβ)) (48) The parameters γα, γβ, θα, θβ, δα, and δβ control com- plexity sensitivity, are learned thresholds, and ...

  59. [67]

    Our adaptation process leveraged the READ 2016 dataset, which provides consistent ground truth across all structural levels from line to triple-column

    Model Selection and Adaptation:We selected mT5- Small (300M parameters) for its robust multilingual ca- pabilities, especially its proficiency in handling German text. Our adaptation process leveraged the READ 2016 dataset, which provides consistent ground truth across all str...

    2016

  60. [68]

    We started by utilizing SentencePiece tokenization, a technique rec- ognized for its prowess in handling subword units across diverse languages

    Tokenization and Preprocessing:In our study focus- ing on historical German texts, we embraced a holistic ap- proach to tokenization and text normalization. We started by utilizing SentencePiece tokenization, a technique rec- ognized for its prowess in handling subword units a...

  61. [69]

    We then cre- ated paired examples of (HADN prediction, ground truth) for each structural level, ranging from sentences to triple- column layouts

    Training Data Preparation:Our process began with generating initial predictions on the READ 2016 dataset using HADN after 1000 epochs of training. We then cre- ated paired examples of (HADN prediction, ground truth) for each structural level, ranging from sentences to triple- ...

    2016

  62. [70]

    Upon this robust foundation, we carefully integrated task-specific adapta- tion layers tailored to the demands of layout-aware error correction

    Model Architecture and Fine-tuning:We began the post-processing step by employing the small model mT5 equipped with 300 million parameters. Upon this robust foundation, we carefully integrated task-specific adapta- tion layers tailored to the demands of layout-aware error corr...

  63. [71]

    Loss Function and Training: We introduce a spe- cialized loss function in Equation 52 to balance error correction with content preservation. L =α·L CE +β·L sim +γ·L layout (52) where:LCE is the cross-entropy loss for correction ac- curacy,Lsim is the cosine similarity to ensur...

  64. [72]

    HADN output processing involves extracting predicted text along with layout in- formation

    Integration and Inference Pipeline: The fine-tuned mT5 model was integrated into our HADN pipeline as a post-processing stage, following these procedural steps as formalized in Algorithm 4. HADN output processing involves extracting predicted text along with layout in- formati...

    2016

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.