Pith. sign in

REVIEW 3 major objections 3 minor 30 references

MMHMER:Multi-viewer and Multi-task for Handwritten Mathematical Expression Recognition

T0 review · 3 major / 3 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read Pairing a Transformer branch with a CNN counting branch improves handwritten math expression recognition on three CROHME benchmarks.

desk verdict A plausible but weakly supported extension: combining PosFormer's transformer branch with CAN-style counting improves CROHME results modestly, but the evidence is undercut by test-set-driven tuning and missing stats. read the letter →

arxiv 2502.05557 v3 pith:DC7SZCBZ submitted 2025-02-08 cs.CV

classification cs.CV
keywords HandwrittenMathematicalExpressionRecognitionmulti-viewerlearningmulti-taskTransformerCNNsymbolcountingpositionforestCROHMEbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes MMHMER, a multi-viewer, multi-task network for Handwritten Mathematical Expression Recognition that runs two decoders in parallel: a Transformer with a position forest (following PosFormer) and a CNN-based counting decoder (following CAN). Its central claim is that training both branches together, with a weighted loss combining symbol recognition, position-forest prediction, and symbol counting, beats the Transformer branch alone, reaching 63.96%, 62.51%, and 65.55% expression-level accuracy on CROHME14/16/19 with scale augmentation. The point of the design is to let the local feature extraction of the CNN and the global sequence modeling of the Transformer compensate for each other during training. A reader would care because HMER accuracy on these public benchmarks has been hard to move, and the paper shows a relatively simple joint-training recipe that improves on a recent state-of-the-art Transformer.

What carries the argument

The central object is the multi-viewer architecture itself: two parallel decoders—a Position-Forest Transformer viewer and a CNN-GRU counting viewer—sharing a DenseNet encoder. The load-bearing identity is the combined loss \(L_{\mathrm{all}} = \lambda_1 L_{\mathrm{rec}} + \lambda_2 L_{\mathrm{pos}} + \lambda_3 L_{\mathrm{counting}}\), which ties the Transformer's symbol recognition and position-forest objectives to the CNN branch's symbol-counting regression (a smooth-L1 loss). The Multi-Scale Counting Module in the CNN branch, with 3x3 and 5x5 convolutional kernels, channel attention, and sum pooling, is what produces the counting signal; the position forest produces the structural signal. The argument is that these two views—local counting and global structure—push the shared backbone to learn features neither decoder would learn alone.

What would settle it

Train the same architecture with \(\lambda_3\) selected on a held-out validation split drawn from the CROHME training data, then evaluate once on the official test sets; if the resulting margins over PosFormer shrink to near zero or become negative, the reported improvement is an artifact of test-set-driven selection. A second check: run the full recipe with several random seeds and report the variance of the ExpRate margins.

Watch

Extended reading notes

Core claim

MMHMER's core claim is that a CNN viewer and a Transformer viewer, trained jointly as two observers of the same input, produce better recognition than the Transformer alone. Concretely, a DenseNet backbone feeds two branches: a Transformer decoder equipped with a position forest and implicit attention correction, and a GRU-based counting-aware decoder. The training objective is \(L_{\mathrm{all}} = \lambda_1 L_{\mathrm{rec}} + \lambda_2 L_{\mathrm{pos}} + \lambda_3 L_{\mathrm{counting}}\), with \(\lambda_1=1\), \(\lambda_2=0.5\), and \(\lambda_3=0.1\) chosen by experiment. On CROHME14/16/19 with scale augmentation the model reports 63.96%, 62.51%, and 65.55% ExpRate, exceeding PosFormer by 1.28, 1.48, and 0.58 percentage points; on the MNE nested-expression set it reports gains of 1.44%, 0.65%, and 0.21% on N1/N2/N3. The paper also reports that adding a third task (Task2, text prediction) degrades performance markedly, which it interprets as evidence that task balance, not task count, is what matters.

Load-bearing premise

The reported gain is only convincing if the CROHME test sets were used solely for final evaluation and not for choosing the loss weight \(\lambda_3\), the task set, or other hyperparameters; the paper does not report a held-out validation split, repeated runs, or significance tests.

Editorial extensions

If this is right

  • If the reported numbers hold, fusing a position-forest Transformer with a counting CNN gives a new state of the art on CROHME14, CROHME16, and CROHME19 under scale augmentation.
  • Symbol counting (Task1) alone gives only small or negative gains on the Transformer baseline, so the multi-viewer joint training—not the counting loss per se—is what the paper credits for the improvement.
  • Adding an additional text-prediction task (Task2) drops accuracy by 5.4–8.0 percentage points, implying that multi-task HMER frameworks must limit the number and type of auxiliary tasks.
  • The gains extend to nested expressions: the paper reports consistent increases on the MNE N1/N2/N3 subsets, suggesting the multi-viewer setup helps structured complexity, not just overall accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence the paper does not test: the same two-viewer recipe could transfer to other image-to-sequence problems (e.g., formula detection or scene text recognition) where a counting branch and a structure-aware branch can be defined.
  • The paper motivates a consistency constraint between the two viewers but the loss function shown contains no explicit consistency term; a testable extension is to add one and measure whether the reported margins grow.
  • Because the gains are reported without variance or significance testing, a cautious reading is that the true effect may be smaller than the point estimates; a re-run with multiple seeds on a held-out validation split would settle the size of the effect.
  • The MNE results suggest the method helps most on shallow nesting (N1: +1.44%) and least on deep nesting (N3: +0.21%), which, if confirmed, points to where the CNN counting view stops adding information.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes MMHMER, a multi-viewer, multi-task framework for handwritten mathematical expression recognition (HMER). It combines a Transformer-based branch following PosFormer with a CNN/GRU branch following CAN that performs symbol counting. The two branches are trained jointly with a combined loss, and the paper claims improved ExpRate over PosFormer on CROHME 2014, 2016, and 2019 (63.96%, 62.51%, 65.55% vs. 62.68%, 61.03%, 64.97%) and on the MNE nested-expression test set. The main contribution is stated as an online collaborative multi-viewer strategy that lets CNN and Transformer complement each other.

Significance. If the reported gains are reliable, the paper would demonstrate a simple and effective way to combine a counting-based CNN decoder with a position-forest Transformer decoder for HMER, a direction that the authors note is underexplored. The work uses standard public benchmarks, reports comparisons with several recent methods, and includes ablations of the counting task and the loss weight. However, the core evidence is empirical and the reported improvements are small (0.58–1.48 percentage points on test sets of roughly 1,000–1,200 expressions). The manuscript does not currently provide the statistical or methodological support needed to distinguish these gains from test-set-driven selection or random variation, so the central claim should be treated as unverified until that support is added.

major comments (3)
  1. [§IV, Tables III and IV] The final configuration appears to be selected on the same CROHME test sets that are then used to report the headline results. Table IV varies λ3 over {0.01, 0.1, 0.5} and reports the best value (0.1) on each test set; Table III adds Task2 and then removes it after observing large drops (5.75, 5.40, and 8.01 points) on those same test sets. No held-out validation split, no multiple-seed runs, and no significance tests are reported. The claimed gains over PosFormer (1.28, 1.48, and 0.58 points) are within the range that could arise from test-set-driven selection, especially given the small test-set sizes. The central claim that MMHMER outperforms PosFormer is therefore not established by the evidence as presented.
  2. [Abstract vs. Table I] The abstract reports 65.46% ExpRate on CROHME19 and an absolute gain of 0.58% over PosFormer, while Table I reports 65.55% for CROHME19 (a 0.58% gain relative to the 64.97% PosFormer result). The abstract numbers are internally inconsistent: 65.46% would imply a 0.49% gain. This discrepancy must be resolved before the reported results can be relied upon.
  3. [§IV, Table III] The ablation shows that the counting task alone ('+Task1') reduces CROHME16 ExpRate from 61.03 to 60.85, while the text states that symbol counting 'contributes to a slight enhancement' in HMER performance. No explanation is given for the negative result on CROHME16. Since the multi-task contribution is a central part of the paper's claim, the interaction between the counting task and the multi-viewer mechanism needs explicit analysis rather than a summary statement over inconsistent numbers.
minor comments (3)
  1. [§III-C, Eq. (7)] Only λ3 is ablated in Table IV; λ1 and λ2 are fixed at 1 and 0.5, respectively, with no sensitivity analysis. A brief discussion or supplementary ablation of these weights would strengthen the paper.
  2. [General] The manuscript contains several typographical and formatting errors, including an author email with an embedded space ('zhong lifan@yahoo.co.jp'), duplicate reference [23] identical to [11], and a repeated value in Table I for PosFormer on CROHME19 (87.24 appears twice in the ≤3 column).
  3. [Figure 2] Figure 2 is referenced in the text but no actual images are visible in the provided manuscript; the caption 'Sample Images from CROHME 2014/2016/2019 Datasets' appears but the figure content is missing.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline CROHME accuracies are selected, not predicted: λ3 and the task set (Task2 dropped) were chosen on the same test sets that produce the Table I gains over PosFormer.

  1. fitted input called prediction [Section IV, Table IV and adjacent ablation text (λ3 selection)]
    "As shown in Table IV, the symbol λ is used to balance the trade-off between tasks for CNNs and transformers in the experiments above. In our setting, λ=0.1 performs the best, whereas a smaller λ would reduce the learning capability of the CNN. On the other hand, a larger λ presents issues and can lead to performance degradation because the network might converge in the wrong direction."

    λ3 was varied over {0.01, 0.1, 0.5} and the value yielding the highest ExpRate on the same CROHME14/16/19 test sets used in Table I was chosen (λ3=0.1). The headline numbers 63.96/62.51/65.55 are therefore the maximum over the tried λ values on those exact test sets. Reporting these as MMHMER's predicted performance is equivalent to reporting the result of fitting a hyperparameter to the evaluation set; the gains over PosFormer are part of the selection criterion, not an independent prediction.

  2. fitted input called prediction [Section IV, Table III and ablation discussion (Task2 removal)]
    "As we integrate additional tasks into our model, such as text prediction designated as Task2, we notice a marked deterioration in the overall performance metrics. Specifically, the inclusion of this extra task results in a decrease in performance by 5.75% on the 2014 dataset, 5.40% on the 2016 dataset, and a more pronounced drop of 8.01% on the 2019 dataset."

    The final architecture reported in Table I was obtained by first adding Task2, observing large drops on the same CROHME14/16/19 test sets, then removing it. The reported configuration is thus selected to maximize performance on the very test sets that later serve as the evidence for the claimed 0.58-1.48% gains over PosFormer. The ablation table is the selection record, and the same numbers are reused as the prediction, so the headline comparison is statistically forced rather than an independent evaluation.

full rationale

No load-bearing self-citation was found: reference [33] (SemiHMER) is a related-work self-citation only and does not justify the central architecture. The CNN/Transformer fusion and the losses in Eq. (7) are not defined in terms of the target ExpRate, so the method has independent empirical content. The circular component is confined to test-set-driven selection: Table IV chooses λ3 because it 'performs the best' on CROHME14/16/19, and Table III removes Task2 after observing drops on those same sets; Table I then reuses those selected configurations as the headline result. There is no held-out validation split, repeated-seed statistics, or significance test, and the abstract's CROHME19 value (65.46%) differs from Table I (65.55%). These issues make the reported gains over PosFormer a selected optimum rather than a verified prediction, giving partial circularity (6/10) rather than a fully circular derivation.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim rests on empirical benchmark performance rather than a mathematical derivation. The most important free parameter is the counting loss weight λ3, which is tuned on the same test sets used for the headline results. No new physical or conceptual entities are introduced.

free parameters (5)
  • λ3 (counting loss weight) = 0.1
    Selected by sweeping values 0.01, 0.1, 0.5 and choosing the best ExpRate on the CROHME test sets (Table IV).
  • λ1 (recognition loss weight) = 1 (default)
    Stated as default in Section III-C; not tuned in the paper.
  • λ2 (position loss weight) = 0.5 (default)
    Stated as default in Section III-C; the paper does not clarify whether the final multi-view model uses it.
  • Transformer decoder layers = 3
    Implementation detail stated in Section IV-C.
  • Counting convolution kernel sizes = 3x3 and 5x5
    Architectural choice for the Multi-Scale Counting Module described in Section III-B.
assumptions (3)
  • domain assumption The CROHME training/test splits and ground truth LaTeX labels are correct and representative of handwritten mathematical expression recognition.
    All results are evaluated on these benchmarks without error analysis or external validation.
  • domain assumption The PosFormer baseline results in Table I are reproduced under the same training conditions.
    The paper says it follows Posformer's training parameters, but no baseline code or logs are provided.
  • standard math Standard cross-entropy and smooth L1 losses are appropriate for the recognition, position, and counting tasks.
    Used in equations (1) through (7).

how reviews work

0 comments
Cite this review

Pith. "Pith review of MMHMER:Multi-viewer and Multi-task for Handwritten Mathematical Expression Recognition." pith.science (2026). https://pith.science/paper/DC7SZCBZ

@misc{pith2026250205557,
  author       = {Pith},
  title        = {Pith review of: MMHMER:Multi-viewer and Multi-task for Handwritten Mathematical Expression Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DC7SZCBZ}},
  note         = {Machine review of arXiv:2502.05557}
}
read the original abstract

Handwritten Mathematical Expression Recognition (HMER) methods have made remarkable progress, with most existing HMER approaches based on either a hybrid CNN/RNN-based with GRU architecture or Transformer architectures. Each of these has its strengths and weaknesses. Leveraging different model structures as viewers and effectively integrating their diverse capabilities presents an intriguing avenue for exploration. This involves addressing two key challenges: 1) How to fuse these two methods effectively, and 2) How to achieve higher performance under an appropriate level of complexity. This paper proposes an efficient CNN-Transformer multi-viewer, multi-task approach to enhance the model's recognition performance. Our MMHMER model achieves 63.96%, 62.51%, and 65.46% ExpRate on CROHME14, CROHME16, and CROHME19, outperforming Posformer with an absolute gain of 1.28%, 1.48%, and 0.58%. The main contribution of our approach is that we propose a new multi-view, multi-task framework that can effectively integrate the strengths of CNN and Transformer. By leveraging the feature extraction capabilities of CNN and the sequence modeling capabilities of Transformer, our model can better handle the complexity of handwritten mathematical expressions.

Figures

Figures reproduced from arXiv: 2502.05557 by the authors.

Figure 1
Figure 1. Overview of multi-viewer and multi-task for Handwritten Mathematical Expression Recognition [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Sample Images from CROHME 2014/2016/2019 Datasets [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 27 canonical work pages

  1. [1]

    Winkler H J. HMM-based handwritten symbol recognition using on- line and off-line features[C]//1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings. IEEE, 1996, 6: 3438-3441

  2. [2]

    Hu L, Zanibbi R. HMM-based recognition of online handwritten math- ematical symbols using segmental k-means initialization and a modified pen-up/down feature[C]//2011 International conference on Document analysis and Recognition. IEEE, 2011: 457-462

  3. [4]

    Hybrid mathematical symbol recognition using sup- port vector machines[C]//Ninth International Conference on Document Analysis and Recognition (ICDAR 2007)

    Keshari B, Watt S. Hybrid mathematical symbol recognition using sup- port vector machines[C]//Ninth International Conference on Document Analysis and Recognition (ICDAR 2007). IEEE, 2007, 2: 859-863

  4. [5]

    Densely connected convo- lutional networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition

    Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convo- lutional networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 4700-4708

  5. [6]

    Learned image compression with mixed transformer-cnn architectures[C]//Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition

    Liu J, Sun H, Katto J. Learned image compression with mixed transformer-cnn architectures[C]//Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition. 2023: 14388-14397

  6. [7]

    CNN-transformer mixed model for object detection

    Li W. CNN-transformer mixed model for object detection[J]. arXiv preprint arXiv:2212.06714, 2022

  7. [8]

    Zhu J, Luo Y , Zheng X, et al. A good student is cooperative and reliable: CNN-transformer collaborative learning for semantic segmen- tation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 11720-11730

  8. [9]

    Zhang, Z., Zhang, Y . (2022). Combining CNN and Transformer as En- coder to Improve End-to-End Handwritten Mathematical Expression Recognition Accuracy. In: Porwal, U., Forn´es, A., Shafait, F. (eds) Fron- tiers in Handwriting Recognition. ICFHR 2022. Lecture Notes in Com- puter Science, vol 13639. Springer, Cham. https://doi.org/10.1007/978- 3-031-21648-0 13

Show all 30 references
  1. [10]

    Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition[J]

    Zhang J, Du J, Zhang S, et al. Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition[J]. Pattern Recognition, 2017, 71: 196-206

  2. [12]

    When counting meets HMER: counting-aware network for handwritten mathematical expression recog- nition[C]//European conference on computer vision

    Li B, Yuan Y , Liang D, et al. When counting meets HMER: counting-aware network for handwritten mathematical expression recog- nition[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 197-214

  3. [13]

    Semantic graph representation learning for handwritten mathematical expression recognition[C]//International conference on document analysis and recognition

    Liu Z, Yuan Y , Ji Z, et al. Semantic graph representation learning for handwritten mathematical expression recognition[C]//International conference on document analysis and recognition. Cham: Springer Nature Switzerland, 2023: 152-166

  4. [14]

    Zhang J, Du J, Dai L. A gru-based encoder-decoder approach with attention for online handwritten mathematical expression recogni- tion[C]//2017 14th IAPR international conference on document analysis and recognition (ICDAR). IEEE, 2017, 1: 902-907

  5. [15]

    Track, attend, and parse (tap): An end-to-end framework for online handwritten mathematical expression recogni- tion[J]

    Zhang J, Du J, Dai L. Track, attend, and parse (tap): An end-to-end framework for online handwritten mathematical expression recogni- tion[J]. IEEE Transactions on Multimedia, 2018, 21(1): 221-233

  6. [16]

    Zhao W, Gao L, Yan Z, et al. Handwritten mathematical expres- sion recognition with bidirectionally trained transformer[C]//Document analysis and recognition–ICDAR 2021: 16th international conference, Lausanne, Switzerland, September 5–10, 2021, proceedings, part II 16. Spring...

  7. [17]

    Comer: Modeling coverage for transformer-based hand- written mathematical expression recognition[C]//European conference on computer vision

    Zhao W, Gao L. Comer: Modeling coverage for transformer-based hand- written mathematical expression recognition[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 392- 408

  8. [18]

    PosFormer: recognizing com- plex handwritten mathematical expression with position forest trans- former[C]//European Conference on Computer Vision

    Guan T, Lin C, Shen W, et al. PosFormer: recognizing com- plex handwritten mathematical expression with position forest trans- former[C]//European Conference on Computer Vision. Springer, Cham, 2025: 130-147

  9. [19]

    In: Proc

    Mouchere, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: Icfhr 2014 competition on recognition of on-line handwritten mathematical expres- sions (crohme 2014). In: Proc. of International Conference on Frontiers in Handwriting Recognition. pp. 791–796 (2014)

  10. [20]

    In: ICFHR

    Mouch `ere, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: Icfhr2016 crohme: Competition on recognition of online handwritten mathematical expressions. In: ICFHR. pp. 607–612 (2016)

  11. [21]

    In: ICDAR

    Mahdavi, M., Zanibbi, R., Mouchere, H., Viard-Gaudin, C., Garain, U.: Icdar 2019 crohme+ tfd: Competition on recognition of handwritten mathematical expressions and typeset formula detection. In: ICDAR. pp. 1533–1538 (2019)

  12. [23]

    Multi-scale attention with dense encoder for hand- written mathematical expression recognition[C]//2018 24th international conference on pattern recognition (ICPR)

    Zhang J, Du J, Dai L. Multi-scale attention with dense encoder for hand- written mathematical expression recognition[C]//2018 24th international conference on pattern recognition (ICPR). IEEE, 2018: 2245-2250

  13. [24]

    Kosmala A, Rigoll G, Lavirotte S, et al. On-line handwritten formula recognition using hidden Markov models and context dependent graph grammars[C]//Proceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR’99 (Cat. No. PR00318). IEEE, 1999: 107-110

  14. [25]

    Elastic structural matching for online handwritten alphanumeric character recognition[C]//Proceedings

    Chan K F, Yeung D Y . Elastic structural matching for online handwritten alphanumeric character recognition[C]//Proceedings. Fourteenth Interna- tional Conference on Pattern Recognition (Cat. No. 98EX170). IEEE, 1998, 2: 1508-1511

  15. [26]

    Towards a web-based progressive hand- writing recognition environment for mathematical problem solving[J]

    Vuong B Q, He Y , Hui S C. Towards a web-based progressive hand- writing recognition environment for mathematical problem solving[J]. Expert Systems with Applications, 2010, 37(1): 886-893

  16. [27]

    Learning phrase represen- tations using RNN encoder-decoder for statistical machine translation[J]

    Cho K, Van Merri ¨enboer B, Gulcehre C, et al. Learning phrase represen- tations using RNN encoder-decoder for statistical machine translation[J]. arXiv preprint arXiv:1406.1078, 2014

  17. [28]

    A tree-based structure-aware transformer decoder for image-to-markup generation[C]//Proceedings of the 30th ACM International Conference on Multimedia

    Zhong S, Song S, Li G, et al. A tree-based structure-aware transformer decoder for image-to-markup generation[C]//Proceedings of the 30th ACM International Conference on Multimedia. 2022: 5751-5760

  18. [29]

    NAMER: Non-Autoregressive Modeling for Handwritten Mathematical Expression Recognition[C]//European Conference on Computer Vision

    Liu C, Pan J, Hu J, et al. NAMER: Non-Autoregressive Modeling for Handwritten Mathematical Expression Recognition[C]//European Conference on Computer Vision. Springer, Cham, 2025: 273-291

  19. [30]

    Attention is all you need[J]

    Vaswani A. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017

  20. [31]

    The pascal visual object classes (voc) challenge[J]

    Everingham M, Van Gool L, Williams C K I, et al. The pascal visual object classes (voc) challenge[J]. International journal of computer vision, 2010, 88: 303-338

  21. [32]

    Faster R-CNN: Towards real-time object detection with region proposal networks[J]

    Ren S, He K, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 39(6): 1137-1149

  22. [33]

    SemiHMER: Semi-supervised Handwritten Mathe- matical Expression Recognition using pseudo-labels[J]

    Chen K, Shen H. SemiHMER: Semi-supervised Handwritten Mathe- matical Expression Recognition using pseudo-labels[J]. arXiv preprint arXiv:2502.07172, 2025

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.