Pith. sign in

REVIEW 4 major objections 4 minor 12 references

Automated LaTeX Code Generation from Handwritten Math Expressions Using Vision Transformer

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A patch-based vision transformer with a transformer decoder converts handwritten math images into LaTeX more accurately than CNN-LSTM and ResNet-LSTM baselines, the paper claims.

desk verdict The central ViT-beats-CNN-LSTM claim is unsupported because the reported BLEU/Levenshtein numbers are computed on a training-set subset, and the paper's own text contradicts its results table. read the letter →

arxiv 2412.03853 v2 pith:MZDOARXN submitted 2024-12-05 cs.CV cs.CL

Jayaprakash Sundararaj , Akhil Vyas , Benjamin Gonzalez-Maldonado This is my paper · ORCID
classification cs.CVcs.CL
keywords visiontransformerhandwrittenmathematicalexpressionrecognitionimagetoLaTeXencoder-decoderarchitectureCNN-LSTMbaselineBLEU-4Levenshteindistanceim2latexdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that replacing the CNN encoder in an image-to-LaTeX system with a vision transformer improves the translation of handwritten and digital mathematical expressions into LaTeX code. It builds a CNN-LSTM baseline, a variant with a pretrained ResNet50 encoder, and a patch-based vision transformer encoder with a transformer decoder, and reports that the vision transformer achieves the best loss, accuracy, masked accuracy, BLEU-4, and Levenshtein distance on its 200,000-sample subset of the im2latex datasets. If the comparison holds up, the result points to self-attention over image patches as a stronger feature extractor for math recognition than the recurrent baselines, with further gains available from tuning layers, heads, and patch size.

What carries the argument

The load-bearing object is the vision transformer encoder: a 50x200 grayscale image is divided into 10x10 patches, giving a sequence of 100 patch embeddings with positional encodings, processed by 8 transformer layers with 4 attention heads and an MLP of 2048/1024 units. The decoder is a standard transformer with 4 layers and 8 heads using both self-attention over generated tokens and cross-attention over encoded image features; this cross-attention is what aligns each LaTeX token to the relevant visual region. The baselines are a three-block 3x3 convolution/max-pool CNN encoder with an LSTM decoder, and the same decoder with a pretrained ResNet50 encoder adapted to grayscale by replicating the channel three times.

What would settle it

Compute BLEU-4, Levenshtein distance, and token accuracy on a held-out split of the same datasets, with the three architectures trained under matched batch sizes and optimization schedules; if the vision transformer does not beat both recurrent baselines on unseen formulas, the paper's central claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that the vision transformer architecture outperforms the CNN-RNN framework on the task of generating LaTeX from math-expression images. On the reported metrics, the vision transformer reaches a loss of 0.520, an accuracy of 0.873, a masked accuracy of 0.641, a BLEU score of 0.557, and a Levenshtein distance of 0.354, compared with CNN-LSTM (0.647, 0.847, 0.600, 0.429, 0.401) and ResNet-LSTM (0.781, 0.819, 0.520, 0.372, 0.435). The authors attribute this to positional embeddings and attention in both encoder and decoder, which let the model relate distant image regions while generating the LaTeX token sequence.

Load-bearing premise

The central claim depends on the reported quality scores being measured on examples the models had not seen during training; the paper states in Section 5.2 that the scores were computed on a subset of the training set, so if that statement is literal the comparison shows memorization rather than generalization and the claimed advantage is not established.

Editorial extensions

If this is right

  • If the reported numbers are representative, vision transformers become a competitive encoder choice for mathematical expression recognition, not just for classification.
  • Masked accuracy and BLEU-4 improvements indicate that more of the predicted token sequence matches the reference LaTeX exactly, while lower Levenshtein distance means the generated strings are closer to correct even when not exact.
  • The paper's own conclusion is that tuning the number of transformer layers, attention heads, and patch size, and training on larger datasets, would improve accuracy further.
  • The ResNet-LSTM comparison suggests that pretrained visual encoders converge and perform differently from scratch-trained encoders, but in this setup neither matches the vision transformer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because Section 5.2 says the BLEU-4 and Levenshtein metrics were computed on a subset of the training set, the generalization comparison in the table may reflect memorization; a held-out evaluation is needed before accepting the ranking.
  • The three architectures differ not only in encoder but also in batch size and optimizer (128 with Adam vs 64 with AdamW and a decaying learning rate), so part of the reported gap could come from training configuration rather than the encoder family.
  • The fixed 10x10 patch grid on a 50x200 image yields a 5-by-20 sequence; testing rectangular patches matched to the wide aspect ratio could change how much spatial structure the encoder preserves.
  • Since the im2latexv2 data mixes printed and handwritten formulas, the same architecture could be evaluated separately on each domain to see where the attention-based encoder helps most.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes a vision-transformer (ViT) encoder with a transformer decoder for converting handwritten mathematical expression images into LaTeX sequences, and compares it against a CNN-LSTM baseline and a ResNet-LSTM baseline on 200,000 samples from Im2latex-100k/230k. The authors report loss, masked loss, accuracy, BLEU-4, and Levenshtein distances, and claim that the ViT architecture 'consistently outperforms' the baseline CNN-RNN framework. However, the key sequence-similarity metrics are stated to be computed on a subset of the training set, and no held-out test evaluation is provided.

Significance. If the claimed results were based on held-out data, the paper would provide a useful empirical comparison of encoder-decoder architectures for math-expression recognition, and the public code repository would support reproducibility. The work also usefully includes a pretrained ResNet50 encoder variant. Unfortunately, the evaluation as described does not measure generalization: the main metric table refers to training-set predictions, and the model comparisons are confounded by different optimizers, batch sizes, and learning-rate schedules. These issues undermine the central conclusion, so the paper in its current form does not constitute a reliable contribution.

major comments (4)
  1. [§5.2, Table 1] The paper states: 'We measure the Levenshtein distance and BLEU-4 score for predicted sequences of a subset of the training set.' Taken literally, this means that the BLEU and Levenshtein values in Table 1 are computed on training examples, so they reflect how well the models memorize their training data rather than how well they recognize unseen handwritten expressions. The central claim of the abstract and §5.4—that the vision transformer outperforms the baseline CNN-RNN framework—is therefore unsupported. The authors must define a held-out test split before training and report all sequence-similarity metrics exclusively on that split.
  2. [§5.1, Table 1] The compared architectures are trained with different hyperparameters: the CNN-LSTM and ResNet-LSTM models use batch size 128 and Adam with learning rate 0.001, while the vision transformer uses batch size 64 and AdamW with a decaying learning rate from 1e-4 to 1e-6. Consequently, the performance differences in Table 1 cannot be attributed solely to the architectural change from CNN to ViT; they could equally result from the optimizer, batch size, or learning-rate schedule. A controlled comparison should keep these factors fixed (or at least run ablations) before drawing conclusions about encoder choice.
  3. [§5.4, Table 1] Each row of Table 1 appears to correspond to a single run. No standard deviations, confidence intervals, or significance tests are reported. With only one seed per condition, the observed differences (e.g., BLEU 0.557 vs 0.429) cannot be distinguished from run-to-run variation. At minimum, the authors should report results over multiple seeds with error bars, or a paired significance test.
  4. [§5.1] The manuscript does not specify how the 200,000 data points are split into training, validation, and test sets, nor does it state whether early stopping used a validation set. This lack of a clear data-split protocol makes it impossible to know whether any model selection occurred on the same examples used for the reported metrics. The authors should provide the exact split sizes and the criterion used for early stopping.
minor comments (4)
  1. [Abstract] The abstract contains a stray superscript '2' after 'domain' ('domain 2'), which appears to be a leftover footnote marker.
  2. [§4.2] The target input size for ResNet50 is given as '254 × 254'; the standard ResNet50 input size is 224 × 224, so the dimension should be checked and corrected.
  3. [References] Several references are unrelated to the topic of this paper (e.g., Jayanth et al. 2015 and Sundararaj et al. 2015 on opinion summarization, and Sundararaj's 2024 thesis). These citations appear to be self-promotional and should be removed or replaced with relevant literature.
  4. [§2, §4.1] The related-work discussion of the CNN-LSTM baseline does not explicitly cite the actual baseline used (e.g., Deng et al. 2016 or Wang and Liu 2019), even though these are included in the reference list; the baseline Section 4.1 should be connected to the cited prior work.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported BLEU/Levenshtein metrics are computed on a training-set subset (§5.2), so the claimed ViT superiority is an in-sample comparison, not a held-out prediction.

  1. fitted input called prediction [§5.1-§5.2, with results reported in §5.4 Table 1]
    "We measure the Levenshtein distance and BLEU-4 score for predicted sequences of a subset of the training set. These metrics were chosen in order to quantify closeness/correctness between sequences beyond a simple binary score that relies on exact matching. ... The learning rate decay strategy was determined through experimentation and was found to enhance the performance and stability of the transformer model during training."

    By §5.2's own statement, the BLEU-4 and Levenshtein numbers in Table 1 are computed on a subset of the training set; therefore they measure how well each model reproduces data it was trained on, not how it generalizes to unseen inputs. The central claim in the abstract and §5.4 that 'the vision transformer architectures outperform the baseline CNN-RNN framework, delivering higher overall accuracy and BLEU scores while achieving lower Levenshtein distances' is based directly on those in-sample numbers. The transformer's learning-rate schedule was also 'determined through experimentation' on the same data (§5.1), and no train/validation/test split is reported.

full rationale

This paper is an empirical architecture comparison rather than a formal derivation, so the classical derivation-circularity patterns (self-definitional equations, imported uniqueness theorems, ansatz smuggling) do not apply. The self-citations to the first author's earlier summarization and thesis work are unrelated to the model comparisons and are not load-bearing. The central claim rests on Table 1, whose BLEU-4 and Levenshtein values are explicitly measured 'for predicted sequences of a subset of the training set' (§5.2), and whose hyperparameters were selected through experimentation on the same data (§5.1). That makes the claimed ViT superiority an in-sample fit rather than a test-set prediction, so the reported 'predictions' reduce by construction to memorization performance. The paper's internal contradictions, such as claiming the ResNet-LSTM model 'demonstrated improved performance' while Table 1 shows it worse on every metric, further undermine the results but are correctness issues rather than circularity. Score 6 reflects that one central evaluation quantity is an in-sample fit; the comparison is not supported without held-out evaluation.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

The paper makes no theoretical derivation, so the ledger records the manually chosen hyperparameters and unstated data assumptions on which the empirical comparison depends. No new entities are introduced.

free parameters (9)
  • Vision transformer patch size = 10 x 10 pixels
    Chosen by hand in §4.3.1; determines 100 patches per 50x200 image and affects what the model can represent.
  • Vision transformer encoder depth and heads = 8 layers, 4 heads, MLP 2048/1024
    Manual configuration in §4.3.1; no ablation or sensitivity analysis is provided.
  • Transformer decoder depth and heads = 4 layers, 8 heads
    Manual configuration in §4.3.2; no ablation.
  • Vision transformer batch size = 64
    Set to 64 due to AWS GPU memory (§5.1), while CNN/ResNet baselines use 128; this confounds the architecture comparison.
  • Vision transformer learning rate schedule = 1e-4 to 1e-6 (AdamW)
    Described as determined through experimentation (§5.1); the baselines use Adam at 0.001, so optimization differs across models.
  • Early stopping patience = 10 epochs
    Applied to all models (§5.1); affects the selected checkpoint and final metrics.
  • Maximum token length / padding = 151
    Sequences padded to 151 tokens (§5.2); masked metrics were introduced to handle padding, but raw accuracy includes it.
  • Baseline batch size = 128
    Used for CNN-LSTM and ResNet-LSTM in §5.1; different from the transformer's 64, so the comparison is not matched.
  • Baseline optimizer learning rate = Adam, 0.001
    Used in §5.1 for CNN-LSTM and ResNet-LSTM; the transformer uses AdamW with decay, creating another confound.
assumptions (5)
  • domain assumption The reported metrics are computed on held-out, representative test data.
    No train/validation/test split is described (§5.1), and §5.2 says BLEU/Levenshtein are for a subset of the training set; the whole generalization claim depends on this.
  • domain assumption The combined Im2latex datasets have compatible preprocessing and no formula leakage.
    §3 introduces two datasets but does not state how they are merged, cleaned, tokenized, or deduplicated.
  • domain assumption The architecture comparison isolates the encoder/decoder choice.
    Models differ in batch size, optimizer, learning-rate schedule, and capacity (§5.1), so observed gaps cannot be attributed solely to architecture.
  • domain assumption Grayscale-to-RGB replication is an adequate adapter for pretrained ResNet50.
    §4.2 replaces one channel with three identical channels; the effect on pretrained feature quality is not tested.
  • domain assumption Early stopping is applied on a validation split independent of the test split.
    Patience 10 is set in §5.1, but the validation protocol is not described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated LaTeX Code Generation from Handwritten Math Expressions Using Vision Transformer." pith.science (2026). https://pith.science/paper/MZDOARXN

@misc{pith2026241203853,
  author       = {Pith},
  title        = {Pith review of: Automated LaTeX Code Generation from Handwritten Math Expressions Using Vision Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MZDOARXN}},
  note         = {Machine review of arXiv:2412.03853}
}
read the original abstract

Transforming mathematical expressions into LaTeX poses a significant challenge. In this paper, we examine the application of advanced transformer-based architectures to address the task of converting handwritten or digital mathematical expression images into corresponding LaTeX code. As a baseline, we utilize the current state-of-the-art CNN encoder and LSTM decoder. Additionally, we explore enhancements to the CNN-RNN architecture by replacing the CNN encoder with the pretrained ResNet50 model with modification to suite the grey scale input. Further, we experiment with vision transformer model and compare with Baseline and CNN-LSTM model. Our findings reveal that the vision transformer architectures outperform the baseline CNN-RNN framework, delivering higher overall accuracy and BLEU scores while achieving lower Levenshtein distances. Moreover, these results highlight the potential for further improvement through fine-tuning of model parameters. To encourage open research, we also provide the model implementation, enabling reproduction of our results and facilitating further research in this domain.

Figures

Figures reproduced from arXiv: 2412.03853 by the authors.

Figure 1
Figure 1. Formulas breakdown by length 4 METHODS 4.1 BASELINE: CNN ENCODER AND LSTM DECODER As a baseline, we employ a CNN encoder to process the input image, which is resized to dimensions of 50×200 pixels with a single channel (grayscale). The architecture utilizes a 3×3 convolutional filter, followed by a 2×2 max-pooling layer. This block is repeated three times to progressively extract features from the image. The final o… view at source ↗
Figure 2
Figure 2. Encoder architecture consists of 3 convolution-max pooling blocks (50,200) -> (25,100) -> [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Pretrained ResNet50 Encoder with LSTM Decoder. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Original latex image and the generated patches [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Transformer encoder architecture 4.3.2 DECODER For the decoder, we use a standard transformer block that incorporates both cross-attention and self-attention mechanisms. The cross-attention component is utilized to identify and focus on specific regions of the image th…
Figure 6
Figure 6. Figure 6: Dataset: Most popular symbols and frequencies. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Dataset: Least popular symbols and frequencies. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 4 canonical work pages

  1. [10]

    Image-to-LaTeX Converter for Mathematical Formulas and Text

    URL https://arxiv.org/abs/2408.04015. Keiron O’Shea and Ryan Nash. An introduction to convolutional neural networks. CoRR, abs/1511.08458,

    work page Pith review arXiv

  2. [11]

    Jayaprakash Sundararaj

    URL http://arxiv.org/abs/1511.08458. Jayaprakash Sundararaj. Thesis: Document summarization with applications to keyword extraction and image retrieval,

    arXiv

  3. [12]

    Thesis: Document Summarization with applications to Keyword extraction and Image Retrieval

    URL https://arxiv.org/abs/2406.00013. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735–1780,

    work page Pith review arXiv

  4. [1997]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu

    doi: 10.1162/neco.1997.9.8.1735. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311–318, Philadelphia, Pennsylvania, USA, July

    doi:10.1162/neco.1997.9.8.1735 1997

  5. [2002]

    doi: 10.3115/1073083.1073135

    Association for Computational Linguistics. doi: 10.3115/1073083.1073135. URL https://aclanthology.org/P02-1040. 8 7 APPENDIX Figure 6: Dataset: Most popular symbols and frequencies. Figure 7: Dataset: Least popular symbols and frequencies. 9

    arXiv

  6. [2016]

    Zelun Wang and Jyh-Charn Liu

    URL http://arxiv.org/abs/1609.04938. Zelun Wang and Jyh-Charn Liu. Translating mathematical formula images to latex sequences using deep neural networks with sequence-level training. CoRR, abs/1908.11415,

    arXiv 1908

  7. [2018]

    Guillaume Genthial and Romain Sauvestre

    URL https: //cs230.stanford.edu/projects_spring_2018/reports/8287695.pdf. Guillaume Genthial and Romain Sauvestre. Image to latex,

    work page arXiv

  8. [2019]

    Jayaprakash Sundararaj, Jayant Jaiswal, and Pushpak Bhattacharya

    URL http://arxiv.org/abs/1908.11415. Jayaprakash Sundararaj, Jayant Jaiswal, and Pushpak Bhattacharya. Opinion summarization using submodular functions: Subjectivity vs relevance trade-off,

    arXiv 1908

Show all 12 references
  1. [2020]

    Daniil Gurgurov and Aleksey Morshnev

    URL https: //arxiv.org/abs/2003.00817. Daniil Gurgurov and Aleksey Morshnev. Image-to-latex converter for mathematical formulas and text,

    2003 arXiv

  2. [2021]

    Anssi Kanervisto

    URL https://arxiv.org/abs/2010.11929. Anssi Kanervisto. im2latex-100k , arxiv:1609.04938, July

    2010 arXiv

  3. [2023]

    URL https://arxiv.org/abs/ 1706.03762. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words...

    arXiv

  4. [2024]

    Jayanth Jayanth, Jayaprakash Sundararaj, and Pushpak Bhattacharyya

    URL https://arxiv.org/abs/2404.10690. Jayanth Jayanth, Jayaprakash Sundararaj, and Pushpak Bhattacharyya. Monotone submodularity in opinion summaries. In Lluís Màrquez, Chris Callison-Burch, and Jian Su, editors, Proceedings of the 2015 Conference on Empirical Methods in Natur...

    2015 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.