{"id":"d3c4bf8e-11bc-4c60-b97f-b826f2754e50","arxiv_id":"2412.03853","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":9,"one_line_summary":"A vision transformer encoder with a transformer decoder beats small CNN-LSTM and ResNet-LSTM baselines on image-to-LaTeX conversion in the authors' reported experiments.","lead":"Researchers compared three neural network designs for turning photos of handwritten math into LaTeX code and found that a vision transformer encoder paired with a transformer decoder gave the best reported scores. The result is a small benchmark comparison, not a new capability, and the paper's evaluation has serious gaps that make the claim hard to trust.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported BLEU and Levenshtein metrics are computed on a training-set subset per §5.2, so the claimed ViT superiority measures memorization rather than generalization; the central claim is unsupported without held-out evaluation.","rationale":"The reader's weakest-assumption analysis and my stress-test converge on the same load-bearing issue: §5.2 explicitly states that BLEU and Levenshtein are computed on a subset of the training set. That is a direct, internal statement, not an inference from external expectations. Since the central claim is comparative, the validity of the comparison depends entirely on evaluating unseen examples. Training-set metrics cannot establish generalization, and without a held-out split the reported accuracy, BLEU, and Levenshtein numbers are consistent with the models having memorized the training data. Additionally, the hyperparameter differences in §5.1 (batch size 64 vs 128, AdamW vs Adam, learning-rate schedules) mean that architecture is not the only variable separating the best model from the baselines. The paper also contains a smaller internal contradiction: §5.4 says the ResNet-LSTM model 'demonstrated improved performance' over CNN-LSTM, yet Table 1 shows ResNet-LSTM is worse on every reported metric. That contradiction further weakens confidence in the written analysis, though it is secondary to the missing held-out evaluation. My recommended verdict is unchanged from the reader's REJECT because the identified flaw undermines the paper's central empirical claim as written. The proposed check is a single, decisive experiment: rerun evaluation on a held-out split and see whether the ViT advantage persists. If it does, the paper could be salvaged with a corrected evaluation protocol; if not, the claim fails outright.","tokens_in":5915,"tokens_out":2698,"duration_ms":83786,"concrete_test":"Inspect the released repository's evaluation script to confirm whether BLEU and Levenshtein are computed on the training split. Then, using a fixed random split of the 200k samples into, say, 160k train and 40k test, retrain all three models under their reported hyperparameters and recompute Table 1 on the held-out test set only. If the ViT's BLEU and Levenshtein advantages over CNN-LSTM persist on test data, the central claim survives this concern; if they shrink or reverse, the claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the ViT encoder with a transformer decoder outperforms the CNN-LSTM baseline on handwritten math-to-LaTeX conversion. For that claim to be valid, the reported comparison must reflect performance on examples the models did not train on. §5.2 states the opposite: 'We measure the Levenshtein distance and BLEU-4 score for predicted sequences of a subset of the training set.' Taken literally, Table 1's BLEU (0.557 vs 0.429) and Levenshtein (0.354 vs 0.401) are training-set fits, which can favor the larger or differently tuned transformer model without implying better recognition on new inputs. The paper provides no train/validation/test split, and §5.1 further confounds architecture with batch size (64 vs 128), optimizer (AdamW vs Adam), and learning-rate schedule. This is not a quibble about metric choice; it is an internal inconsistency about what the reported numbers measure. The natural fix is to hold out a balanced subset before any training and report all metrics on that subset alone. Until that is done, the central claim remains unverified for unseen handwritten expressions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a vision-transformer (ViT) encoder with a transformer decoder for converting handwritten mathematical expression images into LaTeX sequences, and compares it against a CNN-LSTM baseline and a ResNet-LSTM baseline on 200,000 samples from Im2latex-100k/230k. The authors report loss, masked loss, accuracy, BLEU-4, and Levenshtein distances, and claim that the ViT architecture 'consistently outperforms' the baseline CNN-RNN framework. However, the key sequence-similarity metrics are stated to be computed on a subset of the training set, and no held-out test evaluation is provided.","tokens_in":106,"tokens_out":5739,"duration_ms":66733,"significance":"If the claimed results were based on held-out data, the paper would provide a useful empirical comparison of encoder-decoder architectures for math-expression recognition, and the public code repository would support reproducibility. The work also usefully includes a pretrained ResNet50 encoder variant. Unfortunately, the evaluation as described does not measure generalization: the main metric table refers to training-set predictions, and the model comparisons are confounded by different optimizers, batch sizes, and learning-rate schedules. These issues undermine the central conclusion, so the paper in its current form does not constitute a reliable contribution.","major_comments":[{"comment":"The paper states: 'We measure the Levenshtein distance and BLEU-4 score for predicted sequences of a subset of the training set.' Taken literally, this means that the BLEU and Levenshtein values in Table 1 are computed on training examples, so they reflect how well the models memorize their training data rather than how well they recognize unseen handwritten expressions. The central claim of the abstract and §5.4—that the vision transformer outperforms the baseline CNN-RNN framework—is therefore unsupported. The authors must define a held-out test split before training and report all sequence-similarity metrics exclusively on that split.","section":"§5.2, Table 1"},{"comment":"The compared architectures are trained with different hyperparameters: the CNN-LSTM and ResNet-LSTM models use batch size 128 and Adam with learning rate 0.001, while the vision transformer uses batch size 64 and AdamW with a decaying learning rate from 1e-4 to 1e-6. Consequently, the performance differences in Table 1 cannot be attributed solely to the architectural change from CNN to ViT; they could equally result from the optimizer, batch size, or learning-rate schedule. A controlled comparison should keep these factors fixed (or at least run ablations) before drawing conclusions about encoder choice.","section":"§5.1, Table 1"},{"comment":"Each row of Table 1 appears to correspond to a single run. No standard deviations, confidence intervals, or significance tests are reported. With only one seed per condition, the observed differences (e.g., BLEU 0.557 vs 0.429) cannot be distinguished from run-to-run variation. At minimum, the authors should report results over multiple seeds with error bars, or a paired significance test.","section":"§5.4, Table 1"},{"comment":"The manuscript does not specify how the 200,000 data points are split into training, validation, and test sets, nor does it state whether early stopping used a validation set. This lack of a clear data-split protocol makes it impossible to know whether any model selection occurred on the same examples used for the reported metrics. The authors should provide the exact split sizes and the criterion used for early stopping.","section":"§5.1"}],"minor_comments":[{"comment":"The abstract contains a stray superscript '2' after 'domain' ('domain 2'), which appears to be a leftover footnote marker.","section":"Abstract"},{"comment":"The target input size for ResNet50 is given as '254 × 254'; the standard ResNet50 input size is 224 × 224, so the dimension should be checked and corrected.","section":"§4.2"},{"comment":"Several references are unrelated to the topic of this paper (e.g., Jayanth et al. 2015 and Sundararaj et al. 2015 on opinion summarization, and Sundararaj's 2024 thesis). These citations appear to be self-promotional and should be removed or replaced with relevant literature.","section":"References"},{"comment":"The related-work discussion of the CNN-LSTM baseline does not explicitly cite the actual baseline used (e.g., Deng et al. 2016 or Wang and Liu 2019), even though these are included in the reference list; the baseline Section 4.1 should be connected to the cited prior work.","section":"§2, §4.1"}],"recommendation":"reject","confidential_remarks":"This appears to be a preliminary course-project style paper. The most serious issue is that the main evaluation is done on training data, which invalidates the central comparative claim. Even after fixing this, the comparison is confounded by hyperparameter differences, and there is no comparison to established test sets such as CROHME. The citation list also contains several unrelated self-citations. Given the scope of the required changes, I do not see a path to acceptance without fundamentally redoing the experimental evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this before anything else: the paper's headline result is not supported as written. Section 5.2 says the Levenshtein distance and BLEU-4 score are computed on a subset of the training set. If that sentence is literal, Table 1 measures how well the models memorized training examples, not how well they recognize new handwriting. That is a load-bearing flaw, not a quibble. The authors also give no train/validation/test split, no error bars, and they compare architectures under different batch sizes, optimizers, and learning-rate schedules, so architecture is confounded with optimization.\n\nThe paper does have some virtues. The architecture descriptions are clear and honest. The task is a real usability problem, and the choice of the im2latex and MathWriting datasets is sensible. The authors provide a GitHub link and describe their setup in enough detail that someone could reproduce the experiments, which is more than many class-project papers do. They also cite the relevant transformer and image-to-LaTeX literature, including prior work that already applies ViT-style encoders to this task. So the paper is not vacuous; it is a straightforward implementation benchmark.\n\nBut the soft spots are serious. Besides the training-set evaluation, there is an internal contradiction: Section 5.4 says the ResNet-LSTM model demonstrated improved performance over the CNN-LSTM model, but the table shows CNN-LSTM winning on every metric. The absence of any comparison with published image-to-LaTeX systems also weakens the claim of higher overall accuracy in the abstract. And the novelty is low: a ViT encoder with a transformer decoder is a routine combination of well-known components. The only new content is the particular implementation and comparison, which is fine for a technical report but not for a research paper.\n\nWho is this for? A reader who wants a quick, readable account of how a simple ViT encoder compares to CNN-LSTM on formula images might skim it, but they should not trust the numbers. This is not a paper for a serious journal or conference in its current state. My recommendation: reject as is. If the authors re-run the evaluation on a proper held-out test set, fix the contradictory prose, and report error bars, it could become a modest but honest benchmark. As submitted, it does not deserve referee time because the central claim is unverifiable from the reported evidence.","headline":"The central ViT-beats-CNN-LSTM claim is unsupported because the reported BLEU/Levenshtein numbers are computed on a training-set subset, and the paper's own text contradicts its results table.","tokens_in":728,"tokens_out":733,"would_cite":false,"duration_ms":24259,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A patch-based vision transformer with a transformer decoder converts handwritten math images into LaTeX more accurately than CNN-LSTM and ResNet-LSTM baselines, the paper claims.","keywords":["vision transformer","handwritten mathematical expression recognition","image to LaTeX","encoder-decoder architecture","CNN-LSTM baseline","BLEU-4","Levenshtein distance","im2latex dataset"],"falsifier":"Compute BLEU-4, Levenshtein distance, and token accuracy on a held-out split of the same datasets, with the three architectures trained under matched batch sizes and optimization schedules; if the vision transformer does not beat both recurrent baselines on unseen formulas, the paper's central claim is refuted.","tokens_in":5589,"feed_emoji":"✍️","tokens_out":6607,"duration_ms":58735,"temperature":0.7,"pith_summary":"The paper sets out to show that replacing the CNN encoder in an image-to-LaTeX system with a vision transformer improves the translation of handwritten and digital mathematical expressions into LaTeX code. It builds a CNN-LSTM baseline, a variant with a pretrained ResNet50 encoder, and a patch-based vision transformer encoder with a transformer decoder, and reports that the vision transformer achieves the best loss, accuracy, masked accuracy, BLEU-4, and Levenshtein distance on its 200,000-sample subset of the im2latex datasets. If the comparison holds up, the result points to self-attention over image patches as a stronger feature extractor for math recognition than the recurrent baselines, with further gains available from tuning layers, heads, and patch size.","feed_headline":"Vision transformer beats CNN-LSTM at turning math images into LaTeX","feed_subtitle":"Patch-based attention encoder lifts accuracy, BLEU, and edit-distance scores over recurrent baselines in the reported runs.","key_machinery":"The load-bearing object is the vision transformer encoder: a 50x200 grayscale image is divided into 10x10 patches, giving a sequence of 100 patch embeddings with positional encodings, processed by 8 transformer layers with 4 attention heads and an MLP of 2048/1024 units. The decoder is a standard transformer with 4 layers and 8 heads using both self-attention over generated tokens and cross-attention over encoded image features; this cross-attention is what aligns each LaTeX token to the relevant visual region. The baselines are a three-block 3x3 convolution/max-pool CNN encoder with an LSTM decoder, and the same decoder with a pretrained ResNet50 encoder adapted to grayscale by replicating the channel three times.","core_discovery":"The paper's central claim is that the vision transformer architecture outperforms the CNN-RNN framework on the task of generating LaTeX from math-expression images. On the reported metrics, the vision transformer reaches a loss of 0.520, an accuracy of 0.873, a masked accuracy of 0.641, a BLEU score of 0.557, and a Levenshtein distance of 0.354, compared with CNN-LSTM (0.647, 0.847, 0.600, 0.429, 0.401) and ResNet-LSTM (0.781, 0.819, 0.520, 0.372, 0.435). The authors attribute this to positional embeddings and attention in both encoder and decoder, which let the model relate distant image regions while generating the LaTeX token sequence.","pith_inferences":["Because Section 5.2 says the BLEU-4 and Levenshtein metrics were computed on a subset of the training set, the generalization comparison in the table may reflect memorization; a held-out evaluation is needed before accepting the ranking.","The three architectures differ not only in encoder but also in batch size and optimizer (128 with Adam vs 64 with AdamW and a decaying learning rate), so part of the reported gap could come from training configuration rather than the encoder family.","The fixed 10x10 patch grid on a 50x200 image yields a 5-by-20 sequence; testing rectangular patches matched to the wide aspect ratio could change how much spatial structure the encoder preserves.","Since the im2latexv2 data mixes printed and handwritten formulas, the same architecture could be evaluated separately on each domain to see where the attention-based encoder helps most."],"forward_implications":["If the reported numbers are representative, vision transformers become a competitive encoder choice for mathematical expression recognition, not just for classification.","Masked accuracy and BLEU-4 improvements indicate that more of the predicted token sequence matches the reference LaTeX exactly, while lower Levenshtein distance means the generated strings are closer to correct even when not exact.","The paper's own conclusion is that tuning the number of transformer layers, attention heads, and patch size, and training on larger datasets, would improve accuracy further.","The ResNet-LSTM comparison suggests that pretrained visual encoders converge and perform differently from scratch-trained encoders, but in this setup neither matches the vision transformer."],"supporting_citations":[{"why":"Supplies the transformer self-attention architecture used for the decoder and for the attention-based comparison.","marker":"Vaswani et al. [2023]"},{"why":"Provides the Vision Transformer patch-based encoder design that the paper adapts with 10x10 patches.","marker":"Dosovitskiy et al. [2021]"},{"why":"Defines the CNN-encoder/LSTM-decoder image-to-sequence framework used as the paper's baseline.","marker":"Genthial et al. [2016]"},{"why":"Provides the im2latex-100k dataset of image-formula pairs used for training.","marker":"Kanervisto [2016]"},{"why":"Provides the im2latex-230k/im2latexv2 dataset with handwritten examples that broaden the training data.","marker":"Gervais et al. [2024]"},{"why":"Represents the earlier non-attention approaches the paper positions its work against.","marker":"Schechter et al. [2017]"},{"why":"Defines the LSTM unit used in the recurrent decoder baselines.","marker":"Hochreiter and Schmidhuber [1997]"}],"fun_headline_variants":["Vision transformer beats CNN-LSTM on handwritten math to LaTeX","Vision transformer outperforms CNN-LSTM for math-to-LaTeX","ViT edges out CNN-LSTM on transforming math images to LaTeX","Attention wins: vision transformer tops CNN-LSTM for math LaTeX"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the reported quality scores being measured on examples the models had not seen during training; the paper states in Section 5.2 that the scores were computed on a subset of the training set, so if that statement is literal the comparison shows memorization rather than generalization and the claimed advantage is not established.","fun_headline_variants_meta":{"raw":{"variants":["Vision transformer beats CNN-LSTM on handwritten math to LaTeX","Vision transformer outperforms CNN-LSTM for math-to-LaTeX","ViT edges out CNN-LSTM on transforming math images to LaTeX","Attention wins: vision transformer tops CNN-LSTM for math LaTeX"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001038,"raw_usage":{"total_tokens":4337,"prompt_tokens":883,"completion_tokens":3454,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":3380}},"tokens_in":499,"tokens_out":3454,"duration_ms":23735,"temperature":1.0,"reasoning_tokens":3380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:00:42.121093+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute BLEU-4, Levenshtein distance, and token accuracy on a held-out split of the same datasets, with the three architectures trained under matched batch sizes and optimization schedules; if the vision transformer does not beat both recurrent baselines on unseen formulas, the paper's central claim is refuted.","supporting_citations":[],"review_version":1}