REVIEW 3 major objections 4 minor 26 references
Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Rendering each word as a fixed-size image before feeding it to a transformer decoder makes next-token prediction nearly immune to character-level noise and transfers better to non-Latin scripts than an English-trained subword tokenizer.
desk verdict Word-as-image generative LM is a genuinely new architecture with a plausible robustness mechanism, but the headline margin is measured under an assumption the authors admit is false in practice. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the adaptive renderer: each entry of the 32,001-token BPE vocabulary is pre-rendered as a fixed-size grayscale image, with font size scaled so longer words fit the same canvas, and the resulting image bank is stored as an embedding layer. A learnable linear projector maps each image into the hidden dimension of a LLaMA-style decoder. This makes token-to-image conversion an O(1) lookup at training and inference and keeps the next-token cross-entropy loss intact, while giving out-of-vocabulary and noisy tokens a visual neighborhood instead of an empty embedding.
What would settle it
Re-run the LAMBADA noise experiments without freezing token boundaries: tokenize the clean sentence, inject noise, re-tokenize the noised sentence with the same BPE tokenizer, and render the resulting token sequence as pixel images. If perplexity then grows by a similar factor to the token model's, the claimed noise immunity is an artifact of fixed tokenization.
Extended reading notes
Core claim
Under the paper's setup, the discovery is that a visual token representation can carry the next-token prediction objective without any patch slicing, OCR, or text-image modality bridge. Because each word is rendered alone and stored as a lookup table, a corrupted word still produces an image close to the original word's image; the learned linear projection then maps that image near the original word's embedding. The paper reports that this makes perplexity almost flat as character noise increases, and makes an English-pretrained decoder noticeably better at non-Latin script tokens than the same decoder with subword embeddings. The boundary of the claim is equally explicit: the experiments ho
Load-bearing premise
The robustness comparison assumes that adding character noise does not change tokenization; in practice it usually does, and the paper leaves re-tokenized noisy input for token-level pixel methods as an open problem.
Editorial extensions
If this is right
- At fixed token boundaries, character-level noise of 10-50% moves pixel-model perplexity from 139 to at most 485, so pixel embeddings can serve as a noise-stable input layer for standard decoder training.
- A pixel model trained only on English data models Russian, Chinese, Japanese, and Hindi with perplexity one to two orders of magnitude below the same-size subword model, suggesting a tokenizer-bypass route for script diversity.
- Because rendering is a precomputed lookup, the method adds no OCR or generative image decoding to the pipeline; the output head still produces text token IDs.
- The robustness mechanism transfers to downstream tasks: on SST-2 the pixel model's accuracy drop under noise is 0.3 versus 0.6 for the token model.
Reading between the lines
- If real orthographic attacks re-tokenize the input, the reported 3.5x versus 294x margin is an upper bound; the practical margin will depend on how much visual similarity survives across changed token boundaries.
- The same visual-similarity mechanism should extend to other glyph-preserving corruptions such as OCR errors, leetspeak, homoglyphs, and dialectal spellings; testing those would generalize the claim beyond random character replacement.
- A hybrid that renders characters or fixed-size word slices instead of BPE tokens could close the re-tokenization gap while keeping the visual neighborhood; this is a natural next experiment.
- The Latin-script results, where the token model beats the pixel model, suggest the best system might switch representations by script, using pixel embeddings only when the tokenizer lacks coverage.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a pixel-based generative language model that renders each word as a fixed-size image, replaces the token embedding layer of a LLaMA decoder with a pixel embedding layer, and trains with ordinary next-token prediction. The authors evaluate the model on multilingual LAMBADA, WMT24, and SST-2, reporting that the pixel model degrades far less than a standard token-based LLaMA when characters are randomly replaced by lookalike or multilingual characters. They also report better perplexity for non-Latin scripts. The central claim is that word-level pixel embeddings preserve enough visual similarity under character-level noise to make next-token prediction robust, and that the same mechanism transfers to non-Latin scripts.
Significance. If the central claim were established end-to-end, the paper would make a useful contribution: it offers a simple way to bring pixel representations into autoregressive text generation without an OCR module, and it identifies a concrete mechanism (visual similarity of rendered words) that could improve robustness to orthographic variation. The paper also honestly acknowledges a major limitation in Section 5. The method itself is straightforward and the training setup is reproducible in principle from the provided hyperparameters. However, the headline robustness numbers in Table 4 are obtained under an assumption that the authors themselves state is false in practice, so the significance of the claimed robustness result is currently not established for actual orthographic attacks.
major comments (3)
- [Section 5 and Section 4.3, Table 4] The robustness evaluation is not end-to-end. Section 5 states that noise 'inevitably alters the tokenization in most cases' and that re-tokenization is left as an open problem. Under a real attack, the noised sentence would be re-tokenized by the BPE tokenizer, producing a different sequence of tokens with different lengths and composition. The argument in Section 3.4 that P(I(t1),...,I(tn)) ≈ P(I(t1'),...,I(tm')) assumes the token sequence is the same after noise, which contradicts Section 5. The reported 3.49x vs 294x perplexity growth in Table 4 is therefore measured under a fixed-tokenization oracle, not under the actual inference pipeline. The conclusion's claim that robustness is 'demonstrated' is too strong. The authors should either evaluate under re-tokenization (including byte-fallback tokens) or explicitly reframe the robustness claim as applying only to a constrained setting.
- [Section 4.2, Table 2] The multilingual advantage in non-Latin scripts is confounded by the choice of baseline. The token-based LM uses an English-trained BPE tokenizer, which has poor coverage for Russian, Chinese, Japanese, and Hindi; its extreme perplexities (e.g., 424,341 for Chinese) may reflect tokenizer OOV rather than a fundamental limitation of text-based embeddings. The paper attributes the pixel model's smaller relative increases to 'better visual clues,' but the experimental design does not isolate visual modeling from tokenizer coverage. A stronger baseline would use a multilingual tokenizer or a byte-level model. As reported, the evidence supports the weaker claim that the pixel model is more robust than an English-tokenizer LM on non-Latin scripts, not that pixel representations are intrinsically superior for multilingual modeling.
- [Section 3.2 and Section 3.3] The paper describes the pixel approach as reducing dependence on tokenization, but the training pipeline is actually tied to the LLaMA BPE vocabulary: 'we pre-render all tokens from the tokenizer’s vocabulary and store them in a dictionary.' The renderer lookup table is constructed from this fixed vocabulary, and training loss is computed over token IDs. OOV tokens can be rendered at inference, but the model has not been trained on such renderings. This is not by itself an error, but it weakens the claim that the method is 'tokenizer-free' and it makes the robustness behavior on novel tokens an untested extrapolation. The paper should report how often noisy inputs fall outside the pre-rendered vocabulary and how the model behaves when they do.
minor comments (4)
- [Table 1] The header has 'image weight' instead of 'image width'. Also, the table lists hyperparameters but does not report the exact number of trainable parameters for PM and LM; the text says the models are 'comparable' but gives no parameter counts.
- [Figure 3] The caption says cosine similarity is 0.89, while the main text says 0.86 for the same example. Please correct the inconsistency.
- [Section 4.3] The noise injection procedure is only described as 'randomly sampled from a predefined noise dictionary as introduced in PIXAR.' For reproducibility, the exact noise dictionary, replacement sampling process, and whether noise is applied to all characters or only alphabetic characters should be specified.
- [Section 4.2] The text interpreting Table 3 is syntactically ambiguous: 'the language maintains better cross-lingual generalization' should specify whether 'the language' refers to the pixel model or the language model. The table shows the pixel model has larger relative increases for Latin languages, which is the opposite of the wording if 'language' means 'language model.'
Circularity Check
Robustness advantage in Table 4 is measured under the fixed-tokenization assumption admitted in Section 5; the claimed end-to-end robustness in the Conclusion therefore reduces to that assumption.
-
self definitional
[Section 4.3 / Table 4; Section 5 (Limitation); Section 6 (Conclusion)]
"In our robustness experiments against orthographic noise, the performance of the pixel generative language model is evaluated under the assumption that the tokenization of a noised sentence remains identical to that of the original clean sentence. In practice, however, introducing noise inevitably alters the tokenization in most cases."
The headline margin (PM 3.49x vs LM 294x perplexity growth, Table 4) is produced under the Section 5 assumption that token boundaries of clean and noised input are identical. With boundaries fixed, each noised word is rendered as a visually similar image of the same token, so the Section 3.4 approximation P(I(t1),...,I(tn)) ≈ P(I(t1'),...,I(tm')) holds by construction, while the LM must embed corrupted strings that often fall outside its vocabulary. The paper concedes that real noise 'inevitably alters the tokenization,' changing the token sequence (n≠m) and breaking the one-to-one image correspondence on which this approximation rests. The Conclusion's 'demonstrated stronger robustness against orthographic noise' therefore restates the evaluation's fixed-tokenization input condition as a
full rationale
There is no self-citation chain, imported uniqueness theorem, or ansatz-by-citation in this paper. The multilinguality results are measured against external benchmarks (LAMBADA, WMT24, SST-2) and are self-contained experimental comparisons with a matched LLaMA baseline. The method itself is an independent architectural proposal. The circularity is localized to the robustness claim: the evaluation protocol in Section 4.3 implements precisely the scenario that makes pixel representations advantageous (fixed tokenization), and Section 5 explicitly admits that this is not what happens under a real orthographic attack, where re-tokenization is inevitable. The reported 3.49x-vs-294x margin is therefore an artifact of the evaluation's input condition rather than a demonstrated end-to-end robustness property. Because the paper is transparent about the assumption, this is partial circularity rather than hidden equivalence; the central architecture and multilingual findings retain independent content.
Assumptions & free parameters
free parameters (3)
- image height (20 px) =
20
- image width (50 px) =
50
- basic font size (10) =
10
assumptions (3)
- ad hoc to paper Tokenization of a noised sentence is identical to that of the clean sentence (Section 5).
- domain assumption Rendering a word as a fixed-size image preserves sufficient orthographic information for the decoder to learn semantics (Section 3.1).
- domain assumption The LLaMA BPE tokenizer trained on English BookCorpus/Wikipedia is an appropriate common tokenizer for comparing multilingual perplexity (Section 4.1).
Cite this review
Pith. "Pith review of Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach." pith.science (2026). https://pith.science/paper/2NV6IQEJ
@misc{pith2026250821206,
author = {Pith},
title = {Pith review of: Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach},
year = {2026},
howpublished = {\url{https://pith.science/paper/2NV6IQEJ}},
note = {Machine review of arXiv:2508.21206}
}
read the original abstract
Autoregressive language models are vulnerable to orthographic attacks, where input text is perturbed with characters from multilingual alphabets, leading to substantial performance degradation. This vulnerability primarily stems from the out-of-vocabulary issue inherent in subword tokenizers and their embeddings. To address this limitation, we propose a pixel-based generative language model that replaces the text-based embeddings with pixel-based representations by rendering words as individual images. This design provides stronger robustness to noisy inputs, while an extension of compatibility to multilingual text across diverse writing systems. We evaluate the proposed method on the multilingual LAMBADA dataset, WMT24 dataset and the SST-2 benchmark, demonstrating both its resilience to orthographic noise and its effectiveness in multilingual settings.
Figures
Reference graph
Works this paper leans on
-
[1]
Nadav Borenstein, Phillip Rust, Desmond Elliott, and Isabelle Augenstein. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.7 PHD: pixel-based language modeling of historical documents . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 , pages 87--107. Association for Com...
-
[2]
Yekun Chai, Qingyi Liu, Jingwu Xiao, Shuohuan Wang, Yu Sun, and Hua Wu. 2024. https://doi.org/10.18653/V1/2024.EMNLP-MAIN.182 Autoregressive pre-training on pixels and texts . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024 , pages 3106--3125. Association for Compu...
-
[3]
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.58 Revisiting pre-trained models for C hinese natural language processing . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 657--668, Online. Association for Computational Linguistics
-
[4]
Daniel Deutsch, Eleftheria Briakou, Isaac Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, and Markus Freitag. 2025. https://arxiv.org/abs/2502.12404 WMT24++: Expanding the Language Coverage of WMT24 t...
arXiv 2025
-
[5]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, and 1 others. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929
arXiv 2020
-
[6]
Steffen Eger, G \" o zde G \" u l Sahin, Andreas R \" u ckl \' e , Ji - Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. 2019. https://doi.org/10.18653/V1/N19-1165 Text processing like humans do: Visually attacking and shielding NLP systems . In Proceedings of the 2019 Conference of the North American Chapter...
-
[7]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Communications of the ACM, 63(11):139--144
2020
-
[8]
Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. https://doi.org/10.48550/ARXIV.2307.10169 Challenges and applications of large language models . CoRR, abs/2307.10169
Show all 26 references
-
[9]
Girshick
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll \' a r, and Ross B. Girshick. 2022. https://doi.org/10.1109/CVPR52688.2022.01553 Masked autoencoders are scalable vision learners . In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Or...
2022
-
[10]
Taku Kudo and John Richardson. 2018. https://doi.org/10.18653/V1/D18-2012 Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, ...
2018 doi
-
[11]
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. 1989. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541--551
1989
-
[12]
Jonas F Lotz, Hendra Setiawan, Stephan Peitz, and Yova Kementchedjhieva. 2025. Overcoming vocabulary constraints with pixel-level fallback. arXiv preprint arXiv:2504.02122
2025 arXiv
-
[13]
Alberto Mu \ n oz - Ortiz, Verena Blaschke, and Barbara Plank. 2025. https://aclanthology.org/2025.coling-main.427/ Evaluating pixel language models on non-standardized languages . In Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, A...
2025
-
[14]
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. https://doi.org/10.5281/zenodo.2630551 The lambada dataset
2016 doi
-
[15]
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners
2019
-
[16]
Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott
Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. 2023. https://openreview.net/forum?id=FkSp8VW8RjH Language modelling with pixels . In The Eleventh International Conference on Learning Representations, ICLR 2023, Kig...
2023
-
[17]
Elizabeth Salesky, David Etter, and Matt Post. 2021. https://doi.org/10.18653/V1/2021.EMNLP-MAIN.576 Robust open-vocabulary translation from visual text representations . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Vir...
2021 doi
-
[18]
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/V1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, B...
2016 doi
-
[19]
Manning, Andrew Ng, and Christopher Potts
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. https://www.aclweb.org/anthology/D13-1170 Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of the 2013 Conferenc...
2013
-
[20]
Zijun Sun, Xiaoya Li, Xiaofei Sun, Yuxian Meng, Xiang Ao, Qing He, Fei Wu, and Jiwei Li. 2021. https://doi.org/10.18653/V1/2021.ACL-LONG.161 Chinesebert: Chinese pretraining enhanced by glyph and pinyin information . In Proceedings of the 59th Annual Meeting of the Association...
2021 doi
-
[21]
Yintao Tai, Xiyang Liao, Alessandro Suglia, and Antonio Vergari. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-ACL.874 PIXAR: auto-regressive language modeling in pixel space . In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virt...
2024 doi
-
[22]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971
2023 arXiv
-
[23]
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, and 12 o...
2016 arXiv
-
[24]
Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. https://doi.org/10.1109/ICCV.2015.11 Aligning books and movies: Towards story-like visual explanations by watching movies and reading books . In 2015 IEEE I...
2015 doi
-
[25]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[26]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.