REVIEW 4 major objections 5 minor 44 references
A Case Against Implicit Standards: Homophone Normalization in Machine Translation for Languages that use the Ge'ez Script
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Mapping Amharic homophones to one character before training harms MT transfer to Tigrinya and Ge'ez; normalizing after translation instead recovers up to 1.03 BLEU.
desk verdict Honest, useful study of a real preprocessing problem; the headline transfer claim is not backed by Table 5 and needs a rewrite. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The paper's central machinery is the contrast among three normalization settings: No-Norm (unmodified data), H-only (mapping same-sound characters to the most frequent character), and HSL (a more aggressive scheme from prior work that also merges similar-sound and labialized characters, applied only to Amharic). The proposed intervention, post-inference normalization, is the same character-mapping idea applied to model outputs and reference translations only when computing BLEU and ChrF, preserving all spellings in the training data. The cross-lingual transfer protocol, fine-tuning an English-Amharic model and then continuing fine-tuning on English-Tigrinya or English-Ge'ez, is what exposes the implicit-standard problem, because it carries Amharic's merged character identities into languages where those characters are still distinct.
What would settle it
Train NLLB-600M directly on Tigrinya, and train transfer models by first fine-tuning on Amharic (No-Norm versus H-Only) and then fine-tuning on Tigrinya with identical total compute and epochs; if the normalized and unnormalized transfer models produce statistically indistinguishable BLEU scores across several seeds, the claim that normalization hurts transfer is falsified.
Extended reading notes
Core claim
On the paper's own terms: homophone normalization is a preprocessing choice that trades away linguistic flexibility for automatic-metric gains, and the trade is worse than previous work suggested. In monolingual experiments, the from-scratch Transformer performed best with no normalization for both Amharic and Tigrinya, and the aggressive HSL scheme hurt NLLB-600M; only H-only on NLLB for Amharic showed a marginal gain (+0.62 BLEU). In cross-lingual transfer, the Amharic No-Norm model transferred best to Tigrinya with the Transformer, and for Ge'ez the No-Norm and H-only transfer models beat the HSL one; normalized base models tended to repeat words rather than produce translations. The proposed alternative, post-inference normalization, applies the homophone mapping to model predictions and references when scoring, and raised BLEU by 0.24 (Transformer), 0.69 (NLLB-600M), and 1.03 (on the Belay et al. no-norm model) without touching training data. The paper concludes that normalization should be understood as setting an implicit orthographic standard, and offers post-inference normalization as a middle-ground intervention.
Load-bearing premise
The transfer conclusions compare a baseline fine-tuned only on Tigrinya or Ge'ez with models fine-tuned on Amharic first and then on the target language, so normalization effects are entangled with the extra Amharic training step, and the Ge'ez test set is small with near-zero BLEU scores.
Editorial extensions
If this is right
- If correct, MT teams for Amharic can retain most of the automatic-score benefit of normalization by applying it only when scoring outputs, without restricting the spellings the model learns.
- Models trained on normalized Amharic will keep failing on alternative spellings of the same words, which the paper's post-inference approach avoids by preserving diverse spellings in training.
- The aggressive HSL scheme, which merges similar-sound and labialized characters, transfers worst to Ge'ez, indicating that broader normalization amplifies the implicit-standard problem.
- The evaluation metric itself, exact-match n-gram overlap, is part of what motivates normalization, so scoring-time normalization is a direct remedy within the current metric ecosystem.
Reading between the lines
- We infer that post-inference normalization could be adopted in other Ge'ez-script tasks, such as speech recognition or information retrieval, wherever homophone variants are scored.
- The implicit-standard argument extends beyond homophones to any preprocessing choice, such as tone-diacritic removal or forced transliteration, that bakes one variety's conventions into a model shared by related languages.
- A larger human evaluation than the 50-sample inspection reported would show whether the BLEU gains from post-inference normalization correspond to perceived translation quality.
- A direct test with matched fine-tuning budgets and multiple random seeds would separate the normalization effect from the extra Amharic fine-tuning step in the transfer experiments.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates homophone normalization in machine translation for Amharic, Tigrinya, and Ge'ez, three languages that use the Ge'ez script. It evaluates three normalization settings (No-Norm, H-only, and HSL) on monolingual training and cross-lingual transfer, and proposes a post-inference intervention that applies normalization to model predictions and references only at evaluation time. The authors report that homophone normalization negatively affects cross-lingual transfer to Tigrinya and Ge'ez, and that post-inference normalization can improve BLEU scores by up to 1.03 while preserving orthographic variation in training data. The paper argues against implicit standardization of Amharic preprocessing and offers a simple alternative that does not constrain the model's ability to handle diverse spellings.
Significance. If the central claims are supported, the paper makes a valuable contribution to the discussion of language-aware preprocessing for low-resource languages. It is one of the few studies to compare different homophone normalization schemes and to examine their effect on cross-lingual transfer within the Ge'ez script family. The proposed post-inference normalization is a transparent, low-cost intervention that could be adopted by practitioners who want metric gains without altering training data. The qualitative analysis, including the observation that normalized base models tend to repeat characters, adds practical insight. However, as detailed in the major comments, the evidence for the negative effect on cross-lingual transfer is mixed and needs to be either clarified or restricted.
major comments (4)
- [Sec. 4.3.2, Table 5] The claim that homophone normalization negatively affects cross-lingual transfer is not consistently supported by the reported results. For the NLLB-600M model on Ge'ez, the No-Norm transfer setting obtains BLEU 5.22, while H-only and HSL obtain 6.33 and 6.31 respectively, contradicting the claim of a uniform negative effect. For NLLB-600M on Tigrinya, the differences are negligible (21.55 vs. 21.63 and 21.68). The prose in Sec. 4.3.2 states that 'the No-Norm and H-Only setting provides better BLEU ... compared to HSL' for Ge'ez, but this is only true for the from-scratch Transformer; the NLLB row shows the opposite pattern. The authors should either restrict the claim to the settings that support it or provide a substantive explanation for the NLLB Ge'ez result.
- [Sec. 4.3.2, Table 5] No error bars, confidence intervals, or significance tests are reported for any BLEU or ChrF result. The Ge'ez test set is small (1.9k sentences) and BLEU scores range from 1.46 to 6.33, near zero, so the observed differences may be within noise. This is especially concerning for conclusions drawn from small gaps, such as the 0.08 BLEU difference between No-Norm and H-only on NLLB Tigrinya. The paper should include a measure of variability (e.g., bootstrap confidence intervals) or at least explicitly acknowledge this limitation in the interpretation.
- [Sec. 4.3.2] The comparison between the no-transfer baseline and the transfer models is confounded by additional fine-tuning. The no-transfer NLLB model is fine-tuned only on Tigrinya/Ge'ez data, whereas the transfer models are first fine-tuned on Amharic and then fine-tuned again on Tigrinya/Ge'ez. Thus, the statement that 'the model directly finetuned ... performed better than the ones first finetuned on Amharic' reflects a difference in training schedule, not necessarily a normalization effect. The normalization-specific claim should be evaluated only within the transfer settings (No-Norm vs. H-only vs. HSL), where the amount of fine-tuning is matched.
- [Sec. 4.4, Table 6] The post-inference normalization result is reported on a test set that was filtered to homophone-rich sentences (at least 9 homophone characters for Amharic and 17 for Tigrinya). This filtering is appropriate for isolating the effect of normalization, but the claim that post-inference normalization 'can achieve an increase in BLEU score of up to 1.03' should be framed as specific to this filtered subset. The paper does not report whether the gain generalizes to unfiltered test sets, so a caveat is needed to prevent overgeneralization.
minor comments (5)
- [Sec. 4.4] Typo: 'pots-inference' should be 'post-inference'.
- [Sec. 4.3.2] Typo: 'BLUE' should be 'BLEU' in the Ge'ez results paragraph.
- [Sec. 4.3.2] Typo: 'dateset' should be 'dataset' in the character-count analysis.
- [Footnotes and data availability] The footnote on data collection says 'We will add a link to data, code, and models upon publication', while the abstract provides a GitHub URL; please make the availability statement consistent and ensure the link is active.
- [Table 2] Table 2 would be clearer with a note explaining that HSL normalization is only defined for Amharic, which is why Tigrinya and Ge'ez have no HSL row.
Circularity Check
No significant circularity: the paper's claims are empirical comparisons plus a transparent scoring-time intervention.
full rationale
The paper does not derive any first-principles prediction from fitted parameters or from an unverified self-citation. The central transfer and monolingual claims (RQ2/RQ3, Secs. 4.3.1 and 4.3.2) are direct experimental comparisons on held-out test sets, with no parameters fitted to the evaluated data and no equation that reduces a conclusion to its input. The post-inference normalization result (Sec. 4.4) is explicitly a scoring-time intervention: the authors state they 'applied normalization to the reference and predictions after inference,' and the BLEU increase is an arithmetic consequence of computing exact n-gram overlap on normalized strings. This is transparently presented as their proposed intervention rather than as a hidden prediction or a fitted parameter renamed as an outcome. Self-citations to prior work by the same group (e.g., Yimam et al. 2021 for the HSL normalization script, Belay et al. 2022 for a baseline model, Ademtew and Birbo 2024 for the AGE dataset, Nigatu et al. 2024 for framing) are used as tools, data, or background; none of them does load-bearing work as an unverified 'uniqueness theorem' or an ansatz that predetermines the conclusion. The mixed results in Table 5, where normalization sometimes helps NLLB on Ge'ez, are a threat to the strength of the transfer claim but are an empirical-correctness concern, not circularity. The limitations section explicitly acknowledges the scope of human evaluation and reliance on automatic metrics. No circular step meeting the quoted-evidence bar was found.
Assumptions & free parameters
free parameters (3)
- Amharic test-set homophone threshold =
9 characters
- Tigrinya test-set homophone threshold =
17 characters
- Canonical homophone target character =
Most frequent character in dataset
assumptions (5)
- domain assumption Characters that share a sound in Amharic are true homophones and can be mapped to one character without changing meaning.
- domain assumption In Tigrinya and Ge'ez, the same characters have distinct sounds, so normalizing them as in Amharic is linguistically harmful.
- ad hoc to paper The three normalization settings (No-Norm, H-only, HSL) adequately represent preprocessing choices in the literature.
- domain assumption BLEU and ChrF are appropriate evaluation metrics for the study's conclusions.
- ad hoc to paper Filtering test sets to homophone-rich sentences yields a valid evaluation of normalization impact.
Cite this review
Pith. "Pith review of A Case Against Implicit Standards: Homophone Normalization in Machine Translation for Languages that use the Ge'ez Script." pith.science (2026). https://pith.science/paper/ZPLJFQYX
@misc{pith2026250715142,
author = {Pith},
title = {Pith review of: A Case Against Implicit Standards: Homophone Normalization in Machine Translation for Languages that use the Ge'ez Script},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZPLJFQYX}},
note = {Machine review of arXiv:2507.15142}
}
read the original abstract
Homophone normalization, where characters that have the same sound in a writing script are mapped to one character, is a pre-processing step applied in Amharic Natural Language Processing (NLP) literature. While this may improve performance reported by automatic metrics, it also results in models that are not able to understand different forms of writing in a single language. Further, there might be impacts in transfer learning, where models trained on normalized data do not generalize well to other languages. In this paper, we experiment with monolingual training and cross-lingual transfer to understand the impacts of normalization on languages that use the Ge'ez script. We then propose a post-inference intervention in which normalization is applied to model predictions instead of training data. With our simple scheme of post-inference normalization, we show that we can achieve an increase in BLEU score of up to 1.03 while preserving language features in training. Our work contributes to the broader discussion on technology-facilitated language change and calls for more language-aware interventions.
Figures
Reference graph
Works this paper leans on
-
[1]
Solomon Teferra Abate, Michael Melese, Martha Yifiru Tachbelie, Million Meshesha, Solomon Atinafu, Wondwossen Mulugeta, Yaregal Assibie, Hafte Abera, Binyam Ephrem, Tewodros Abebe, Wondimagegnhue Tsegaye, Amanuel Lemma, Tsegaye Andargie, and Seifedin Shifaw. 2018. Parallel Corpora for bi-lingual English - Ethiopian Languages Statistical Machine Translation
work page 2018
-
[2]
Solomon Teferra Abate, Martha Yifiru Tachbelie, and Tanja Schultz. 2020. https://doi.org/10.21437/Interspeech.2020-2856 Multilingual Acoustic and Language Modeling for Ethio - Semitic Languages . In Interspeech 2020, pages 1047--1051. ISCA
-
[3]
Ife Adebara and Muhammad Abdul-Mageed. 2022. https://doi.org/10.18653/v1/2022.acl-long.265 Towards afrocentric NLP for A frican languages: Where we are and where we can go . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3814--3841, Dublin, Ireland. Association for Computational Li...
-
[4]
Henok Ademtew and Mikiyas Birbo. 2024. https://doi.org/10.18653/v1/2024.loresmt-1.14 AGE : Amharic , Ge ’ez and English Parallel Dataset . In Proceedings of the Seventh Workshop on Technologies for Machine Translation of Low - Resource Languages ( LoResMT 2024) , pages 139--145, Bangkok, Thailand. Association for Computational Linguistics
-
[5]
https://library.bu.edu/amharic/Home Research: Language Learning - Amharic : Home
Gabe Adugna. https://library.bu.edu/amharic/Home Research: Language Learning - Amharic : Home
-
[6]
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.614 Do All Languages Cost the Same ? Tokenization in the Era of Commercial Language Models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 9904--9923...
-
[7]
Amsalu Aklilu. 2010. Problems of Writing Homophones without care and its Solution [ Title translated from Amhairc ]
work page 2010
-
[8]
Shaik Johny Basha, Duggineni Veeraiah, Boddu Venkat Charan, Wiltrud Sahithi Joyce Yeddu, and Devalla Ganesh Babu. 2023. https://doi.org/10.1109/ICICT57646.2023.10134103 Detection and Comparative Analysis of Handwritten Words of Amharic Language to English using CNN - Based Frameworks . In 2023 International Conference on Inventive Computation Technologies...
arXiv 2023
Show all 44 references
-
[9]
Tadesse Destaw Belay, Abinew Ali Ayele, Getie Gelaye, Seid Muhie Yimam, and Chris Biemann. 2021. https://doi.org/10.1109/ICT4DA53266.2021.9672229 Impacts of Homophone Normalization on Semantic Models for Amharic . In 2021 International Conference on Information and Communicati...
2021
- [10]
-
[11]
Yohanens Biadgligne and Kamel Smaïli. 2021. https://doi.org/10.1007/978-3-030-85030-2_37 Parallel Corpora Preparation for English - Amharic Machine Translation . In Advances in Computational Intelligence , pages 443--455. Springer, Cham. ISSN: 1611-3349
2021 doi
-
[12]
Yohannes Biadgligne and Kamel Smaili. 2022. https://doi.org/10.1109/ICICT55905.2022.00030 Offline Corpus Augmentation for English - Amharic Machine Translation . In 2022 5th International Conference on Information and Computer Technologies ( ICICT ) , pages 128--135
2022
-
[13]
Sidsel Boldsen and Patrizia Paggio. 2022. https://doi.org/10.18653/v1/2022.acl-long.463 Letters from the past: Modeling historical sound change through diachronic character embeddings . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ...
2022 doi
-
[14]
Adane Kasie Chekole, Tesfa Tegegne Asfaw, Tesfahun Nurrie Mengestie, Belayneh Teshome Kebie, Mengistu Kinfe Negia, and Yohannes Abinet Worku. 2024. https://doi.org/10.1109/ICT4DA62874.2024.10777148 Effect of Parallel Data Processing Model on Bi - Directional English - Khimtagn...
2024
-
[15]
Fitehalew Ashagrie Demilew. 2019. ANCIENT GEEZ SCRIPT RECOGNITION USING DEEP CONVOLUTIONAL NEURAL NETWORK . Software Engineering
2019
-
[16]
Ige, Atnafu Lambebo Tonja, Chiamaka Chukwuneke, Bonaventure F
Chris Emezue, Hellina Nigatu, Cynthia Thinwa, Helper Zhou, Shamsuddeen Muhammad, Lerato Louis, Idris Abdulmumin, Samuel Oyerinde, Benjamin Ajibade, Olanrewaju Samuel, Oviawe Joshua, Emeka Onwuegbuzia, Handel Emezue, Ifeoluwatayo A. Ige, Atnafu Lambebo Tonja, Chiamaka Chukwunek...
-
[17]
Abebawu Eshetu. 2022. https://github.com/Abe2G/Amharic-Simple-Text-Preprocessing-Usin-Python Amharic- Simple - Text - Preprocessing - Usin - Python . Original-date: 2019-08-05T09:30:04Z
2022
-
[18]
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2021. Beyond en...
2021
-
[19]
Negasi Haile, Nuredin Ali, and Asmelash Teka Hadgu. 2023. ERROR ANALYSIS OF TIGRINYA – ENGLISH MACHINE TRANSLATION SYSTEMS
2023
-
[20]
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. https://doi.org/10.18653/v1/2020.acl-main.560 The State and Fate of Linguistic Diversity and Inclusion in the NLP World . In Proceedings of the 58th Annual Meeting of the Association for Co...
2020 doi
- [21]
-
[22]
Shreya Khare, Ashish Mittal, Anuj Diwan, Sunita Sarawagi, Preethi Jyothi, and Samarth Bharadwaj. 2021. https://doi.org/10.21437/Interspeech.2021-2062 Low Resource ASR : The Surprising Effectiveness of High Resource Transliteration . In Interspeech 2021, pages 1529--1533. ISCA
2021 doi
-
[23]
Bushra Kidanemariam. 2019. https://github.com/Bushra-KB/Amharic-NLP-Tools-in-JAVA/tree/main Amharic- NLP - Tools -in- JAVA
2019
- [24]
- [25]
-
[26]
Daniel Mekuriaw and Arman Cohan. 2024. https://github.com/danielmekuriaw/mT5-PEFT-Amharic-Text-Summarization BENCHMARK DATASET AND PARAMETER - EFFICIENT CROSS - LINGUAL TRANSFER LEARNING FOR AMHARIC TEXT SUMMARIZATION . Technical report
2024
-
[27]
https://www.merriam-webster.com/dictionary/homophone Definition of HOMOPHONE
Merriam-Webster. https://www.merriam-webster.com/dictionary/homophone Definition of HOMOPHONE
-
[28]
Abraham Negash. 2017. https://www.asmarino.com/eng/93-books/4860-the-origin-and-development-of-tigrinya-language-publications-1886-1991-volume-one The Origin and Development of Tigrinya Language Publications (1886
2017
-
[29]
Hellina Hailu Nigatu, Atnafu Lambebo Tonja, Benjamin Rosman, Thamar Solorio, and Monojit Choudhury. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.983 The Zeno ’s Paradox of ‘ Low - Resource ’ Languages . In Proceedings of the 2024 Conference on Empirical Methods in Natural...
2024 doi
-
[30]
Rubungo Andre Niyongabo, Qu Hong, Julia Kreutzer, and Li Huang. 2020. https://doi.org/10.18653/v1/2020.coling-main.480 KINNEWS and KIRNEWS : Benchmarking cross-lingual text classification for K inyarwanda and K irundi . In Proceedings of the 28th International Conference on Co...
2020 doi
- [31]
-
[32]
Structural Irregularities within the English Language : Implications for Teaching and Learning in Second Language Situations
Jane Chinelo Obasi. Structural Irregularities within the English Language : Implications for Teaching and Learning in Second Language Situations
-
[33]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...
2002
-
[34]
Maja Popovi \'c . 2015. https://doi.org/10.18653/v1/W15-3049 chr F : character n-gram F -score for automatic MT evaluation . In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392--395, Lisbon, Portugal. Association for Computational Linguistics
2015 doi
-
[35]
Matt Post. 2018. https://doi.org/10.18653/v1/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Brussels, Belgium. Association for Computational Linguistics
2018 doi
- [36]
-
[37]
Martha Yifiru Tachbelie, Solomon Teferra Abate, and Laurent Besacier. 2014. https://doi.org/10.1016/j.specom.2013.01.008 Using different acoustic, lexical and language modeling units for ASR of an under-resourced language – Amharic . Speech Communication, 56:181--194
2014 doi
-
[38]
van Dijk, Merel van Witteloostuijn, Nada Vasić, Sergey Avrutin, and Elma Blom
Chantal N. van Dijk, Merel van Witteloostuijn, Nada Vasić, Sergey Avrutin, and Elma Blom. 2016. https://doi.org/10.1371/journal.pone.0152409 The Influence of Texting Language on Grammar and Executive Functions in Primary School Children . PLoS ONE, 11(3):e0152409
2016 doi
-
[39]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need
2017
-
[40]
Kimi Wenzel and Geoff Kaufman. 2024. https://doi.org/10.1145/3613904.3642900 Designing for Harm Reduction : Communication Repair for Multicultural Users ' Voice Interactions . In Proceedings of the CHI Conference on Human Factors in Computing Systems , pages 1--17, Honolulu HI...
2024
-
[41]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...
2020 doi
-
[42]
Seid Muhie Yimam, Abinew Ali Ayele, Gopalakrishnan Venkatesh, Ibrahim Gashaw, and Chris Biemann. 2021. https://doi.org/10.3390/fi13110275 Introducing Various Semantic Models for Amharic : Experimentation and Evaluation with Multiple Tasks and Datasets . Future Internet, 13(11)...
2021 doi
-
[43]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
-
[44]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.