Pith. sign in

REVIEW 4 major objections 6 minor 26 references

Reconsidering SMT Over NMT for Closely Related Languages: A Case Study of Persian-Hindi Pair

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On a one-million-sentence Persian-Hindi corpus, phrase-based statistical machine translation beats a Transformer neural system, 66.32 to 53.7 BLEU, suggesting architecture choice should follow language-pair structure.

desk verdict A useful new Persian-Hindi corpus and a first MT system, but the SMT-vs-NMT claim is confounded by extra LM data and an untuned baseline. read the letter →

arxiv 2412.16877 v1 pith:CF4CERKS submitted 2024-12-22 cs.CL

Waisullah Yousofi , Pushpak Bhattacharyya This is my paper · ORCID
classification cs.CL
keywords phrase-basedstatisticalmachinetranslationneuralPersian-Hindicloselyrelatedlanguagesmoderate-resourceTransformerBLEUlanguagetypology
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that phrase-based statistical machine translation can still beat Transformer-based neural machine translation when the language pair is closely related and the parallel corpus is moderate-sized, about one million sentences. Using a filtered Persian-to-Hindi corpus, the paper reports a BLEU score of 66.32 for the phrase-based system against 53.7 for the Transformer trained on the same parallel data. The result matters because it challenges the default assumption that neural models are always the best choice and points to cheaper, architecture-aware alternatives. It also shows that artificial changes to the source text, like romanization or reversing the writing direction, hurt translation quality.

What carries the argument

The central mechanism is the phrase-based statistical translation pipeline: word alignment built by expectation-maximization, extraction of a phrase table from that alignment, a target-side 5-gram language model with back-off smoothing, and a beam-search decoder. The argument depends on this pipeline's ability to exploit near-one-to-one word correspondences between Persian and Hindi, so that a moderate parallel corpus suffices for reliable phrase pairs. The Transformer baseline, by contrast, must learn the same correspondences through many layers of attention, and the paper's claim is that it needs more data than one million sentences to do so.

What would settle it

Retrain the Transformer on the same one-million-sentence corpus plus the 13.7 million Hindi sentences used by the SMT language model, with dev-based early stopping and a small hyperparameter search; if BLEU reaches or exceeds 66.32, the claimed architectural advantage collapses.

Watch

Extended reading notes

Core claim

For Persian-to-Hindi, the paper claims, a classic phrase-based statistical system is not merely competitive but superior: 66.32 versus 53.7 BLEU on the same one-million-sentence corpus. The explanation offered is structural closeness: more than 72 percent of parallel sentences differ in length by three tokens or fewer, the languages share lexicon and SOV word order, and alignments are near one-to-one, which is exactly the regime phrase-based alignment can exploit with less data. The paper further reports that romanizing the Persian input drops BLEU to 51.21 and reversing Persian script to left-to-right order drops it to 48.74, concluding that the natural orthography and word order of the source language carry information the model should keep.

Load-bearing premise

The load-bearing premise is that the Transformer baseline is a fair, fully trained representative of NMT rather than an undertuned configuration, and that the SMT score does not owe its margin to the 13.7 million extra Hindi sentences used only in the language model.

Editorial extensions

If this is right

  • For closely related, moderate-resource language pairs, SMT should be treated as a serious baseline rather than written off as obsolete.
  • A carefully filtered parallel corpus of roughly one million sentences is enough for high BLEU scores on Persian-Hindi with phrase-based SMT.
  • Typological features such as sentence-length similarity, shared vocabulary, and consistent SOV order can determine which architecture wins.
  • Because the winning SMT system trains with far less compute than the 300K-step Transformer, comparable quality can come with a smaller energy footprint, a point the paper makes explicitly.
  • Script-transformation tricks such as romanization or direction reversal are not safe ways to force more similarity between related languages; both hurt BLEU.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fairer architecture test would give the Transformer the same 13.7 million Hindi sentences the SMT language model uses; the reported gap may overstate the architectural difference.
  • If the SMT advantage is real, it likely extends to other closely related Indo-Aryan and Iranian pairs with moderate corpora, such as Urdu-Hindi or Dari-Hindi, where similar typological closeness holds.
  • The 66.32 versus 53.7 gap is a single metric; human evaluation or chrF could show a different ordering, and the boundary corpus size at which NMT overtakes SMT remains untested.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports a Persian-to-Hindi machine translation study in which a phrase-based SMT system (Moses) achieves BLEU 66.32 and a Transformer-based NMT system (OpenNMT) achieves BLEU 53.7 on a corpus of about 1M parallel sentences. The authors interpret this as evidence that PBSMT can outperform Transformer NMT for closely related, moderate-resource language pairs, and they also report experiments with Romanized input and reversed word order. The paper contributes a filtered Persian-Hindi parallel corpus, a Moses/SRILM SMT pipeline, and a fixed-configuration OpenNMT baseline.

Significance. If the headline comparison were cleanly controlled, the paper would be a useful addition to the systems literature on architecture selection for related languages, and the corpus-construction details could be reused by other researchers. The authors should be credited for building and filtering a sizeable parallel corpus, for reporting SMT results as a 4-fold cross-validation average, and for making the preprocessing pipeline (LABSE filtering, language-specific tokenizers, uroman) explicit. However, the central claim is not established by the reported experiments: the SMT system uses 13.7M extra Hindi sentences for its language model, whereas the NMT system uses only the parallel corpus, and the NMT result is a single fixed-configuration run without significance testing. The evidence therefore does not support the abstract's claim that SMT 'significantly' exceeds NMT on the same dataset.

major comments (4)
  1. [Section 3, Table 2; Section 4.1-4.2] The SMT language model is trained on 13.7M+ unfiltered Hindi monolingual sentences (Table 2), while the NMT Transformer is trained only on the 1M-sentence parallel corpus (Table 1). This contradicts the abstract's phrase 'ingesting the same dataset' and means the 12.6-BLEU gap cannot be attributed to the SMT architecture. A 5-gram Kneser-Ney LM over 13.7M sentences plausibly supplies substantial target-side fluency that the NMT system never receives. A controlled comparison should either remove the external LM data from the SMT system or provide the NMT system with comparable target-side text (e.g., through back-translation), and both conditions should be reported.
  2. [Section 4.2 and Section 5] The NMT score of 53.7 comes from one training run with a fixed hyperparameter configuration (8 encoder/decoder layers, 8 attention heads, embedding size 512, 300K steps, initial learning rate 2) and no reported dev-set checkpoint selection, early stopping, dropout, or validation metric. By contrast, the SMT score 66.32 is the average of four folds (67.32, 66.32, 64.90, 66.74). The word 'significantly' in the abstract is never supported by a statistical test, confidence intervals, or multiple seeds. The comparison should report NMT with hyperparameter tuning and repeated runs, ideally on the same folds, with error bars.
  3. [Section 3, Table 1; Section 4.3] The test set contains only 3,000 sentences and is drawn from the same LABSE-filtered pool as the training data. A 3,000-sentence test set yields wide BLEU confidence intervals, and using the same filter for training and test likely selects easy, well-aligned sentences, inflating scores for all systems. Additionally, Section 4.3 says the authors use sacreBLEU, but the footnote points to Moses' internal mteval-v13a.pl script and no sacreBLEU signature is reported. Please report an independent or carefully split test set, confidence intervals, and the exact BLEU signature used.
  4. [Section 5, Table 4; Section 6] The internal reporting of results is inconsistent. The text states that the Romanized SMT 'dropped to 51.21 from 66.7,' but Table 4 lists the initial SMT model as 64.91 and the best SMT model as 66.32, with no 66.7 value. The conclusion also says romanization 'showed a beneficial effect compared to the inversion experiment,' which is misleading because Romanization is 15 BLEU points below the standard SMT baseline and only less harmful than inversion. Furthermore, the general conclusion that 'SMT outperforms NMT for closely related languages' is drawn from a single language pair; at minimum, the claim should be restricted to this pair and framed as a case study.
minor comments (6)
  1. [Section 1] The sentence 'the performance of attention-based MT systems reaches BLEU scores of almost one' is ambiguous because BLEU is usually reported on a 0-100 scale; please state the intended scale explicitly.
  2. [Section 2] The claim that no prior comparative analysis of NMT and SMT exists for structurally similar languages should be softened or supported by a broader literature search, since the related work covers only a few selected references.
  3. [Section 3, Figure 1] The text says 'more than 72%' of parallel sentences have a length difference of less than three tokens, but the exact count and the tokenization used to compute the difference should be reported, and the donut chart should include precise percentages.
  4. [Section 4.3] The paper states that sacreBLEU is used but does not report the tokenizer or the 'sacrebleu' signature string; without this information, the BLEU scores are not fully reproducible.
  5. [Appendix A.2] The inversion example shows two Persian lines that look identical in the printed text, with different English glosses; please clarify the actual reversal or provide a clearer example that makes the word-order change visible.
  6. [Section 1 and Section 6] The paper motivates the work by computational efficiency and carbon footprint, but no runtime, energy, or parameter-count measurements are reported; either add such measurements or state that efficiency is a qualitative motivation only.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the SMT-vs-NMT comparison is an empirical benchmark, not a derivation that reduces to its inputs.

full rationale

The paper makes an empirical claim that PBSMT (BLEU 66.32) outperforms a Transformer-NMT system (BLEU 53.7) for Persian-to-Hindi translation. No equation in the paper defines the SMT score in terms of the NMT score, and no fitted parameter is renamed as a prediction. The BLEU scores are measured outputs of separately trained systems on held-out test sentences, so the central comparison is self-contained rather than circular. The self-citations that appear (e.g., Batheja and Bhattacharyya 2022 for corpus-filtering help, Kunchukuttan and Bhattacharyya 2017 in related work, and Bhattacharyya 2015 for linguistic divergence examples) are not load-bearing: none of them is invoked to justify the SMT-superiority result or to forbid alternative architectures. Valid concerns about the comparison exist, but they are fairness and validity concerns, not circularity. In particular, the SMT language model is trained on 13.7M+ additional unfiltered Hindi sentences (Section 3, Table 2), while the NMT system is reported as a single fixed configuration (Section 4.2) without reported dev-set model selection or early stopping; also, the SMT score is a 4-fold average while the NMT score appears to be a single run. These confounds weaken the causal attribution of the BLEU gap to architecture choice, but they do not make the result true by construction. Accordingly, no circular step is identified and the score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central result is an empirical benchmark comparison. The key free parameters are the filtering threshold, BPE merges, NMT hyperparameters, and LM order. The load-bearing unstated assumption is that the SMT's 13.7M-sentence Hindi language model is not responsible for the BLEU gap; the paper never controls for this. It also assumes the LABSE-filtered test set represents general-domain translation.

free parameters (5)
  • LABSE filtering threshold = 0.9
    Hand-selected threshold used to filter 10.9M raw parallel sentences down to 1.01M; it determines the corpus for both systems and may bias toward easy, near-literal sentence pairs.
  • BPE merge operations = 32K
    Hand-selected BPE merge count for NMT tokenization; no tuning reported.
  • NMT hyperparameters = 8 layers, 8 heads, 512 embedding, batch 4096, 300K steps, lr 2, warmup 8000
    A single untuned configuration; the reported NMT BLEU depends on these choices, and no model selection on a dev set is described.
  • SMT language model order = 5-gram with Kneser-Ney smoothing
    Standard choice; the LM is trained on 13.7M extra Hindi sentences, which is a major data asymmetry not controlled for.
  • Cross-validation fold count = 4
    SMT result is the average over 4 folds; the choice of averaging (vs. single run, as for NMT) affects the reported comparison.
assumptions (4)
  • domain assumption The LABSE-filtered test set is representative of general-domain Persian-Hindi translation.
    Section 3 filters all parallel data with LABSE at 0.9; if this favors short, literal sentences, the reported BLEU scores, for both SMT and NMT, may not reflect real-world translation difficulty.
  • domain assumption The SMT system's use of 13.7M target-side Hindi sentences for its language model does not confound the comparison with NMT.
    Section 3 and Table 2 describe the extra monolingual data; the paper never tests an NMT system with equivalent data, so the architecture comparison is not controlled.
  • domain assumption Sentence length difference of three tokens or less indicates an optimal one-to-one alignment and structural closeness.
    Section 3: 'We assumed that if the majority of sentences exhibit a difference of three tokens or less, the alignment achieved through mgiza would represent an optimal one-to-one correspondence.' This assumption is used to conclude Persian-Hindi are structurally similar.
  • domain assumption BLEU is a sufficient metric for comparing MT architectures.
    The paper evaluates all models using BLEU only; BLEU correlates with but does not fully capture translation adequacy and fluency.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reconsidering SMT Over NMT for Closely Related Languages: A Case Study of Persian-Hindi Pair." pith.science (2026). https://pith.science/paper/CF4CERKS

@misc{pith2026241216877,
  author       = {Pith},
  title        = {Pith review of: Reconsidering SMT Over NMT for Closely Related Languages: A Case Study of Persian-Hindi Pair},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CF4CERKS}},
  note         = {Machine review of arXiv:2412.16877}
}
read the original abstract

This paper demonstrates that Phrase-Based Statistical Machine Translation (PBSMT) can outperform Transformer-based Neural Machine Translation (NMT) in moderate-resource scenarios, specifically for structurally similar languages, like the Persian-Hindi pair. Despite the Transformer architecture's typical preference for large parallel corpora, our results show that PBSMT achieves a BLEU score of 66.32, significantly exceeding the Transformer-NMT score of 53.7 on the same dataset. Additionally, we explore variations of the SMT architecture, including training on Romanized text and modifying the word order of Persian sentences to match the left-to-right (LTR) structure of Hindi. Our findings highlight the importance of choosing the right architecture based on language pair characteristics and advocate for SMT as a high-performing alternative, even in contexts commonly dominated by NMT.

Figures

Figures reproduced from arXiv: 2412.16877 by the authors.

Figure 1
Figure 1. Categories the differences of lengths of paral [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Architecture of Persian to Hindi SMT Model. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 13 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Hadi Abdi Khojasteh, Ebrahim Ansari, and Mahdi Bohlouli. 2020. https://aclanthology.org/2020.lrec-1.776 LSCP : Enhanced large scale colloquial P ersian language understanding . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6323--6327, Marseille, France. European Language Resources Association

    work page 2020

  4. [4]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473

    arXiv 2014

  5. [5]

    Akshay Batheja and Pushpak Bhattacharyya. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.361 Improving machine translation with phrase pair injection and corpus filtering . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5395--5400, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics

    work page doi:10.18653/v1/2022.emnlp-main.361 2022

  6. [6]

    Pushpak Bhattacharyya. 2015. Machine translation. CRC Press

    work page 2015

  7. [7]

    Bonnie Jean Dorr. 1993. Machine translation: a view from the lexicon. MIT press

    work page 1993

  8. [8]

    Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Parteek Sharma, Fan Chen, and Lei Jiang. 2023. Llmcarbon: Modeling the end-to-end carbon footprint of large language models. arXiv preprint arXiv:2309.14393

    arXiv 2023

Show all 26 references
  1. [9]

    Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic bert sentence embedding. arXiv preprint arXiv:2007.01852

    2020 arXiv

  2. [10]

    Ulf Hermjakob, Jonathan May, and Kevin Knight. 2018. https://doi.org/10.18653/v1/P18-4003 Out-of-the-box universal R omanization tool uroman . In Proceedings of ACL 2018, System Demonstrations , pages 13--18, Melbourne, Australia. Association for Computational Linguistics

    2018 doi

  3. [11]

    Inigo Jauregi Unanue, Lierni Garmendia Arratibel, Ehsan Zare Borzeshi, and Massimo Piccardi. 2018. https://aclanthology.org/L18-1141 E nglish- B asque statistical and neural machine translation . In Proceedings of the Eleventh International Conference on Language Resources and...

    2018

  4. [12]

    Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810

    2017 arXiv

  5. [13]

    Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In Proceedings of the 1995 International Conference on Acoustics, Speech, and Signal Processing (ICASSP), volume 1, pages 181--184. IEEE

    1995

  6. [14]

    Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting ...

    2007

  7. [15]

    Anoop Kunchukuttan. 2020. The indic nlp library. https://github.com/anoopkunchukuttan/indic_nlp_library

    2020

  8. [16]

    Anoop Kunchukuttan and Pushpak Bhattacharyya. 2017. https://doi.org/10.18653/v1/W17-4102 Learning variable length units for SMT between related languages via byte pair encoding . In Proceedings of the First Workshop on Subword and Character Level Models in NLP , pages 14--24, ...

    2017 doi

  9. [17]

    Anoop Kunchukuttan, Maulik Shah, Pradyot Prakash, and Pushpak Bhattacharyya. 2017. https://aclanthology.org/I17-2048 Utilizing lexical similarity between related, low-resource languages for pivot-based SMT . In Proceedings of the Eighth International Joint Conference on Natura...

    2017

  10. [18]

    Franz Josef Och and Hermann Ney. 2000. Improved statistical alignment models. In Proceedings of the 38th annual meeting of the Association for Computational Linguistics, pages 440--447. Association for Computational Linguistics

    2000

  11. [19]

    Romina Oji, Seyedeh Fatemeh Razavi, Sajjad Abdi Dehsorkh, Alireza Hariri, Hadi Asheri, and Reshad Hosseini. 2021. Parsinorm: A persian toolkit for speech processing normalization. In 2021 7th International Conference on Signal Processing and Intelligent Systems (ICSPIS), pages...

    2021

  12. [20]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  13. [21]

    Matt Post. 2018. https://doi.org/10.18653/v1/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Brussels, Belgium. Association for Computational Linguistics

    2018 doi

  14. [22]

    Miguel Rios and Serge Sharoff. 2015. https://doi.org/10.18653/v1/W15-3410 Obtaining SMT dictionaries for related languages . In Proceedings of the Eighth Workshop on Building and Using Comparable Corpora, pages 68--73, Beijing, China. Association for Computational Linguistics

    2015 doi

  15. [23]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2016 doi

  16. [24]

    J \"o rg Tiedemann. 2016. Opus-parallel corpora for everyone. Baltic Journal of Modern Computing, 4(2)

    2016

  17. [25]

    Antonio Toral and Andy Way. 2015. https://doi.org/10.3115/v1/W15-0714 Translating literary text between related languages using SMT . In Proceedings of the Fourth Workshop on Computational Linguistics for Literature, pages 123--132, Denver, Colorado, USA. Association for Compu...

    2015 doi

  18. [26]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.