{"id":"1e08f662-1e8b-4a5d-99ce-0b3aa4664ed4","arxiv_id":"2412.16877","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"On a filtered Persian-Hindi parallel corpus, phrase-based SMT yields a BLEU of 66.3, ahead of the transformer NMT's 53.7.","lead":"This paper tests an older translation method, phrase-based statistical machine translation, against a modern neural transformer method for translating Persian into Hindi. It reports the older method scoring higher on a one-million-sentence test, suggesting classical approaches still matter for related languages with limited data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SMT-vs-NMT comparison is confounded: SMT uses 13.7M extra Hindi LM sentences while NMT is a single fixed-config run, so the 12.6-BLEU gap does not isolate architecture.","rationale":"The strongest empirical claim is the 12.6-BLEU SMT advantage, and the manuscript itself discloses both confounds: Table 2 shows 13.7M sentences used for the SMT LM, and Section 4.2 reports a single NMT configuration with no tuning or early stopping. No experiment controls for data access or model selection. Because the SMT LM component directly leverages these extra sentences, the comparison conflates architecture with data resources. This is not a disagreement with consensus; it is an internal inconsistency in the reported setup: the abstract says 'same dataset' while Section 3 describes extra target monolingual data for SMT. A single retraining of Moses with the LM restricted to the parallel corpus would settle the data-asymmetry part of the concern. I credit the paper for using standard toolkits and reporting detailed configurations, but no released code or external reproducibility evidence offsets the missing controlled comparison. The reader's REJECT verdict and weak-assumption analysis align with this concern, so I make no adjustment to the verdict.","tokens_in":7014,"tokens_out":6547,"duration_ms":62074,"concrete_test":"Retrain the Moses SMT pipeline with the 5-gram LM built only from the Hindi side of the 1.01M parallel training sentences in Table 1, excluding the 13.7M Table-2 monolingual sentences; keep MGIZA++/Moses settings and preprocessing identical, and score the same 3,000-sentence test set with sacreBLEU. If the SMT BLEU drops from 66.32 to roughly the NMT level of 53.7, the reported superiority is attributable to the extra monolingual LM data rather than to SMT architecture. If the score stays near 66, the remaining NMT-tuning confound must be tested separately.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5's central claim ('SMT outperforms NMT for closely related languages') requires a comparison in which architecture is the only relevant variable. That condition is not met. Section 3/Table 2 states that the SMT language model was built from 13.7M+ unfiltered Hindi sentences, while Section 4.2 describes a single OpenNMT Transformer (8 layers, 8 heads, embedding 512, 300K steps, LR 2) with no reported dev-set selection or early stopping. Thus the abstract's phrase 'ingesting the same dataset' is not literally true: SMT consumes an additional 13.7M target-language sentences, and the NMT result is one fixed configuration. For a 1M-sentence parallel corpus, a 5-gram Kneser-Ney LM on 13.7M sentences is a substantial fluency boost, so a 12.6 BLEU gap cannot be attributed to the PBSMT architecture. The comparison is also muddied by reporting SMT as a 4-fold cross-validation average but NMT as a single model's score.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a Persian-to-Hindi machine translation study in which a phrase-based SMT system (Moses) achieves BLEU 66.32 and a Transformer-based NMT system (OpenNMT) achieves BLEU 53.7 on a corpus of about 1M parallel sentences. The authors interpret this as evidence that PBSMT can outperform Transformer NMT for closely related, moderate-resource language pairs, and they also report experiments with Romanized input and reversed word order. The paper contributes a filtered Persian-Hindi parallel corpus, a Moses/SRILM SMT pipeline, and a fixed-configuration OpenNMT baseline.","tokens_in":7268,"tokens_out":5318,"duration_ms":47768,"significance":"If the headline comparison were cleanly controlled, the paper would be a useful addition to the systems literature on architecture selection for related languages, and the corpus-construction details could be reused by other researchers. The authors should be credited for building and filtering a sizeable parallel corpus, for reporting SMT results as a 4-fold cross-validation average, and for making the preprocessing pipeline (LABSE filtering, language-specific tokenizers, uroman) explicit. However, the central claim is not established by the reported experiments: the SMT system uses 13.7M extra Hindi sentences for its language model, whereas the NMT system uses only the parallel corpus, and the NMT result is a single fixed-configuration run without significance testing. The evidence therefore does not support the abstract's claim that SMT 'significantly' exceeds NMT on the same dataset.","major_comments":[{"comment":"The SMT language model is trained on 13.7M+ unfiltered Hindi monolingual sentences (Table 2), while the NMT Transformer is trained only on the 1M-sentence parallel corpus (Table 1). This contradicts the abstract's phrase 'ingesting the same dataset' and means the 12.6-BLEU gap cannot be attributed to the SMT architecture. A 5-gram Kneser-Ney LM over 13.7M sentences plausibly supplies substantial target-side fluency that the NMT system never receives. A controlled comparison should either remove the external LM data from the SMT system or provide the NMT system with comparable target-side text (e.g., through back-translation), and both conditions should be reported.","section":"Section 3, Table 2; Section 4.1-4.2"},{"comment":"The NMT score of 53.7 comes from one training run with a fixed hyperparameter configuration (8 encoder/decoder layers, 8 attention heads, embedding size 512, 300K steps, initial learning rate 2) and no reported dev-set checkpoint selection, early stopping, dropout, or validation metric. By contrast, the SMT score 66.32 is the average of four folds (67.32, 66.32, 64.90, 66.74). The word 'significantly' in the abstract is never supported by a statistical test, confidence intervals, or multiple seeds. The comparison should report NMT with hyperparameter tuning and repeated runs, ideally on the same folds, with error bars.","section":"Section 4.2 and Section 5"},{"comment":"The test set contains only 3,000 sentences and is drawn from the same LABSE-filtered pool as the training data. A 3,000-sentence test set yields wide BLEU confidence intervals, and using the same filter for training and test likely selects easy, well-aligned sentences, inflating scores for all systems. Additionally, Section 4.3 says the authors use sacreBLEU, but the footnote points to Moses' internal mteval-v13a.pl script and no sacreBLEU signature is reported. Please report an independent or carefully split test set, confidence intervals, and the exact BLEU signature used.","section":"Section 3, Table 1; Section 4.3"},{"comment":"The internal reporting of results is inconsistent. The text states that the Romanized SMT 'dropped to 51.21 from 66.7,' but Table 4 lists the initial SMT model as 64.91 and the best SMT model as 66.32, with no 66.7 value. The conclusion also says romanization 'showed a beneficial effect compared to the inversion experiment,' which is misleading because Romanization is 15 BLEU points below the standard SMT baseline and only less harmful than inversion. Furthermore, the general conclusion that 'SMT outperforms NMT for closely related languages' is drawn from a single language pair; at minimum, the claim should be restricted to this pair and framed as a case study.","section":"Section 5, Table 4; Section 6"}],"minor_comments":[{"comment":"The sentence 'the performance of attention-based MT systems reaches BLEU scores of almost one' is ambiguous because BLEU is usually reported on a 0-100 scale; please state the intended scale explicitly.","section":"Section 1"},{"comment":"The claim that no prior comparative analysis of NMT and SMT exists for structurally similar languages should be softened or supported by a broader literature search, since the related work covers only a few selected references.","section":"Section 2"},{"comment":"The text says 'more than 72%' of parallel sentences have a length difference of less than three tokens, but the exact count and the tokenization used to compute the difference should be reported, and the donut chart should include precise percentages.","section":"Section 3, Figure 1"},{"comment":"The paper states that sacreBLEU is used but does not report the tokenizer or the 'sacrebleu' signature string; without this information, the BLEU scores are not fully reproducible.","section":"Section 4.3"},{"comment":"The inversion example shows two Persian lines that look identical in the printed text, with different English glosses; please clarify the actual reversal or provide a clearer example that makes the word-order change visible.","section":"Appendix A.2"},{"comment":"The paper motivates the work by computational efficiency and carbon footprint, but no runtime, energy, or parameter-count measurements are reported; either add such measurements or state that efficiency is a qualitative motivation only.","section":"Section 1 and Section 6"}],"recommendation":"reject","confidential_remarks":"The core experimental comparison is confounded by asymmetric data and asymmetric evaluation, and the current manuscript does not provide the additional experiments needed to establish the headline claim. The authors have done useful corpus-construction work, and a future submission with a controlled comparison (e.g., matching data access, tuned NMT baselines, significance testing, and an independent test set) could be worth considering, but the present version does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is a legitimate first system for Persian-Hindi MT and builds a clean filtered corpus, but its central claim—that PBSMT beats Transformer NMT on the same data—is not supported by the experiment as reported. The SMT system uses an additional 13.7M Hindi sentences for its language model, and the NMT run is a single fixed configuration with no dev-set selection or significance testing. So the abstract's 'same dataset' is not literally true.\n\nWhat is genuinely useful: the corpus construction effort, LABSE filtering at 0.9, the length-difference analysis of Persian/Hindi, and the cross-validation for the SMT results. The Romanized and inverted-script experiments are sensible negative results. The writing is clear and the experimental setup is transparent enough that the confound is visible rather than hidden.\n\nThe confound is load-bearing. A 5-gram Kneser-Ney LM on 13.7M sentences gives a real fluency advantage to a system translating into Hindi; that alone could account for a large part of the 12.6 BLEU gap. The NMT baseline also looks under-resourced relative to standard practice: one configuration, no early stopping or checkpoint selection, no hyperparameter search. Reporting SMT as a 4-fold average while comparing it to a single NMT score makes the gap look artificially stable. No confidence intervals, no paired significance tests. The use of unfiltered monolingual data that may overlap with the test set is another open question; the paper doesn't specify where the monolingual sentences came from. Also, 'significantly exceeding' in the abstract is not backed by any statistical test.\n\nThis should go to peer review, not because the current claim is acceptable, but because the underlying question is real and the fix is straightforward: match the training data (or give NMT comparable extra data via back-translation or LM initialization), tune the NMT properly, and report variance. If SMT still wins after that, it's an interesting case study for closely related languages and resource-efficient MT. If not, the corpus alone is still a contribution. Who is it for: MT researchers working on related-language pairs, and anyone evaluating SMT-vs-NMT under data constraints. I would not cite it in its current form; I would cite the corpus if it were released, but no data release is mentioned.","headline":"A useful new Persian-Hindi corpus and a first MT system, but the SMT-vs-NMT claim is confounded by extra LM data and an untuned baseline.","tokens_in":7788,"tokens_out":4415,"would_cite":false,"duration_ms":40393,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On a one-million-sentence Persian-Hindi corpus, phrase-based statistical machine translation beats a Transformer neural system, 66.32 to 53.7 BLEU, suggesting architecture choice should follow language-pair structure.","keywords":["phrase-based statistical machine translation","neural machine translation","Persian-Hindi","closely related languages","moderate-resource translation","Transformer","BLEU","language typology"],"falsifier":"Retrain the Transformer on the same one-million-sentence corpus plus the 13.7 million Hindi sentences used by the SMT language model, with dev-based early stopping and a small hyperparameter search; if BLEU reaches or exceeds 66.32, the claimed architectural advantage collapses.","tokens_in":6832,"feed_emoji":"🌐","tokens_out":7983,"duration_ms":63857,"temperature":0.7,"pith_summary":"This paper tries to establish that phrase-based statistical machine translation can still beat Transformer-based neural machine translation when the language pair is closely related and the parallel corpus is moderate-sized, about one million sentences. Using a filtered Persian-to-Hindi corpus, the paper reports a BLEU score of 66.32 for the phrase-based system against 53.7 for the Transformer trained on the same parallel data. The result matters because it challenges the default assumption that neural models are always the best choice and points to cheaper, architecture-aware alternatives. It also shows that artificial changes to the source text, like romanization or reversing the writing direction, hurt translation quality.","feed_headline":"SMT beats Transformer for Persian-Hindi: 66.3 vs 53.7 BLEU","feed_subtitle":"On one million parallel sentences, the statistical system beats the transformer for closely related languages.","key_machinery":"The central mechanism is the phrase-based statistical translation pipeline: word alignment built by expectation-maximization, extraction of a phrase table from that alignment, a target-side 5-gram language model with back-off smoothing, and a beam-search decoder. The argument depends on this pipeline's ability to exploit near-one-to-one word correspondences between Persian and Hindi, so that a moderate parallel corpus suffices for reliable phrase pairs. The Transformer baseline, by contrast, must learn the same correspondences through many layers of attention, and the paper's claim is that it needs more data than one million sentences to do so.","core_discovery":"For Persian-to-Hindi, the paper claims, a classic phrase-based statistical system is not merely competitive but superior: 66.32 versus 53.7 BLEU on the same one-million-sentence corpus. The explanation offered is structural closeness: more than 72 percent of parallel sentences differ in length by three tokens or fewer, the languages share lexicon and SOV word order, and alignments are near one-to-one, which is exactly the regime phrase-based alignment can exploit with less data. The paper further reports that romanizing the Persian input drops BLEU to 51.21 and reversing Persian script to left-to-right order drops it to 48.74, concluding that the natural orthography and word order of the source language carry information the model should keep.","pith_inferences":["A fairer architecture test would give the Transformer the same 13.7 million Hindi sentences the SMT language model uses; the reported gap may overstate the architectural difference.","If the SMT advantage is real, it likely extends to other closely related Indo-Aryan and Iranian pairs with moderate corpora, such as Urdu-Hindi or Dari-Hindi, where similar typological closeness holds.","The 66.32 versus 53.7 gap is a single metric; human evaluation or chrF could show a different ordering, and the boundary corpus size at which NMT overtakes SMT remains untested."],"forward_implications":["For closely related, moderate-resource language pairs, SMT should be treated as a serious baseline rather than written off as obsolete.","A carefully filtered parallel corpus of roughly one million sentences is enough for high BLEU scores on Persian-Hindi with phrase-based SMT.","Typological features such as sentence-length similarity, shared vocabulary, and consistent SOV order can determine which architecture wins.","Because the winning SMT system trains with far less compute than the 300K-step Transformer, comparable quality can come with a smaller energy footprint, a point the paper makes explicitly.","Script-transformation tricks such as romanization or direction reversal are not safe ways to force more similarity between related languages; both hurt BLEU."],"supporting_citations":[{"why":"Supplies the phrase-based SMT toolkit used for all SMT experiments.","marker":"(Koehn et al., 2007)"},{"why":"Provides the word-alignment models that learn the Persian-Hindi one-to-one mappings.","marker":"(Och and Ney, 2000)"},{"why":"Gives the smoothing method for the 5-gram language model used by the SMT decoder.","marker":"(Kneser and Ney, 1995)"},{"why":"Defines the Transformer architecture that serves as the NMT baseline.","marker":"(Vaswani et al., 2017)"},{"why":"Supplies the OpenNMT toolkit used to train the Transformer baseline.","marker":"(Klein et al., 2017)"},{"why":"Defines BLEU, the evaluation metric on which the comparison rests.","marker":"(Papineni et al., 2002)"},{"why":"Provides the refined BLEU implementation used for reporting final scores.","marker":"(Post, 2018)"},{"why":"Supplies the sentence-embedding filter used to reduce the noisy raw corpus to the high-confidence parallel set.","marker":"(Feng et al., 2020)"}],"fun_headline_variants":["Persian-Hindi: SMT beats Transformer, 66.3 vs 53.7 BLEU","Classic phrase-based MT outperforms Transformer for Persian-Hindi","For Persian-Hindi, statistical MT beats neural by 12.6 BLEU","Moderate-resource Persian-Hindi: SMT outperforms NMT","Persian-Hindi structure explains SMT advantage over NMT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Transformer baseline is a fair, fully trained representative of NMT rather than an undertuned configuration, and that the SMT score does not owe its margin to the 13.7 million extra Hindi sentences used only in the language model.","fun_headline_variants_meta":{"raw":{"variants":["Persian-Hindi: SMT beats Transformer, 66.3 vs 53.7 BLEU","Classic phrase-based MT outperforms Transformer for Persian-Hindi","For Persian-Hindi, statistical MT beats neural by 12.6 BLEU","Moderate-resource Persian-Hindi: SMT outperforms NMT","Persian-Hindi structure explains SMT advantage over NMT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000911,"raw_usage":{"total_tokens":3865,"prompt_tokens":848,"completion_tokens":3017,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":2928}},"tokens_in":464,"tokens_out":3017,"duration_ms":20459,"temperature":1.0,"reasoning_tokens":2928,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T06:01:10.261852+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the Transformer on the same one-million-sentence corpus plus the 13.7 million Hindi sentences used by the SMT language model, with dev-based early stopping and a small hyperparameter search; if BLEU reaches or exceeds 66.32, the claimed architectural advantage collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the word-alignment models that learn the Persian-Hindi one-to-one mappings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the smoothing method for the 5-gram language model used by the SMT decoder."}],"review_version":1}