{"id":"271bae4e-f08b-4446-80bc-c20e24fcea3a","arxiv_id":"2505.14553","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Hindi-pivot transfer gives a 14.2 SacreBLEU on Nepali-English devtest, beating the fully supervised direct baseline by 6.6 points, but with no code, no error bars, and a non-controlled baseline.","lead":"This paper tests using Hindi as a bridge language to translate Nepali into English, since direct Nepali-English training data is scarce and low-quality. On a standard test set, the bridge approach scores 14.2 SacreBLEU, 6.6 points above the direct fully-supervised baseline, but the comparison is not fully controlled.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 6.6-point gain is not yet established: the transfer system is compared to a baseline score taken on different data and an unreproduced metric, with a single run and no variance reported.","rationale":"The reader's weakest assumption is the same as mine: the central claim rests on a single, uncontrolled comparison to a baseline that was neither reproduced nor shown to use the same metric and preprocessing. I find no reason to escalate to rejection or to downgrade to unverified, because the result is plausible and the paper is transparent about its limited training budget and noisy backtranslation experiment. The concern is not that the authors are wrong, but that the published evidence does not yet establish a 6.6-point improvement attributable to pivoting. The concrete test would settle the issue by controlling for data, metric, and variance; until then, the conditional verdict is appropriate. I also note that the semi-supervised underperformance is explained by noise rather than tested, which further underscores the need for seed-level replication, but this is secondary to the uncontrolled baseline comparison.","tokens_in":7658,"tokens_out":3875,"duration_ms":37377,"concrete_test":"Re-run the Guzmán et al. fully supervised baseline with the paper's exact pipeline: train the §6.2 Transformer on the same 569K Nepali-English pairs with the §6.1 SentencePiece BPE-5000 preprocessing, evaluate on the FLORES devtest with detokenized SacreBLEU, and repeat both baseline and the transfer pipeline over at least 3 seeds. If the reproduced baseline differs from 7.6 by more than ~1 BLEU, or if the transfer-vs-reproduced-baseline gap is within seed variance, the reported 6.6-point improvement is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result in the abstract, Section 1, and Table 5 depends on comparing one 14.2 SacreBLEU devtest score from the Hindi-pivot transfer pipeline to the 7.6 \"fully supervised\" baseline taken directly from Guzmán et al. (2019). That comparison is not controlled in three ways that bear directly on the claim. First, the transfer pipeline consumes 1.49M Hindi-English pairs plus 284K Nepali-Hindi pairs (Tables 1–2), whereas the baseline sees only 569K Nepali-English pairs; the +6.6 could equally come from the extra parallel data and diverse ILTPDC domain as from pivoting. Second, Table 5 reports \"detokenized SacreBLEU\" for English outputs, but no evidence is given that the copied Guzmán baseline was computed with the same detokenized SacreBLEU version; a tokenized BLEU vs. SacreBLEU mismatch alone can move scores by several points. Third, the 14.2 is from a single training run with no seeds, and Section 8 itself invokes noise to explain the 2.4-point semi-supervised drop (14.2→10.7 on devtest), showing run-to-run or pipeline variation is material. The paper also notes in §6.2 that the Hindi-English model ran only 55 of 100 epochs, so the reported transfer score is an under-trained snapshot. None of these facts proves the claim false, but they make the 6.6-point margin load-bearing on unverified comparability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using Hindi as a pivot language for Nepali-to-English machine translation. The authors construct a Nepali-Hindi parallel corpus (~284K sentence pairs) from OPUS, Bible, and ILTPDC sources; use the IIT Bombay Hindi-English parallel corpus (~1.49M pairs); and apply two transfer-method variants: a fully-supervised pipeline that translates Nepali to Hindi and then Hindi to English with n=m=1, and a semi-supervised pipeline that inserts one iteration of backtranslation into the Nepali-Hindi leg. The main claim is that the fully-supervised transfer method achieves 14.2 detokenized SacreBLEU on the Nepali-English devtest set, which is 6.6 points higher than the fully-supervised baseline of 7.6 reported by Guzmán et al. (2019), while the semi-supervised variant reaches 10.7, below the 15.1 semi-supervised baseline. The paper also discusses linguistic reasons why Hindi is a suitable pivot, including shared script, word order, and morphology.","tokens_in":7907,"tokens_out":6671,"duration_ms":56104,"significance":"If the headline comparison were controlled, the result would be a useful contribution to low-resource MT: it demonstrates that a simple Hindi-pivot transfer, using relatively small additional parallel data, can outperform a direct supervised baseline on the Nepali-English pair, and it provides a new Nepali-Hindi corpus (~284K pairs) that the community could reuse. The paper is also transparent about the evaluation metric (SacreBLEU for English outputs, tokenized BLEU for Nepali-Hindi) and openly notes that the Hindi-English model was trained for only 55 of 100 epochs. It fails, however, to establish that the 6.6-point gain is due to pivoting rather than to the additional 1.49M Hindi-English and 284K Nepali-Hindi training pairs, different preprocessing, or run-to-run variation; the absence of a rerun of the Guzmán et al. baseline in the same pipeline, of variance estimates, and of code/data makes the central claim unverifiable as presented. These are experimental-rigor issues that can be addressed within the manuscript's scope, but they must be fixed before publication.","major_comments":[{"comment":"The 6.6-point improvement of 'OUR-Transfer Method (Fully-Supervised)' over the 'BASELINE (Fully-Supervised)' is not a controlled comparison. The baseline 7.6 is imported from Guzmán et al. (2019), while the transfer pipeline consumes 284K Nepali-Hindi pairs (Table 1) and 1.49M Hindi-English pairs (Table 2) that the baseline never saw, and the preprocessing is described only as 'similar' to Guzmán et al. (§6.1). To support the claim that pivoting through Hindi is responsible for the gain, the authors must either rerun the fully-supervised direct Nepali-English baseline under their own preprocessing and evaluation pipeline, or provide an ablation that trains the transfer pipeline on comparable amounts of direct data.","section":"§7, Table 5"},{"comment":"The reported devtest score of 14.2 is from a single training run, with no seeds, no standard deviation, and no significance test. The paper itself shows that such comparisons are noisy: §8 attributes the 2.4-point drop from 14.2 to 10.7 in the semi-supervised variant to harmful backtranslation and related noise, and run-to-run variance in low-resource NMT is substantial. The authors should report multiple runs (at least 3) and confidence intervals for both the transfer and baseline scores.","section":"§7, Table 5; §8"},{"comment":"The Hindi-to-English Transformer stopped at 55 of 100 epochs 'due to time constraints,' and §9 itself states that the score 'can be improved simply by letting our Hindi-English Transformer train for a few more days until we reach convergence.' The headline 14.2 is therefore an under-trained snapshot, not a converged or representative estimate. The paper should state whether the 47.2 and 16.8 intermediate scores are from checkpoints at 55 epochs and, if so, add a converged checkpoint or at least show a learning curve so the reader can judge how much the final score could change.","section":"§6.2; §9"},{"comment":"The paper reports 'detokenized SacreBLEU' for {Ne,Hi}→En, but the Guzmán et al. baseline of 7.6 is simply taken from the published table, and no evidence is given that the two numbers were produced with the same SacreBLEU version, tokenization, and detokenization settings. Since SacreBLEU scores can differ by several points across versions and settings, the authors must recompute the baseline with the same evaluation script (and ideally the same preprocessing) before claiming a 6.6-point gap.","section":"§7, Table 5 footnote"},{"comment":"The paper does not provide code or data for the corpus-construction pipeline (cleaning, deduplication, splitting of the Nepali-Hindi corpus) or for the training/evaluation pipeline, so the reported 14.2 cannot be independently reproduced. For a claim that rests on a single numerical comparison, this is a load-bearing reproducibility gap; at minimum, the corpus statistics and all preprocessing scripts should be released.","section":"§2.1; §6"}],"minor_comments":[{"comment":"The abstract and Section 1 refer to a 'devtest Set SacreBLEU score of 14.2' and a 'Test Set BLEU score of 14.2,' while the footnote to Table 5 clarifies that the English-side scores are detokenized SacreBLEU and the Nepali-Hindi scores are tokenized BLEU; the terminology should be aligned throughout.","section":"Abstract; §1; §7"},{"comment":"Table 5 shows a 'Ne → En dev' column with no baseline entries; the paper should state why dev scores are not compared and how the dev/devtest split relates to Guzmán et al.'s official split.","section":"§7, Table 5"},{"comment":"Section 4.2 says 300K Hindi sentences were randomly subsampled for backtranslation, but Section 8 does not report the quality of the backtranslated Nepali sentences; adding the BLEU of the backtranslation model and a few sample outputs would make the 'noise' explanation more convincing.","section":"§4.2; §8"},{"comment":"The paper does not specify how the four Nepali-Hindi sources (GNOME/KDE/Ubuntu, Bible, Agriculture/Entertainment) were cleaned, deduplicated, and sentence-aligned; these details are needed to reuse the corpus.","section":"§2.1, Table 1"},{"comment":"The formula uses the intersection 'TEH ∩ THN' without defining the phrase-table notation; the reader has to infer that TEH and THN are sets of phrase pairs from Hindi-English and Nepali-Hindi, respectively.","section":"§3.3"},{"comment":"The claim that 'a high amount of reordering is required' is supported by one anecdotal example; a quantitative reordering metric over the corpora would be more compelling.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"This is a very short paper with a single headline number, and the current version reads more like a workshop or course-project report than a journal article. The most important fix is to rerun the Guzmán et al. baseline in the authors' own pipeline and to report multi-seed variance; without that, the 6.6-point gain is not established. The corpus contribution (a 284K Nepali-Hindi corpus assembled from three sources) is real, and the authors should be encouraged to release it. I am open to reconsidering if the experimental section is substantially strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on Nepali or Hindi MT, but don't take the headline at face value. The paper assembles a new Nepali-Hindi parallel corpus (OPUS, Bible, ILTPDC, 284K pairs) and applies the standard transfer method with n=m=1: a Nepali-Hindi Transformer and a Hindi-English Transformer, concatenated at inference. That is a legitimate extension, and the specific result—14.2 SacreBLEU on FLORES Ne→En devtest—is new. The authors also write clearly about why Hindi is a plausible pivot (shared script, SOV order, gendered morphology) and are honest about the semi-supervised backtranslation experiment hurting (10.7 vs 15.1 baseline).\n\nThe soft spot is the central comparison. The stress-test note's three points hold: the 7.6 baseline is copied from Guzmán et al. and was trained only on the 569K Nepali-English parallel corpus; the transfer pipeline consumes 1.49M Hindi-English pairs plus 284K Nepali-Hindi pairs, so the +6.6 could just be more parallel data and different domains, not pivoting. Table 5 says the authors' scores are detokenized SacreBLEU, but no one has shown Guzmán's 7.6 was computed with the same SacreBLEU version; a tokenization mismatch alone can move BLEU by several points. And the 14.2 is from one run—no seeds, no variance. The authors themselves invoke noise in Section 8 to explain the semi-supervised drop, which cuts against confidence in the main score. I also note that the Hindi-English model only ran 55 of 100 epochs (Section 6.2), so the reported transfer score is from an undertrained component.\n\nNo code, data, or training scripts are released, so the new corpus cannot be reused. The paper is essentially a readable, well-motivated course project (it says CS 287 Final Project) with an unverified headline result.\n\nWho this is for: low-resource MT practitioners working with Hindi-Nepali–related pairs might get a useful pointer, but the lack of artifacts limits practical value. I would not cite it in the next year. As an editor, I would desk reject at a main venue; a workshop might give useful feedback if the authors supply artifacts and reproduce the baseline under the same pipeline. As it stands, the central claim is not established.","headline":"A readable course project with a new Nepali-Hindi corpus and a plausible transfer result, but the headline 6.6-point gain over the prior baseline is not controlled and the artifacts are missing.","tokens_in":8491,"tokens_out":4559,"would_cite":false,"duration_ms":43936,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Routing Nepali through a Hindi pivot lifts fully supervised translation from 7.6 to 14.2 SacreBLEU on the devtest set.","keywords":["low-resource machine translation","pivot language","transfer method","backtranslation","Nepali","Hindi","SacreBLEU","Transformer"],"falsifier":"Train the same fully supervised transfer system five times with different random seeds and compare the spread of devtest SacreBLEU scores against the 7.6 baseline; if any run lands at or below 7.6, or the gap is within the run-to-run variance, the claimed 6.6-point improvement is not established.","tokens_in":7408,"feed_emoji":"🔄","tokens_out":11332,"duration_ms":92692,"temperature":0.7,"pith_summary":"The paper sets out to show that a related, higher-resource language can substitute for missing parallel data in low-resource machine translation. Using Hindi as a pivot, the authors assemble a 284K-sentence Nepali-Hindi corpus, combine it with a 1.49M-sentence Hindi-English corpus, and train two Transformers that are chained at inference: Nepali to Hindi, then Hindi to English. This fully supervised transfer pipeline scores 14.2 SacreBLEU on the Nepali-English devtest set, 6.6 points above the fully supervised baseline. A semi-supervised variant that backtranslates Hindi monolingual data into the Nepali-Hindi pair reaches only 10.7, which the paper attributes to synthetic source noise eroding the natural similarity between Nepali and Hindi. The paper presents this as a practical blueprint for low-resource pairs that have a close high-resource relative.","feed_headline":"Hindi pivot lifts Nepali-English BLEU by 6.6 points","feed_subtitle":"Chaining Nepali→Hindi→English Transformers beats the fully supervised baseline, but backtranslation cuts the gain.","key_machinery":"The carrying mechanism is the transfer method with $n=m=1$: at inference, the source sentence is decoded once by a Nepali-to-Hindi Transformer to produce one Hindi hypothesis, and that hypothesis is decoded once by a Hindi-to-English Transformer to produce the final English sentence, eliminating any rescoring stage. The method leans on two properties: the close structural relatedness of Nepali and Hindi (same Devanagari script, high lexical overlap, shared SOV word order and gendered morphology, so word reordering between them is minimal) and the availability of a large, diverse Hindi-English corpus of about 1.49M sentences plus a newly assembled 284K-sentence Nepali-Hindi corpus from three sources. All models are Transformers with the hyperparameters of the baseline and joint byte-pair-encoding vocabularies of 5,000 symbols.","core_discovery":"The paper's central claim is that for Nepali-to-English translation, routing the translation through Hindi as a pivot—training separate Nepali-to-Hindi and Hindi-to-English Transformers and chaining them with a single candidate each—yields a devtest SacreBLEU score of 14.2, exceeding the fully supervised direct baseline of 7.6 reported by Guzmán et al. (2019) by 6.6 points. A semi-supervised variant that applies one iteration of backtranslation to the Nepali-Hindi pair scores 10.7, below the semi-supervised baseline of 15.1; the paper attributes this to backtranslation injecting noise into a source pair that was already easy because of Hindi-Nepali similarity. The paper also argues that Hindi is a suitable pivot because of its historical and lexical overlap with Nepali, shared Devanagari script, matching SOV word order, and shared morphological features such as gender agreement.","pith_inferences":["My inference: the 6.6-point gain conflates the pivot method with extra training data, because the transfer system draws on 1.49M Hindi-English and 284K Nepali-Hindi parallel pairs that the direct baseline never saw; a direct baseline trained on the same total data would isolate the pivot's contribution.","My inference: the reported gain is specific to the devtest split; the paper's own dev-set score for the fully supervised transfer model is 11.3 and no dev-set baseline is given, so the advantage may not be uniform across evaluation sets.","My inference: the same relatedness criteria—shared script, matching word order, and morphological overlap—could select pivots for other low-resource Indic pairs, such as Sinhala-English, making the recipe testable beyond Nepali."],"forward_implications":["A fully supervised pivot chain can beat direct fully supervised training for a low-resource pair without adding any new source-target parallel text.","End-to-end English quality is bounded by the pivot-to-target leg; the paper's Hindi-to-English model stopped at 55 epochs and the authors expect more training to raise the final score.","Backtranslation can hurt on a closely related pair: adding one iteration to Nepali-Hindi dropped the devtest score from 14.2 to 10.7, below the semi-supervised baseline of 15.1.","The transfer method is portable, since the paper frames it as applicable to other low-resource pairs and identifies the data requirements for building the two legs."],"supporting_citations":[{"why":"Supplies the Nepali-English evaluation dataset and the fully supervised (7.6) and semi-supervised (15.1) baseline scores that the paper's 14.2 is measured against.","marker":"Guzmán et al., 2019"},{"why":"Introduces the transfer method of training source-to-pivot and pivot-to-target models separately and chaining them at inference, which is the paper's core approach.","marker":"Wu & Wang, 2009"},{"why":"Provides the pivot scoring and triangulation framework the paper uses to motivate the transfer method and to show why the neural version is intractable.","marker":"Utiyama & Isahara, 2007"},{"why":"Introduces backtranslation, the semi-supervised data-augmentation technique the paper applies to the Nepali-Hindi leg.","marker":"Sennrich et al., 2015"},{"why":"Provides the IIT Bombay Hindi-English parallel corpus and Hindi monolingual data used to train the pivot leg and to sample backtranslation source sentences.","marker":"Kunchukuttan et al., 2017"},{"why":"Contributes OPUS parallel data to the newly assembled Nepali-Hindi corpus.","marker":"Tiedemann, 2012"},{"why":"Contributes the Bible's massively parallel corpus as a second source of Nepali-Hindi sentence pairs.","marker":"Christodouloupoulos & Steedman, 2015"},{"why":"Defines the Transformer architecture and training setup used for all models in the paper.","marker":"Vaswani et al., 2017"},{"why":"Establishes SacreBLEU as the comparable metric in which the paper's 14.2 devtest score is reported.","marker":"Post, 2018"}],"fun_headline_variants":["Hindi pivot nets Nepali-English a 6.6 BLEU gain","Via Hindi, Nepali-English BLEU jumps 6.6 points","Chained Hindi beats direct Nepali-English by 6.6 BLEU","Pivot Hindi raises Nepali-English score 6.6 points","Hindi-as-pivot beats full supervision with 14.2 BLEU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single reported 14.2 devtest SacreBLEU score is a reliable, comparable estimate—that the 6.6-point gap over the Guzmán et al. baseline reflects the Hindi-pivot method rather than the extra 1.77 million training sentences, different preprocessing, or run-to-run variation.","fun_headline_variants_meta":{"raw":{"variants":["Hindi pivot nets Nepali-English a 6.6 BLEU gain","Via Hindi, Nepali-English BLEU jumps 6.6 points","Chained Hindi beats direct Nepali-English by 6.6 BLEU","Pivot Hindi raises Nepali-English score 6.6 points","Hindi-as-pivot beats full supervision with 14.2 BLEU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000607,"raw_usage":{"total_tokens":2807,"prompt_tokens":904,"completion_tokens":1903,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":1803}},"tokens_in":520,"tokens_out":1903,"duration_ms":13468,"temperature":1.0,"reasoning_tokens":1803,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:32:01.932831+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same fully supervised transfer system five times with different random seeds and compare the spread of devtest SacreBLEU scores against the 7.6 baseline; if any run lands at or below 7.6, or the gap is within the run-to-run variance, the claimed 6.6-point improvement is not established.","supporting_citations":[],"review_version":1}