{"id":"82dbe16a-428e-4c82-a5e3-a92acd622b03","arxiv_id":"2501.00529","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning M2M100 on 9,000 parallel examples yields a Singlish-to-Sinhala transliterator that strongly outperforms a rule-based baseline on held-out shared-task test sets.","lead":"This paper compares two ways to convert Romanized Sinhala (Singlish) back into Sinhala script: a hand-written rule table and a fine-tuned M2M100 transformer. The transformer produces far higher translation quality scores (BLEU 0.53 vs 0.02) while the rules run thousands of times faster.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central BLEU gap depends on the shared-task test sets being disjoint from the 10k Dakshina/Swa-Bhasha training pairs; no overlap check is reported, so hidden contamination could explain the entire advantage.","rationale":"The reader's weakest assumption correctly identifies the train/test overlap risk, and I see no more load-bearing issue. The central claim is empirical: a 9k-example fine-tune of M2M100 beats a hand-written rule table by a wide margin on shared-task test sets. For that claim to hold, the test sets must be external to the training data. Section 4.1 describes training data from Dakshina and Swa-Bhasha and test data from the IndoNLP shared task, but no overlap check is reported. The paper's own augmentation in Table 3 produces ad-hoc variants that resemble the description of Test Set 02, so near-duplicate contamination is a concrete possibility rather than a generic worry. The reader's conditional verdict is the right disposition: the paper should be accepted only if the authors release and run an overlap analysis, or the evaluation should be redone on a verified disjoint test set. I did not find a stronger internal inconsistency: the methodology is coherent, the metrics are standard, and the qualitative examples in Table 6 are consistent with the numerical results. The lack of error bars is a weakness but not decisive given the magnitude of the gap. Because my concern matches the reader's and does not move the verdict, I recommend keeping the conditional acceptance.","tokens_in":9841,"tokens_out":3389,"duration_ms":35996,"concrete_test":"Compute exact and fuzzy overlap between the 9k training/1k validation pairs from Section 4.1 and both shared-task test sets. Normalize both sides by removing diacritics and collapsing repeated characters, then check exact string equality, token-level equality, and near-duplicate patterns such as vowel deletion (e.g., 'kawd' vs 'kawada' and 'oy' vs 'oya'). Report the number of overlapping or near-duplicate sentence pairs per test set. Additionally, inspect the IndoNLP shared-task repository to determine whether its test sets were derived from Dakshina or Swa-Bhasha; if so, the 'held-out' status is false and the headline results must be recomputed. If exact overlap is zero and near-duplicate overlap is negligible, the central claim survives; otherwise the reported BLEU/WER/CER comparison is not valid evidence for the paper's conclusion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for the paper's central claim is that the IndoNLP 2025 shared-task test sets are genuinely held out from the fine-tuning data. Section 4.1 says the training/validation data were built from Dakshina and Swa-Bhasha, and the evaluation used the shared-task test sets, but the paper reports no deduplication or overlap analysis. Both sources are public South Asian transliteration corpora, so overlap or near-duplicate forms are plausible. This matters most for Test Set 02, which is described as containing ad-hoc Romanized patterns; the authors' own augmentation in Table 3 generates exactly such variants (e.g., 'kawd' from 'kawada', 'oy' from 'oya'), so training examples could coincide with test entries not just exactly but in the no-vowel/abbreviated forms the model is supposed to learn. If even a few hundred of the 15,000 test sentences duplicate or closely resemble training pairs, the reported BLEU gap (0.5268/0.4384 vs 0.0177/0.0163) could be largely an artifact of memorization rather than generalization. The absence of error bars or significance testing is secondary: the gap is so large that only systematic contamination, not sampling noise, is a realistic alternative explanation. I am not alleging intentional leakage; accidental overlap is easy when both sides draw from the same small set of public Sinhala resources, which is exactly why the check must be explicit before the claimed comparison is trusted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two approaches for transliterating Romanized Sinhala (Singlish) into Sinhala script: a rule-based transliteration system and a fine-tuned multilingual sequence-to-sequence model (M2M100 418M). The rule-based method uses a hand-authored character mapping table; the deep learning method is fine-tuned on 9,000 parallel training pairs constructed from the Dakshina and Swa-Bhasha corpora with additional augmentation that simulates ad-hoc typing patterns. Both methods are evaluated on the IndoNLP 2025 shared-task test sets (10,000 general and 5,000 ad-hoc entries) using BLEU, WER, and CER. The reported results show a large advantage for the deep learning approach (BLEU 0.5268 and 0.4384 on the two test sets versus 0.0177 and 0.0163 for the rule-based system), while the rule-based system is far faster. The paper also discusses limitations and efficiency trade-offs, and provides a public code repository.","tokens_in":10136,"tokens_out":4312,"duration_ms":43296,"significance":"If the comparison is valid, this is a useful empirical data point for low-resource transliteration: it shows that fine-tuning a large multilingual seq2seq model on a modest parallel corpus can substantially outperform hand-written rules on noisy, ad-hoc Romanized input, and the use of two external shared-task test sets makes the evaluation more credible than a self-constructed test would be. The code release supports reproducibility. The significance is tempered, however, by the absence of any train/test overlap analysis, the lack of variance or significance estimates, and an underspecified rule-based baseline; these issues do not necessarily overturn the qualitative conclusion, but they prevent the reader from trusting the specific numbers and the strength of the claim until they are addressed.","major_comments":[{"comment":"The central comparison assumes that the IndoNLP 2025 shared-task test sets are genuinely held out from the 9,000 training pairs built from Dakshina and Swa-Bhasha, but the paper reports no deduplication or overlap analysis. This is load-bearing because Swa-Bhasha is a public corpus of Romanized Sinhala containing ad-hoc forms, and the augmentation procedure in Table 3 explicitly generates vowel-dropped and shortened variants (e.g., 'kawd' from 'kawada', 'oy' from 'oya') that are exactly the kind of patterns described for Test Set 02. If even a small fraction of the 15,000 test sentences closely resembles training examples, the reported BLEU gap in Table 5 could be inflated by memorization rather than generalization. The authors should add an explicit overlap check (exact string matching as well as normalized matching after vowel removal and other augmentation-style transformations) between the training/validation data and both test sets, and report the number of overlapping or near-duplicate entries and any actions taken.","section":"Section 4.1 and Section 5, Table 5"},{"comment":"The evaluation reports a single run of the fine-tuned model and a single deterministic rule-based system, without error bars, multiple seeds, or significance tests. While the BLEU gap is large enough that sampling noise is unlikely to reverse the qualitative ordering, the specific values in Table 5 and the claim that the deep learning approach is 'more robust' (Section 5) need supporting evidence such as standard deviations across multiple runs or bootstrap confidence intervals for the metrics. Without this, the reader cannot assess the stability of the reported averages.","section":"Section 5, Table 5"},{"comment":"The rule-based baseline is described as an extension of Tennage et al. (2018) with 'a few additions,' but the paper does not specify which rules were added, provide the complete mapping table, or list the exact version of the rule set used. Since the paper's core claim is a comparative evaluation, the baseline must be fully specified or unambiguously pinned to a released configuration (e.g., a tagged version in the GitHub repository) so that readers can reproduce or assess the strength of the rule-based system. As written, it is unclear whether this is a reasonable state-of-the-art rule-based baseline or a minimal character-level matcher.","section":"Section 3.1 and Table 1"}],"minor_comments":[{"comment":"The use of 'Singlish' to mean Romanized Sinhala is confusing because the same term is commonly used for Singapore Colloquial English; the paper acknowledges this but should use a less ambiguous term or consistently write 'Singlish (Romanized Sinhala)' throughout.","section":"Footnote 1"},{"comment":"Several rows in the qualitative comparison table contain garbled or inconsistently ordered Sinhala text (e.g., 'ෙම් ෙද්'), which could be a PDF rendering issue. If the text is corrupted, the table should be regenerated; if not, it undermines the qualitative comparison of the pre-trained models.","section":"Table 2"},{"comment":"The sentence 'The original datasets were not well structured' is ambiguous: it is unclear whether it refers to the Dakshina and Swa-Bhasha training sources, the shared-task test sets, or both. Please clarify and describe the conversion to CSV more precisely.","section":"Section 4.1"},{"comment":"The pseudocode does not state what happens to Latin characters that do not match any rule. The text says they are 'added as is,' but it is unclear whether these characters are inserted into the Sinhala output as Latin characters and whether this is consistent with the evaluation setting.","section":"Algorithm 1"},{"comment":"The paper attributes the first Sinhala-to-English transliteration system and the rule-based system used as a baseline to Tennage et al. (2018), but that reference describes a Tamil-to-Sinhala neural machine translation system using transliteration and byte-pair encoding, not a Sinhala-to-English transliteration tool. Please correct or rephrase this citation.","section":"Section 2.1 and Section 3.1"},{"comment":"The speed comparison reports TPS values but does not specify the hardware for the CPU row (the GPU rows are labeled, but the CPU model is not), nor the exact input used for the measurement. Please add these details so the efficiency comparison is reproducible.","section":"Table 7"}],"recommendation":"major_revision","confidential_remarks":"The main concern is the potential train/test overlap between the training set built from Dakshina and Swa-Bhasha and the IndoNLP shared-task test sets. This is a test-set integrity issue rather than an allegation of wrongdoing; accidental overlap is easy when both sources draw from the same small set of Sinhala resources. If the overlap check the referee requests comes back clean, the paper would likely be acceptable after addressing the variance and baseline-specification issues. I would not reject at this stage because the qualitative direction of the result is plausible and the external evaluation is a strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, modest empirical paper that will be useful to anyone working on Sinhala transliteration, but the headline result—neural M2M100 beating rule-based by a huge margin—currently rests on an unverified assumption that the training and test sets do not overlap. That needs to be fixed before I'd trust the numbers.\n\nWhat's new: the paper is one of the first to fine-tune M2M100 for Singlish-to-Sinhala transliteration and evaluate on the IndoNLP 2025 shared-task test sets. The rule table extends Tennage et al. with a handful of new mappings, and the data augmentation in Table 3 is a nice practical touch. The methodology is standard but clearly described, hyperparameters are given, code is on GitHub, and the limitations section is honest about ambiguity and data-quality issues. The speed comparison, while rough, makes the accuracy/efficiency tradeoff concrete.\n\nSoft spots, in order of severity. The load-bearing concern is train/test overlap. The training data is built from Dakshina and Swa-Bhasha, both public South Asian transliteration corpora, and the shared-task test sets are exactly the kind of ad-hoc Romanized text that the augmentation in Table 3 simulates. The paper reports no deduplication or overlap check. If even a few hundred test entries resemble training pairs, the BLEU gap could be inflated substantially. The gap is so large that sampling noise is not the worry; systematic contamination is. Second, there are no error bars, significance tests, or multiple runs—minor, but worth a sentence. Third, the rule-based baseline is simple longest-match; it's fine as a baseline, but it's not the strongest possible rule-based system. The speed table lacks methodology (tokenizer, batch size, etc.), so treat the TPS numbers as order-of-magnitude only.\n\nI don't think these are fatal. The paper's central claim is plausible, and the authors seem aware of their limitations. But the overlap check is quick to do and must be in the revision. If they can show no significant overlap, this is a solid contribution to low-resource NLP.\n\nVerdict: send to peer review. It's a useful data point for Sinhala and for the broader question of whether fine-tuned multilingual seq2seq models are worth the compute for transliteration. I'd cite it once the evaluation is tightened.","headline":"Useful benchmark for Sinhala transliteration, but the BLEU gap needs an overlap check before the headline numbers are trustworthy.","tokens_in":10671,"tokens_out":2543,"would_cite":true,"duration_ms":24470,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a multilingual translation model beats hand-written rules for Singlish-to-Sinhala transliteration.","keywords":["Sinhala transliteration","Singlish","rule-based transliteration","sequence-to-sequence","M2M100","low-resource NLP","reverse transliteration","BLEU"],"falsifier":"Re-run the evaluation after checking every training and validation sentence against the two shared-task test sets and removing any near-duplicate overlap; if the neural BLEU drops substantially from 0.5268 and 0.4384 when overlaps are excluded, the claimed advantage is partly memorization. A second check is to test the same fine-tuned model on a freshly collected, non-overlapping Singlish corpus, which would show whether the model generalises beyond the benchmark's distribution.","tokens_in":9664,"feed_emoji":"🔤","tokens_out":7978,"duration_ms":67456,"temperature":0.7,"pith_summary":"This paper tries to establish that Romanized Sinhala (Singlish) to Sinhala script is better handled by a fine-tuned sequence-to-sequence model than by hand-authored transliteration rules. The authors compare a rule-based character-mapping system with a fine-tuned M2M100 model, trained on just 9,000 parallel examples, and evaluate both on two shared-task test sets. The neural model achieves average BLEU of 0.5268 on the general test set and 0.4384 on the ad-hoc set, while the rule-based system scores 0.0177 and 0.0163. The authors argue this matters because Singlish typing is highly variable, code-mixed, and often drops vowels, making fixed rules brittle. If correct, the result offers a practical recipe: a small fine-tune of an existing multilingual translation model can serve as a reverse transliterator for a low-resource language without designing new linguistic rules.","feed_headline":"M2M100 fine-tune beats rule-based Singlish-to-Sinhala transliteration","feed_subtitle":"With only 9,000 training pairs, the neural model scores BLEU 0.53/0.44; the rules score under 0.02.","key_machinery":"The deep-learning system is a fine-tuned M2M100 encoder-decoder transformer, a multilingual sequence-to-sequence model pretrained for translation; the authors reuse its English tokenizer by treating Romanized Sinhala as 'English' and Sinhala as the target language. Fine-tuning is done with a learning rate of 2e-5 for three epochs on 9,000 training pairs. The rule-based baseline is a longest-match lookup table that extends the mapping rules of an existing Sinhala transliteration system with additional two- and three-character sequences; it greedily consumes the longest matched substring at each position. The M2M100 model is the machinery that carries the accuracy claim, while the table carries the baseline comparison.","core_discovery":"The paper's central claim is that the reverse transliteration task for Singlish to Sinhala should be treated as a machine translation problem rather than a lookup over fixed character mappings. Fine-tuning the 418M-parameter M2M100 model on 9,000 parallel examples yields an average BLEU of 0.5268 on test set 01 and 0.4384 on test set 02, compared with 0.0177 and 0.0163 for a rule-based system that extends an existing mapping table with additional two- and three-character rules. The neural model also reports lower word error rate and character error rate on both test sets, and handles vowel-dropped and ad-hoc spellings that the rule-based approach gets wrong. The paper concludes that the deep-learning approach is more robust to language variability, while the rule-based approach remains faster on CPU and unsuitable for real-time use without GPU acceleration.","pith_inferences":["Editorial inference: even a substantial training/test overlap would need to be very large to erase the reported gap, but a deduplicated benchmark is needed to know the true effect size.","Editorial inference: the paper's choice to encode Romanized Sinhala with the English language token may under-use M2M100's multilingual capacity; a dedicated Romanized-Sinhala token could improve handling of Sinhala-specific phonetics.","Editorial inference: a hybrid pipeline that applies rule-based lookup to clear tokens and the neural model only to ambiguous ones could preserve most of the accuracy gain while reducing GPU dependence."],"forward_implications":["With only 9,000 parallel training examples, a fine-tuned multilingual translation model can outperform a hand-tuned transliteration table by a large margin on held-out test sets.","Because the sequence-to-sequence model uses context, it can resolve ambiguous and vowel-dropped spellings that fixed character mappings cannot.","On GPU the neural approach is real-time (about 65 tokens per second in fp16), while the rule-based approach is much faster on CPU but far less accurate.","The same training recipe should transfer to other low-resource languages with Romanized typing conventions, since it only needs a parallel corpus rather than language-specific rules.","The lower BLEU on test set 02 (0.4384 versus 0.5268) shows that ad-hoc typing remains harder, so accuracy on extreme spellings is still an open issue."],"supporting_citations":[{"why":"Provides the original Sinhala-to-English transliteration mapping table that the rule-based baseline extends with two- and three-character rules.","marker":"Tennage et al. (2018)"},{"why":"Supplies the framing of transliteration as a character-level translation task, justifying the sequence-to-sequence formulation.","marker":"Deselaers et al. (2009)"},{"why":"Introduces M2M100, the pretrained multilingual encoder-decoder model the paper fine-tunes for Singlish-to-Sinhala.","marker":"Fan et al. (2020)"},{"why":"Contributes the Dakshina parallel dataset used to build the 10k training and validation entries.","marker":"Roark et al. (2020)"},{"why":"Provides the Swa-Bhasha hybrid transliteration approach and dataset that supplies additional parallel training data.","marker":"Sumanathilaka et al., 2023"},{"why":"Publishes the Swa-Bhasha Romanized-Sinhala to Sinhala ad-hoc transliteration corpus used for training.","marker":"Sumanathilaka et al., 2024"},{"why":"Offers comparative evidence that transformer encoder-decoder architectures outperform recurrent ones for transliteration, motivating the model choice.","marker":"Moran and Lignos (2020)"}],"fun_headline_variants":["Neural transliteration beats rules: BLEU 0.53 vs 0.02","Fine-tuned M2M100 wins on Singlish-to-Sinhala transliteration","9k examples: neural translit BLEU 0.53 vs rule 0.02","Seq2Seq crushes rule-based for Sinhala transliteration","Transformer translit: 0.53 BLEU vs 0.02 rule-based"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the shared-task test sets are an external benchmark that does not overlap with the 9,000 training and 1,000 validation pairs built from Dakshina and Swa-Bhasha; no deduplication check is reported.","fun_headline_variants_meta":{"raw":{"variants":["Neural transliteration beats rules: BLEU 0.53 vs 0.02","Fine-tuned M2M100 wins on Singlish-to-Sinhala transliteration","9k examples: neural translit BLEU 0.53 vs rule 0.02","Seq2Seq crushes rule-based for Sinhala transliteration","Transformer translit: 0.53 BLEU vs 0.02 rule-based"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000545,"raw_usage":{"total_tokens":2589,"prompt_tokens":910,"completion_tokens":1679,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":1566}},"tokens_in":526,"tokens_out":1679,"duration_ms":10510,"temperature":1.0,"reasoning_tokens":1566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:47:36.695102+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation after checking every training and validation sentence against the two shared-task test sets and removing any near-duplicate overlap; if the neural BLEU drops substantially from 0.5268 and 0.4384 when overlaps are excluded, the claimed advantage is partly memorization. A second check is to test the same fine-tuned model on a freshly collected, non-overlapping Singlish corpus, which would show whether the model generalises beyond the benchmark's distribution.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the original Sinhala-to-English transliteration mapping table that the rule-based baseline extends with two- and three-character rules."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the framing of transliteration as a character-level translation task, justifying the sequence-to-sequence formulation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces M2M100, the pretrained multilingual encoder-decoder model the paper fine-tunes for Singlish-to-Sinhala."},{"cited_title":"Mielke, Cibu Johny, I s in Demir s ahin, and Keith Hall","cited_arxiv_id":null,"evidence_quote":"Contributes the Dakshina parallel dataset used to build the 10k training and validation entries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Offers comparative evidence that transformer encoder-decoder architectures outperform recurrent ones for transliteration, motivating the model choice."}],"review_version":1}