{"id":"92d96caa-35bc-444c-89e4-e3a3d6a418d6","arxiv_id":"2412.09957","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A Bi-LSTM attention model for romanized-to-Malayalam transliteration achieves 7.4% CER on standard patterns but degrades to 22.7% when vowels are omitted.","lead":"This paper trains an attention-based Bi-LSTM model on 4.3 million romanized-to-native Malayalam word pairs and reports 7.4% character error on a standard test set and 22.7% on an adhoc test set with missing vowels. A generalist might read it to see how far simple sequence-to-sequence models go on Indic transliteration and where they fail on informal typing patterns.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing baseline comparison makes the reported 7.4% CER uninterpretable; without it the central claim is unverified rather than wrong.","rationale":"The reader identified the word-level vs sentence-level gap as the weakest assumption; that is a legitimate limitation the authors acknowledge. My concern is more load-bearing because it affects interpretation of every reported number: without a baseline, the central claim that this architecture achieves 7.4% CER cannot be evaluated. This is not an internal inconsistency, but it is a correctness/interpretability risk. The missing error analysis and hyperparameters compound it. I agree with the conditional verdict — the work is plausible but not independently verifiable as an advance over existing models. The concrete test is inexpensive and would settle whether the architecture or the data drives the result.","tokens_in":5127,"tokens_out":1219,"duration_ms":15450,"concrete_test":"Run the released model on the IndoNLP test sets and compare against two controls with identical training data: (1) IndicXlit or another transformer baseline fine-tuned on the same 4.3M pairs, and (2) a non-attention Bi-LSTM with the same hyperparameters. Report CER/WER/BLEU with bootstrap confidence intervals. If the attention model does not beat the no-attention control by a statistically significant margin, the reported quality is attributable to data scale, not architecture.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper reports 7.4% CER on general romanized Malayalam, but never benchmarks against any existing system. IndicXlit, a transformer trained on the same Aksharantar corpus, is cited in related work but no comparison is given. The reader cannot tell whether the Bi-LSTM architecture contributes anything beyond the 4.3M training pairs. The WER of 34.5% alongside 7.4% CER implies roughly 4-5 character errors per wrong word, which suggests systematic failure modes (missing vowel signs, chillu/virama handling) rather than isolated typos — contrary to the discussion that WER/BLEU merely penalize minor variations. Section 5 error analysis stops mid-sentence, so no error-type breakdown supports the claim that the model is 'strong' on standard patterns beyond the aggregate CER. Hyperparameters, training details, and error bars are all absent, so the headline numbers cannot be independently checked.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a reverse transliteration system that converts romanized Malayalam into native Malayalam script using an attention-based Bi-LSTM encoder-decoder architecture. The model is trained on 4.3 million word-level transliteration pairs from the publicly available Dakshina and Aksharantar datasets. The authors evaluate on the IndoNLP-2025 shared task test sets, reporting a character error rate (CER) of 7.4% on general typing patterns (Test Set-1) and 22.7% on adhoc typing patterns with missing vowel indicators (Test Set-2), with correspondingly higher word error rates and lower BLEU scores. The paper discusses the limitations of the word-level pipeline and the absence of a language model, and it releases the training code under an MIT license.","tokens_in":5253,"tokens_out":5458,"duration_ms":55188,"significance":"The reported performance is plausible but currently uninterpretable without comparison to existing systems. The main strength is the reproducible setup: public training data, a shared-task test set, and released code. If the 7.4% CER holds up against baselines, the model would represent a functional word-level transliterator for standard typing patterns; however, the paper does not demonstrate that this outperforms or even matches the transformer-based IndicXlit, which is trained on the same corpus. The sentence-level reconstruction approach and the acknowledged absence of a language model limit the generality of the claims, especially for adhoc patterns. Overall, the work is a credible system description but does not yet establish a scientific contribution beyond its own reported numbers.","major_comments":[{"comment":"The claimed performance is not situated relative to any baseline. The related work discusses IndicXlit (Madhani et al., 2023), an encoder-decoder transformer trained on Aksharantar, but no comparison is provided, nor is any rule-based or statistical baseline reported. Without a baseline, the 7.4% CER on Test Set-1 is uninterpretable; it could be strong or weak relative to the state of the art. The authors should at minimum evaluate IndicXlit on the same test sets and report pairwise comparisons, ideally with statistical significance tests across multiple runs or bootstrapped confidence intervals.","section":"Section 4, Table 2"},{"comment":"The evaluation methodology is underspecified. The paper does not report the number of sentences in each test set, how CER/WER/BLEU are computed (e.g., tokenization at character/word level, which tools are used), or the variance of the metrics. No error bars or confidence intervals are given, and no multiple-seed training is described. In addition, key training hyperparameters such as batch size, number of epochs, optimizer, learning rate, regularization, and number of LSTM layers are omitted; the code link is not enough if the paper's central numbers are to be independently checked. These details are necessary to assess the reliability of the headline results.","section":"Section 3.2 and Section 4"},{"comment":"The sentence-level evaluation is based on a word-level model with naive reconstruction: sentences are split into words, transliterated individually, and reassembled. As the authors acknowledge in Limitations, the model lacks a language model, and word boundaries are unclear in adhoc patterns with missing vowels. The paper does not analyze how often this reconstruction fails or whether sentence context would resolve the observed errors. Since the shared task test sets are sentence-level, the reported CER/WER/BLEU reflect this pipeline, not just the transliteration model. This design choice should be characterized quantitatively, for example by comparing against a context-aware baseline.","section":"Section 3 and Limitations"},{"comment":"The error analysis is incomplete and the interpretation of WER/BLEU is not supported. The sentence 'Specifically, the model frequently misclassifies character pairs such as' is cut off, so no actual error types are identified. The claim that the gap between CER (7.4%) and WER (34.5%) on Test Set-1 is due to WER/BLEU penalizing minor variations is not demonstrated; with an average of roughly 4–5 character errors per wrong word, the errors may be systematic (e.g., missing vowel signs or chillu/virama mishandling). The authors should provide a breakdown of error types and examples, and quantify how many errors are single-character versus multi-character.","section":"Section 5 (Discussion)"}],"minor_comments":[{"comment":"The section heading contains a typo: 'Datsets' should be 'Datasets'.","section":"Section 3.1"},{"comment":"The sentence 'Aksharamukha script converter is is another rule-based systems' contains a duplicated 'is' and a subject-verb agreement error; it should be 'Aksharamukha is another rule-based system'.","section":"Section 2"},{"comment":"The word 'severly' in the caption should be 'severely'.","section":"Table 1 caption"},{"comment":"The attribution of the Marian-based NMT transliteration system to Kunchukuttan et al. is unclear; please cite the specific reference and verify that the claim matches the cited work.","section":"Section 2"},{"comment":"The sentence 'The context vector extracts from this processed sequence' is grammatically awkward; rephrase to clarify what the context vector is and how it is computed.","section":"Section 3.2"},{"comment":"The captions refer to distributions of evaluation metrics, but the figures are not included in the visible text; please ensure they have labeled axes and are properly referenced.","section":"Figures 1 and 2"},{"comment":"Some reference entries are incomplete (e.g., Baby et al. lacks venue details, James et al. lacks the arXiv identifier). Please complete all references.","section":"References"},{"comment":"The phrase '4.100 million pairs' is ambiguous; use '4.1 million' or '4,100,000'.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a system-description-style contribution for a shared task. The strengths are the public code and use of standard datasets. The main weakness is the absence of any comparison to existing systems, which is a serious omission for a journal submission. I would encourage the editor to require a substantial revision that adds baselines, statistical rigor, and a completed error analysis before considering it for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a shared-task system description, not a research advance. The model is a standard Bi-LSTM encoder-decoder with attention, trained on Dakshina + Aksharantar (4.3M pairs), and evaluated on the two IndoNLP-2025 Malayalam test sets. The specific CER/WER/BLEU numbers on those sets are new, and the reported degradation on adhoc vowel-omitted typing (CER 7.4% -> 22.7%) is a useful empirical observation.\n\nWhat it does well: the data handling is clean, the evaluation follows the shared task protocol, and the Limitations section is honest — it admits the lack of a language model and the training-data bias toward standard typing. The code is linked (though I couldn't access the repo directly from the paper). For a workshop paper this is acceptable.\n\nSoft spots: the biggest issue is the absence of any baseline comparison. IndicXlit is cited and was trained on the same Aksharantar data, and the reader cannot tell whether the Bi-LSTM architecture adds anything over a strong transformer baseline. The 7.4% CER is uninterpretable without knowing what IndicXlit scores on the same test set. Also missing: hyperparameters like batch size, learning rate, dropout, early stopping; no error bars or significance tests; and the error analysis in Section 5 stops mid-sentence, so we never see the promised breakdown of confusable character pairs. The WER/BLEU vs CER discrepancy is plausibly explained by a few character errors per word, but the paper's dismissal of WER/BLEU as 'inherently limited' is a bit glib — a real analysis would show whether the errors are systematic (vowel signs, chillu) or random.\n\nThe stress-test note about the missing baseline is on point. The note's inference that roughly 4-5 char errors per wrong word suggests systematic failure modes is reasonable, and it means the model is 'strong' only relative to nothing. None of these issues make the central claim wrong — the model likely does what it says — but they do make the significance unverifiable.\n\nWho is this for? People building Indic transliteration systems or comparing shared-task entries. It's a legitimate system paper. I'd give it a conditional accept: require a baseline comparison to IndicXlit (or at least a discussion of why none is possible) and complete the error analysis. It doesn't need heavy statistical machinery, but a simple significance test or confidence interval on CER would help.","headline":"A modest but honest shared-task system description; the architecture is textbook, the numbers are plausible, but the missing baseline makes the headline CER hard to interpret.","tokens_in":5832,"tokens_out":2525,"would_cite":false,"duration_ms":24115,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Bi-LSTM encoder-decoder trained on 4.3 million romanized-Malayalam word pairs converts standard typing to native script at 7.4% character error, and degrades sharply to 22.7% when vowel signs are missing.","keywords":["Malayalam transliteration","romanized to native script","encoder-decoder","Bi-LSTM","attention mechanism","character error rate","vowel omission","Indic transliteration"],"falsifier":"Take the adhoc test sentences, restore the word boundaries that the preprocessing assumes, and feed the model each sentence with spaces inserted according to the ground truth. If CER stays near 22.7% even though the segmentation is now correct, the bottleneck is the missing-vowel character mapping; if CER drops sharply, the reported error is partly an artifact of the word-extraction pipeline. A second check is to rerun training after adding vowel-dropping paraphrases to the corpus and see whether Test Set-2 CER moves materially.","tokens_in":4892,"feed_emoji":"🔤","tokens_out":8980,"duration_ms":92625,"temperature":0.7,"pith_summary":"The paper claims that a character-level sequence-to-sequence model, an attention-based bidirectional LSTM encoder-decoder, can convert romanized Malayalam into native Malayalam script well enough for everyday standard typing. Trained on 4.3 million curated word pairs from two public Indic transliteration corpora, it reports a 7.4% character error rate on the general test set. The same model reaches only 22.7% character error on adhoc typing patterns in which vowel signs are mostly omitted, so the claim is carefully bounded to standard typing. A sympathetic reader should care because romanized input is how most Malayalam speakers type on phones and computers, while formal documents need native script; an automatic converter would remove a manual retyping burden. The paper's design choice is word-level transliteration with sentence reconstruction, and the authors state the model has no language model to exploit word context.","feed_headline":"Romanized Malayalam to native script: 7.4% character error","feed_subtitle":"The same model drops to 22.7% on vowel-missing informal typing, so the win is bounded.","key_machinery":"The load-bearing mechanism is the attention-based bidirectional LSTM encoder-decoder. The encoder reads up to 57 romanized characters as 64-dimensional embeddings and processes them with a Bi-LSTM whose hidden states are concatenated into 256 dimensions and compressed to 128 per timestep; a context vector is built from these states. The decoder repeats that context vector for every output timestep, runs it through an LSTM, and an attention layer lets each decoding step focus on the relevant encoder positions. The concatenated LSTM and attention representations pass through a time-distributed dense layer to produce probabilities over 76 Malayalam characters, including vowels, vowel signs, consonants, anuswaram, visargam, virama, and chillu. The companion mechanism is the sentence pipeline: strip non-alphabetic characters, transliterate each word independently, then reinsert punctuation and numbers; this pipeline is what makes the reported CER a word-level claim rather than a sentence-level one.","core_discovery":"On its own terms, the paper's discovery is that an attention-enabled Bi-LSTM encoder-decoder, trained on a merged corpus of 4.344 million word-level transliteration pairs, learns the romanized-to-Malayalam character mapping well for standard typing patterns, achieving 7.4% CER on the general test set. On the adhoc set with frequent vowel omissions, CER rises to 22.7%, with WER at 66.9% and BLEU at 7.5%. The authors interpret the high CER on adhoc input as a consequence of training data that covers standard patterns but misses irregular vowel-dropping styles. They also argue that WER and BLEU misrepresent quality because a few character errors can make whole words count as wrong, and they identify phonetically similar Malayalam characters that share a romanized form as a recurring error source. The system is word-level: sentences are split into words, transliterated independently, and reassembled; the authors note this omits cross-word dependencies and limits sentence-level quality.","pith_inferences":["A fair extension would be to rerun the same architecture on sentences rather than isolated words, or to add a language-model rescoring step; the paper's own limitation section suggests this could close much of the 22.7% gap, but that is an inference, not a demonstrated result.","Real-world romanized Malayalam often lacks consistent spaces and vowel signs at the same time; if so, the reported Test Set-2 CER may still overstate usability, because the word-extraction preprocessing assumes detectable word boundaries.","The error analysis hints that confusable character pairs sharing one romanized spelling are a main error source; a testable fix is to augment the training data with minimal pairs that force the model to use surrounding characters."],"forward_implications":["If the reported 7.4% CER is representative, a standard-pattern romanized Malayalam sentence will need roughly one character correction per fourteen characters typed, making the model usable as a real-time transliteration aid.","The 22.7% CER on vowel-dropping input means the model is not reliable for the informal SMS-style typing where vowels are skipped; users would see about one in four characters wrong.","Because WER on Test Set-1 is 34.5%, far above the CER, even good transliterations will show many whole-word mismatches under word-based scoring, so CER is the metric that reflects the character-level nature of the task.","Adding adhoc vowel-dropping patterns to the training data is the paper's stated next step and should directly reduce the Test Set-2 error if the architecture is sound.","The word-level design implies that punctuation and numbers are preserved exactly, but any ambiguity resolvable only through neighboring words is invisible to the model."],"supporting_citations":[{"why":"Supplies the Dakshina corpus of 244 thousand word transliteration pairs and the observation that sentence-level transliteration differs from word-level.","marker":"(Roark et al., 2020)"},{"why":"Provides the Aksharantar corpus of 4.1 million pairs that forms most of the training data.","marker":"(Madhani et al., 2023)"},{"why":"Documents large-scale evaluation and mining of Indic transliteration data that Aksharantar builds on.","marker":"(Kunchukuttan et al., 2021)"},{"why":"Describes context-aware transliteration using language models, the approach the paper cites as the sentence-context mechanism it does not implement.","marker":"Kirov et al."},{"why":"Argues for character error rate over WER and BLEU in multilingual evaluation, which the paper uses to justify its primary metric.","marker":"(James et al., 2024)"},{"why":"Defines the Malayalam character set, including vowels, vowel signs, consonants, and special characters, used as the decoder's target vocabulary.","marker":"(Manohar et al., 2022)"}],"fun_headline_variants":["Malayalam script from romanized text: 7.4% CER on standard typing","Encoder-decoder maps romanized Malayalam, but informal typing hurts","Bi-LSTM transliteration: 7.4% error, but 22.7% on vowel-light text","Word-level transliteration to Malayalam: standard typing works, ad hoc fails","Attention Bi-LSTM for Malayalam transliteration: CER jumps with missing vowels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach assumes word-by-word transliteration is enough, so sentence context never needs to influence how a word is converted; the paper itself notes the model has no language model to capture word dependencies.","fun_headline_variants_meta":{"raw":{"variants":["Malayalam script from romanized text: 7.4% CER on standard typing","Encoder-decoder maps romanized Malayalam, but informal typing hurts","Bi-LSTM transliteration: 7.4% error, but 22.7% on vowel-light text","Word-level transliteration to Malayalam: standard typing works, ad hoc fails","Attention Bi-LSTM for Malayalam transliteration: CER jumps with missing vowels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1250,"prompt_tokens":920,"completion_tokens":330,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":220}},"tokens_in":536,"tokens_out":330,"duration_ms":3942,"temperature":1.0,"reasoning_tokens":220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:30:30.676619+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the adhoc test sentences, restore the word boundaries that the preprocessing assumes, and feed the model each sentence with spaces inserted according to the ground truth. If CER stays near 22.7% even though the segmentation is now correct, the bottleneck is the missing-vowel character mapping; if CER drops sharply, the reported error is partly an artifact of the word-extraction pipeline. A second check is to rerun training after adding vowel-dropping paraphrases to the corpus and see whether Test Set-2 CER moves materially.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Aksharantar corpus of 4.1 million pairs that forms most of the training data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents large-scale evaluation and mining of Indic transliteration data that Aksharantar builds on."}],"review_version":1}