{"id":"b48127d2-b61c-44d5-8e9e-c4d3633dc021","arxiv_id":"2412.04351","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A claimed 2B-parameter multi-task translation model for 36 Indian languages, built from pivoted and synthetic corpora, evaluated without baselines and with inconsistent reported numbers.","lead":"This paper describes a 2-billion-parameter translation model trained on billions of parallel sentences across 36 Indian languages, plus corpora for grammar correction, post-editing, quality scoring, and error marking. The authors claim broad multilingual coverage, but release no model weights and provide no comparison with existing translation systems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Table 1 reports 325 parallel pairs and 'over 1B' sentences, not the abstract's 36x36=1296 directions and 10B parallel sentences; the stated evaluation size also cannot cover all 1296 directions.","rationale":"The reader's weakest_assumption focuses on synthetic-data noise. That is a real risk, but I do not make it the primary attack because it depends on external quality judgments and could in principle be repaired by releasing the corpus. The more decisive problem is internal: the paper's own Table 1 cannot support the claimed 36x36, 10B resource. The central claim uses '36 x 36 language directions' and '10 billion parallel corpora' in the abstract, introduction, and conclusion, and the introduction says this requires corpora for all 36 x 36 pairs. Table 1 provides 325 pairs and describes the corpus as over 1B; it is the only general parallel-corpus table. No additional table supplies the missing ~935 directions, and no released model or corpus link lets a reader verify the totals. Table 10's evaluation size is also inconsistent with full 36x36 coverage. If the authors were to release an expanded table or the model, the concern could be resolved; as written, the central resource claim is unsupported by the manuscript's own evidence. This is not an external-consensus disagreement; it is an internal contradiction. The paper does deserve credit for using FLORES/IN22 rather than only self-built test sets, and for attempting at least some human validation of aligned data, but those positives do not repair the coverage arithmetic.","tokens_in":30919,"tokens_out":14525,"duration_ms":147430,"concrete_test":"Transcribe the entries in Table 1 into a list of (source_lang, target_lang, count) triples. Count the number of distinct directed pairs, check whether reverse directions are present or absent, and sum the counts; compare the totals with the claimed 1296 directions and 10B parallel sentences. Separately, divide 902,198 by the per-direction sizes of FLORES and IN22 to infer how many directions were actually evaluated. If the pair count is not 1296, or the sum is not 10B, or the evaluated-direction count is not 1296, then the 36x36/10B claim is not supported by the paper's own evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the model uses '10 billion parallel corpora for 36 x 36 language directions' is contradicted by the paper's own data tables. With 36 languages there are 1296 directed source-target pairs (or 1260 excluding identity), and the abstract and Section 1 explicitly require corpora for all 36 x 36 pairs. Table 1, the sole general-purpose parallel-corpus table, reports only 325 language pairs and describes the total as 'over 1B' sentences, not 10B. It lists pairs one direction at a time, so it does not silently contain the reverse directions. Table 10 then reports evaluation on '36 * 36 Languages' using FLORES + IN22 with 517,638 + 384,560 = 902,198 sentences; even FLORES alone for 36 languages across all non-identity directions would need about 36 x 35 x 1012 = 1,275,120 sentence pairs, so the stated evaluation size cannot be a full 36x36 aggregate. Since the 36x36 coverage and 10B-corpus magnitude are headline contributions, the manuscript's own tables contradict the claim as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BhashaVerse, a 2-billion-parameter multilingual, multi-task encoder-decoder model claimed to support translation and auxiliary tasks (grammar correction, post-editing, quality estimation, error identification) across 36 Indian languages. Corpus creation combines existing resources, web-mined alignments filtered by COMET-QE, human post-edited domain data, and synthetic pivot/back-translation data. The MT component is evaluated on FLORES and IN22 with reported average BLEU 25.45, CHRF3 53.81, and COMET-22 0.8138; the remaining tasks are evaluated on held-out portions of the training data. The paper also describes publicly released corpora on Hugging Face.","tokens_in":31180,"tokens_out":6010,"duration_ms":62386,"significance":"If the claims were fully supported, the paper would offer a valuable resource: a single medium-sized multilingual model spanning many under-resourced Indic languages, together with released parallel corpora and multi-task training data. The use of external FLORES/IN22 benchmarks for MT is a strength, as is the explicit attention to low-resource languages such as Khasi, Santali, and Ho. The multi-task JSON interface is a practical design choice. However, the central quantitative claims are undermined by internal contradictions and by the absence of baselines, per-language evaluation, and independent test sets for all non-MT tasks. As it stands, the paper does not establish the 36x36-direction or 10-billion-sentence contributions that motivate the work.","major_comments":[{"comment":"The abstract and conclusion state that the model uses '10 billion parallel corpora' and supports '36 x 36 language directions', but Section 3 and Table 1 report 'more than 1B parallel corpora' covering only 325 language pairs. A complete 36x36 directed set would require 1296 pairs (or 1260 excluding identity). These are incompatible statements about the paper's headline contribution; the authors must reconcile the corpus size, the number of language pairs, and the exact sense in which all 36x36 directions are supported.","section":"Abstract, Section 3, Table 1, Section 13"},{"comment":"The MT evaluation reports an average over FLORES + IN22 for '36 * 36 Languages' with 517,638 + 384,560 sentences. Full 36x36 coverage on FLORES alone would require on the order of 36 x 35 x 1012 = 1,275,120 sentence pairs, so the stated evaluation size cannot represent all directed pairs. The paper must specify exactly which source-target directions were evaluated, report per-language and per-direction results, and include strong baselines such as IndicTrans2, NLLB-200, or Google Translate on the same splits. Without baselines, error bars, and a direction-by-direction breakdown, the average BLEU and COMET scores are not interpretable and do not support the cross-lingual coverage claim.","section":"Section 12, Table 10"},{"comment":"All non-MT tasks (grammar correction, post-editing, direct assessment, quality estimation, error identification) are evaluated on 'reserved development corpora' carved from the training data. This does not measure generalization to unseen data, and the synthetic DA/QE labels described in Section 5 are themselves derived from COMET scores and perturbation heuristics. If the development corpora contain or resemble these synthetic labels, the reported Spearman correlations of 0.46 and 0.47 and the F1/BLEU scores for error-related tasks are partly circular. The authors need independent held-out benchmarks or human evaluation, plus task-appropriate baselines, before claiming multi-task effectiveness.","section":"Section 12, Table 10, Section 5"},{"comment":"The corpus-construction pipeline relies heavily on synthetic translations generated by SSMT and IndicTrans2 pivoting through English/Hindi, filtered by per-pair average COMET-QE thresholds, with only 10% of sampled alignments human-validated. The paper reports no inter-annotator agreement, no per-pair precision of the COMET-QE filter, and no analysis of how pivot noise propagates into the final model. Since the 36x36 claim would depend on these synthetic pairs for most low-resource directions, the authors must provide validation statistics and ideally release the filtered/unfiltered data so the quality of the training corpus can be independently assessed.","section":"Section 3, 'Pivoted Parallel Corpora' and 'Iterative Backward Translation'"}],"minor_comments":[{"comment":"The metric is written as 'CHAF3' in Table 10 and in the text; it should be CHRF3.","section":"Table 10, Section 12"},{"comment":"The language inventory is inconsistent: the abstract says '36 Indian languages' but includes English in the list, and Table 9 contains 38 script-specific entries (e.g., kas_Arab and kas_Deva, snd_Arab and snd_Deva). The paper should define whether scripts are counted as separate entries and should align all counts with a single language-code table.","section":"Abstract, Section 8.2, Table 9"},{"comment":"The example output for Hindi uses 'hin_Latn' although Table 9 defines Hindi as hin_Deva; the example input/output JSON structures should use the same language codes defined in Table 9.","section":"Section 10.1.1, Table 9"},{"comment":"The text refers to 'Appendix Section 7' for perturbation techniques, but the perturbation descriptions are in Section 7 and there is no appendix; the cross-references should be corrected.","section":"Section 4, Section 7"},{"comment":"The prose says the human APE corpus includes 'English and 11 Indian languages', but Table 4 lists fewer target languages and the count is not consistent with other tables; the language counts should be reconciled.","section":"Section 4, Table 4"}],"recommendation":"reject","confidential_remarks":"The manuscript reads more like a resource announcement or technical report than a peer-reviewed systems paper in its current form. The internal contradictions between the abstract's 10B/36x36 claims and the body's 1B/325-pair tables, together with the absence of baselines and independent task evaluations, are too central to the paper's contribution to be addressed by minor revision. I would not oppose a future resubmission if the authors substantially rework the claims and provide the missing evaluations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline claim is overstated in a way that matters. The abstract says 10 billion parallel sentences and 36x36 directions; Table 1 reports 325 pairs and \"over 1B\" sentences. Those are not interchangeable, and Table 10's evaluation size (902,198 sentences) cannot cover 1,296 directions either. The paper never addresses the gap, so the central scale claim, as written, is contradicted by its own tables.\n\nWhat is real: the authors assembled a large parallel corpus collection for a broad set of Indian languages, including low-resource ones, added human post-edited health and education corpora, and trained a 2B-parameter encoder-decoder in a JSON multi-task format. The MT evaluation uses external FLORES and IN22 benchmarks, so that part is not circular. The corpus tables are granular and the data is released on HuggingFace. That is genuine infrastructure work.\n\nThe soft spots are significant. No baselines are reported, so an average BLEU of 25.45 and COMET-22 of 0.8138 have no interpretation without comparison systems. Non-MT tasks are evaluated on development sets carved from the same training data, so those scores (e.g., F1 41.23, Spearman ~0.46) are not evidence of generalization. The model and code are not released, only the data. The synthetic direct-assessment and QE labels are COMET-derived, which means the model's scores on those tasks partly reflect the COMET model used to create the training labels. And Table 9 lists 38 language-script entries, not 36, which adds to the impression of unreconciled reporting.\n\nWho gets value: anyone building parallel corpora or MT systems for Indian languages, especially if the HuggingFace release is complete and usable. The paper deserves a serious referee because the resource could be useful, but the claims need to match the data: per-language results, baselines, and a clearer statement of actual coverage and corpus size. I would not cite the current version, but I would look again after a revision that fixes the numbers and releases the model.","headline":"A genuine resource-building effort whose headline numbers (10B sentences, 36x36 directions) are contradicted by its own Table 1, so the paper needs heavy revision before its central claims can be trusted.","tokens_in":31704,"tokens_out":2587,"would_cite":false,"duration_ms":25253,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single 2-billion-parameter model can translate among 36 languages of the Indian subcontinent and, in the same JSON task format, correct grammar, post-edit translations, and score translation quality.","keywords":["machine translation","multilingual model","Indian languages","multi-task learning","parallel corpora","low-resource languages","quality estimation","post-editing"],"falsifier":"Take a random sample of about 2,000 sentence pairs from the pivot-generated Indian-to-Indian corpora, have independent translators mark each pair as aligned and adequate, and compute the error rate; if a substantial fraction, say over 20%, are mistranslations or misalignments, the corpus-quality premise fails and the reported scores cannot be taken as evidence for a usable model.","tokens_in":30684,"feed_emoji":"🌐","tokens_out":9994,"duration_ms":95392,"temperature":0.7,"pith_summary":"This paper tries to establish that one medium-sized encoder-decoder model can serve as a translation and language-processing backbone for the Indian subcontinent. The authors assemble a parallel corpus that combines existing resources, automatically aligned web and book content, human post-edited education and health data, and synthetic pairs created by pivoting through English and Hindi, then train a 2-billion-parameter multilingual multi-task model on it. On the FLORES and IN22 benchmarks they report an average BLEU of 25.45, CHRF3 of 53.81, and COMET-22 of 0.8138 across the covered language directions. They also report that the same model, without separate heads, handles grammar correction, post-editing, direct assessment, quality estimation, and error-span marking. If the claim holds, a single deployable system could lower the cost of cross-lingual services in education, health, and governance for dozens of low-resource languages.","feed_headline":"A single 2B-parameter model translates among 36 subcontinent languages","feed_subtitle":"It reports BLEU 25.45 and COMET-22 0.81 on FLORES+IN22, plus grammar correction and post-editing in one system.","key_machinery":"The load-bearing mechanism is the combination of a shared script-aware subword tokenizer and a JSON-structured multi-task training format. The tokenizer, a subword model with 48,000 tokens trained on mixed parallel data, maps Devanagari, Bengali, Tamil, Gurmukhi, Ol Chiki, and other scripts into one vocabulary so that related languages share representations. The JSON envelope specifies the task (translation, grammar correction, post-editing, direct assessment, quality estimation, error marking) and the domain in both input and output, which lets a single 18-layer encoder and 18-layer decoder be trained on all tasks at once. The authors use the same format at inference time, so the model's output type is chosen by the task string in the prompt.","core_discovery":"The central claim is that a 2-billion-parameter transformer encoder-decoder, trained from scratch with a shared subword vocabulary of 48,000 tokens, is enough to cover translation between English and 35 Indian languages in all 36 by 36 directions. The paper's evidence is a reported average of 25.45 BLEU, 53.81 CHRF3, and 0.8138 COMET-22 on the FLORES and IN22 benchmarks, together with development-set results for grammar correction, post-editing, quality estimation, direct assessment, and error identification. The authors attribute this to multi-task training in a JSON input-output format that explicitly names the task, source and target languages, and domain, so one decoder can emit different output types. The training corpus is built by combining cleaned existing corpora with sentence alignment scored by a COMET-QE model, pivot translation through English and Hindi, iterative back-translation, and paragraph-level back-translation, with human validation on a sample of the aligned data.","pith_inferences":["A reader could test whether the synthetic pivot data is the main driver of the reported scores by training a version on only the human-validated and directly curated subsets; the paper does not report such an ablation.","The paper's quality-estimation Spearman correlations, around 0.46 to 0.47, are modest, which suggests the score-prediction tasks may benefit more from a dedicated regression formulation or calibration than from the shared decoder alone.","Because grammar correction and post-editing are evaluated on held-out development sets drawn from the same domains as the training data, the high scores are not yet evidence for out-of-domain text; an external benchmark in legal or conversational text would settle that.","If the corpus-creation recipe transfers, the same pivot-plus-filtering pipeline could be applied to other low-resource language clusters outside India, such as African or Southeast Asian language groups, but the paper does not test that."],"forward_implications":["If the paper is right, a single 2-billion-parameter model can replace separate systems for English-to-Indian and Indian-to-Indian translation, cutting deployment and maintenance cost for multilingual services.","The JSON task format is extensible: a new task or domain can be added by defining a new task string and supplying formatted training pairs, rather than redesigning the architecture.","Shared subword vocabularies across scripts mean low-resource languages that share Devanagari or Bengali script can inherit representations from higher-resource relatives, which is the mechanism the authors rely on for Khasi, Santali, and Ho.","Because the same model outputs quality scores and post-edited text, translation pipelines can include automatic quality gatekeeping and correction without calling additional systems.","The public release of the parallel, post-editing, and evaluation corpora would give other researchers a common base for comparing models on Indian language translation."],"supporting_citations":[{"why":"Supplies the BPCC parallel corpus and the IN22 benchmark used for training and evaluation.","marker":"Gala et al. 2023"},{"why":"Samanantar is the main large-scale English-Indic and Indic-Indic parallel corpus the work builds on.","marker":"Ramesh et al. 2022"},{"why":"COMET-22 is the metric used for filtering corpus pairs and for the reported COMET score.","marker":"Rei et al. 2022a"},{"why":"The COMET-QE model is the neural aligner used to score sentence similarity and set alignment thresholds.","marker":"Rei et al. 2022b"},{"why":"FLORES is the evaluation benchmark for multilingual low-resource translation.","marker":"Goyal et al. 2021"},{"why":"Provides the transformer encoder-decoder architecture behind the model.","marker":"Vaswani et al. 2017"},{"why":"Defines BLEU, the primary reported translation metric.","marker":"Papineni et al. 2002"},{"why":"Defines CHRF, the character-level metric reported alongside BLEU.","marker":"Popović 2015"}],"fun_headline_variants":["2B model handles translation for 36 Indian languages","One 2B transformer covers 36 subcontinent languages","BhashaVerse: 2B model speaks all 36 Indian languages","36 Indian languages, one 2B model, all directions","Single 2B model translates across 36 languages in India"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that sentence pairs passing the COMET-QE threshold used for alignment and filtering are genuine parallel translations, even though only 10% of sampled alignments were human-checked; if the threshold keeps systematically noisy translations, the training signal and the reported scores collapse.","fun_headline_variants_meta":{"raw":{"variants":["2B model handles translation for 36 Indian languages","One 2B transformer covers 36 subcontinent languages","BhashaVerse: 2B model speaks all 36 Indian languages","36 Indian languages, one 2B model, all directions","Single 2B model translates across 36 languages in India"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1406,"prompt_tokens":1065,"completion_tokens":341,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":255}},"tokens_in":681,"tokens_out":341,"duration_ms":3461,"temperature":1.0,"reasoning_tokens":255,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:30:53.833640+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of about 2,000 sentence pairs from the pivot-generated Indian-to-Indian corpora, have independent translators mark each pair as aligned and adequate, and compute the error rate; if a substantial fraction, say over 20%, are mistranslations or misalignments, the corpus-quality premise fails and the reported scores cannot be taken as evidence for a usable model.","supporting_citations":[],"review_version":1}