{"id":"82997b17-490e-4884-b4b0-7ad036ea627f","arxiv_id":"2501.15747","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A machine-translated version of MMLU-Pro in nine Indic languages is released as a benchmark, with baseline accuracy scores for multilingual LLMs.","lead":"The authors translated the English MMLU-Pro question set into nine Indian languages using an automatic translation system and ran several language models on the result. The paper is a potential resource for measuring how well AI models understand Indian languages, but the reported quality checks cover only three of the nine languages and the human review scores are not shown.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Translation fidelity is the load-bearing assumption; published validation covers only 3 of 9 languages, shows meaning-changing errors, and the authors concede gaps—so the benchmark's validity as an evaluation tool is not established.","rationale":"The reader's weakest_assumption is exactly the translation-fidelity assumption, and the manuscript's own evidence contains the most damaging admissions: metrics for six languages are missing, expert scores are absent, and published examples demonstrate meaning-changing translation errors. These are not hypothetical risks; they are concrete counterexamples to the claim that the benchmark 'maintains the integrity of the original MMLU-Pro.' The correct answer to an item about the embryological origin of the hyoid bone cannot be determined from a Tamil question about its thematic appearance. If such errors are widespread, the benchmark measures translation artifacts rather than language understanding, and the reported model rankings (Tables 1–2) could be driven by translation quality rather than model capability. The concern is load-bearing because the entire contribution is the dataset; without validated translations, the benchmark lacks the property that makes it useful. The CONDITIONAL verdict remains appropriate: the dataset and scripts are public, and the missing evidence is obtainable. The proposed concrete_test directly measures answer preservation, which is the property that matters for a multiple-choice benchmark. If the test passes, the benchmark's validity is substantially restored; if it fails, the paper's central claim would need to be revised. The paper's own limitation statement in Section 3.3—that comprehensive conclusions cannot be drawn for six of nine languages—reinforces this concern and should have been given more weight in the reader's rationale, though the overall verdict direction is correct.","tokens_in":18338,"tokens_out":3067,"duration_ms":28704,"concrete_test":"Take a stratified random sample of 100 questions per language (900 total) from the released dataset. For each, two native-speaker annotators (not authors) attempt the translated multiple-choice item without seeing the English original; compute agreement between the selected option and the English gold label. Also re-translate the same sample to English with an independent MT system (e.g., NLLB-200 or Google Translate) and compute answer agreement with the English MMLU-Pro labels. If per-language answer-preservation is below 95%, or if the independent re-translation changes the correct option for more than 5% of items, the benchmark's validity as a faithful adaptation is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that IndicMMLU-Pro is a high-quality benchmark preserving MMLU-Pro's integrity—depends on translations preserving each question's meaning and correct answer in all nine languages. This is not established. Section 3.3 reports back-translation metrics for only Hindi, Gujarati, and Tamil; the authors explicitly state data for the other six languages is missing and metrics are 'being compiled' for future publications. Expert proofreading (Section 2.6) involved 13 reviewers on 9,000 pairs, but no scores or inter-annotator agreement are reported, so the reader cannot verify the claimed assurance. The published examples undercut the claim: Figure 4 shows Tamil 'embryological origin of the hyoid bone' back-translated as 'thematic appearance of the hyoid bone', and Figures 8–9 show 'pupil of your eye' rendered as 'eyelid' in Gujarati and Tamil. These are not mere style shifts; they change the referent of the question and can flip the correct option. Because the gold labels are inherited from the English MMLU-Pro, any such meaning-changing error invalidates that item. The back-translation QA is also circular: both passes use IndicTrans2, so systematic errors cancel. LaBSE cosine similarities (Table 4, ~0.91–0.93) are too coarse to detect meaning-changing confusions; a sentence about 'thematic appearance' can embed close to one about 'embryological origin' in a multilingual embedding space. Thus the load-bearing assumption is not merely under-supported; available evidence contains concrete counterexamples.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces IndicMMLU-Pro, a nine-language (Hindi, Bengali, Gujarati, Marathi, Kannada, Punjabi, Tamil, Telugu, Urdu) benchmark created by translating the English MMLU-Pro dataset with IndicTrans2 while retaining the original multiple-choice structure and inherited gold labels. It describes a quality-assurance pipeline based on back-translation metrics (chrF++, BLEU, METEOR, TER, SacreBLEU), LaBSE cosine similarity, and an expert proofreading exercise, and it reports zero-shot baseline accuracy for a range of multilingual and Indic-specific models. The central claim is that this systematic pipeline ensures a high-quality multilingual benchmark preserving MMLU-Pro's integrity.","tokens_in":18623,"tokens_out":5569,"duration_ms":51033,"significance":"If the translation-fidelity claim were fully supported, this would be a useful public benchmark filling a clear gap for Indic-language evaluation, with broad model coverage and a plausible template for adapting MMLU-Pro. The authors make the dataset publicly available on Hugging Face, describe their process transparently, and include a wider range of baseline models than many prior Indic benchmark papers. However, as submitted, the load-bearing validation is incomplete: reported QA metrics cover only three of nine languages, expert proofreading scores are never presented, and the paper's own examples contain meaning-changing translation errors. The current version therefore does not yet establish the benchmark's validity for the comprehensive claims made in the title, abstract, and Figure 1 caption.","major_comments":[{"comment":"The quality assessment that underpins the 'high-quality benchmark' claim is reported for only three of nine languages. The text explicitly states that metrics for Bengali, Punjabi, Kannada, Telugu, Urdu, and Marathi are missing and 'being compiled.' Because the dataset is released for all nine languages and the abstract and Figure 1 caption assert comprehensive coverage, the central claim is not supported for six languages. Complete per-language metrics and, ideally, per-item pass/fail counts under the stated thresholds are needed.","section":"§3.3, Table 3"},{"comment":"The back-translation QA is circular with respect to the translation model: forward translation and back-translation both use IndicTrans2. Any systematic translation errors that IndicTrans2 makes in both directions will be invisible to chrF++, BLEU, METEOR, TER, and SacreBLEU. The independent check described in §2.6 (expert proofreading of 9,000 sentence pairs by 13 reviewers) would break this circularity, but no scores, no per-language results, and no inter-annotator agreement are reported. As a result, the manuscript does not currently provide the independent evidence needed to support the claim that meaning and answer labels are preserved.","section":"§2.1, §3.3"},{"comment":"The paper's own published examples contradict the semantic-fidelity claim. Figure 4 translates 'embryological origin' into Tamil as 'thematic appearance,' and Figures 8 and 9 render 'pupil of your eye' as 'eyelid' in both Gujarati and Tamil. These are referent-changing errors rather than stylistic shifts; for a multiple-choice benchmark whose gold labels are inherited from the English source, such errors can invalidate individual items. The authors need to audit the released dataset for these and similar errors and either correct the translations or explicitly remove/repair the affected items; aggregate back-translation scores do not address the validity of individual items.","section":"Figure 4; Figures 8–9"},{"comment":"The LaBSE cosine similarities (~0.91–0.93) are too coarse to establish meaning preservation. A sentence about 'thematic appearance' can be close in embedding space to one about 'embryological origin,' particularly when the surrounding context is similar, so these scores do not detect the referent-changing errors shown in Figures 4, 8, and 9. The table should be presented as a coarse supplementary signal, not as evidence that answer-carrying content is preserved.","section":"Table 4"},{"comment":"The baseline protocol for MuRIL and XLM-RoBERTa uses a non-standard pairwise sequence-classification scoring heuristic (question paired with each option separately, then argmax over the resulting scores), which the appendix itself concedes is 'not the standard way' and 'may not be as accurate or reliable' as a properly fine-tuned multiple-choice model; IndicBART is scored with a generation-length heuristic. Because §3.2 and §5 draw tier conclusions and cross-model comparisons from these numbers, the baselines should either be replaced by a consistent fine-tuned multiple-choice protocol or be explicitly labeled as heuristic scores and excluded from comparative claims such as 'XLM-RoBERTa significantly outperforms other models.'","section":"§7.2, Tables 1–2"},{"comment":"Figure 1 and §2.2 state that 'only translations meeting quality thresholds proceed to the final dataset,' but the manuscript reports no information about how many items failed the thresholds in each language, whether thresholds were applied per item or per aggregate corpus, or how rejected items were handled. Without this information, the dataset construction process is not reproducible, and the claim that the thresholds enforce quality cannot be verified.","section":"Figure 1, §2.2"}],"minor_comments":[{"comment":"The threshold lists give BLEU as '>25-30%' and METEOR as '50-60%,' but the reported scores are on a 0–1 scale (0.54–0.59 BLEU, 0.52–0.56 METEOR); the units should be harmonized to avoid ambiguity.","section":"§2.2, §3.3"},{"comment":"The caption contains the misspellings 'Dravidan' and 'Telegu,' and the table does not explain why certain model–language combinations are marked N/A.","section":"Table 2 caption"},{"comment":"The introduction lists 'task taxonomy' as a contribution, but no explicit task taxonomy section or enumeration is provided in the paper; either add it or soften the claim.","section":"§1, §2"},{"comment":"The text says that data for the other six languages 'is missing from Table 2,' but the intended reference is Table 3 (the back-translation metrics table), not Table 2 (model accuracy).","section":"§3.3"},{"comment":"The axis label '#' is ambiguous; clarifying that it represents a count would improve readability.","section":"Figure 6"},{"comment":"The model list in §2.5 includes Airavata, OpenHathi, TamilLlama, and MahaMarathi, but Tables 1–2 report results for only a subset of these models; the availability or applicability conditions for each model should be stated in one place.","section":"§2.5, Tables 1–2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the authors are unusually transparent about their own limitations, and the public dataset release is a useful community resource. However, the paper's central claim currently rests on incomplete and partially circular validation, and several published examples show meaning-changing translation errors. I would not reject outright because the dataset can in principle be corrected and re-validated; with complete per-language metrics, reported expert scores, an item-level error audit, and a more cautious framing, this could become a solid benchmark contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a real dataset release, but the paper overclaims what it has validated. The useful part is clear—translated MMLU-Pro for nine Indic languages, publicly hosted, with a consistent set of baselines. That fills a genuine gap, and the core idea is not exotic: take a strong English benchmark, translate with IndicTrans2, check quality, run models. It is an extension, not a new paradigm, and that is fine.\n\nWhat is done well: the dataset structure mirrors MMLU-Pro; the language coverage is diverse in scripts and families; the baselines span a sensible range of general and Indic-focused models; and the paper is transparent about the fact that QA metrics are only shown for three languages. Running a human proofreading exercise with 13 native speakers was the right instinct.\n\nThe soft spots are not minor. The paper's central claim is that the benchmark \"maintains the integrity\" of MMLU-Pro, but the evidence does not support that yet. Back-translation scores are reported for Hindi, Gujarati, and Tamil only; the other six languages are explicitly missing. The line that \"preliminary assessments suggest similar trends\" is not data. The expert proofreading scores are never reported, so we cannot know whether the 9,000-pair review caught the errors visible in the appendix. Those errors are concrete: \"pupil of your eye\" becomes \"eyelid\" in Gujarati and Tamil, and \"embryological origin\" becomes \"thematic appearance\" in Tamil. For a benchmark inheriting English gold labels, these are not style shifts; they change the referent and can change the correct answer. The back-translation QA is also circular because both passes use the same model, so systematic translation errors cancel. LaBSE cosine similarities around 0.92 are too coarse to catch confusions of this kind.\n\nSmaller issues: the paper mentions \"prior work on IndicMMLU\" without citing it, which makes the novelty claim hard to assess, and the scoring heuristics for MuRIL and XLM-RoBERTa are non-standard and acknowledged as such in the appendix. Those are fixable or at least reportable.\n\nWho this is for: anyone working on Indic-language evaluation. The dataset could become useful, but only after the missing QA metrics and expert scores are released, and ideally after the problematic items are corrected or filtered. I would not cite it in its current form.\n\nRecommendation: send it to peer review, conditional. The artifact is worth referee time, but reviewers should require the full QA data, a rerun or explanation of the suspect translations, and a clear statement of novelty relative to the earlier IndicMMLU work.","headline":"A genuinely useful dataset release buried under an overclaimed validation story: the benchmark deserves referee time, but only with the missing QA evidence and the acknowledged translation errors addressed.","tokens_in":19180,"tokens_out":2036,"would_cite":false,"duration_ms":19837,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IndicMMLU-Pro translates the MMLU-Pro benchmark into nine Indic languages, reports that GPT-4o is the best of thirteen tested models with top accuracy of 44.8% in Hindi, and argues that back-translation checks and expert proofreading…","keywords":["Indic languages","multilingual benchmark","MMLU-Pro","machine translation","large language models","evaluation","natural language understanding"],"falsifier":"Take a random sample of questions from Bengali, Punjabi, Kannada, Telugu, Urdu, and Marathi, have native speakers independently identify the intended answer from the translated options, and compare with the English key; if a substantial fraction change or become unanswerable — as the printed Tamil example already shows 'embryological origin' becoming 'thematic appearance' — the dataset's integrity claim fails for those languages.","tokens_in":18144,"feed_emoji":"🌐","tokens_out":5203,"duration_ms":42614,"temperature":0.7,"pith_summary":"IndicMMLU-Pro is a benchmark that takes the English MMLU-Pro multiple-choice dataset and translates it into nine Indian languages — Hindi, Bengali, Gujarati, Marathi, Kannada, Punjabi, Tamil, Telugu, and Urdu — using the IndicTrans2 machine translation model. The paper argues that with back-translation quality checks, expert proofreading, and embedding-based similarity scores, the translated dataset preserves the meaning and difficulty of the original while adapting to Indian linguistic and cultural contexts. It then reports baseline accuracy for thirteen multilingual and Indic-specific models, finding that GPT-4o leads every language with scores between 38.46% and 44.80%, and that most other models stay near chance level. If the benchmark is sound, it gives researchers a standardized, publicly available way to measure progress in Indic language understanding.","feed_headline":"AI tops out near 45% on nine-language Indic benchmark","feed_subtitle":"IndicMMLU-Pro translates MMLU-Pro into Hindi, Tamil, Urdu and six more languages; GPT-4o leads all models but no one breaks 45%.","key_machinery":"The load-bearing object is the translation-and-back-translation validation loop. IndicTrans2 produces the nine Indic datasets from English MMLU-Pro; quality is asserted by translating a sample back to English and thresholding five automatic metrics (chrF++ above 0.5, BLEU/SacreBLEU 25–30, METEOR 0.5–0.6, TER below 0.4–0.5), then scoring 9,000 sentence pairs by 13 expert reviewers on semantic accuracy, fluency, and stylistic appropriateness, and finally measuring LaBSE embedding cosine similarity between languages. This loop is what the paper relies on to claim the dataset maintains the integrity of the original MMLU-Pro.","core_discovery":"The central claim is that a high-quality, publicly available Indic-language equivalent of MMLU-Pro can be built by machine translation rather than by native human authoring. The paper shows the full pipeline: MMLU-Pro is translated by IndicTrans2 into nine languages, a subset is back-translated to English and scored with chrF++, BLEU, METEOR, TER, and SacreBLEU, thirteen native experts proofread 9,000 sentence pairs on meaning, fluency, and style, and LaBSE cosine similarity between Indic and English items is used to confirm semantic alignment. On the resulting benchmark, GPT-4o is the strongest model in all nine languages, with Hindi the highest at 44.80% accuracy, while older multilingual encoders cluster near 10–13%. The paper interprets these numbers as baseline results and a performance tier structure for future Indic LLM work.","pith_inferences":["The paper reports quality metrics for only three of nine languages, so a natural extension is to back-translate and expert-check the other six; until then, statements about the whole dataset's quality are extrapolations.","Because translation errors can change the correct option, cross-language accuracy differences (for example, Tamil's lower scores) may reflect translation artifacts as much as model capability — a confound the benchmark does not decompose.","A diagnostic extension would pair English MMLU-Pro with IndicMMLU-Pro for the same models: the per-category score drop in each language would map where translation or reasoning fails.","The dataset could also serve as a stress test for machine translation quality itself, since answer-key consistency across languages is a measurable, high-stakes signal of semantic preservation."],"forward_implications":["GPT-4o's 38.46–44.80% range across the nine languages becomes the baseline that future Indic LLMs must beat.","The large gap between GPT-4o and the 10–13% scores of IndicBERT, MuRIL, XLM-RoBERTa, and similar models quantifies how far current open and Indic-specific models lag on multitask understanding.","Because the dataset preserves MMLU-Pro's 14 categories and structure, model rankings on IndicMMLU-Pro can be compared directly with English MMLU-Pro rankings to isolate language-specific degradation.","The public Hugging Face release lets any team reproduce baselines or add new models without building a translation pipeline.","The absence of complete quality metrics for six of the nine languages means a full-dataset quality statement will require the promised future publication of those numbers."],"supporting_citations":[{"why":"Supplies the source benchmark MMLU-Pro, whose structure, categories, and questions the paper translates into nine Indic languages.","marker":"Wang et al., 2024"},{"why":"Provides IndicTrans2, the machine translation model used to produce all nine Indic-language versions of the dataset.","marker":"Gala et al., 2023"},{"why":"Defines chrF++, the primary back-translation metric reported for Hindi, Gujarati, and Tamil.","marker":"Popović, 2017"},{"why":"Defines BLEU, used along with SacreBLEU to threshold translation quality in back-translation checks.","marker":"Papineni et al., 2002"},{"why":"Defines METEOR, used to measure semantic similarity in the quality assurance step.","marker":"Banerjee and Lavie, 2005"},{"why":"Defines SacreBLEU, a standardized BLEU variant used as one of the quality thresholds.","marker":"Post, 2018"},{"why":"Defines TER, the translation edit rate used in the paper's evaluation metrics.","marker":"Snover et al., 2006"}],"fun_headline_variants":["Indic benchmark: GPT-4o leads, but 45% ceiling","Nine-language AI test: best score barely 45%","IndicMMLU-Pro: multiling AI hits 45% max","GPT-4o tops Indic benchmark, still under half","New benchmark gauges AI on 9 Indic languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's value rests on the assumption that IndicTrans2's translations into all nine languages preserve each question's meaning, difficulty, and correct answer; the paper directly verifies this for only Hindi, Gujarati, and Tamil.","fun_headline_variants_meta":{"raw":{"variants":["Indic benchmark: GPT-4o leads, but 45% ceiling","Nine-language AI test: best score barely 45%","IndicMMLU-Pro: multiling AI hits 45% max","GPT-4o tops Indic benchmark, still under half","New benchmark gauges AI on 9 Indic languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000583,"raw_usage":{"total_tokens":2738,"prompt_tokens":936,"completion_tokens":1802,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":1715}},"tokens_in":552,"tokens_out":1802,"duration_ms":11858,"temperature":1.0,"reasoning_tokens":1715,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:57:55.912425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of questions from Bengali, Punjabi, Kannada, Telugu, Urdu, and Marathi, have native speakers independently identify the intended answer from the translated options, and compare with the English key; if a substantial fraction change or become unanswerable — as the printed Tamil example already shows 'embryological origin' becoming 'thematic appearance' — the dataset's integrity claim fails for those languages.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines BLEU, used along with SacreBLEU to threshold translation quality in back-translation checks."}],"review_version":1}