{"id":"e4af44e5-aa4c-4a7b-a067-2bbcbf966d97","arxiv_id":"2412.12997","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A machine-translated Urdu MS MARCO dataset and fine-tuned mT5 reranker achieve MRR@10 0.248 and Recall@10 0.438, outperforming zero-shot baselines and providing first baselines for Urdu IR.","lead":"This paper creates the first Urdu version of the MS MARCO retrieval dataset by machine-translating it with IndicTrans2, then fine-tunes a multilingual reranker on it. The fine-tuned model beats zero-shot and BM25 baselines on Urdu retrieval, suggesting machine translation can bootstrap information retrieval for low-resource languages.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains may be adaptation to machine-translation artifacts, because the Urdu dev set's relevance labels are inherited from English without human validation; a human-translated dev subset would settle this.","rationale":"The reader's weakest assumption is that the machine-translated Urdu dev set preserves relevance labels well enough to support genuine Urdu retrieval evaluation. My stress-test pass reaches the same conclusion: this is the least secure link in the argument. The paper's own Section 5.2 concedes translation errors, and Section 6 defers quantitative translation-quality analysis to future work. Because the fine-tuned model is trained and evaluated on the same translation pipeline, label corruption from translation could inflate its apparent advantage over zero-shot and BM25, which have not been adapted to translationese. The concrete test I propose directly validates the dev-set labels with human judgment and checks whether the observed improvement survives on a human-verified subset. This is a strengthening, not a rejection, of the central claim; the paper's contribution as a dataset and baseline remains useful, and the requested validation is consistent with the authors' own stated future work. Therefore the reader's CONDITIONAL verdict remains appropriate, and no verdict change is needed.","tokens_in":5769,"tokens_out":3178,"duration_ms":33752,"concrete_test":"Take a random sample of 200 dev queries and their labelled relevant passages. Have two fluent Urdu speakers independently verify the existing Urdu translations of each query and passage, flagging cases where the translated query and translated passage are no longer semantically aligned with the original English label. Build a small human-validated Urdu dev set retaining the original binary relevance judgments. Re-run BM25, zero-shot mMARCO, and Urdu mT5-mMARCO on this subset and compare MRR@10 and Recall@10, with bootstrap 95% confidence intervals over the 200 queries. If the fine-tuned model's advantage over zero-shot persists on the human-validated subset, the headline claim is genuine; if the advantage collapses or reverses, the reported gains are an artifact of translationese.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Urdu mT5-mMARCO genuinely beats zero-shot mMARCO rests on the untested assumption that the IndicTrans2 translation of each query and passage preserves the original English relevance relation on the dev set. Section 5.2 admits translation errors (e.g., the word 'native' omitted in a passage), and Section 6 states that a quantitative analysis of translation quality is future work. If systematic translation errors break the query-passage relevance relation, then the fine-tuned model, which is trained on exactly the same style of translationese as the dev set, can appear to improve simply by learning to exploit translation artifacts rather than by understanding Urdu. This risk is asymmetric: BM25 and zero-shot mMARCO have not been adapted to translationese, so the comparison in Table 2 is not a fair test of genuine Urdu retrieval ability. The paper reports no human evaluation, no label-preservation rate, and no confidence intervals; the abstract/table discrepancy (MRR 0.247 vs 0.248) and the absence of significance testing make the reported advantage (0.204 to 0.248) fragile. These limitations are acknowledged in spirit in Section 5.2, but the central claim cannot be fully verified until the dev-set labels are validated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper creates the first large-scale Urdu IR dataset by translating the English MS MARCO passage ranking dataset with IndicTrans2, and establishes retrieval baselines on it. The authors evaluate BM25, a zero-shot mMARCO re-ranker, and a fine-tuned Urdu mT5-mMARCO re-ranker on the Urdu-translated development set. They report that fine-tuning improves MRR@10 from 0.204 (zero-shot) and 0.121 (BM25) to 0.248, and Recall@10 from 0.408 and 0.247 to 0.438. The dataset, model, and code are released publicly.","tokens_in":6002,"tokens_out":2548,"duration_ms":25831,"significance":"If the reported results are reliable, the paper makes a useful resource contribution: it provides a large-scale Urdu IR dataset, reproducible baseline numbers, and an open-source fine-tuned model, which can support future work on Urdu and similar low-resource languages. The paper is transparent about the machine-translation provenance of the data and openly describes its limitations. However, the central empirical claim rests on an unvalidated assumption that the Urdu-translated development set preserves the English relevance labels; no human evaluation, label-preservation analysis, significance testing, or confidence intervals are provided. The contribution is therefore more of a promising first step than a fully verified benchmark, and this distinction needs to be made explicit in the paper.","major_comments":[{"comment":"The dev set is entirely machine-translated with IndicTrans2, and relevance labels are inherited from English without human validation. Because the fine-tuned model is trained on the same translation style, its advantage over zero-shot mMARCO could partly reflect adaptation to translation artifacts rather than genuine Urdu retrieval ability. The paper itself acknowledges this risk (Section 5.2) and postpones a quantitative translation-quality analysis to future work (Section 6). To make the central claim load-bearing, I ask for a human-translated validation subset (e.g., 200-500 queries with human or independently adjudicated relevance labels) and a report of the same metrics on that subset, or at minimum a quantitative label-preservation analysis on a sample of the translated dev set.","section":"Section 3.2 and Section 5.2"},{"comment":"The zero-shot mMARCO row is missing MAP@10 and NDCG@10, even though Section 4.3 states that these metrics are included in the evaluation. Since the paper's main comparison is between zero-shot and fine-tuned reranking, the table should report all four metrics for every row. Without the missing values and without any measure of uncertainty (e.g., bootstrap confidence intervals or paired significance tests over the 6,980 queries), the reported MRR gain from 0.204 to 0.248 and Recall gain from 0.408 to 0.438 cannot be assessed as statistically meaningful.","section":"Table 2 and Section 4.3"},{"comment":"The abstract reports MRR@10 of 0.247 and Recall@10 of 0.439, while Table 2 lists 0.248 and 0.438 for the same Urdu mT5-mMARCO model. The paper should reconcile these numbers and ensure the abstract matches the final table, since a discrepancy of 0.001 on the headline numbers undermines reader confidence in the precision of the reported results.","section":"Abstract and Table 2"}],"minor_comments":[{"comment":"The phrase 'significant improvements' is used in the abstract and Section 5.1 without any statistical testing; consider replacing it with 'improvements' or adding the appropriate significance evidence, as noted in the major comments.","section":"Abstract"},{"comment":"The model name is spelled inconsistently as mMARCO, mMarco, and mMarco; please standardize to a single spelling (mMARCO) throughout.","section":"Section 2 and Section 4.2"},{"comment":"Table 1 is difficult to read in the current format: the highlighted omission of the word 'native' is mentioned in Section 5.2, but the table itself is not self-contained and the highlighting is not described in the caption. Please add a note about the coloring and ensure the Urdu examples are clearly aligned with their English counterparts.","section":"Table 1"},{"comment":"The sentence 'MIRACL addresses this by providing a diverse set of 18 languages' is slightly misleading because MIRACL is primarily a retrieval dataset rather than an initiative that 'addresses' low-resource IR in the same sense as this paper; consider rephrasing to describe what MIRACL actually offers.","section":"Section 2"},{"comment":"Reference [11] is cited as 'Xue, L.: mt5: A massively multilingual pre-trained text-to-text transformer' but the arXiv identifier 2010.11934 corresponds to the mT5 paper; the citation should include the full author list and the correct title format.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be a useful resource contribution for the Urdu IR community, and the authors are transparent about limitations. The main issue is that the headline empirical claim (fine-tuning improves over zero-shot on Urdu MS MARCO) is evaluated on a machine-translated dev set with no validation of label preservation and no statistical uncertainty quantification. This is fixable within the scope of the manuscript by adding a human-translated sample evaluation or a label-preservation analysis, plus significance testing. I also note that the paper is quite short and reads like a workshop contribution; for a full journal or major conference venue, the evaluation section would need to be substantially expanded."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plain take: this is a dataset-and-baselines paper, not a methods paper. The Urdu MS MARCO translation is new and fills a real gap; the baselines are honest and the resources are open. The central result — fine-tuning on translated Urdu data beats zero-shot — is credible but rests on an unvalidated machine-translated dev set, so the exact numbers should be treated as provisional.\n\nWhat it does well: the authors translated the full MS MARCO corpus with IndicTrans2, fine-tuned the mMARCO mT5 reranker, and report BM25 and zero-shot comparisons. They give concrete training details (batch size, learning rate, hours on which GPU). They also flag in Section 5.2 that translation errors can break query-passage relevance, and they make data, model, and code public. That is the right way to ship a resource.\n\nThe soft spots are real but mostly minor. There are no confidence intervals or significance tests, so the gap between zero-shot MRR 0.204 and fine-tuned 0.248 could be noise. The zero-shot row omits MAP and NDCG; the abstract says 0.247 MRR and 0.439 Recall while Table 2 says 0.248 and 0.438. The bigger validity question is whether the dev set, translated from English, preserves relevance labels well enough. Since the model trains on the same translation style as the dev set, some of the apparent gain may be adaptation to translationese rather than real Urdu understanding. The stress-test note has this right. The authors acknowledge the issue but do not quantify it; Section 6 lists human verification and translation-quality analysis as future work. That is fair for a first dataset, but it means the headline numbers are not a final verdict on Urdu retrieval quality.\n\nOne thing the stress-test note does not mention: the BM25 baseline probably uses a generic tokenizer, which is a known weak spot for Urdu morphology. So the BM25 numbers are likely pessimistic, not just the neural numbers being optimistic.\n\nWho this is for: IR researchers working on low-resource languages, especially South Asian languages. It is a useful starting resource and a reasonable baseline reference. It deserves a serious referee — the dataset itself is a contribution — but the review should push for a human-validated dev subset and error bars before the numbers are taken as definitive.\n\nRecommendation: engage with it; cite it if you work in multilingual IR; send it to peer review with requested revisions.","headline":"A useful dataset-and-baselines paper for Urdu IR: honest, open, and worth engaging with, but the headline numbers rest on an unvalidated machine-translated dev set and should be treated as provisional.","tokens_in":6489,"tokens_out":1947,"would_cite":true,"duration_ms":17771,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that translating MS MARCO into Urdu with IndicTrans2 and fine-tuning mMARCO on that translation yields a retrieval model with MRR@10 of 0.248 and Recall@10 of 0.438, clearly above zero-shot and BM25 baselines.","keywords":["Urdu information retrieval","MS MARCO","machine translation","low-resource languages","mMARCO","mT5","BM25","passage reranking"],"falsifier":"Take a random sample of the 6,980 Urdu dev queries, have native Urdu speakers judge the relevance of the top passages returned by Urdu mT5-mMARCO, and compare those judgments with the machine-translated labels; strong disagreement would indicate that the reported MRR and Recall are inflated by translation artifacts, while agreement would confirm the scores.","tokens_in":1461,"feed_emoji":"🔍","tokens_out":2335,"duration_ms":65050,"temperature":0.7,"pith_summary":"This paper gives Urdu its first large-scale information-retrieval dataset by machine-translating the English MS MARCO passage-ranking corpus with IndicTrans2, then establishes retrieval baselines on it. A fine-tuned Urdu mT5-mMARCO reranker reaches MRR@10 of 0.248 and Recall@10 of 0.438, beating both a zero-shot mMARCO reranker and BM25. The authors argue that language-specific fine-tuning on machine-translated data is a practical route for low-resource-language retrieval, and they release the dataset, model, and code so others can build on the baselines.","feed_headline":"Fine-tuned Urdu search model lifts MRR@10 to 0.248","feed_subtitle":"Machine-translated MS MARCO plus fine-tuning beats zero-shot and BM25 baselines on 6,980 dev queries.","key_machinery":"The load-bearing object is the Urdu mT5-mMARCO reranker: an mT5 sequence-to-sequence model, initialized from the multilingual mMARCO checkpoint, fine-tuned on 39 million machine-translated query-relevant-passage and query-nonrelevant-passage triples. At inference it scores each passage by the softmax probability of the Urdu word for \"yes\" versus \"no,\" producing binary relevance decisions, and those scores rerank the top 1,000 passages retrieved by BM25. The other carrying component is the translation step itself: IndicTrans2 converts both queries and passages into Urdu while preserving the original relevance labels, which turns MS MARCO into a usable training and evaluation resource for Urdu.","core_discovery":"The central discovery is that translating an existing high-resource retrieval dataset and fine-tuning a multilingual reranker on the translation gives a substantial retrieval-quality boost for a low-resource language. On the Urdu dev set, BM25 achieves MRR@10 of 0.121 and Recall@10 of 0.247, zero-shot mMARCO achieves 0.204 and 0.408, and the fine-tuned Urdu mT5-mMARCO model achieves 0.248 and 0.438, with MAP@10 of 0.159 and NDCG@10 of 0.340. The authors take this as evidence that even imperfect machine-translated training data can adapt a multilingual model to a language it was never trained on, and that the resulting benchmark and baselines are a useful foundation for future Urdu IR work.","pith_inferences":["Editorial inference: because the evaluation set is itself machine-translated, the reported gains may partly measure how well the model adapts to translationese; a human-annotated Urdu test set would separate genuine language ability from translation-artifact adaptation.","Editorial inference: the same translate-then-finetune recipe may work for other low-resource languages, but its payoff probably depends on translation quality, so pairing it with post-editing or confidence-based filtering of translated training triples would be a natural next test.","Editorial inference: Urdu-specific tokenization and morphological normalization would likely raise the BM25 baseline itself, which would change the comparison point for all neural rerankers evaluated on this dataset.","Editorial inference: the framework's reliance on MS MARCO's labels means it cannot measure whether retrieval quality holds for natural Urdu queries outside the translated domain; a small, natively written Urdu query set would test that."],"forward_implications":["Urdu IR now has a public benchmark and reproducible baselines, so future systems can be trained and evaluated in Urdu instead of relying on English or zero-shot multilingual transfer.","Fine-tuning on machine-translated data raises ranking quality above zero-shot transfer in this low-resource setting, suggesting a general recipe for languages that lack native IR datasets.","The two-stage pipeline of BM25 candidate retrieval plus a fine-tuned reranker transfers to Urdu without architectural changes, keeping the method simple and extensible.","Because dataset, model, and code are released, other South Asian languages with adequate translation models can follow the same path to establish their own first-pass benchmarks.","The fine-tuned model's gains over zero-shot mMARCO show that language-specific adaptation matters even when the underlying model is massively multilingual."],"supporting_citations":[{"why":"Supplies the MS MARCO passage-ranking dataset, the source corpus whose queries and passages are translated to create the Urdu dataset.","marker":"[3]"},{"why":"Provides IndicTrans2, the machine-translation model used to produce the Urdu translations of MS MARCO queries and passages.","marker":"[5]"},{"why":"Gives the mMARCO multilingual reranker and training recipe that this paper extends by fine-tuning on Urdu-translated data.","marker":"[4]"},{"why":"Defines mT5, the massively multilingual text-to-text transformer that serves as the underlying architecture for the fine-tuned reranker.","marker":"[11]"},{"why":"Defines the BM25 retrieval method used as the non-neural baseline and as the first-stage candidate retriever.","marker":"[9]"},{"why":"Supplies the setup used to retrieve the top 1,000 passages per query before reranking.","marker":"[7]"}],"fun_headline_variants":["Urdu retrieval MRR rises from 0.204 to 0.248 with fine-tuned mT5","First large-scale Urdu IR dataset from machine-translated MS MARCO","Fine-tuned Urdu mT5 beats zero-shot and BM25 on new benchmark","Translating MS MARCO creates Urdu IR benchmark with retrieval gains","Low-resource Urdu sees MRR boost via fine-tuned multilingual model"],"cache_read_input_tokens":8576,"weakest_assumption_plain":"The entire Urdu dataset is created by machine translation, so the scores depend on the assumption that IndicTrans2's translations preserve the original relevance labels well enough that retrieval metrics reflect genuine Urdu retrieval quality rather than adaptation to translation artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Urdu retrieval MRR rises from 0.204 to 0.248 with fine-tuned mT5","First large-scale Urdu IR dataset from machine-translated MS MARCO","Fine-tuned Urdu mT5 beats zero-shot and BM25 on new benchmark","Translating MS MARCO creates Urdu IR benchmark with retrieval gains","Low-resource Urdu sees MRR boost via fine-tuned multilingual model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000317,"raw_usage":{"total_tokens":1775,"prompt_tokens":911,"completion_tokens":864,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":760}},"tokens_in":527,"tokens_out":864,"duration_ms":7802,"temperature":1.0,"reasoning_tokens":760,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:30:27.768386+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 6,980 Urdu dev queries, have native Urdu speakers judge the relevance of the top passages returned by Urdu mT5-mMARCO, and compare those judgments with the machine-translated labels; strong disagreement would indicate that the reported MRR and Recall are inflated by translation artifacts, while agreement would confirm the scores.","supporting_citations":[{"cited_title":"Nist Special Publication Sp pp","cited_arxiv_id":null,"evidence_quote":"Defines the BM25 retrieval method used as the non-neural baseline and as the first-stage candidate retriever."}],"review_version":1}