Pith. sign in

REVIEW 4 major objections 5 minor 18 references

Multilingual Open QA on the MIA Shared Task

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Zero-shot question-generation likelihood can re-rank retrieved passages in multilingual open QA, improving Korean and Japanese recall with no training while hurting Finnish and Bengali.

desk verdict An honest but overclaimed shared-task report: the one clear Korean re-ranking gain is likely noise, and the 'high-resource' explanation contradicts the paper's own table. read the letter →

arxiv 2501.04153 v1 pith:H6AQJSGA submitted 2025-01-07 cs.CL cs.LG

Navya Yarrabelly , Saloni Mittal , Ketan Todi , Kimihiro Hasegawa This is my paper · ORCID
classification cs.CLcs.LG
keywords cross-lingualinformationretrievalquestiongenerationre-rankingopenansweringzero-shotlearningmultilinguallanguagemodelsdensepassageXOR-TyDiQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a simple, training-free re-ranking step improves passage retrieval in multilingual open question answering. The re-ranker takes the top-50 passages from an existing dense retriever, mDPR, and re-scores each one by the average log-likelihood that a pretrained multilingual model assigns to the question given that passage — a zero-shot question-generation score. On the XOR-TyDi QA benchmark, this mBART-based score raises Recall@5 and Recall@15 for Korean and Japanese, but it lowers recall for Finnish and Bengali. The authors therefore claim the method helps for reasonably high-resource languages when the pretrained model generates well, and that it can be layered on any base retriever without labelled retrieval data. The paper's second contribution, machine-translation data augmentation, did not yield consistent gains in answer generation.

What carries the argument

The central object is the question-generation re-ranking score, QGPR, which is $\log p(q|z)$: the average per-token log-likelihood of the question $q$ when a pretrained multilingual model generates it conditioned on passage $z$. This score is computed by a zero-shot mBART-50 model, with a language tag for the target language, over the top-50 passages produced by mDPR; passages are then re-ranked solely by this likelihood, ignoring the original retrieval score. The mechanism is token-level cross-attention between question and passage inside the language model, which the authors argue is more expressive than the dense inner-product matching of mDPR. It lets the same pretrained model serve as a training-free relevance scorer across languages.

What would settle it

On the XOR-TyDi dev sets, compute the QGPR score for each language's ground-truth positive passages and for sampled negative passages, then check whether positives rank above negatives; the paper's own recall drops for Finnish and Bengali already suggest that in those languages the score is not functioning as a cross-lingual relevance signal.

Watch

Extended reading notes

Core claim

On the paper's terms, the central discovery is that the conditional probability of the question given a retrieved passage, estimated by a zero-shot pretrained multilingual model, carries enough relevance signal to re-rank mDPR's top-50 passages in a completely unsupervised way. The authors instantiate this as QGPR, computing $\log p(q|z) = \frac{1}{|q|} \sum_t \log p(q_t | q_{<t}, z; \Theta)$ with mBART, and report Recall@5 improvements from 12.1 to 16.0 (Korean) and 5.5 to 5.6 (Japanese), with Recall@15 improvements from 21.4 to 25.6 and 9.1 to 9.7, against the mDPR baseline. The same procedure with mT5, and with either questions or passages machine-translated into the other language, failed to beat the baseline. The authors interpret the gains as evidence that question generation forces the model to explain every token in the question through rich cross-attention, and they qualify the claim to languages where the zero-shot generator is competent: Korean and Japanese improve, Finnish and Bengali degrade.

Load-bearing premise

The load-bearing premise is that the average token log-likelihood $\log p(q|z)$ from a pretrained multilingual model ranks relevant passages above irrelevant ones even when the question and passage are in different languages and the model was never trained for question generation — a premise the paper's own Finnish and Bengali results violate.

Editorial extensions

If this is right

  • Any open-QA retriever, sparse or dense, can be coupled with this re-ranker, so languages without labelled retrieval data can still get a ranking signal from a pretrained multilingual model.
  • The benefit is conditional on the quality of the zero-shot generator: Korean and Japanese gain, Finnish and Bengali lose, so the re-ranker should be deployed selectively rather than uniformly.
  • Because re-ranking ignores the base retriever's scores, it provides an orthogonal signal that can improve the ordering of passages already retrieved, but it cannot recover positives that the base retriever failed to include in its top-50.
  • The machine-translation data augmentation approach does not reliably improve QA F1, and the paper attributes this mainly to input-length limits in the reader model that keep most translated contexts out.
  • The paper's MRR analysis indicates the re-ranking helps mainly for same-language positives, while cross-lingual MRR improves in some languages but remains low in absolute terms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run is to use the QGPR score with a question-generation model that is itself strong in Finnish and Bengali; if the gain reappears, the bottleneck is per-language generator quality rather than the scoring principle.
  • The near-zero cross-lingual MRR values in the paper's Table 4 suggest the method is mostly reordering same-language passages, so the cross-lingual retrieval problem remains largely open even if this re-ranker is adopted.
  • The same average-likelihood criterion could in principle re-rank passages in other retrieval-heavy tasks, such as fact verification or entity linking, wherever a multilingual generator exists, but nothing in this paper tests that transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper, submitted to the MIA shared task, proposes two modifications to a CORA-style multilingual open QA pipeline (mDPR retrieval followed by mT5 generation). The first contribution, Question-Generation based Re-ranking (QGPR), re-scores the mDPR top-50 passages with a zero-shot question-generation likelihood log p(q|z) computed by mBART or mT5, with additional variants that translate the question or passage before scoring. The second contribution is machine-translation-based data augmentation of Korean and Bengali training triples for the reader. Retrieval results are reported for Korean, Japanese, Finnish, and Bengali, and QA F1 results for seven languages. The paper claims that QGPR outperforms the mDPR baseline for 'reasonably high resource languages like Korean and Japanese' and that the augmentation approach gives only marginal, inconsistent gains.

Significance. If the QGPR claim were robust, a training-free reranker applicable to any retrieval method would be a useful contribution to low-resource CLIR, because it avoids labeled retrieval data. The paper has genuine positive features: the method is simple, the reranking score in Eq. (4) is a model likelihood rather than a fitted value, the evaluation is against external XOR-TyDi ground truth, and the authors report negative results as well as positive ones. However, the evidence offered does not establish the central claim. The only non-trivial improvement is for Korean; Japanese gains are tiny, Finnish and Bengali degrade, and the paper's own resource-based explanation is contradicted by its dataset statistics. The QA augmentation comparison is additionally confounded by a change in reader input length.

major comments (4)
  1. [§6.1, Tables 2–3] The central claim rests on a single language. The conclusion narrows the claim to 'reasonably high resource languages like Korean and Japanese', but Table 1 lists Korean with the fewest training examples (1,856) of all seven languages, fewer than Bengali (2,428), where QGPR-mBART degrades most. Japanese gains are +0.1 R@5 and +0.6 R@15 on a dev set of 693 questions, which is within plausible sampling noise. No significance tests, confidence intervals, or error bars are reported anywhere. The paper therefore provides no statistical support for the claim that the method is effective, and the proposed high-resource explanation is a post hoc assertion contradicted by the paper's own table.
  2. [§6.1, metrics paragraph] Recall@K is defined with 'the ground truth positives in the top-50 results obtained by mDPR as our total positives set'. This means QGPR can only reorder passages that mDPR already retrieved; any passage missed by mDPR is invisible to the evaluation. The abstract and introduction describe the method as improving 'passage retrieval', but the experiments only measure re-ranking within the mDPR top-50. The claims should be scoped accordingly, and the metric definition should be stated in the abstract or introduction if retrieval recall is discussed.
  3. [§5.2, §6.2, Table 5] The QA comparison is confounded by the reader input length. Section 5.2 states that the maximum input length was reduced from the baseline default of 1000 to 600 tokens, so the augmented reader sees fewer passages than the baseline system. Table 5 therefore does not provide a controlled comparison of the data augmentation method. The authors themselves note that they 'do not see any consistent tendency' and attribute non-augmented-language changes to training hyperparameters. As written, no conclusion about the augmentation approach can be drawn from this experiment.
  4. [§4.1, Eq. (4)] The method's core assumption is that a pretrained mBART likelihood p(q|z) is a reliable cross-lingual relevance signal. The paper gives no analysis of when this assumption holds: the Finnish and Bengali degradations show it fails for some languages, but the paper does not examine characteristics such as passage-language mix, question answerability, or language relatedness. Without such analysis, the 'high-resource language' caveat is an ad hoc explanation rather than a tested condition.
minor comments (5)
  1. [Throughout] There are numerous typographical errors, including 'Japaneese', 'vice-e-versa', 'mulltingual', 'space space', 'had to tracks', and 'doesn't require'. A careful proofreading pass is needed.
  2. [§3.1] The sentence describing the 768-dimensional passage representation is duplicated: 'mBERTp(p)' and the following sentence both introduce the mBERT encoder. One copy should be removed.
  3. [References] The citation style is inconsistent: the paper alternates between 'XOR-TYDI QA', 'XOR QA', and 'XOR-TYDI', and Asai et al. 2020/2021a appear to refer to the same work. The 'F oundations' spacing in the Robertson and Zaragoza reference should also be fixed.
  4. [§6.1] The text says the mBART variant gave an improvement of '4% in Recall@5 and Recall@15' for Korean. This is consistent with Table 3 (+3.9, +4.2), but the preceding sentence about 'Positives@K and Recall@K' should make clear which metric is being quoted, since Table 2 shows much smaller absolute changes in P@K.
  5. [Table 1] The table caption calls the dataset 'XOR-TyDi-QA' and includes English with 91,876 training examples, but English has no development split in the table. Clarify whether English is used only for training and how the 'target dataset' label applies.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: QGPR score is a fixed pretrained likelihood, not a fitted parameter, and evaluation is external against mDPR and XOR-TyDi ground truth.

full rationale

The paper's central claim is that reranking mDPR top-50 passages by zero-shot question-generation likelihood log p(q|z) (Equation 4, Section 4.1) improves retrieval for Korean and Japanese. This score is computed by a fixed pretrained mBART checkpoint (Section 5.1); no parameter is fitted on XOR-TyDi, no hyperparameter is tuned on the reported dev sets, and no constant is chosen to force the result. The evaluation is external: it compares against the mDPR baseline and uses ground-truth positives, only within the top-50 candidate set that mDPR returned. That candidate-set constraint is an evaluation limitation, not a circularity, because the reranking procedure is not defined in terms of the Recall@K metric. The only citation used as motivation, Sachan et al. (2022), is external prior work and is not by the present authors. The paper itself reports negative results for Finnish and Bengali, so the method is not fitted to the conclusion; the post-hoc 'high-resource language' explanation is empirically shaky because Korean has the fewest training examples in Table 1, but that is a correctness and statistical concern, not a circular reduction. Consequently no circular step can be exhibited, and the score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the pretrained models' zero-shot question-generation quality, on the fixed mDPR top-50 candidate set, and on the reliability of machine translation. None of these is derived or validated inside the paper.

free parameters (3)
  • top-k candidate set size (k=50) = 50
    Reranking is applied only to the top-50 mDPR results; this set defines the ground-truth positive set and caps the achievable recall (Section 5.1, Section 6.1).
  • reader max input length = 600 tokens
    Chosen due to GPU memory limits, but differs from the baseline's 1000-token input, confounding the data augmentation comparison (Section 5.2).
  • augmentation volume = 5k examples, 3 translated paragraphs each
    Heuristic choice for computational cost; no sensitivity analysis (Section 5.2).
assumptions (3)
  • domain assumption A pretrained multilingual model's question likelihood log p(q|z) correlates with passage relevance in a zero-shot, cross-lingual setting.
    Central to QGPR; introduced in Section 4.1 (Eq. 4) and never validated independently.
  • domain assumption The mDPR top-50 retrieval outputs shared by the organizer are a fixed, reliable candidate set, and any relevant passage outside this set is unrecoverable by reranking.
    Evaluation is limited to this set in Section 5.1 and Section 6.1.
  • domain assumption Machine translation (mBART-50) preserves question, passage, and answer semantics well enough for either reranking or training augmentation.
    Relied on in Section 4.2 and in the translation-based QGPR variants of Section 4.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multilingual Open QA on the MIA Shared Task." pith.science (2026). https://pith.science/paper/H6AQJSGA

@misc{pith2026250104153,
  author       = {Pith},
  title        = {Pith review of: Multilingual Open QA on the MIA Shared Task},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H6AQJSGA}},
  note         = {Machine review of arXiv:2501.04153}
}
read the original abstract

Cross-lingual information retrieval (CLIR) ~\cite{shi2021cross, asai2021one, jiang2020cross} for example, can find relevant text in any language such as English(high resource) or Telugu (low resource) even when the query is posed in a different, possibly low-resource, language. In this work, we aim to develop useful CLIR models for this constrained, yet important, setting where we do not require any kind of additional supervision or labelled data for retrieval task and hence can work effectively for low-resource languages. \par We propose a simple and effective re-ranking method for improving passage retrieval in open question answering. The re-ranker re-scores retrieved passages with a zero-shot multilingual question generation model, which is a pre-trained language model, to compute the probability of the input question in the target language conditioned on a retrieved passage, which can be possibly in a different language. We evaluate our method in a completely zero shot setting and doesn't require any training. Thus the main advantage of our method is that our approach can be used to re-rank results obtained by any sparse retrieval methods like BM-25. This eliminates the need for obtaining expensive labelled corpus required for the retrieval tasks and hence can be used for low resource languages.

Figures

Figures reproduced from arXiv: 2501.04153 by the authors.

Figure 1
Figure 1. Block Diagram of our model architecture. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 3 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. https://doi.org/10.18653/v1/2020.acl-main.421 On the cross-lingual transferability of monolingual representations . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623--4637, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.421 2020

  4. [5]

    Akari Asai, Jungo Kasai, Jonathan H Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2020. Xor qa: Cross-lingual open-retrieval question answering. arXiv preprint arXiv:2010.11856

    arXiv 2020

  5. [6]

    Akari Asai, Xinyan Yu, Jungo Kasai, and Hanna Hajishirzi. 2021 b . One question answering model for many languages with cross-lingual dense passage retrieval. Advances in Neural Information Processing Systems, 34

    work page 2021

  6. [7]

    Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454--470

    2020

  7. [8]

    Zhuolin Jiang, Amro El-Jaroudi, William Hartmann, Damianos Karakos, and Lingjun Zhao. 2020. Cross-lingual information retrieval with bert. arXiv preprint arXiv:2004.13005

    arXiv 2020

  8. [10]

    Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020 b . Dense passage retrieval for open-domain question answering. ArXiv, abs/2004.04906

    arXiv 2020

Show all 18 references
  1. [11]

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for C...

    2019

  2. [12]

    Kyungjae Lee, Kyoungho Yoon, Sunghyun Park, and Seung-won Hwang. 2018. https://aclanthology.org/L18-1437 Semi-supervised training data generation for multilingual question answering . In Proceedings of the Eleventh International Conference on Language Resources and Evaluation ...

    2018

  3. [13]

    Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. https://doi.org/10.48550/ARXIV.1910.07475 Mlqa: Evaluating cross-lingual extractive question answering

    2019 doi arXiv

  4. [14]

    Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019. http://arxiv.org/abs/1909.07005 Korquad1.0: Korean QA dataset for machine reading comprehension . CoRR, abs/1909.07005

    2019 arXiv

  5. [15]

    Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. http://arxiv.org/abs/2001.08210 Multilingual denoising pre-training for neural machine translation

    2020 arXiv

  6. [16]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. http://arxiv.org/abs/1606.05250 Squad: 100, 000+ questions for machine comprehension of text . CoRR, abs/1606.05250

    2016 arXiv

  7. [17]

    Stephen Robertson and Hugo Zaragoza. 2009. https://doi.org/10.1561/1500000019 The probabilistic relevance framework: Bm25 and beyond . Foundations and Trends® in Information Retrieval, 3(4):333--389

    2009 doi

  8. [18]

    Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen tau Yih, Jo \"e lle Pineau, and Luke Zettlemoyer. 2022. Improving passage retrieval with zero-shot question generation. ArXiv, abs/2204.07496

    2022 arXiv

  9. [19]

    Peng Shi, Rui Zhang, He Bai, and Jimmy Lin. 2021. Cross-lingual training of dense retrievers for document retrieval. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 251--253

    2021

  10. [20]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934

    2020 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.