{"id":"e7c1044e-5aad-45b3-a44e-25c6e7bcf56c","arxiv_id":"2507.16011","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"mRAKL reformulates multilingual knowledge graph completion as question answering and shows that retrieving context from Wikipedia improves tail-entity prediction for Tigrinya and Amharic, with gains up to 8.79 points in an idealized retrieval setting.","lead":"This paper builds a retrieval-augmented question-answering system that fills in missing facts in knowledge graphs for the low-resource languages Tigrinya and Amharic. A smart generalist might read it because it shows how Wikipedia text in better-resourced languages like Arabic and English can help complete knowledge graphs in languages with very little structured data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline accuracy gains (4.92/8.79) come from an oracle retriever that inserts the gold tail into the context; the real BM25 pipeline yields roughly 1.2/2.2 point gains, and the Amharic number uses a weaker baseline.","rationale":"The central claim of the paper is that RAG improves mKGC for low-resource languages. The BM25 results in Table 5 do provide evidence for this, but the marquee numbers in the abstract and conclusion come from an oracle retriever that searches for the gold tail entity, making the task extractive with the answer present. This is a real weakness because the headline improvements do not reflect the deployed retrieval pipeline. The paper is transparent in §3.1 about the oracle nature, and the abstract qualifies the numbers as 'with an idealized retrieval system,' but the conclusion and the overall-finding sentence in §4 drop that qualifier, which is misleading. The baseline inconsistency for Amharic is a separate concrete error: using KGT5-No-Context (32.58) instead of the best prior KGT5-Description (32.91) inflates the gain from 8.46 to 8.79. Neither issue invalidates the core direction, but they must be fixed before the numbers are taken at face value. The reader's CONDITIONAL verdict already captures this; I agree with the reader's weakest assumption.","tokens_in":23311,"tokens_out":4828,"duration_ms":42039,"concrete_test":"Recompute the reported percentage-point gains in §4.3 using (a) the best KGT5 baseline for each language (KGT5-No-Context for Tigrinya, KGT5-Description for Amharic) and (b) the BM25 retriever results from Table 5 instead of the (im)perfect retriever. If the Amharic gain falls from 8.79 to 8.46 and the real-pipeline gains are about 1.2 and 2.2 points, the abstract and conclusion overstate the benefit of the deployed system.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim, 'mRAKL improves accuracy by 4.92 and 8.79 percentage points for Tigrinya and Amharic' (Abstract, and restated unqualified in the Conclusion), is computed with the '(im)perfect retriever' described in §3.1: for each test triple, the retriever searches the head entity's Wikipedia article for the gold tail entity and returns up to two sentences containing it. This leaks the answer into the retrieved context, so the generator is evaluated under an oracle that is not available at deployment. The actual deployed retriever, BM25 (§4.1), produces much smaller improvements over no-context: from Table 5, Tigrinya H@1 goes 12.97 to 14.15 (+1.18 points) and Amharic goes 33.12 to 35.27 (+2.15 points). The abstract does label these as 'with an idealized retrieval system,' but the Conclusion and Section 4's 'over a 4.9 percentage point increase' drop the qualifier. Additionally, the Amharic 8.79 point gain is computed against KGT5-No-Context (32.58) rather than the best prior baseline KGT5-Description (32.91); the correct difference against the best prior method is 41.37 minus 32.91, or 8.46 points, overstating the claim by 0.33 points. These issues together inflate the central claim: the real system's improvement is small, and the headline numbers are tied to an oracle.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes mRAKL, a retrieval-augmented generation system that reformulates multilingual knowledge graph construction (mKGC) as question answering. The authors construct KGs for Tigrinya (3.5k triples) and Amharic (34k triples) from Wikidata, verbalize triples into question–answer pairs using manually prepared templates in four languages, and fine-tune AfriTeVa with LoRA as the generator. They evaluate BM25 and LaBSE retrievers against a no-context setting and also an '(im)perfect retriever' that selects context by searching the head entity's Wikipedia article for the gold tail entity. The paper reports that with this idealized retriever, mRAKL improves H@1 by 4.92 and 8.79 percentage points over prior baselines for Tigrinya and Amharic, and that BM25 retrieval improves over no-context. It also presents cross-lingual link prediction results and a bias-focused analysis of the constructed data.","tokens_in":23562,"tokens_out":8178,"duration_ms":73726,"significance":"The paper contributes new resources (Tigrinya and Amharic KGs, question templates, and a trained retriever/generator pipeline) and a sensible reformulation of mKGC as QA, which could be useful for other low-resourced languages. The result that BM25-based RAG improves over no-context—though modest—plus the analysis of cross-lingual context transfer are valuable. The method is described in enough detail to be reproduced in principle, with hyperparameters and retrieval settings given, though code and data are not yet released. However, the central accuracy claims are tied to an oracle retriever that leaks the answer into the retrieved context; the real deployed pipeline's gains are much smaller. These claims need to be reframed and the arithmetic corrected before publication.","major_comments":[{"comment":"The headline improvements of 4.92 and 8.79 percentage points are computed with the '(im)perfect retriever' of §3.1, which selects context by searching the head entity's Wikipedia article for the gold tail entity and keeping up to two sentences that contain it. This leaks the answer into the retrieved context, so these numbers describe an oracle upper bound rather than the deployed system. With the actual BM25 retriever, Table 5 shows gains of only 1.18 points for Tigrinya (H@1 from 12.97 to 14.15) and 2.15 points for Amharic (H@1 from 33.12 to 35.27). The Abstract states these gains as the headline result (even with the qualifier 'with an idealized retrieval system'), and the Conclusion restates them without any qualifier. I recommend reporting the BM25-based gains as the primary accuracy result and presenting the (im)perfect-retriever numbers strictly as an upper-bound ablation in all summaries, including the Abstract and Conclusion.","section":"Abstract; §3.1; §4.3; Table 5"},{"comment":"The 8.79-point Amharic gain is computed as 41.37 − 32.58, i.e., against KGT5-No-Context, not against the best prior baseline. The best KGT5 variant for Amharic is KGT5-Description with H@1=32.91 (Table 4), so under the paper's own definition in footnote 11 ('best-performing methods for each work'), the correct gain is 8.46 points. This 0.33-point overstatement propagates to the Abstract and Conclusion and should be corrected.","section":"§4.3, Table 4, footnote 11"},{"comment":"All results are reported from a single inference run (§4), while the key BM25-versus-no-context differences in Table 5 are small (1.18 and 2.15 points for Tigrinya and Amharic H@1). Without multiple seeds, confidence intervals, or significance tests, these differences may not be stable. At minimum the paper should acknowledge this uncertainty, and ideally report variance across runs or a significance test (e.g., a paired bootstrap) to support the claim that BM25 retrieval improves over no-context.","section":"§4.1, §4.4, Table 5"}],"minor_comments":[{"comment":"The sentence 'In this section, we provide figures and graphs to support the results reported in Sec.' contains an empty cross-reference; it should point to the relevant results section (e.g., §4.4 or §4.5).","section":"Appendix A.2"},{"comment":"The caption of Table 8 spells 'Hyperparameters' as 'Hyperparamters', and §B.1 contains the typo 'teh training data point' for 'the training data point'.","section":"Table 8 caption; §B.1"},{"comment":"The set notation '∀ tϵ{T igrinya, Amharic, English, Arabic}' has a stray space in 'T igrinya' and should use the proper membership symbol '∈'; the same issue appears in the second occurrence of this formula.","section":"§3.2 input representation"},{"comment":"The model name is spelled 'AfriTeVa' in most of the paper but 'AfriTeva' in Table 3 and in the surrounding text of §4.2; please unify the spelling.","section":"Table 3 and throughout"},{"comment":"The reference to Vrandečić and Krötzsch (2014) appears twice with slightly different formatting; the duplicate should be removed.","section":"References"},{"comment":"The term 'Afro-Semitic languages' is not standard; consider 'Afro-Asiatic (Semitic) languages' or simply 'Semitic languages'.","section":"§1 Languages of Study"}],"recommendation":"major_revision","confidential_remarks":"The paper's core contribution is modest but real: a useful dataset and a reasonable QA-based RAG formulation for low-resourced mKGC. The main problem is that the headline accuracy gains are produced by an oracle retriever that uses the gold tail to select context, and the paper does not consistently distinguish this from the deployed BM25 pipeline. The Amharic baseline comparison is also inconsistent with the stated best-baseline rule. These issues are fixable with rewriting and reanalysis, so I recommend major revision rather than rejection. I would also note that the paper promises code/data release 'soon'; for a resources contribution, availability of the data and templates at publication time is important."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a dataset-and-pipeline paper with a reporting problem. The new Tigrinya and Amharic KGs (3.5k and 34k triples) plus the question templates are genuinely useful, and framing mKGC as QA with retrieval is a reasonable way around the lack of structured data. The BM25 results, though modest (roughly +1.2 and +2.2 H@1 points), do support the central claim that retrieving from monolingual text helps over no-context.\n\nWhat the paper does well: it is transparent about the (im)perfect retriever being an upper bound, at least in Section 3.1. The cross-lingual setup is thoughtful, and the analysis of when Arabic helps versus English is a nice touch.\n\nWhere it gets soft: the headline numbers in the abstract and conclusion (4.92 and 8.79 percentage points) come from that oracle retriever, which searches the head entity's Wikipedia article for the gold tail and keeps sentences containing it. That leaks the answer into the context. The abstract does say \"idealized retrieval system,\" but the conclusion drops the qualifier, and the numbers read as the main result. The real BM25 gains are about a quarter of that. I also noticed the Amharic gain is computed against KGT5-No-Context rather than the stronger KGT5-Description baseline; the difference is 8.46 instead of 8.79, not huge but sloppy. Single runs with no error bars and no released code or data at submission time make it hard to rely on the exact figures.\n\nNone of this kills the paper. The datasets alone are a contribution, and the approach is sound. But the authors need to separate oracle numbers from deployed-pipeline numbers everywhere, fix the baseline comparison, report variance or multiple runs, and release the data and templates. Recommendation: send to peer review, but insist on those changes before acceptance.","headline":"Useful new datasets and a sensible RAG pipeline for low-resource KGC, but the headline gains come from an oracle retriever and the real BM25 wins are small.","tokens_in":24202,"tokens_out":1624,"would_cite":true,"duration_ms":17342,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that knowledge-graph completion for low-resourced languages can be reframed as retrieval-augmented question answering, so missing tail entities are read from Wikipedia context rather than recalled from scarce structured…","keywords":["retrieval-augmented generation","multilingual knowledge graph construction","low-resourced languages","Tigrinya","Amharic","cross-lingual link prediction","question answering","multilingual transfer"],"falsifier":"Run mRAKL's trained generator on the test split using contexts produced by a retriever that never sees the gold tail — BM25 as in the paper, or a dense retriever that covers Tigrinya — and score each retrieved context for whether it mentions the true tail. If H@1 conditioned on the tail actually appearing in context stays far below the 59-78% the (im)perfect retriever reaches, and if overall gains remain near the observed 1-2 point BM25 level, the central accuracy claim would hold only for the oracle setup rather than the deployed pipeline.","tokens_in":23044,"feed_emoji":"📚","tokens_out":19038,"duration_ms":168161,"temperature":0.7,"pith_summary":"mRAKL argues that knowledge-graph construction (KGC) for low-resourced languages should be reframed as question answering: each (head, relation, tail) triple becomes a template question, and a generator predicts the tail using Wikipedia context retrieved at inference time. The paper contributes new KGs and QA datasets for Tigrinya (3.5k triples) and Amharic (34k triples), with Arabic and English as transfer languages, and motivates the design by showing that the structured context prior methods rely on — entity descriptions and one-hop neighbors — is rarely available for these languages and contains the answer less than 2% of the time. Its headline result is that the context-augmented setup beats the KGT5 sequence-to-sequence baselines by 4.92 percentage points on Tigrinya and 8.79 on Amharic, measured as the share of test triples whose single top prediction equals the true tail. Those numbers, however, come from an idealized '(im)perfect' retriever that finds the gold tail inside the head entity's Wikipedia article; the deployed BM25 retriever improves over the no-context setting by only about 1.2 and 2.2 points, so how much of the gain survives real retrieval remains an open question.","feed_headline":"Retrieval-augmented QA lifts KG completion for Tigrinya and Amharic","feed_subtitle":"Unstructured Wikipedia text supplies missing facts where structured KG data is scarce.","key_machinery":"The load-bearing mechanism is the language-tagged QA input sequence. Every triple is converted into a template question Q (head entity and relation) with the tail as answer, the retriever supplies up to two context sentences C, and the generator input is [C-LANt]C | [Q-LANt]Q? [A-LANt], where LANt is the ISO language code, '?' ends the question, '|' ends the context, and the final [A-LAN] token selects the output language. This one format lets a LoRA-finetuned AfriTeVa generator train across all four languages at once and makes the answer language a decodable choice, which is what converts cross-lingual retrieval into implicit entity alignment: an Arabic context paired with a Tigrinya question and answer forces the model to treat the Arabic phrase and the Tigrinya label as the same entity. The '(im)perfect' retriever — a heuristic that searches the head entity's Wikipedia lead for the gold tail and keeps at most two sentences containing it — generates the training contexts and doubles as an upper bound on what a deployed retriever could achieve.","core_discovery":"The paper's central claim is that the binding constraint for multilingual KGC in low-resourced languages is context, not model capacity: for Tigrinya and Amharic, structured contextual data covers at most half of the triples (49.8% description coverage for Tigrinya, 6.3% for Amharic) and contains the tail entity under 2% of the time, whereas unstructured Wikipedia text in the target or transfer languages frequently does contain the answer. mRAKL exploits this by converting each triple to a question-answer pair, tagging context and question with language tokens, and fine-tuning one AfriTeVa sequence-to-sequence model so a single generator serves Tigrinya, Amharic, Arabic, and English and can be told which language to answer in. The experiments show that context beats no context, multilingual context beats monolingual (a 4.69-point gain for Tigrinya), and cross-lingual retrieval acts as implicit entity alignment: when only the Arabic context contains the tail, the model still returns the correct Amharic or Tigrinya label. Against prior work, adding context flips the comparison — KGT5 beats mRAKL by 1.78 and 3.43 points in the no-context setting, but mRAKL beats KGT5 and its structured-context variants by 4.92 and 8.79 points once Wikipedia context is supplied.","pith_inferences":["The gap between the (im)perfect retriever's 59-78% H@1 and BM25's 14.15% (Tigrinya) and 35.27% (Amharic) suggests retrieval quality is the highest-leverage investment for these languages; the paper's upper-bound framing implies this, though the paper never states it as a conclusion.","A diagnostic the paper does not run: compute the tail-mention rate of each retriever's contexts and plot H@1 conditional on whether the context contains the tail. If tail-mention rate drives accuracy, retrieval recall becomes a cheap proxy metric for mKGC in any low-resourced language where end-to-end evaluation is expensive.","The zero-shot results — at most 5.83% H@1 on Amharic and 2.23% on Tigrinya even for GPT-4 — imply that parametric memory is not a viable channel for these languages; by extension, grounding through retrieval may be the default strategy for any fact-oriented task in low-resourced languages, not just knowledge-graph completion."],"forward_implications":["Knowledge-graph completion for low-resourced languages can be driven by unlabeled Wikipedia text: retrieval supplies the fact the model's parameters do not contain, replacing the large structured training sets prior mKGC work requires.","Cross-lingual entity alignment emerges implicitly from multilingual context: the generator answers in the target language even when only a transfer-language context carries the tail, so explicit alignment supervision is not needed.","The released resources — the 3.5k-triple Tigrinya KG, the 34k-triple Amharic KG, and question templates in four languages — give future work a benchmark for mKGC on these languages.","Transfer languages contribute complementary coverage: Arabic context supports region-specific and Middle East/Asia queries that English context misses and vice versa, so transfer-language choice should reflect cultural coverage, not just resource abundance.","An idealized retriever lifts H@1 to roughly 59-78% on queries where it finds context, indicating that the deployed retrievers, not the generator, are the limiting component of the pipeline."],"supporting_citations":[{"why":"Establishes the dominant mKGC paradigm — pretraining on very large structured triple datasets (52M triples) — which the paper argues is infeasible for low-resourced languages, motivating the shift to unstructured retrieval.","marker":"Zhou et al. (2022)"},{"why":"KGT5, the sequence-to-sequence KG-completion baseline whose no-context and context settings mRAKL is compared against; the 4.92 and 8.79 point claims are measured against this line of work.","marker":"Saxena et al. (2022a)"},{"why":"KGT5-context, the description- and one-hop-augmented variant that the paper shows degrades on low-resourced languages because structured context rarely contains the tail.","marker":"Kochsiek et al. (2023)"},{"why":"AfriTeVa, the sequence-to-sequence model that mRAKL fine-tunes with LoRA as its generator base.","marker":"Jude Ogundepo et al. (2022)"},{"why":"LaBSE, the dense sentence retriever used in the deployed pipeline and fine-tuned with contrastive loss on (im)perfect-retriever contexts.","marker":"Feng et al. (2022)"},{"why":"BM25, the lexical retriever whose real-world contexts produce the paper's non-oracle accuracy gains.","marker":"Robertson and Zaragoza (2009)"},{"why":"Wikidata, the source of the Tigrinya and Amharic KGs, the 120 relations, and the cross-lingual entity labels that the whole QA reformulation depends on.","marker":"Vrandečić and Krötzsch (2014)"},{"why":"Supplies the premise that monolingual unstructured data is far more available than labeled structured data for low-resourced languages, the justification for the RAG design.","marker":"Joshi et al. (2020)"}],"fun_headline_variants":["Context beats capacity in multilingual KG completion","Retrieval over Wikipedia lifts KG completion in Tigrinya and Amharic","mRAKL: Unstructured text fills missing knowledge graph links","For low-resource KGC, context matters more than model size","RAG with Wikipedia context lifts Tigrinya and Amharic KG accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline accuracy gains come from an '(im)perfect' retriever that already knows the gold tail: it searches the head entity's own Wikipedia article for the tail and returns sentences containing it, so the context leaks the answer, while the deployed BM25 retriever, which must find the answer without being told it, produces far smaller gains.","fun_headline_variants_meta":{"raw":{"variants":["Context beats capacity in multilingual KG completion","Retrieval over Wikipedia lifts KG completion in Tigrinya and Amharic","mRAKL: Unstructured text fills missing knowledge graph links","For low-resource KGC, context matters more than model size","RAG with Wikipedia context lifts Tigrinya and Amharic KG accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1734,"prompt_tokens":1023,"completion_tokens":711,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":622}},"tokens_in":639,"tokens_out":711,"duration_ms":6891,"temperature":1.0,"reasoning_tokens":622,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:19:37.008231+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run mRAKL's trained generator on the test split using contexts produced by a retriever that never sees the gold tail — BM25 as in the paper, or a dense retriever that covers Tigrinya — and score each retrieved context for whether it mentions the true tail. If H@1 conditioned on the tail actually appearing in context stays far below the 59-78% the (im)perfect retriever reaches, and if overall gains remain near the observed 1-2 point BM25 level, the central accuracy claim would hold only for the oracle setup rather than the deployed pipeline.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the dominant mKGC paradigm — pretraining on very large structured triple datasets (52M triples) — which the paper argues is infeasible for low-resourced languages, motivating the shift to unstructured retrieval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AfriTeVa, the sequence-to-sequence model that mRAKL fine-tunes with LoRA as its generator base."}],"review_version":1}