{"id":"ec06108d-3873-47d9-b5e6-a9414513c0e4","arxiv_id":"2412.14304","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new seven-language ophthalmology benchmark shows LLMs are less accurate in LMIC languages, and an agentic translation-plus-RAG pipeline reduces the gap.","lead":"This paper builds a 1,184-question ophthalmology exam translated into seven languages and tests six AI models, finding they answer much worse in Filipino, Hindi, and Mandarin than in English. It then offers a multi-step AI pipeline, CLARA, that raises accuracy in every language and shrinks the language gap.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's central cross-lingual comparison is unvalidated: translation equivalence is asserted, not demonstrated, so the reported language gaps and CLARA's gap reductions could be artifacts of translation difficulty rather than model bias.","rationale":"Reader's weakest assumption identified translation equivalence; I agree. I considered alternatives: missing significance tests and underspecified CLARA weights are real but secondary—the main LMIC gaps (11–13 points) are large enough to survive reasonable binomial error, and missing code affects reproducibility, not the validity of the central comparison. The more foundational issue is that every cross-lingual claim is a comparison of translations, not of the same questions in different languages, unless translation quality is controlled. The paper asserts expert review but gives no quantitative equivalence evidence, and its own qualitative examples show translation-sensitive ambiguities. A back-translation and equivalence-restricted re-analysis directly tests whether the observed bias and debiasing effects are properties of models or of the benchmark's translated items. My read does not change the reader's CONDITIONAL verdict: the resource is promising, but the headline quantitative claims should not be accepted without this validation and the other stated conditions (data/code release, uncertainty).","tokens_in":14167,"tokens_out":4292,"duration_ms":42738,"concrete_test":"Release a stratified sample of 50 items per language (350 items) from the benchmark. Have two independent professional translators, blinded to the original Portuguese, back-translate each non-English version to Portuguese or English; have board-certified ophthalmologists rate semantic equivalence and difficulty on a 3-point scale and compute inter-rater agreement (Cohen's kappa). Then re-run the GPT-4 direct-inference and CLARA conditions from Table 3 restricted to items where both back-translations are judged equivalent. If the Filipino-English gap and CLARA's gain shrink materially on that subset, the translation-equivalence concern lands; if they persist unchanged, the cross-lingual bias claim is robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—LLMs are biased across languages and CLARA reduces the gap—rests on the paired translations in Multi-OphthaLingua. The benchmark section states that questions were 'originally written in Portuguese' and 'manually translated' into six languages, then 'reviewed and curated by board-certified native-speaker ophthalmologists.' It does not report back-translation, inter-annotator agreement, or per-language difficulty calibration. That is the load-bearing assumption. If, for example, the Filipino version is more literal or uses ambiguous terms (the paper itself notes 'namamaga' can mean either 'swollen' or 'inflamed' in Filipino and 'mancha' can mean 'spot' or 'stain' in Portuguese), then the observed GPT-4 gap of 11.6 points between English and Filipino, and CLARA's reduction of that gap to 5.1 points, may reflect differences in translation difficulty rather than model language bias. The risk is amplified because CLARA's first step translates every query to English: if low-resource-language versions are systematically degraded translations, CLARA's improvement may simply be repairing translation artifacts, not debiasing the model. The hypothesis is plausible enough to require validation before the benchmark's headline comparisons can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Multi-OphthaLingua, a parallel multilingual ophthalmology multiple-choice benchmark in seven languages (English, Spanish, Portuguese, Filipino, Mandarin, Hindi, French), evaluates six LLMs, and proposes CLARA, an inference-time pipeline that combines translation, weighted RAG, web search, and query rewriting. The main empirical claims are that LLM accuracy is systematically lower in LMIC-representative languages such as Filipino, Hindi, and Mandarin, and that CLARA improves absolute accuracy in all seven languages while narrowing the gap relative to English, with the largest reported effect for GPT-4 in Filipino (51.8% direct to 67.1% with CLARA; gap reduced from 11.6 to 5.1 points, Table 3).","tokens_in":14404,"tokens_out":4466,"duration_ms":43367,"significance":"If the benchmark is valid, it is a useful contribution: it appears to be the first paired multilingual ophthalmology QA dataset, it is expert-curated, a sample is publicly available, and the evaluation covers six models and seven languages. The CLARA system is compared against Translate-COT, Web-ToolCall, and an ablation ladder, so the reported gains are not simply fit to the answer key, and the qualitative observations about ambiguous terms such as 'namamaga' and 'mancha' are informative. However, the central quantitative comparisons currently rest on unvalidated translation equivalence and unreported variance; the significance of the headline claims cannot be assessed until those are addressed.","major_comments":[{"comment":"The cross-lingual comparisons are load-bearing but translation equivalence is asserted, not demonstrated. The text states that questions were 'originally written in Portuguese' and 'manually translated' into six languages, then 'reviewed and curated by board-certified native-speaker ophthalmologists,' but no back-translation, inter-annotator agreement, or per-language difficulty calibration is reported. The paper itself shows that ambiguous terms exist in the benchmark ('namamaga' in Filipino, 'mancha' in Portuguese), so without evidence that all seven versions have equivalent difficulty and meaning, the language accuracy gaps in Tables 2 and 3, and the gap reductions attributed to CLARA, could reflect translation artifacts rather than model bias. Please provide translation-equivalence checks, per-language item statistics, and a protocol for verifying cultural neutrality, or explicitly report these as limitations.","section":"Benchmark Construction"},{"comment":"The quantitative results are reported only as 8-run averages without standard deviations, confidence intervals, or significance tests. Claims such as 'significant disparities' and 'significantly reduces the multilingual bias gap' are therefore not statistically supported. Additionally, no item counts are given for the per-language or per-subgroup cells; with 1184 questions split across at least ten subgroups and seven languages, some cells are likely small, and accuracy values such as 65.9% versus 70.5% in the Basic-Sciences rows may not be distinguishable. Please report per-cell sample sizes, variance across runs, and appropriate tests or confidence intervals for the gap comparisons.","section":"Benchmark Results and LLM-Failure Analysis, Tables 2 and 3"},{"comment":"The method description leaves free parameters unspecified: the query-part weights w_j in Eq. (1), the jargon weights w_k in Eq. (2), the maximum iteration limit (given as 5), and the evaluation thresholds for translation certainty and retrieval relevance. It is not stated how these weights are set, whether they were tuned on the benchmark, or whether they are fixed ahead of time. This affects reproducibility and raises the risk of optimistic results if any component was selected on the test set. Please specify the parameter-setting procedure and clarify whether any development data or validation split was used.","section":"CLARA, Eqs. (1) and (2)"}],"minor_comments":[{"comment":"The Hindi column is labeled 'HI' in Table 2 and 'HIN' in Table 3; please unify the notation.","section":"Tables 2 and 3"},{"comment":"The text states that model configuration uses temperature = 0, while Table 2 reports results averaged over 8 runs; please clarify whether the runs varied and report the observed variance, since deterministic decoding would make averaging redundant.","section":"Experimental Setup"},{"comment":"There is a typo in the abstract: 'we propose CLARA' should be capitalized consistently as 'We propose CLARA'.","section":"Abstract"},{"comment":"The sentence 'comprised of 1184 questions across English, Spanish, Filipino, Portuguese, Mandarin, French, and Hindi' is ambiguous: it should state whether 1184 is the number of unique question stems with translations per language or the total number of items across all languages.","section":"Benchmark Construction"},{"comment":"The paper claims questions were 'carefully crafted to ensure question neutrality across regions,' but no procedure for verifying regional neutrality is described; please provide the protocol or soften the claim.","section":"Benchmark Construction"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for an NLP and health applications venue, and the overall idea is timely. The main risk is not methodological circularity but insufficient validation of the language-equivalence assumption and the lack of statistical reporting. I would support a major revision that adds translation validation, per-language difficulty checks, item counts, and variance/significance measures, and that clarifies the CLARA parameter-setting procedure. I do not see grounds for rejection if these issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper delivers a real resource: the first paired seven-language ophthalmology QA benchmark, letting you compare the same questions across English, Spanish, Portuguese, Filipino, Mandarin, Hindi, and French. That pairing is genuinely new, and it is the main reason to read the paper. The evaluation is broad—six models, seven languages, subgroup breakdowns—and the qualitative analysis gives concrete examples of how models stumble on ambiguity (namamaga, mancha). The CLARA pipeline is a reasonable assembly of existing ideas (pre-translation, RAG, web search, self-verification), and the ablation study gives credit where it is due: each component adds a small but consistent gain. The core finding, that LLMs perform worse in LMIC languages, is consistent across all six models and matches prior multilingual work, so I trust the direction even if the exact numbers are soft.\n\nThe soft spots are real but not fatal. The load-bearing assumption is that the seven translations are equivalent in difficulty and meaning. The questions were originally in Portuguese and manually translated, but the paper reports no back-translation, no inter-annotator agreement, and no per-language difficulty calibration. This matters because CLARA first translates everything to English; if some language versions are systematically more awkward or ambiguous, part of the observed bias and part of CLARA's improvement could be translation repair rather than genuine debiasing. The paper itself supplies examples of ambiguous terms, so this is not a hypothetical worry. The second issue is statistical: results are reported as 8-run averages without standard deviations or significance tests, and the per-subgroup cells (basic vs. surgery) are small enough that some gaps could be noise. Third, CLARA's weights, thresholds, and iteration limits are described but not fully specified, and only a sample of data is public—the full release on Physionet is promised but not available. These are fixable issues.\n\nWho is this for? Anyone working on multilingual medical QA, fairness in clinical NLP, or benchmark design for low-resource languages. It deserves a serious referee. The resource is valuable, the method is sensible, and the central conclusion is likely correct. The revisions I would ask for are: validate translation equivalence (back-translation plus agreement or difficulty checks), report uncertainty, and release the full code and data. With those in place, the benchmark will be a solid contribution.","headline":"A genuinely new paired multilingual ophthalmology benchmark and a sensible debiasing pipeline, but the translation-equivalence assumption is unvalidated, so the exact gaps and CLARA's gains are provisional.","tokens_in":15019,"tokens_out":2102,"would_cite":true,"duration_ms":20728,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM eye-care answers are systematically worse in LMIC languages, and a new agentic pipeline narrows the gap.","keywords":["multilingual benchmark","ophthalmology","large language models","cross-lingual bias","low- and middle-income countries","retrieval-augmented generation","question answering","debiasing"],"falsifier":"Have independent professional translators back-translate all 1,184 questions from each language and have ophthalmologists rate equivalence of meaning, difficulty, and cultural neutrality. If the languages with the largest measured gaps (Filipino, Hindi, Mandarin) also show the largest translation-equivalence problems, the central bias claim would lose support; if equivalence is high, the claim is strengthened.","tokens_in":13973,"feed_emoji":"🩺","tokens_out":10242,"duration_ms":82878,"temperature":0.7,"pith_summary":"Large language models answer ophthalmology multiple-choice questions less accurately in Filipino, Hindi, and Mandarin than in English, with the gap reaching 11.6 to 20.1 percentage points depending on model and language. This paper introduces Multi-OphthaLingua, a benchmark of 1,184 ophthalmology questions translated and expert-reviewed in parallel across seven languages, which allows those gaps to be measured against identical content. It then proposes CLARA, an inference-time pipeline that translates questions, retrieves and critiques medical context, searches the web when retrieval fails, and rewrites complex cases. The paper reports that CLARA raises accuracy in all seven languages and narrows the Filipino-English gap; GPT-4 in Filipino goes from 51.8% to 67.1%, and the gap shrinks from 11.6 to 5.1 percentage points. If the results hold, they would show that language bias in clinical question answering is systematic but partly correctable without retraining.","feed_headline":"LLM eye-care answers lose accuracy in LMIC languages","feed_subtitle":"Seven-language benchmark shows the gap; CLARA pipeline cuts it while raising scores everywhere","key_machinery":"CLARA (Cross-Lingual Reflective Agentic system) is the load-bearing mechanism, a multi-agent inference-time pipeline. A translation agent converts the query to English; an evaluation agent judges translation quality and the model's certainty; a knowledge agent runs weighted retrieval over biomedical abstracts, medical textbooks, and general encyclopedic articles, reweighting relevance scores by confidence in each part of the question and by expanded ophthalmology jargon; a second evaluation agent critiques retrieved documents and can trigger a web search; and a rewriting agent decomposes complex queries before another pass. The pipeline is the mechanism that turns the paper's diagnosis of language bias into a correction.","core_discovery":"The paper's central discovery is that cross-lingual bias in ophthalmological question answering is systematic: every model evaluated scores lower in Filipino, Hindi, and Mandarin than in English, and the worst gaps appear in precisely the languages most relevant to low- and middle-income countries. The authors trace the failures to three causes—limited language proficiency, shallow ophthalmology-specific knowledge, and difficulty with linguistic nuance and cultural context—and document that existing fixes (English chain-of-thought, pre-translation, web search, plain RAG) are inconsistent. CLARA, the proposed system, combines translation, weighted corrective retrieval, web search, iterative relevance verification, and query rewriting; it improves accuracy in every tested language and cuts the largest LMIC gaps roughly in half. The paired question design is what makes these comparisons interpretable, since the same question content is evaluated across languages.","pith_inferences":["I would predict that most of CLARA's benefit comes from the translation step plus English-centric retrieval; isolating that would require an oracle condition in which the model receives perfect English translations with no retrieval.","The same paired-question methodology could be extended to other specialties with high LMIC burden, such as obstetrics or tropical medicine, to expose analogous language gaps.","Because the paper documents expert review but no back-translation or inter-annotator equivalence scoring, some portion of the measured gap may reflect translation artifacts; a human equivalence audit of the 1,184 items would settle that.","It remains untested whether CLARA's gains survive domain shift to new questions, other dialects, or spoken-language triage; running it on an independent multilingual medical exam set would be the next check."],"forward_implications":["Deploying LLMs for ophthalmology triage, patient education, or documentation in Filipino, Hindi, or Mandarin without debiasing would systematically under-serve speakers of those languages relative to English speakers.","CLARA's accuracy gains of roughly 10–15 percentage points in lower-resourced languages can be achieved at inference time, without any fine-tuning, making equitable multilingual deployment more feasible.","None of the standard single-component fixes—translation, chain-of-thought, web search, or basic RAG—closes the gap on its own; the ablation shows each CLARA component adds a small but consistent gain.","The released paired benchmark gives other groups a reusable protocol for auditing new models for cross-lingual medical bias rather than relying on English-only evaluations."],"supporting_citations":[{"why":"Provides MedQA, the English and Mandarin medical exam benchmark that Multi-OphthaLingua contrasts with in Table 1.","marker":"Jin et al. 2021"},{"why":"Documents early challenges of LLM integration in Philippine ophthalmology, motivating the inclusion of Filipino as an LMIC language.","marker":"Dychiao et al. 2024"},{"why":"Defines retrieval-augmented generation, the basic RAG baseline and the technique CLARA builds on.","marker":"Lewis et al. 2020"},{"why":"Supplies the EN-COT cross-lingual evaluation baseline and motivates the translation-based comparison.","marker":"Ruder et al. 2021"},{"why":"Examines direct inference versus pre-translation in multilingual LLM applications, the baseline CLARA's translation agent extends.","marker":"Intrator et al. 2024"},{"why":"Cited for Translate-COT, the pre-translation reasoning baseline that CLARA's pipeline refines.","marker":"Shi et al. 2022"},{"why":"Self-RAG, which contributes the retrieve-critique-and-revise loop reused in CLARA's iterative evaluation agent.","marker":"Asai et al. 2023"},{"why":"MedCPT, the medical-domain retriever used to encode and retrieve from the RAG corpora in CLARA.","marker":"Jin et al. 2023"}],"fun_headline_variants":["Benchmark reveals LLM eye-care bias across languages","CLARA method slashes language gap in ophthalmology QA","Multilingual eye-QA: LLMs underperform in LMIC tongues","New benchmark exposes cross-lingual bias in eye-care LLMs","CLARA reduces LLM bias in ophthalmology for global use"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's cross-lingual comparisons assume the manually translated questions preserve meaning, difficulty, and cultural neutrality across all seven languages.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark reveals LLM eye-care bias across languages","CLARA method slashes language gap in ophthalmology QA","Multilingual eye-QA: LLMs underperform in LMIC tongues","New benchmark exposes cross-lingual bias in eye-care LLMs","CLARA reduces LLM bias in ophthalmology for global use"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1245,"prompt_tokens":959,"completion_tokens":286,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":200}},"tokens_in":575,"tokens_out":286,"duration_ms":3179,"temperature":1.0,"reasoning_tokens":200,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:21:07.469635+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent professional translators back-translate all 1,184 questions from each language and have ophthalmologists rate equivalence of meaning, difficulty, and cultural neutrality. If the languages with the largest measured gaps (Filipino, Hindi, Mandarin) also show the largest translation-equivalence problems, the central bias claim would lose support; if equivalence is high, the claim is strengthened.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents early challenges of LLM integration in Philippine ophthalmology, motivating the inclusion of Filipino as an LMIC language."},{"cited_title":"C.; Yeganova, L.; Wilbur, W","cited_arxiv_id":null,"evidence_quote":"MedCPT, the medical-domain retriever used to encode and retrieve from the RAG corpora in CLARA."}],"review_version":1}