{"id":"b9837093-b153-4691-b8d7-99fd16199ca4","arxiv_id":"2608.11002","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multilingual text-to-image benchmark shows current models are far less accurate in non-English languages and that prompt language systematically shifts culture, style, and demographic bias in generated images.","lead":"This paper introduces LingT2I, a 33,000-prompt benchmark that tests text-to-image models in ten languages for both image content and text rendering. It finds large gaps between English and other languages, especially for non-Latin scripts, and shows that prompt language itself changes the style, culture, and demographic skew of generated images.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline inequality could be an artifact of the single translation pipeline: all non-English prompts come from Gemini 2.5 Pro, and the 5% human check validates fluency/semantics, not T2I prompt usability.","rationale":"In good faith, the paper does a lot well: it releases a substantial benchmark, uses a multilingual CLIP variant rather than English-only CLIP, includes some human quality control, and reports consistent large gaps across many models. The descriptive pattern that English and Latin-script languages outperform others is likely robust. However, the single most load-bearing assumption is that the translated prompts are equivalent as T2I instructions. The reader identified this as the weakest assumption, and I agree. The 5% human check is helpful but does not fully settle it, because it targets semantic fidelity, cultural appropriateness, and fluency, not whether the translated prompt is as easy for a T2I model to follow as the original English prompt. A translation can be fluent and semantically consistent yet still be translationese that degrades generation quality in lower-resource languages. Since all non-English prompts share one translation model, this is a systematic confound rather than random noise. The proposed test with independent human translations directly targets this confound. If the gap persists under independent translations, the central inequality claim is supported; if it shrinks substantially, the headline needs to be softened. Secondary concerns about OCR and MLLM-judge language bias are real, but they are partly separable from the content-generation results and would be a second stress-test; the translation pipeline is the broader threat to the paper's central claim. The reader's conditional verdict is appropriate, and this concern does not change it, so I recommend UNCHANGED.","tokens_in":33996,"tokens_out":6871,"duration_ms":75117,"concrete_test":"Randomly sample a few hundred English prompts per language (e.g., 300 total per language across both tasks). Have independent native-speaker translators, who do not use the Gemini pipeline, produce translations with the same explicit constraint: preserve all objects, attributes, spatial relations, style cues, and prompt usability. Generate images with the same models (at minimum Qwen-Image and SD3.5) using these independent translations, under the same seed and settings, and recompute the Table 2 CLIPScore values and text-rendering precision. If the mean gap between English and low-resource languages shrinks by more than 0.05 CLIPScore (or the cross-language std drops by more than 25%), the reported inequality is substantially a translation artifact; if scores are statistically indistinguishable from the original Gemini translations, the translation concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that multilingual T2I is linguistically unequal depends on the assumption that the nine translated versions of each English prompt are equivalent not only semantically but also as text-to-image instructions. In Section 3.4 and Appendix B.3, all non-English prompts are produced by Gemini 2.5 Pro, with a random 5% native-speaker check that judged 98% of samples semantically consistent and GPT-5 flagging only 1.3% as problematic. This is meaningful evidence, but semantic consistency and fluency are not the same as prompt usability for image generation. A translation can be grammatical and faithful yet still differ in word order, ambiguity, lexical choice, or the way it structures spatial relations and style cues; these differences can change how a diffusion model parses the prompt, and they plausibly vary by language and by the target language's resource level. Because every model is evaluated on the same machine-translated prompts, any per-language bias in Gemini 2.5 Pro is baked into all 17 models' scores, so the observed inequality, trade-offs, and language-dependent patterns in Section 4 are not independent of the translation step. The paper's Section 5 assertion that the gaps are too large to be explained by these factors is not supported by any quantitative sensitivity analysis, such as re-running a subset with independent human translations. If Gemini produces less 'prompt-like' translations for Hindi, Arabic, or Korean, the headline linguistic inequality would be inflated, and the causal comparisons in Sections 4.3 and 4.4 would be confounded.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces LingT2I, a multilingual text-to-image benchmark covering ten languages with 33K prompts for content generation and text rendering, and evaluates 17 recent models. The authors report three main findings: general-purpose models exhibit linguistic inequality favoring high-resource Indo-European languages; non-Latin writing systems remain a major bottleneck in text rendering; and language-specific cultural and typological factors create trade-offs across evaluation dimensions. They also present causal analyses (transliteration, tokenization, alignment-conditioned bias) and culture/demographic pattern analyses, and they release code and data.","tokens_in":34167,"tokens_out":14274,"duration_ms":118931,"significance":"If the findings hold, LingT2I is a valuable community resource: it is large, covers both content generation and text rendering, uses a multilingual encoder (MetaCLIP2) for CLIPScore, and is accompanied by publicly released code and dataset. The observation that native multilingual architectures such as Qwen-Image achieve better fairness than post-hoc adapters, and the tokenization correlation, are interesting and falsifiable. The descriptive cross-lingual scores are plausible, and the paper is transparent about some limitations. However, the reliability of several specific claims depends on the translation pipeline, the OCR evaluator, and the LLM-based judges, which need further validation before the stronger conclusions can be fully accepted.","major_comments":[{"comment":"The central inequality finding in Content Generation assumes that the nine Gemini 2.5 Pro translations of each English prompt are equivalent as text-to-image instructions. The 5% human sample and the 1.3% GPT-5 flag rate validate semantic consistency and fluency, not prompt usability for T2I, and no per-language quality breakdown is given. The Section 5 assertion that the gaps are 'large and consistent' and 'unlikely to be explained by these factors' is not supported by any quantitative sensitivity analysis. I ask for a robustness check with independent human or professional translations on a subset, plus per-language back-translation error rates, to rule out translation-induced inflation of the inequality.","section":"Section 3.4 and Appendix B.3; Section 5"},{"comment":"Text-rendering precision is computed from OCR outputs of Gemini 2.5 Flash compared against ground truth. If the OCR model is less accurate for Arabic, Hindi, or Devanagari, the reported cross-lingual precision gaps would partly reflect OCR bias rather than rendering errors. The paper does not validate OCR accuracy on a human-annotated sample or use a second OCR engine. Without such validation, the claim that 'non-Latin writing systems remain a major bottleneck' (Section 4.1.2) is not fully established.","section":"Section 3.3, Appendix C.2.1"},{"comment":"The GPT-5-mini judge for cultural tendency is constrained to a fixed tag set (en, zh, hi, es, ar, fr, pt, ru, ja, ko) that exactly matches the ten prompt languages, plus mixed/other/unclear. Because the tag granularity equals the prompt-language granularity, the reported language-culture match rates (79.6%, 90.2%) may be inflated by the response format. The later 'Western culture' figure (3.7%) is a post-hoc union of five tags, which is sensitive to the tag design. This analysis should be repeated with an open label set or human annotation, and agreement statistics should be reported.","section":"Section 4.3, Appendix E (Figure 21)"},{"comment":"The paper generates only one image per prompt (seed 42). T2I generation is stochastic, and for text rendering a single typo flips sentence-level accuracy from 1 to 0. None of the tables (Tables 2, 3, 6, 7) provide confidence intervals or multi-seed variance. The large cross-linguistic gaps are likely robust, but the fine-grained dimension trade-offs in Section 4.2 (differences of 0.01–0.05) may be within noise. I request multi-seed runs on a subset (e.g., 100–200 prompts per condition) with reported standard deviations, or explicit acknowledgment of this limitation's effect on the trade-off claims.","section":"Appendix D"},{"comment":"The dimensional evaluation uses TRIGScore with Qwen-2.5-VL 72B as the judge, while Section 4.1.1 highlights Qwen-Image as the best model and notes it is built upon Qwen-2.5-VL. This shared backbone creates a risk of evaluator bias in Table 6 and Figure 4, where Qwen-Image is used for the trade-off analysis. Please validate the dimensional scores with an independent judge (e.g., Gemini 2.5 Flash) on a subset, report agreement, and declare the potential conflict with the first author's prior TRIGScore work.","section":"Section 3.2, Appendix C.1.2"},{"comment":"The tokenization analysis reports a Spearman correlation of ρ=-0.89 between prompt fragmentation and CLIPScore across ten languages. With one point per language, the correlation's p-value and confidence interval are not reported, and the metric definition ('prompt-fragmentation score') is not precise. A correlation at the language level cannot establish that inefficient tokenization is a 'systematic text-side bottleneck' without controlling for language family, prompt length, or encoder effects. Either weaken the causal claim or provide a within-language manipulation.","section":"Section 4.4, Figure 11"}],"minor_comments":[{"comment":"The language label 'Korea' should be 'Korean' for consistency with Table 1 and the rest of the paper.","section":"Tables 6 and 7"},{"comment":"The Nano Banana reference is given as '[66]' but should be '[58]'.","section":"Appendix D.1.2"},{"comment":"The paper uses 'Indo-Iranian' in the text while Table 1 labels Hindi as 'Indo-Aryan'; please unify the terminology.","section":"Section 4.1.1 and Table 1"},{"comment":"The order of subfigures in the caption (a, b, e, f, d, c) is confusing and should be reordered.","section":"Figure 4 caption"},{"comment":"In the FLUX model description, the phrase 'with with a resolution' contains a duplicated word and should be fixed.","section":"Appendix D.1.1"},{"comment":"The abbreviation 'TA-C' is used without definition; please define it at first use (likely Task Alignment - Content).","section":"Section 4.4"},{"comment":"The claim of 'first comprehensive cross-lingual analysis' may overstate novelty given the cited related works (e.g., [15, 23, 67]); please temper or justify the claim.","section":"Section 1"},{"comment":"In the PEA<FLUX> row, the numbers '0.680.600.63' are unspaced; please fix the formatting.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of a major multimedia or NLP venue as a benchmark and analysis contribution. I would recommend that the editor require the authors to address the translation-robustness and evaluator-validation concerns before acceptance. Note that the first author is a co-creator of TRIGScore, which is used as the evaluation metric in this paper; this is not disqualifying but should be declared and ideally mitigated by an independent judge."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LingT2I is a real contribution: a 33K-prompt, 10-language benchmark with both content-generation and text-rendering tasks, plus a broad evaluation of 17 models. The descriptive headline—English and Latin-script languages do better, non-Latin scripts break in text rendering, and multilingual-enhanced models are fairer—is probably right. The paper ships code, data, and uses an external multilingual encoder (MetaCLIP2) for the main CLIPScore, which is the right call. The tokenization-fragmentation correlation (rho = -0.89) and the transliteration control are interesting and go beyond prior narrow case studies.\n\nThe main problem is the translation pipeline. Every non-English prompt comes from Gemini 2.5 Pro. The 5% human check validated semantic consistency and fluency, not prompt usability for T2I. A translation can be faithful as text yet parse differently as an image prompt—different word order, ambiguity, lexical choice—and that could plausibly vary by language. Since all 17 models are run on the same translated prompts, any per-language bias in Gemini is baked into every score. Section 5 says the gaps are too large to be explained by this, but that is an assertion, not a sensitivity analysis. An independent human-translated subset for, say, Hindi, Arabic, and Korean would settle it.\n\nThe dimensional and cultural analyses lean on LLM judges in ways that deserve caution. TRIGScore is adapted from the first author's prior work, Qwen-2.5-VL judges Qwen-Image, and the GPT-5-mini culture-tag prompt uses fixed tags that exactly match the ten prompt languages—that could nudge the counts. And there is one image per prompt with a fixed seed and no confidence intervals, so the per-language per-dimension numbers have unknown noise.\n\nNone of this kills the benchmark. The core inequality finding is likely robust, and the resource is reusable. But the causal claims—tokenization bottleneck, native-architecture advantage, cultural priors—are hypotheses until the translation and judge confounds are addressed.\n\nWho is this for: anyone building or evaluating multilingual T2I models. It deserves a serious referee; I would accept it conditionally and push for a human-translated subset, multi-seed or CI reporting, and human validation of a sample of image judgments. I would cite it for the benchmark and the descriptive results.","headline":"LingT2I is a genuinely useful benchmark and the headline inequality finding is likely real, but the single-translation-pipeline confound and LLM-judge circularity make the causal claims hypotheses.","tokens_in":34840,"tokens_out":2304,"would_cite":true,"duration_ms":21787,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multilingual text-to-image systems are not language-neutral: a new 33K-prompt benchmark across 10 languages shows consistent performance drops for non-English prompts, especially in non-Latin scripts, and language-driven shifts in style…","keywords":["text-to-image generation","multilingual evaluation","cross-lingual consistency","linguistic inequality","text rendering","writing systems","benchmark","cultural bias"],"falsifier":"A native-prompt re-run would settle the main claim: recruit native speakers to write the same prompts directly in Hindi, Arabic, and Japanese without English as an intermediate, then compare rankings on the same models. If those languages stop trailing English once translation is removed, the linguistic-inequality conclusion would collapse into a translation artifact.","tokens_in":33710,"feed_emoji":"🌐","tokens_out":5153,"duration_ms":47912,"temperature":0.7,"pith_summary":"This paper argues that current text-to-image (T2I) models are not language-neutral: their output quality, reliability, and failure modes shift systematically with the language of the prompt. To make this case, it introduces LingT2I, a benchmark of 33K prompts across 10 languages spanning Latin, Han, Arabic, Devanagari, Cyrillic, and mixed scripts, and evaluates 17 models on both content generation and in-image text rendering. The results show a consistent linguistic hierarchy, with English, Spanish, French, and Portuguese leading while Hindi, Arabic, Japanese, and Korean trail, and with non-Latin scripts producing broken or unreadable rendered text across nearly all models. The paper also documents language-dependent trade-offs between dimensions such as aesthetics and style, and shows that the same prompt yields culturally specific imagery depending on its language. If these findings hold, English-only evaluation substantially overstates the multilingual competence of current T2I systems.","feed_headline":"33K-prompt test: multilingual image models skew toward English","feed_subtitle":"The LingT2I benchmark shows Hindi and Arabic lag badly and non-Latin rendered text often breaks, across 17 models.","key_machinery":"The load-bearing mechanism is the controlled parallel-prompt benchmark itself. LingT2I builds 30K content-generation prompts by extracting dimension-specific annotations from DOCCI captions and translating them into nine languages, and 3K text-rendering prompts by keeping the English background fixed and translating only the short text string to be rendered, so that language variation is isolated in the text-rendering task. On top of this, the paper adapts evaluation instruments—MetaCLIP-2 for CLIPScore, a Qwen-2.5-VL based TRIGScore, and mT5 tokenization for precision—to make scores comparable across languages. The tokenization fragmentation measurement, the transliteration control, and the culture-tag analysis are the analytical tools that turn the benchmark into evidence about writing systems, tokenizers, and cultural priors.","core_discovery":"Across 17 models, the benchmark finds that general-purpose T2I systems exhibit severe linguistic inequality: average content-generation alignment for English is 0.78, while Hindi and Arabic fall to 0.38, and standard deviations across languages reach up to 0.21 for text-rendering models. Non-Latin writing systems are a persistent bottleneck; for instance, Arabic rendering precision is near 0.12 under some general-purpose models and only 0.43 to 0.46 even for a dedicated rendering model. The analysis further shows that linguistic inequality is not a single-axis gap: languages reshape trade-offs between evaluation dimensions, and the same prompt generates culturally specific imagery aligned with the prompt language, with 79.6% of culture-tagged samples matching the language's region. A tokenization analysis finds a strong negative correlation (rank correlation -0.89) between prompt fragmentation and image-text alignment, pointing to inefficient sub-word tokenization as one systematic text-side bottleneck.","pith_inferences":["If the translation assumption is the main threat, a natural extension is to build LingT2I-style prompts authored natively in each language rather than translated from English; the paper's conclusions predict the same language hierarchy would persist.","The -0.89 fragmentation correlation suggests a testable intervention: retrain or augment a model's tokenizer for under-resourced scripts and measure whether the alignment gap narrows accordingly.","The culture-tag results imply prompt language acts as an implicit cultural control knob; this could be harnessed for culturally aware generation, but it also raises the risk of reinforcing regional stereotypes when prompting in a given language.","The failure-pattern taxonomy (semantic substitution, structural errors, and romanization bias) suggests that script-specific rendering modules and language-family-organized training data would attack different failure modes, and their relative effectiveness could be measured separately."],"forward_implications":["English-only evaluation of T2I models substantially overstates their multilingual competence.","Non-Latin text rendering is a major failure mode, not a marginal case; dedicated rendering models only partially mitigate it.","Post-hoc multilingual adapters can reduce cross-language variance but tend to lower quality in high-resource languages, whereas native multilingual architectures offer a more effective path to fairness.","Because trade-offs are language-dependent, optimizing one evaluation dimension in one language can worsen outcomes in another language or dimension.","Tokenizer efficiency is a measurable text-side bottleneck: languages with more fragmented token sequences show weaker image-text alignment."],"supporting_citations":[{"why":"Supplies the DOCCI captions whose textual component anchors the Content Generation subset.","marker":"[40]"},{"why":"Provides the English raw prompts for the Text Rendering subset and the MLLM-as-judge evaluation protocol the paper adapts.","marker":"[35]"},{"why":"Performs the multilingual translations and serves as the evaluation model for OCR and dimensional text-rendering judgments.","marker":"[13]"},{"why":"Provides the multilingual CLIP encoder used to compute CLIPScore fairly across languages.","marker":"[12]"},{"why":"Supplies the TRIGScore dimensional evaluation method and the trade-off analysis perspective the paper extends to cross-lingual settings.","marker":"[77]"},{"why":"Powers the adapted TRIGScore as the 72B Qwen-2.5-VL judge for dimension scores.","marker":"[60]"},{"why":"Provides the mT5 tokenizer used for token-level precision, prompt length statistics, and fragmentation analysis.","marker":"[73]"}],"fun_headline_variants":["Multilingual T2I models show severe bias toward English","Text-to-image AI: Hindi and Arabic scores drop to 0.38","LingT2I: Non-Latin scripts break text rendering across 17 models","Tokenization gap links to text-to-image quality, New benchmark finds","Cross-lingual text-to-image: Culture, tokenization, and inequality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's cross-language comparisons assume that machine translations from English preserve semantic meaning, cultural neutrality, and prompt usability closely enough that measured gaps reflect model capability rather than translation artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Multilingual T2I models show severe bias toward English","Text-to-image AI: Hindi and Arabic scores drop to 0.38","LingT2I: Non-Latin scripts break text rendering across 17 models","Tokenization gap links to text-to-image quality, New benchmark finds","Cross-lingual text-to-image: Culture, tokenization, and inequality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000732,"raw_usage":{"total_tokens":3258,"prompt_tokens":909,"completion_tokens":2349,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":2252}},"tokens_in":525,"tokens_out":2349,"duration_ms":17037,"temperature":1.0,"reasoning_tokens":2252,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:20:47.372547+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A native-prompt re-run would settle the main claim: recruit native speakers to write the same prompts directly in Hindi, Arabic, and Japanese without English as an intermediate, then compare rankings on the same models. If those languages stop trailing English once translation is removed, the linguistic-inequality conclusion would collapse into a translation artifact.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DOCCI captions whose textual component anchors the Content Generation subset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the TRIGScore dimensional evaluation method and the trade-off analysis perspective the paper extends to cross-lingual settings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the mT5 tokenizer used for token-level precision, prompt length statistics, and fragmentation analysis."}],"review_version":1}