Pith. sign in

REVIEW 3 major objections 7 minor 29 references

Jako Tako or Fluent? Presenting PoVisLE: A Polish Vision-Language Evaluation

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces PoVisLE, the first monocultural Polish vision-language benchmark, and claims current VLMs are far from fluent in Polish cultural understanding.

desk verdict A genuinely useful and transparently built Polish cultural VLM benchmark, but its own Table 3 undercuts the abstract's claim about question-free MCQ performance, so it needs revision rather than rejection. read the letter →

arxiv 2608.07763 v1 pith:ES3LFI7U submitted 2026-08-07 cs.CL cs.CV

Anna Ko{l}os , Grzegorz Statkiewicz , Karolina Seweryn , Katarzyna Kowol , Karolina Piosek , Wojciech Kusa This is my paper · ORCID
classification cs.CLcs.CV
keywords PoVisLEvision-languagebenchmarkPolishculturalcompetencevisualquestionansweringdialectandregionalismgroundingmultilingualevaluationVQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that Polish cultural and linguistic competence in vision-language models can be measured, and that current models are far from fluent in it. It introduces PoVisLE, a benchmark of 1,117 images and 2,366 hand-written questions in Polish that require connecting what is visible in a picture with shared cultural knowledge, dialect, slang, history, or wordplay. On this benchmark the strongest of 16 evaluated models reaches 71.45 percent accuracy, and questions about dialects and regionalisms are the hardest category. The paper argues that because removing the image cuts accuracy sharply, the benchmark really tests grounded multimodal understanding rather than text memorization.

What carries the argument

The load-bearing artifact is the PoVisLE dataset itself, organized around a hierarchical taxonomy with seven main categories: Art and entertainment, Language, Geography and nature, History and society, Culture and tradition, Image understanding, and Visual reasoning. Construction rests on three annotation rules: questions must be unambiguous with a single correct answer, must rely on visual content, and must carry strict output-format instructions. The evaluation harness adds circular option rotation for multiple-choice items, exact-match scoring with Polish diacritics and capitalization, and two ablations, removing the image and removing the question, plus Polish, English, and German prompt translation.

What would settle it

Take a random sample of PoVisLE test questions and have independent Polish annotators from different regions provide answers without seeing the gold standard; if agreement on the single expected answer falls well below the level needed for deterministic scoring, the no-ambiguity premise fails. A simpler check: if any future model scores near full accuracy with images removed, the visual-grounding claim is refuted.

Watch

Extended reading notes

Core claim

PoVisLE is presented as the first monocultural vision-language benchmark built specifically for Polish cultural and linguistic competence. Its 2,366 question-answer pairs are manually written, template-free, and designed so that each question requires the image: even language-focused items such as the riddle that asks how a Pole answers "how are you?" by rhyming with the pseudonym "Taco Hemingway" (the answer being "jako tako") depend on seeing who is in the picture. The paper reports that the best model, Qwen3.5-397B-A17B in its thinking mode, scores 71.45 percent macro accuracy, that dialect and regionalism questions are the weakest category, and that removing the image lowers accuracy by 32 to 48 percentage points while removing the question leaves open-ended accuracy at zero. These results are used to claim that PoVisLE provides a valid, grounded evaluation of culturally situated Polish multimodal understanding.

Load-bearing premise

The load-bearing premise is that each question has exactly one correct, non-debatable answer that all Poles would agree on; the paper's own limitations section concedes cases where culturally canonical answers are privileged over correct alternatives.

Editorial extensions

If this is right

  • If correct, the benchmark gives a controlled yardstick for Polish cultural VQA, with a held-out test split of 1,960 pairs available for future model comparisons.
  • Current best models leave roughly 29 percentage points on the table, so culturally grounded Polish understanding is far from solved.
  • Dialects and regionalisms are the hardest category, meaning intra-language variation is a specific weakness of current VLMs.
  • The large accuracy drop when the image is removed confirms that high scores cannot be achieved from textual priors alone on this benchmark.
  • Prompt-language results suggest that stronger models exploit the original Polish formulation, while weaker models can gain up to 8.29 points from English prompts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the benchmark's difficulty holds up, it could be used to track progress of Polish-specific vision-language models, which currently sit below 37 percent accuracy.
  • The single-answer rule is a deliberate trade-off; a future version with flexible semantic matching might reveal that models are more culturally fluent than exact-match scoring admits.
  • The monocultural design suggests a template for other mid-resource languages: manual, template-free questions anchored in private images could expose cultural gaps that broad multicultural benchmarks miss.
  • Items like the "jako tako" riddle indicate the benchmark tests pragmatic wordplay, so passing it requires more than object recognition or encyclopedic knowledge.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces PoVisLE, a Polish vision-language benchmark with 1,117 images and 2,366 manually authored VQA pairs organized into a hierarchical taxonomy of Polish cultural and linguistic categories. The authors benchmark 16 open-weight and proprietary VLMs, reporting that Qwen3.5-397B-A17B (Thinking) achieves 71.45% macro accuracy, and run ablations that remove either the image or the question, as well as a prompt-language translation study. The central claims are that PoVisLE is the first monocultural Polish vision-language benchmark for cultural and linguistic competence and that the ablations confirm the benchmark cannot be solved from textual priors or answer-option artefacts alone.

Significance. If the construct validity of the benchmark holds, PoVisLE is a valuable resource for Polish cultural and linguistic multimodal evaluation. The dataset construction is unusually transparent: manual template-free annotation, cross-validation by a second annotator, an expert super-annotator, iterative LLM-based grounding checks, image-cluster bootstrap confidence intervals, detailed annotation guidelines in an appendix, and a public release. The strongest models score around 71%, leaving substantial headroom, and the per-category results identify dialect and regionalism questions as the weakest area. These properties make the resource potentially useful for the community. However, the stated interpretation of the ablation results is inconsistent with the reported data, and the answer-unambiguity requirement is in tension with the limitations discussion.

major comments (3)
  1. [Section 5, Table 3, Abstract] The abstract claims that the ablations 'confirm that the benchmark cannot be solved from textual priors or answer-option artefacts alone.' Table 3 shows that for the strongest model, Qwen3.5-397B-A17B (Thinking), removing the question while keeping the image and answer options yields 55.99% MCQ accuracy, only 9.95 percentage points below the full-input MCQ accuracy of 65.94%, and far above the random baseline of 0.38% under circular evaluation. Since MCQs constitute 36.1% of the test split, a large share of the benchmark can be answered without reading the question. Section 5 itself acknowledges that 'the image and answer options are often sufficient to identify the expected answer without access to the question.' This contradicts the abstract's claim. The 'without image' condition also shows residual textual-prior leakage: MCQ accuracy drops to 29.08%, still far above random. The claim as written is not supported by the reported data; the authors should either revise the abstract to describe the relative rather than absolute nature of the ablation effects, or restrict the claim to the open-ended and yes/no components, or report results on the subset of items that are genuinely not answerable without the question.
  2. [Section 2.3 and Limitations] The annotation guidelines require questions to allow 'a single clearly defined and non-debatable answer,' yet the Limitations section describes a 'perspective asymmetry' in which culturally canonical answers are privileged over 'alternative, semantically correct interpretations' and 'over-localization' where models 'may be penalized for producing correct but non-prototypical answers.' These two positions are in direct tension. If the latter effects occur in the test set, then the benchmark may measure alignment with annotator prototypes rather than objective cultural knowledge. The paper should provide evidence that the 'No ambiguity' rule was effective: for example, inter-annotator agreement on the answer key, the number of items revised or removed due to ambiguity during cross-validation, and a discussion of how many test items could admit defensible alternative answers. Without such evidence, the construct validity of PoVisLE as a measure of culturally grounded understanding remains uncertain.
  3. [Section 2.4] The LLM-based grounding check flagged MCQs as 'too easy' only when models achieved a success rate of approximately above 60% in the question-free setting. However, the final evaluation includes Qwen3.5-397B-A17B (Thinking), which achieves 55.99% question-free MCQ accuracy, just below this threshold. If the 11 LLMs used in validation were weaker than the evaluated models, the threshold would have failed to remove items that the strongest models can answer from the image and options alone. The authors should report the question-free MCQ accuracy of the validation LLMs, justify the 60% threshold, or use a model-relative or item-level criterion (e.g., flag items answerable by any model above chance). This is load-bearing because it directly affects the benchmark's guarantee that questions require the linguistic input.
minor comments (7)
  1. [Title] The title contains a typo: 'Jako Takoor' should be 'Jako Tako'.
  2. [Table 1] The subcategory 'Colloqual speech and slang' is misspelled; it should be 'Colloquial speech and slang'.
  3. [Section 4.2 and throughout] The model name 'LLaVA' is frequently rendered as 'LLaV A' with a stray space; this should be corrected globally.
  4. [Section 5 and Figure 2] The sentence 'Figure 2 present model performance' should be 'Figure 2 presents model performance'; the same issue appears in the caption of Figure 2.
  5. [Appendix B.2] The paragraph beginning 'To further illustrate the distribution of complexity...' is duplicated verbatim; one copy should be removed.
  6. [Section 2.3 vs Appendix D.5] Section 2.3 states that a controlled subset was annotated by 12 auxiliary annotators, while Appendix D.5 and Table 6 report 13 auxiliary annotators; the numbers should be reconciled.
  7. [Section 4.3] The main evaluation metric is described as 'macro-averaged accuracy over question type,' but the main text does not explicitly state that overall accuracy is the unweighted mean of the three question-type accuracies (as implied by Table 3). This should be stated explicitly, with a justification for equal weighting given the different sample sizes across question types.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PoVisLE is a manually constructed benchmark with independent human annotation, and the reported evaluations do not reduce to the benchmark's construction inputs.

full rationale

The paper is a benchmark construction and evaluation study rather than a derivation. The central dataset is built from 1,117 images and 2,366 manually authored VQA pairs produced under explicit annotation guidelines requiring visual grounding and unambiguous answers; model performance is then measured against these human gold answers. No equation or procedural step defines the evaluation outcome in terms of the benchmark's own construction choices, and no fitted parameter is relabeled as a prediction. The self-citations to StyloMetrix and to the Polish-oriented LLaVA models are real but not load-bearing for the paper's central claims: StyloMetrix is used descriptively for stylometric feature reporting, and the LLaVA models are merely among the systems evaluated, not invoked as evidence that the benchmark measures what it claims to measure. The LLM-assisted quality-control filtering used a subset of model families that also appear in the later evaluation; this is a legitimate external-validity concern, but it does not make the reported accuracies equivalent to the filtering step by construction. In fact, the paper's own Table 3 reports question-free MCQ accuracy of up to 55.99% for the strongest model, which empirically undercuts the abstract's claim that ablations rule out answer-option artefacts; that is a correctness or construct-validity issue, not a circularity. No step in the paper reduces, by definition or by self-citation chain, to its own inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No invented physical or conceptual entities are introduced. The central resource depends on domain assumptions about Polish cultural consensus and answer objectivity, plus two hand-set construction parameters. These are disclosed but not independently verified.

free parameters (2)
  • MCQ difficulty revision threshold = approximately above 60%
    Hand-set threshold in Section 2.4; questions that LLMs answered above roughly 60% were revised or converted, shaping benchmark difficulty.
  • Initial target share of annotator-provided images = 39.5% targeted; 28.74% final
    Hand-set construction target in Section 2.2; affects the fraction of non-public images in the benchmark.
assumptions (4)
  • domain assumption A single clearly defined and non-debatable answer exists for every question.
    Stated in Section 2.3 and contested in the Limitations; if false, scoring penalizes valid alternative answers.
  • domain assumption A shared baseline of Polish cultural knowledge exists among annotators and target users.
    Section 2 operationalizes culture as shared artifacts and practices; regional variation is acknowledged but assumed not to break answer unambiguity.
  • domain assumption A concept is culturally relevant if it meets one of four criteria: widely recognized, taught, present in media discourse, or needed to interpret culturally grounded scenes.
    Section 2 defines relevance; this determines which items enter the benchmark and what counts as cultural competence.
  • domain assumption LLM judgments can identify questions that are too easy or insufficiently grounded, and filtering them improves the benchmark.
    Section 2.4 uses LLM-based checks to revise or remove items; if those judgments are unreliable, the composition of the benchmark changes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Jako Tako or Fluent? Presenting PoVisLE: A Polish Vision-Language Evaluation." pith.science (2026). https://pith.science/paper/ES3LFI7U

@misc{pith2026260807763,
  author       = {Pith},
  title        = {Pith review of: Jako Tako or Fluent? Presenting PoVisLE: A Polish Vision-Language Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ES3LFI7U}},
  note         = {Machine review of arXiv:2608.07763}
}
read the original abstract

Vision-language models (VLMs) have achieved strong performance on tasks such as image captioning, visual question answering, and image-to-text generation. However, they are predominantly trained on English-centric data, which limits their ability to handle culturally grounded visual understanding and leads to failures in interpreting region-specific meanings, symbolic content, and context-dependent visual cues. Existing benchmarks for cultural competence are often template-driven and focused on surface-level recognition, making them insufficient for evaluating deeper linguistic and pragmatic understanding in culturally situated settings. We introduce PoVisLE, a monocultural vision-language benchmark for Polish designed to evaluate culturally grounded multimodal understanding under a grounded evaluation paradigm, where language is interpreted in interaction with visual context. The dataset contains 1,117 images and 2,366 manually annotated VQA pairs. Overall, our dataset provides a controlled and challenging resource for assessing culturally grounded vision-language understanding beyond surface-level recognition.

Figures

Figures reproduced from arXiv: 2608.07763 by the authors.

Figure 1
Figure 1. Above: Overview of the two-stage PoVisLE dataset construction process. Below: a single example from our dataset, with reasoning steps provided for clar￾ity. More examples are included in Appendix A. ing Polish cultural and linguistic competence. Nearly 30% of the images come from annota￾tors’ private collections and are absent from web￾scale training corpora, providing a stricter test of generalization. • We adapt a… view at source ↗
Figure 2
Figure 2. Overall macro accuracy on the test split with [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Translation effect by model. Macro accuracy [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Examples from the dataset illustrating multiple-choice, binary (yes/no), and open-ended questions across [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Distribution of 2,198 culturally grounded questions across categories and subcategories in the dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Distribution of question lengths (in tokens) across the dataset. The dashed vertical line indicates the mean [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Overall macro accuracy on the validation split [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]
Figure 8
Figure 8. Figure 8: Macro accuracy by category with 95% image-cluster bootstrap confidence intervals on the test split. The [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 22 canonical work pages

  1. [1]

    Dhananjay Ashok, Ashutosh Chaubey, Hirona J Arai, Jonathan May, and Jesse Thomason. 2025. https://aclanthology.org/2025.findings-emnlp.850.pdf Can vlms recall factual associations from visual references? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 15691--15708

    work page 2025

  2. [2]

    Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. 2024. https://doi.org/10.18653/v1/2024.acl-long.555 Artifacts or abduction: How do LLM s answer multiple-choice questions without the question? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10308--10330, Bangkok, Thailan...

    doi:10.18653/v1/2024.acl-long.555 2024

  3. [3]

    Micha Ciesi \'o ka and Filip Grali \'n ski. 2025. revision: A polish benchmark for evaluating vision-language models on multimodal national exam data. In 2025 20th Conference on Computer Science and Intelligence Systems (FedCSIS), pages 665--673. IEEE

    work page 2025

  4. [4]

    S awomir Dadas, Ma gorzata Grebowiec, Micha Pere kiewicz, and Rafa Po \'s wiata. 2025. https://arxiv.org/pdf/2503.00995 Evaluating polish linguistic and cultural competency in large language models . In International Conference on Artificial Intelligence and Soft Computing, pages 60--71. Springer

    work page Pith review arXiv 2025

  5. [5]

    V. A. Elisi \'a rio and W. M. Watanabe. 2025. https://www.scitepress.org/publishedPapers/2025/136738/pdf/index.html Multimodal large language models for portuguese alternative text generation for images . In Proceedings of the 21st International Conference on Web Information Systems and Technologies (WEBIST 2025), pages 493--501. SCITEPRESS -- Science and...

    work page 2025

  6. [6]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. https://openaccess.thecvf.com/content_cvpr_2017/papers/Goyal_Making_the_v_CVPR_2017_paper.pdf Making the v in vqa matter: Elevating the role of image understanding in visual question answering . In Proceedings of the IEEE conference on computer vision and pattern recognition...

    work page 2017

  7. [7]

    Hsin Yi Hsieh, Shang-Wei Liu, Chang-Chih Meng, Chien-Hua Chen, Shuo-Yueh Lin, Hung-Ju Lin, Hen-Hsen Huang, I Wu, and 1 others. 2026. https://papers.nips.cc/paper_files/paper/2025/file/1c27e0352b819d61fbd6b65eef125b23-Paper-Datasets_and_Benchmarks_Track.pdf Taiwanvqa: Benchmarking and enhancing cultural understanding in vision-language models . Advances in...

    work page 2026

  8. [8]

    Krzysztof Jassem, Micha Ciesi \'o ka, Filip Grali \'n ski, Piotr Jab o \'n ski, Jakub Pokrywka, Marek Kubis, Monika Jab o \'n ska, and Ryszard Staruch. 2025. LLMzSzŁ: a comprehensive LLM benchmark for Polish . arXiv preprint arXiv:2501.02266

    arXiv 2025

Show all 29 references
  1. [9]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. 2017. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning . In Proceedings of the IEEE Conference on Computer Vision and Pattern Re...

    2017

  2. [10]

    Karima Kadaoui, Hanin Atwany, Hamdan Al-Ali, Abdelrahman Mohamed, Ali Mekky, Sergei Tilga, Natalia Fedorova, Ekaterina Artemova, Hanan Aldarmaki, and Yova Kementchedjhieva. 2026. https://aclanthology.org/2026.findings-eacl.18.pdf Jeem: Vision-language understanding in four ara...

    2026

  3. [11]

    Jind r ich Libovick \`y , Jind r ich Helcl, Andrei Manea, and Gianluca Vico. 2025. https://arxiv.org/pdf/2507.22752 Cus-qa: Local-knowledge-oriented open-ended question answering dataset . arXiv preprint arXiv:2507.22752

    2025

  4. [12]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. https://arxiv.org/pdf/2311.02709 Microsoft coco: Common objects in context . In European conference on computer vision, pages 740--755. Springer

    2014 arXiv

  5. [13]

    Moritz M \"a hr and Moritz Twente. 2025. https://anthology.ach.org/volumes/vol0003/seeing-history-unseen-evaluating-vision-language/10.63744@njQVYcLndSPE.pdf Seeing history unseen: Evaluating vision-language models for wcag-compliant alt-text in digital heritage collections . ...

    2025

  6. [14]

    Arijit Maji, Raghvendra Kumar, Akash Ghosh, Nemil Shah, Abhilekh Borah, Vanshika Shah, Nishant Mishra, Sriparna Saha, and 1 others. 2025. https://aclanthology.org/2025.emnlp-main.68.pdf Drishtikon: A multimodal multilingual benchmark for testing language models’ understanding ...

    2025

  7. [15]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. https://arxiv.org/pdf/1906.00067 Ok-vqa: A visual question answering benchmark requiring external knowledge . In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, page...

    2019 arXiv

  8. [16]

    Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hendricks, Karolina Sta \'n czak, and Aishwarya Agrawal. 2024. https://aclanthology.org/2024.emnlp-main.329.pdf Benchmarking vision language models for cultural understanding . In Proceeding...

    2024

  9. [17]

    Inez Okulska, Daria Stetsenko, Anna Ko os, Agnieszka Karli \'n ska, Kinga G a bi \'n ska, and Adam Nowakowski. 2023. https://arxiv.org/pdf/2309.12810 Stylometrix: An open-source multilingual tool for representing stylometric vectors . arXiv preprint arXiv:2309.12810

    2023 arXiv

  10. [18]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. https://proceedings.mlr.press/v139/radford21a/radford21a.pdf Learning transferable visual models from natural...

    2021

  11. [19]

    David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, and 1 others. 2024. https://arxiv.org/pdf/2406.05967 Cvqa: Culturally-diverse multilingual visual question...

    2024 arXiv

  12. [20]

    Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. 2019. https://ojs.aaai.org/index.php/AAAI/article/view/4915 Kvqa: Knowledge-aware visual question answering . In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8876--8884

    2019

  13. [21]

    Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Soujanya Poria, and Rada Mihalcea. 2024. https://doi.org/10.18653/v1/2024.naacl-long.316 Understanding the capabilities and limitations of large language models for cultural commonsense . In Proceedings of the 2024 Con...

    2024 doi

  14. [22]

    Grzegorz Statkiewicz, Alicja Dobrzeniecka, Karolina Seweryn, Aleksandra Krasnod e bska, Karolina Piosek, Katarzyna Bogusz, Sebastian Cygert, and Wojciech Kusa. 2026. Annotation-efficient vision-language model adaptation to the polish language using the llava framework. In Proc...

    2026

  15. [23]

    Bryan Chen Zhengyu Tan, Weihua Zheng, Zhengyuan Liu, Nancy Chen, Hwaran Lee, Kenny Tsu Wei Choo, and Roy Ka-Wei Lee. 2026. https://aclanthology.org/2026.eacl-long.215.pdf Blend-vis: Benchmarking multimodal cultural understanding in vision language models . In Proceedings of th...

    2026

  16. [24]

    S pela Vintar, Taja Kuzman Punger s ek, Mojca Brglez, and Nikola Ljube s i \'c . 2025. https://arxiv.org/pdf/2510.24450 Charting the european llm benchmarking landscape: A new taxonomy and a set of best practices . arXiv preprint arXiv:2510.24450

    2025

  17. [25]

    Yuxuan Wang, Yijun Liu, Fei Yu, Chen Huang, Kexin Li, Zhiguo Wan, Wanxiang Che, and Hongyang Chen. 2025. https://ojs.aaai.org/index.php/AAAI/article/view/32884 Cvlue: A new benchmark dataset for chinese vision-language understanding evaluation . In Proceedings of the AAAI Conf...

    2025

  18. [26]

    Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Wang Yutong, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, and 1 others. 2025. https://aclanthology.org/2025.naacl-long.167.pdf Worldcuisines: A massive-sc...

    2025

  19. [27]

    Srishti Yadav, Lauren Tilton, Maria Antoniak, Taylor Arnold, Jiaang Li, Siddhesh Milind Pawar, Antonia Karamolegkou, Stella Frank, Zhaochong An, Negar Rostamzadeh, and 1 others. 2025. https://arxiv.org/pdf/2505.22793 Evaluation of cultural competence of vision-language models ...

    2025 arXiv

  20. [28]

    Amber Yijia Zheng, Jae Joong Lee, Bedrich Benes, and Raymond A Yeh. 2025. https://arxiv.org/pdf/2602.03850 Webaccessvl: Making an accessible web via violation-conditioned vlm . arXiv preprint arXiv:2602.03850

    2025

  21. [29]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations, volume 2024, pages 19426--19454

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.