Pith. sign in

REVIEW 2 major objections 6 minor 49 references

What Does Your Short-Answer VQA Score Actually Measure? Evaluator-Dependent Instability in Multimodal Short-Answer Benchmarks

T0 review · 2 major / 6 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Short-answer VQA scores mix semantic correctness with surface-form compliance, and on text-rich benchmarks up to half of official errors are acceptable answers rejected only for wording.

desk verdict Solid measurement paper: up to half of official errors on text-rich short-answer VQA are surface-form false negatives, structured by answer contract, with dual-judge audit and partial CPU repair. read the letter →

arxiv 2607.10240 v1 pith:XBFJ4D62 submitted 2026-07-11 cs.CV cs.MM

Guanhua Ye , Niu Jingbin , Yan Li , Meiyu Liang , Zhe Xue , Yingxia Shao , Yawen Li This is my paper · ORCID
classification cs.CVcs.MM
keywords visualquestionansweringevaluationmethodologyvision-languagemodelsbenchmarkanalysisanswercontractsshort-answerVQAevaluatorinstability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Short-answer visual question answering benchmarks still score models mainly by string matching, so a leaderboard number confuses two different things: whether the answer is right in meaning, and whether it matches the form the automatic scorer expects. Auditing more than 37,000 official errors from six vision–language models on six benchmarks with a human-validated semantic judge shows that this gap is large on text-rich datasets: as much as about half of the marked errors are answers humans and the judge accept as correct. The problem is not uniform. It tracks answer type—extractive readouts and multi-span answers are far more sensitive than simple numbers or yes/no—and even mild prompt rewrites flip official item-level outcomes at high rates without changing the task. A cheap deterministic repair of answer and reference forms recovers part of the undercount. The practical claim is that official short-answer VQA scores stay useful only if they are reported with semantic audits and answer-type breakdowns.

What carries the argument

The answer contract: the implicit agreement about what form an acceptable answer must take, split operationally into scalar, extractive-readout, identifier-like, and multi-span buckets so that official score can be read as semantic acceptability filtered by contract compatibility.

What would settle it

A large human re-audit of the same official-error pool that found most judge-accepted false negatives were actually wrong or visually unsupported, and that extractive and multi-span items were not more often falsely rejected than scalar items, would overturn both the undercount size and the contract-structure claim.

Watch

Extended reading notes

Core claim

Across six vision–language models and six short-answer VQA benchmarks, a human-validated text-only semantic judge finds that 21–48% of official errors on text-rich datasets are semantically acceptable answers rejected purely for surface-form mismatch, producing multi-point score undercounts. The undercount follows answer contract more than benchmark name: extractive readout and multi-span answers are much more evaluator-sensitive than scalar answers. Benign prompt and context rewrites further flip official item correctness at substantial rates, and CPU-only contract repair recovers a measurable fraction of the false negatives.

Load-bearing premise

The claim depends on treating a text-only language-model judge—given only the question, gold answer, and model output, never the image—as a faithful proxy for whether an answer is semantically acceptable.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper argues that short-answer VQA leaderboard scores conflate semantic correctness with surface-form survival under automatic evaluators. Across six VLMs and six benchmarks, the authors audit over 37k official errors with a text-only LLM semantic judge (human-validated at 97.6% precision, 95.5% recall, κ=0.917) and a second independent judge that reproduces the same benchmark-level false-negative pattern. On text-rich sets such as ST-VQA, up to ~48% of official errors are judged semantically acceptable; the undercount is structured by an operational four-bucket answer-contract taxonomy (scalar, readout, identifier, multi-span), with extractive and multi-span answers far more evaluator-sensitive than scalar ones. Benign prompt/context rewrites flip item-level official correctness at substantial rates, and a deterministic CPU-only bidirectional contract repair recovers about +1.34 pp overall without re-running models. The authors conclude that official scores should be reported with semantic audits and contract-mix diagnostics.

Significance. If the measured undercount and contract structure hold, the paper is a useful measurement contribution for multimodal evaluation: it shows that adjacent score gaps and official error pools on text-rich short-answer VQA can mix genuine failures with surface-form rejections, and that this mixture is not uniform across answer types. Strengths include scale (37k+ audited errors), dual-judge robustness with near-perfect per-dataset FN-rate correlation (r=0.999), stratified human validation, an operational contract taxonomy anchored partly in benchmark-native metadata, and a released deterministic repair pipeline that recovers a non-trivial fraction of the undercount without LLM judges. The practical reporting template and the separation of official, true-overlay, and judge layers are actionable for both benchmark users and dataset designers.

major comments (2)
  1. [§3.3, §5.1] §3.3 and §5.1: The central undercount claim treats text-only semantic equivalence (question + gold + output, no image) as the operational definition of “semantically acceptable.” Human validation and the second judge also operate on this text-side interface, so the audit correctly measures official false negatives relative to gold references, not re-verified visual grounding. That is a defensible diagnostic quantity, but the main results still occasionally read as if they quantify pure task success. Please state more explicitly in §5.1 (not only §6) that judge-accepted errors are gold-equivalent surface variants, and that residual risk of gold-reference incompleteness or ungrounded but gold-matching paraphrases is not eliminated by the audit.
  2. [§5.2, Appendix B] Appendix B vs. §5.2: For raw judge-FN separation, dataset-only grouping has higher η² (0.050) than the contract taxonomy (0.024), while contracts are stronger for perturbation flips. The paper already notes this, but the main-text claim that instability “follows answer contract rather than benchmark name alone” (RQ2 / abstract) should be tightened to match the evidence: contracts explain a useful cross-benchmark structure and perturbation sensitivity, but do not dominate dataset identity for baseline FN rates. A short quantitative caveat in §5.2 would prevent over-reading Table 4–5 as a full replacement for benchmark-level analysis.
minor comments (6)
  1. [Table 3] Table 3: “Acc. Err.” is easy to misread as an accuracy-style error rate; the footnote helps, but renaming the column (e.g., “Judge-accepted share of official errors”) would reduce confusion.
  2. [Table 7, Figure 3] Table 7 and surrounding text: “ST-VAQ” / “TextVAQ” appear as typos for ST-VQA / TextVQA in the repair table and Figure 3 captions.
  3. [§4] §4 Perturbation protocol: B3 is described as an adversarial control excluded from main flip-rate analysis; a one-line statement of what B3 does (empty/minimal instruction) would make the exclusion criterion clearer without appendix diving.
  4. [§3.1] Eq. (2) is conceptual (S_off = f(S_sem, κ)) and never operationalized beyond the diagnostic split; either keep it clearly labeled as notation-only or briefly say that f is the benchmark’s string/ANLS/relaxed rule rather than a fitted function.
  5. [Table 3, Appendix G] Closed-source rows in Table 3 use stratified samples; pointing readers more visibly from the table caption to Appendix G CIs would help avoid treating those Acc. Err. figures (e.g., DocVQA GPT-5.4 77.8%) as full-pool estimates.
  6. [References] A few reference entries appear only loosely related to VQA evaluation methodology (e.g., multi-sensor tracking, image fusion, federated GNN). Trimming or relocating peripheral citations would tighten the related-work signal.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: undercount, contract structure, and repair are measured against external official metrics and human-validated audits, not forced by definition or self-citation.

full rationale

This is an empirical measurement paper, not a first-principles derivation. Official scores are external benchmark anchors (exact match, ANLS, relaxed numeric accuracy). Semantic undercount is estimated by auditing the official error pool with a text-only judge validated against a stratified 570-item human majority (97.6% precision, κ=0.917) and replicated by a second independent judge (per-dataset FN-rate r=0.999). Answer-contract buckets are coarse operational groupings from question form, answer shape, and native metadata; they organize observed sensitivity and are not parameters fitted so that undercount holds by construction. Heuristic RA-Eval and CPU-only bidirectional contract repair recover a fraction of the same pool without relying on the judge as a closed loop. Self-citations appear only as ordinary related-work pointers and do not carry the central claim. Nothing reduces Eq. (2) or the reported FN rates to their own inputs by definition.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central undercount claim rests on a small set of operational choices rather than free physical constants: a text-only semantic-equivalence definition, a coarse four-bucket answer-contract taxonomy, a conservative substring heuristic, and fixed decoding/prompt conventions. No continuous parameters are fitted to produce the headline percentages; the numbers are measured. The main invented construct is the answer-contract decomposition used for explanation.

free parameters (3)
  • RA-Eval substring length threshold = >3 characters
    Heuristic recovers a gold answer only when |g_norm| > 3 characters appears as a word-boundary substring of the normalized output; the cutoff is chosen by hand and affects the lower-bound recoverable count.
  • Closed-source sample sizes = 500 / 200 items
    GPT-5.4 and Claude Sonnet 4.6 are evaluated on stratified samples of 500 (ST-VQA/TextVQA) or 200 (remaining) items rather than full pools; sample size is a design choice that limits precision of closed-source deltas.
  • Human validation set size = 570 items
    Judge precision/recall are estimated on a stratified 570-item review; the size and stratification are design choices that bound the reported 97.6% precision.
assumptions (4)
  • domain assumption Text-only semantic equivalence between model output and gold reference(s), judged without the image, is a valid diagnostic of answer acceptability for undercount estimation.
    Stated in §3.3 and used for all 37k audits; supported by human validation but still an assumption that visual grounding failures are not systematically mislabeled as acceptable.
  • domain assumption Greedy decoding under a fixed short-answer instruction represents the model's answer realization for scoring purposes.
    §4 Experimental Setup; excludes self-consistency and sampling, so measured instability is under a single decoding policy.
  • ad hoc to paper The four operational answer-contract buckets (scalar, readout, identifier, multi-span) plus residual other-open are sufficiently stable explanatory units for evaluator sensitivity.
    Table 1 and §3.2; rule-based and intentionally coarse; authors treat them as explanatory, not a universal ontology.
  • domain assumption Official benchmark metrics (exact match, ANLS, 5% relaxed numeric accuracy) are the correct external anchors for defining official errors.
    Throughout; the paper measures undercount relative to these anchors rather than replacing them.
invented entities (2)
  • Answer contract (four-bucket operational taxonomy)
    purpose: Organize evaluator-sensitive answer regimes so that undercount and flip rates can be attributed to answer type rather than only to benchmark name.
    Introduced in §3.2 / Table 1; buckets are rule-based groupings of question form, answer shape, and native metadata. Independent evidence is partial: TextVQA OCR spans and InfographicVQA answer-type labels anchor two buckets, but the full four-way split is paper-defined.
  • S_off = f(S_sem, κ) measurement decomposition
    purpose: Separate semantic acceptability from benchmark-contract compatibility so official score can be diagnosed rather than treated as pure success.
    §3.1 conceptual framework; diagnostic notation, not a fitted physical law. Useful organizing device with no external existence claim beyond the paper's analyses.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Does Your Short-Answer VQA Score Actually Measure? Evaluator-Dependent Instability in Multimodal Short-Answer Benchmarks." pith.science (2026). https://pith.science/paper/XBFJ4D62

@misc{pith2026260710240,
  author       = {Pith},
  title        = {Pith review of: What Does Your Short-Answer VQA Score Actually Measure? Evaluator-Dependent Instability in Multimodal Short-Answer Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XBFJ4D62}},
  note         = {Machine review of arXiv:2607.10240}
}
read the original abstract

Short-answer VQA benchmarks conflate two distinct quantities: whether a model's answer is semantically correct, and whether that answer matches the surface form expected by the automatic evaluator. We study this conflation across six vision--language models and six benchmarks, using a human-validated semantic judge (97.6% precision) to audit over 37k official errors. A second text-only judge reproduces the same benchmark-level false-negative pattern, showing that the effect is not an artifact of a single audit model. On text-rich benchmarks, up to half of these errors are semantically acceptable answers penalized purely for surface-form mismatch. This instability is structured by answer type: extractive and multi-span answers are far more evaluator-sensitive than scalar answers. Benign prompt and context rewrites further destabilize official outcomes, flipping item-level correctness at substantial rates without changing the underlying task. A deterministic CPU-only contract repair confirms that the undercount is partially recoverable. These findings imply that official short-answer VQA scores should be accompanied by semantic audits and answer-type diagnostics to remain interpretable.

Figures

Figures reproduced from arXiv: 2607.10240 by the authors.

Figure 1
Figure 1. Evaluator-sensitive undercount. Short-answer VQA benchmarks have driven much of the progress in text-rich multimodal understanding over the past decade. ST-VQA [4] and TextVQA [32] stress scene text, DocVQA [27] and InfographicVQA [26] extend the challenge to documents and infographics, and ChartQA [25] adds visual￾logical reasoning over charts. Despite these differences, they share one evaluation habit inherited fr… view at source ↗
Figure 2
Figure 2. A summarization of benchmark undercount, contract structure, and perturbation-induced instability. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Representative output-side repairs inside the broader bidirectional pipeline. All four examples start as official rejects [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 7 linked inside Pith

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Milli- can, Malcolm Reynolds, et al. 2022. Flamingo: a Visual Language Model for Few-Shot Learning. InAdvances in Neural Information Processing Systems, Vol. 35

    2022

  2. [2]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh

  3. [3]

    InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    VQA: Visual Question Answering. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2425–2433

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Shuai Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025. Qwen2.5-VL Technical Report.arXiv preprint arXiv:2502.13923(2025)

    arXiv 2025

  5. [5]

    Ali Furkan Biten, Rubèn Tito, Andrés Mafla, Lluís Gomez, Marçal Rusiñol, C. V. Jawahar, Ernest Valveny, and Dimosthe- nis Karatzas. 2019. Scene Text Visual Question Answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 4291–4301

    2019

  6. [6]

    Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or LLMs as the Judge? A Study on Judgement Bias. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 8301–8327. doi:10.18653/v1/2024.emnlp-main.474 9

    doi:10.18653/v1/2024.emnlp-main.474 2024

  7. [7]

    Mengshi Chen, Yuxiang Sun, Tengchao Li, Jianwei Wang, Kai Wang, Xuemin Lin, Ying Zhang, and Wenjie Zhang. 2026. Em- powering tabular data preparation with language models: Why and how?. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 228–246

    2026

  8. [8]

    Yihao Ding, Zhe Huang, Runlin Wang, YanHang Zhang, Xianru Chen, Yuzhong Ma, Hyunsuk Chung, and Soyeon Caren Han

Show all 49 references
  1. [9]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    V-Doc: Visual Questions Answers With Documents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 21492–21498

  2. [10]

    Jiantong Du, Yan Li, Yawen Li, Liwen Liao, Zhihao Zhao, and Guanhua Ye. 2026. MedFuse: a multi-source data fusion frame- work for diabetic retinopathy lesion segmentation.Frontiers of Computer Science20, 3 (2026), 2003625

    2026

  3. [11]

    Gemma Team. 2025. Gemma 3 Technical Report.arXiv preprint arXiv:2503.19786(2025)

    2025 arXiv

  4. [12]

    Bravo, and Thomas Brox

    Simon Ging, Maria A. Bravo, and Thomas Brox. 2024. Open- Ended VQA Benchmarking of Vision-Language Models by Ex- ploiting Classification Datasets and Their Semantic Hierarchy. arXiv preprint arXiv:2402.07270(2024)

    2024 arXiv

  5. [13]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. 2024. The Llama 3 Herd of Models.arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  6. [14]

    Zeli Guan, Yawen Li, Zhe Xue, Yuxin Liu, Hongrui Gao, and Yingxia Shao. 2021. Federated graph neural network for cross- graph node classification. In2021 IEEE 7th International Con- ference on Cloud Computing and Intelligent Systems (CCIS). IEEE, 418–422

    2021

  7. [15]

    Huishan Ji, Qingyi Si, Zheng Lin, Yanan Cao, and Weiping Wang

  8. [16]

    InFind- ings of the Association for Computational Linguistics: EMNLP

    Towards One-to-Many Visual Question Answering. InFind- ings of the Association for Computational Linguistics: EMNLP

  9. [17]

    Association for Computational Linguistics, 16931–16943

  10. [18]

    Huishan Ji, Qingyi Si, Zheng Lin, and Weiping Wang. 2024. To- wards Flexible Evaluation for Generative Visual Question Answer- ing. InProceedings of the 32nd ACM International Conference on Multimedia (ACM MM). ACM, 38–47

    2024

  11. [19]

    Chaoya Jiang, Hongrui Jia, Mengfan Dong, Wei Ye, Haiyang Xu, Ming Yan, Ji Zhang, and Shikun Zhang. 2024. Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models. InProceedings of the 32nd ACM International Conference on Multim...

    2024

  12. [20]

    Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2024. Benchmarking Cognitive Biases in Large Language Models as Evaluators. InFindings of the Association for Computational Linguistics: ACL 2024. 517–545. doi:10.18653/v1/2024.findings-acl.29

    2024 doi

  13. [21]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lian- min Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Lan- guage Model Serving with PagedAttention. InProceedings of the ACM SIGOPS 29th Symposium on Operatin...

    2023

  14. [22]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping Language-Image Pre-Training with Frozen Image Encoders and Large Language Models. InProceedings of the International Conference on Machine Learning (ICML)

    2023

  15. [23]

    Wenling Li, Yingmin Jia, Junping Du, and Fashan Yu. 2013. Gaussian mixture PHD filter for multi-sensor multi-target tracking with registration errors.Signal Processing93, 1 (2013), 86–99

    2013

  16. [24]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating Object Hallucination in Large Vision-Language Models. InProceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP)

    2023

  17. [25]

    Yawen Li, Liu Yang, Bohan Yang, Ning Wang, and Tian Wu

  18. [26]

    Application of interpretable machine learning models for the intelligent decision.Neurocomputing333 (2019), 273–283

    2019

  19. [27]

    Yawen Li, Mengyu Zhuang, Guanhua Ye, Yan Li, Junheng Wang, Jinyi Zhou, and Pengfei Zhang. 2025. FeBT: A Feature Balancing Transformer for Corporate ESG Forecasting.IEEE Transactions on Knowledge and Data Engineering(2025)

    2025

  20. [28]

    Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. 2024. Split and Merge: Aligning Position Biases in LLM-based Evaluators. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 11084–11108. doi:10.18...

    2024 doi

  21. [29]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pon- tus Stenetorp. 2022. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (A...

    2022

  22. [30]

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. In Findings of the Association for Computational Linguistics: ACL

    2022

  23. [31]

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V. Jawahar. 2022. InfographicVQA. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 5497–5507

    2022

  24. [32]

    Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. 2021. DocVQA: A Dataset for VQA on Document Images. InProceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2200–2209

    2021

  25. [33]

    Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2020. AmbigQA: Answering Ambiguous Open-Domain Questions. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 5783–5797. doi:10.18653/v1/2020.emnlp-main.466

    2020 doi

  26. [34]

    David Peer, Peter Schopf, Volker Nebendahl, Alexander Rietzler, and Sebastian Stabinger. 2024. ANLS* – A Universal Document Processing Metric for Generative Large Language Models.arXiv preprint arXiv:2402.03848(2024)

    2024 arXiv

  27. [35]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Com- prehension of Text. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2383–2392

    2016

  28. [36]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design with Sensitivity Decomposition. InProceedings of the International Conference on Learning Representations (ICLR)

    2024

  29. [37]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA Models That Can Read. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR). 8317–8326

    2019

  30. [38]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

  31. [39]

    InProceedings of the International Conference on Learning Representations (ICLR)

    Self-Consistency Improves Chain of Thought Reasoning in Language Models. InProceedings of the International Conference on Learning Representations (ICLR)

  32. [40]

    Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. 2022. Retro- MAE: Pre-training retrieval-oriented language models via masked auto-encoder. InProceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing. 538–548

    2022

  33. [41]

    Liang Xu, Junping Du, and Qingping Li. 2013. Image fusion based on nonsubsampled contourlet transform and saliency-motivated pulse coupled neural networks.Mathematical Problems in Engi- neering2013, 1 (2013), 135182

    2013

  34. [42]

    Guanhua Ye, Jifeng He, Yan Li, Junping Du, Zhe Xue, Yingxia Shao, Meiyu Liang, and Yawen Li. 2026. Rethink representation learning for questionnaire data. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 27782–27790

    2026

  35. [43]

    Weld, and Jonathan Berant

    Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel S. Weld, and Jonathan Berant. 2023. Answering Questions by Meta- Reasoning over Multiple Chains of Thought. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). 5942–5966. doi:10...

    2023 doi

  36. [44]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. InProceedings of the 41st International Conference on Machine Learning (ICML). PMLR, 57730–57754

    2024

  37. [45]

    Yuting Zhang, Yi Han, Kai Wang, Wei Ni, Angela Bonifati, and Wenjie Zhang. 2026. Are Large Language Models Suitable for Graph Computation? Progress and Prospects.arXiv preprint arXiv:2606.06865(2026)

    2026 arXiv

  38. [46]

    Xing, et al

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, et al. 2023. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena. InAdvances in Neural Information 10 Processing Systems, Vol. 36

    2023

  39. [47]

    Kun Zhou, Hui Yu, Wayne Xin Zhao, and Ji-Rong Wen. 2022. Filter-enhanced MLP is all you need for sequential recommen- dation. InProceedings of the ACM web conference 2022. 2388– 2399

    2022

  40. [48]

    Mo Zhou, Jianwei Wang, Xuanmeng Zhang, Dylan Campbell, Kai Wang, Long Yuan, Wenjie Zhang, and Xuemin Lin. 2026. ProbDiffFlow: an efficient learning-free framework for probabilistic single-image optical flow estimation.Frontiers of Computer Science20, 8 (2026), 2008342

    2026

  41. [49]

    official reversal

    Jinguo Zhu, Weiyun Chen, Zhe Wang, et al. 2025. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open- Source Multimodal Models.arXiv preprint arXiv:2504.10479 (2025). 11 A Raw-Split Support Two raw benchmark fields directly support the heuristic split. In Text...

    2025 arXiv

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.