Pith. sign in

REVIEW 4 major objections 5 minor 39 references

ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reading text from images costs vision-language models accuracy, and the loss grows sharply with multipage contexts.

desk verdict Useful new benchmark for VLM visual text reading; qualitative trend is credible, but the unvalidated 35-example sampling means the precise numbers should not yet be trusted. read the letter →

arxiv 2505.19091 v1 pith:DA5WZ6BC submitted 2025-05-25 cs.CL cs.AIcs.CVcs.LG

Benjamin Clavié , Florian Brand This is my paper · ORCID
classification cs.CLcs.AIcs.CVcs.LG
keywords vision-languagemodelsmultimodalbenchmarkdensetextreadingdocumentunderstandinglong-contextvisualreasoningtext-imagedegradationresolutionrobustnessretrieval-augmentedgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces ReadBench, a benchmark that turns the context passages of established text-only QA and multiple-choice benchmarks into images of text while keeping questions and prompts as text. Using it on nine current vision-language models, the authors try to establish that reading from images causes universal but uneven performance loss: small on short one-page inputs, sharp on multipage contexts, and largely unaffected by image resolution. They also report that the specific questions on which models lose accuracy differ strongly across models, arguing against a single universal failure trigger. If correct, ReadBench gives a practical measurement of a capability that existing multimodal benchmarks do not isolate: dense text reading from visually presented documents.

What carries the argument

The central object is ReadBench itself: a benchmark constructed by rendering the context passages of five text-only benchmarks (MMLU-Redux, MMLU-Pro, GPQA-Diamond, BABILong, and four LongBench subsets) as images in A4 format at 92.9 PPI with 12-point Arial text, while leaving instructions and questions in text. Short-context benchmarks have only their answer options rendered as images; long-context benchmarks have the entire context rendered across pages, with partially filled pages cropped. Models receive the same prompts in text-only and multimodal conditions, so accuracy differences isolate the cost of reading text visually. Scores are averaged over three runs, and each subset is evaluated on up to 35 sampled examples.

What would settle it

Run the ReadBench evaluation on the full dataset (or a much larger sample) for at least one model and one subset, comparing per-subset accuracy with the 35-example estimate; the central result would be weakened if the full-data degradation is small where the sample showed a large drop, or if the model ranking on degradation changes materially.

Watch

Extended reading notes

Core claim

On ReadBench, every evaluated vision-language model degrades when the context it must read is presented as images instead of text. The degradation is minimal or even slightly negative for short contexts, with GPT-4o and Qwen2.5-VL 7B sometimes matching or beating their text-only scores, but becomes large on inputs of two or more A4 pages, with Gemini 1.5 Pro dropping by around 30 percent relative to text-only performance while GPT-4o degrades the least. Degradation also tracks task difficulty, appearing more strongly on GPQA-Diamond than on MMLU-Redux. Varying the rendering resolution from 72 to 300 PPI changes Gemini 2.0 Flash scores negligibly, and mismatch analysis shows that failures on individual questions are mostly model-specific rather than shared across models.

Load-bearing premise

The benchmark's conclusions rest on the claim that sampling up to 35 examples per subset gives results highly correlated with evaluating the full dataset, but the correlation experiment is not shown in the paper.

Editorial extensions

If this is right

  • For short documents of up to about one page, current vision-language models can answer from rendered text almost as reliably as from plain text, so image-based reading is a viable interface at that scale.
  • For multipage visual contexts, no evaluated model currently maintains text-level accuracy, so applications that feed scanned or rendered documents to VLMs should budget for a substantial quality loss.
  • Raising image resolution will not recover that loss, since performance is roughly flat from 72 to 300 PPI, at least for Gemini 2.0 Flash.
  • Because model failures on multimodal inputs are largely model-specific, combining or routing among vision-language models may recover more accuracy than any single-model fix.
  • ReadBench provides a reusable measurement protocol for tracking whether future vision-language models close this gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the multipage drop reflects a visual-reading limit or the model's overall context-window mechanics; reproducing the same page count as token-heavy text in a text-only baseline would separate these.
  • A natural extension is to test whether the same degradation appears when the images are screenshots of real PDFs rather than cleanly rendered A4 pages, since document noise and layout variation may interact with the resolution findings.
  • The sampling assumption (35 examples per subset correlate with the full dataset) is stated but not demonstrated; rerunning one subset at full size would settle whether the reported ranking and degradation magnitudes are stable.
  • The mismatch results suggest that a model ensemble could be tested against single models on ReadBench; the paper does not evaluate that possibility.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ReadBench, a benchmark that converts contexts from five text-only benchmarks (MMLU-Redux, MMLU-Pro, GPQA-Diamond, BABILong, and four LongBench subsets) into images of text while keeping prompts and questions in text form. The authors evaluate nine current VLMs on this mixed-modality format and report universal performance degradation relative to text-only baselines, with small drops on short contexts but sharp declines on multi-page inputs. They also report a mismatch analysis suggesting that failures are largely model-specific, and a resolution ablation on a single model suggesting that PPI has negligible effect. The benchmark and evaluation code are released publicly.

Significance. If the central claims are validated, ReadBench fills a genuine gap: most existing multimodal benchmarks focus on visual reasoning or OCR, whereas ReadBench isolates the ability to read and reason over dense textual documents, which is directly relevant to document QA and VisRAG applications. The paper's strengths are its transparent construction from permissively licensed standard benchmarks, the evaluation across nine models of different sizes and families, the public release of data and code, and the explicit acknowledgment of limitations in a dedicated section. However, the validity of the quantitative conclusions rests on a sampling strategy whose stated validation is not shown, and on a resolution ablation from one model only. These issues are load-bearing because every reported degradation magnitude, model ordering, and overlap statistic inherits them.

major comments (4)
  1. [Section 2.3] The claim that sampling up to 35 examples per subset "yields results highly correlated with evaluating the full dataset" is unsupported: no correlation analysis, plot, or numeric comparison with full-set evaluations is provided. For a binary metric, n=35 gives a standard error of up to roughly 8.5 percentage points at p=0.5, so the three-run averaging reported in Section 3 reduces only run-to-run variance, not sampling variance. As a consequence, the headline numbers (e.g., "around 30%" relative degradation for Gemini 1.5 Pro, "GPT-4o fares the best", the difficulty-correlation claim, and the 20–35% Jaccard overlap in Figure 4a) may be seed effects. Please either provide the full-set comparison, or add confidence intervals/bootstrap resampling over sampled questions for all reported aggregates and demonstrate that the model ordering is stable across random seeds.
  2. [Section 2.2.1 / Figure 2] The conclusion that text resolution has negligible effects on multimodal performance is drawn entirely from one model, Gemini 2.0 Flash, across PPI settings. Since the main benchmark evaluation spans nine models with different vision encoders and input-resolution handling, the abstract and conclusion overgeneralize this finding. Please extend the resolution ablation to at least a few representative models (e.g., one open-weight and one additional closed-weight model), or explicitly restrict the claim to Gemini 2.0 Flash and rephrase the abstract/conclusion accordingly.
  3. [Section 3] The paper's central claim of "universal performance degradation" is not literally supported by its own data for short contexts: the text reports that GPT-4o and Qwen2.5-VL 7B maintain or slightly improve on multimodal GPQA. Given the small sample sizes and absence of error bars in Figure 3, the claim needs a statistical qualifier (e.g., "on average, most models degrade on most subsets") or explicit per-subset significance tests. Please add error bars or confidence intervals to Figure 3 and report the per-subset, per-model differences with uncertainty rather than aggregate trend lines alone.
  4. [Section 4 / Figure 4] The mismatch analysis is computed on the same 35-example subsets as the main results, and the pairwise Jaccard similarities (20–35%) and the 3.1% statistic for questions affecting more than five models are presented without any uncertainty quantification. With sampling noise of the magnitude described above, the conclusion that failures "primarily arise from model-specific challenges" rather than universally problematic inputs may be an artifact of the subset. Please report confidence intervals for the Jaccard values (e.g., via bootstrap over sampled questions) or validate the overlap structure on a full-set comparison before drawing this qualitative conclusion.
minor comments (5)
  1. [Section 1] Typo: "examplified" should be "exemplified".
  2. [Acknowledgements] Typo: "reproduceable" should be "reproducible".
  3. [Appendix A.4] Typo in the NarrativeQA prompt: "asconcisely" should be "as concisely" (two occurrences).
  4. [Figure 2] The y-axis label is "multi-modal scores" but the metric is not defined; please state whether this is accuracy, exact-match, or the binary overlap metric used elsewhere, and add run details or error bars.
  5. [Section 2.1.2] The phrase "We sample from all ten questions it contains" for BABILong is ambiguous; these are ten question templates, not ten fixed questions. Please rephrase to make the sampling unit clear.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ReadBench's degradation findings are empirical measurements against external text-only baselines, not derivations that reduce to their own inputs.

full rationale

ReadBench's central claims are measurements, not derivations from fitted parameters. The benchmark transposes contexts from established text-only datasets (MMLU-Redux, MMLU-Pro, GPQA-Diamond, BABILong, LongBench) into images while keeping prompts and questions textual; the text-only baseline is the same model on the same item in its original modality. No quantity used in the headline findings is defined in terms of the finding itself. The only assertion that could look like an unsupported calibration is Section 2.3's statement that 35 sampled examples "experimentally found to yield results highly correlated with evaluating the full dataset," since no correlation analysis is shown. But that is a sampling-design claim, not a circular derivation: the sample size is not a parameter fitted to the degradation numbers, and the paper does not use a full-set result as an input to predict itself. Concerns about representativeness, standard errors, and seed effects belong to correctness and statistical robustness, not to circularity. There is also no load-bearing self-citation: the paper cites external datasets and model reports, and its conclusions are independently checkable against the released evaluation suite. Therefore no circular step is identified; the benchmark construction is self-contained with respect to its conclusions.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims depend on several domain assumptions about rendering fidelity and sample representativeness, plus two ad hoc choices (35-sample subset, single-model resolution ablation) that the paper does not fully justify.

free parameters (4)
  • sample_size_per_subset = 35
    Authors say they 'experimentally found this size to yield results highly correlated with evaluating the full dataset', but the correlation experiment is not shown; this hand-chosen size bounds the precision of every headline claim.
  • pixels_per_inch = 92.9
    Design choice mimicking common screen resolutions; resolution is later claimed to have negligible effect based on a one-model ablation, so the value is a fixed rendering parameter.
  • font_size_pt = 12
    Chosen per W3C accessibility recommendations; affects text readability and thus the measured degradation.
  • max_token_bin = 8k
    Long-context subsets truncated to 8k tokens, averaging 12 rendered pages; limits the generality of the long-context finding.
assumptions (4)
  • domain assumption Rendering A4 text at 92.9 PPI with Arial 12pt approximates real-world documents closely enough for benchmark conclusions.
    Section 2.2 states this mimics real-world document scenarios; the resolution-negligibility claim rests on this rendering fidelity.
  • ad hoc to paper 35 random examples per subset are representative of each full benchmark.
    Section 2.3 asserts this without showing the correlation experiment; the claim is load-bearing for all degradation statistics.
  • domain assumption Averaging 3 runs reduces variance sufficiently to support cross-model comparisons.
    Section 3 states results are averages of 3 runs, but no variance or significance is reported.
  • ad hoc to paper Resolution effects on one model (Gemini 2.0 Flash) generalize to all evaluated VLMs.
    Section 2.2.1 draws a global conclusion from a single-model ablation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models." pith.science (2026). https://pith.science/paper/DA5WZ6BC

@misc{pith2026250519091,
  author       = {Pith},
  title        = {Pith review of: ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DA5WZ6BC}},
  note         = {Machine review of arXiv:2505.19091}
}
read the original abstract

Recent advancements in Large Vision-Language Models (VLMs), have greatly enhanced their capability to jointly process text and images. However, despite extensive benchmarks evaluating visual comprehension (e.g., diagrams, color schemes, OCR tasks...), there is limited assessment of VLMs' ability to read and reason about text-rich images effectively. To fill this gap, we introduce ReadBench, a multimodal benchmark specifically designed to evaluate the reading comprehension capabilities of VLMs. ReadBench transposes contexts from established text-only benchmarks into images of text while keeping textual prompts and questions intact. Evaluating leading VLMs with ReadBench, we find minimal-but-present performance degradation on short, text-image inputs, while performance sharply declines for longer, multi-page contexts. Our experiments further reveal that text resolution has negligible effects on multimodal performance. These findings highlight needed improvements in VLMs, particularly their reasoning over visually presented extensive textual content, a capability critical for practical applications. ReadBench is available at https://github.com/answerdotai/ReadBench .

Figures

Figures reproduced from arXiv: 2505.19091 by the authors.

Figure 1
Figure 1. An example MMLU-Redux converted input format, converting contexts into images while pre￾serving textual prompts and questions. Evaluating state-of-the-art VLMs, we observe universal per￾formance degradation in multimodal versus purely textual scenarios. While performance degradation varies across models, the general trend shows that it is minimal for short inputs but significant for multi-page contexts, often causin… view at source ↗
Figure 3
Figure 3. Performance degradation overview across datasets for all models [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Consistency of multimodal–text disagreements across models. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 16 canonical work pages

  1. [1]

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, and 1 others. 2024. Pixtral 12b. arXiv preprint arXiv:2410.07073

    arXiv 2024

  2. [2]

    Artificial Analysis. 2024. https://x.com/ArtificialAnlys/status/1859614633654616310 Wait - is the new gpt-4o a smaller and less intelligent model? X.com. Accessed: 2025-05-06

    work page arXiv 2024

  3. [3]

    Maliha Ashraf, Alexandre Chapiro, and Rafa K Mantiuk. 2024. Resolution limit of the eye: how many pixels can we see? arXiv preprint arXiv:2410.06068

    work page arXiv 2024

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

    arXiv 2025

  5. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, and 1 others. 2024. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119--3137

    work page 2024

  6. [6]

    Lucas Beyer. 2024. On the speed of ViTs and CNNs . http://lb.eyer.be/a/vit-cnn-speed.html

    work page 2024

  7. [7]

    Lucas Beyer, Andreas Steiner, Andr \'e Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, and 1 others. 2024. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726

    arXiv 2024

  8. [8]

    BIA . 2023. Accessibility tips: Let users control font size. https://www.boia.org/blog/accessibility-tips-let-users-control-font-size. Accessed on 2025-05-13

    work page 2023

Show all 39 references
  1. [9]

    ukasz Borchmann, Micha Pietruszka, Tomasz Stanislawek, Dawid Jurkiewicz, Micha Turski, Karolina Szyndler, and Filip Grali \'n ski. 2021. Due: End-to-end document understanding benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmark...

    2021

  2. [10]

    Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics

    2017

  3. [11]

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, and 1 others. 2024. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of ...

    2024

  4. [12]

    Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, C \'e line Hudelot, and Pierre Colombo. 2024. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations

    2024

  5. [13]

    Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini...

    2025 arXiv

  6. [14]

    Team Gemini, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    2024 arXiv

  7. [15]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. http://dblp.uni-trier.de/db/conf/iclr/iclr2021.html#HendrycksBBZMSS21 Measuring massive multitask language understanding. In ICLR. OpenReview.net

    2021

  8. [16]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. https://doi.org/10.18653/v1/2020.coling-main.580 Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps . In Proceedings of the 28th International Conference on Computational Li...

    2020 doi

  9. [17]

    HuggingFace. 2024. Multimodal RAG using document retrieval, reranker and VLMs . https://huggingface.co/learn/cookbook/en/multimodal_rag_using_document_retrieval_and_reranker_and_vlms. Accessed on 2025-05-06

    2024

  10. [18]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  11. [19]

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904...

    2021

  12. [20]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ass...

    2017

  13. [21]

    Greg Kamradt. 2023. LLMTest\_NeedleInAHaystack: Doing simple retrieval from LLM models at various context lengths to measure accuracy . https://github.com/gkamradt/LLMTest_NeedleInAHaystack

    2023

  14. [22]

    Tom \'a s Ko c isk \`y , Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, G \'a bor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317--328

    2018

  15. [23]

    Yury Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. Advances in Neural Information Processing Systems, 37:106519--106554

    2024

  16. [24]

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. 2024. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):220102

    2024

  17. [25]

    Zhiheng Lyu, Xueguang Ma, and Wenhu Chen. 2025. Pixelworld: Towards perceiving everything as pixels. arXiv preprint arXiv:2501.19339

    2025

  18. [26]

    Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. 2024. Unifying multimodal retrieval via document screenshot embedding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6492--6505

    2024

  19. [27]

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200--2209

    2021

  20. [28]

    Microsoft . 2009. Pixel density and usability. https://learn.microsoft.com/en-us/previous-versions/windows/desktop/mpc/pixel-density-and-usability. Accessed: 2025-05-13

    2009

  21. [29]

    Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A Rossi, Seunghyun Yoon, and Hinrich Sch \"u tze. 2025. Nolima: Long-context evaluation beyond literal matching. arXiv preprint arXiv:2502.05167

    2025 arXiv

  22. [30]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine...

    2021

  23. [31]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling (CoLM)

    2024

  24. [32]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317--8326

    2019

  25. [33]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574

    2024 arXiv

  26. [34]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language ...

    2018

  27. [35]

    Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, and 1 others. 2024. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. arXiv preprint arXiv:2410.10594

    2024 arXiv

  28. [36]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Co...

    2024

  29. [37]

    Junyuan Zhang, Qintong Zhang, Bin Wang, Linke Ouyang, Zichen Wen, Ying Li, Ka-Ho Chow, Conghui He, and Wentao Zhang. 2024. Ocr hinders rag: Evaluating the cascading impact of ocr on retrieval-augmented generation. arXiv preprint arXiv:2412.02592

    2024 arXiv

  30. [38]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.