Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

VARCO-VISION: Expanding Frontiers in Korean Vision-Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An open 14B Korean-English vision-language model claims to beat same-size open models on its own translated Korean benchmarks and to match much larger proprietary systems, while keeping its language scores intact.

desk verdict Honest, useful resource-paper: releases a 14B Korean-English VLM and five Korean benchmarks, but the headline performance claims on the author-built Korean benchmarks still need an independent yardstick. read the letter →

arxiv 2411.19103 v1 pith:F4IT654T submitted 2024-11-28 cs.CV cs.CL

Jeongho Ju , Daeyoung Kim , SunYoung Park , Youngjune Kim This is my paper · ORCID
classification cs.CVcs.CL
keywords Koreanvision-languagemodelbilingualmultimodalfour-stagetrainingDirectPreferenceOptimizationOCRgroundingandreferringbenchmarksopen-sourceVLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VARCO-VISION-14B is an open-weight bilingual vision-language model that the authors train in four stages so it can handle Korean and English images and text without losing the language ability it starts with. The paper argues that adding visual skills gradually — feature alignment first, then broad instruction tuning, then harder reasoning, then preference optimization — lets one 14B model do visual question answering, OCR, grounding, and referring in both languages. To support this claim, the authors translate three English multiple-choice benchmarks and one open-ended generation benchmark into Korean, human-edit the translations, and build a new document, table, and chart benchmark from scratch. On these released benchmarks the model reports the highest scores among open models of similar size, and on several closed-set tests it is competitive with much larger open models and with proprietary APIs. The paper also releases the five Korean evaluation datasets themselves, so future models can be compared on the same ground.

What carries the argument

The load-bearing mechanism is a four-stage training schedule that progressively unfreezes the model. Stage 1 trains only the randomly initialized MLP projector on image-caption pairs; Stage 2 unfreezes all layers and fine-tunes on six task families including OCR, grounding, referring, captioning, document/table/chart/math, and text-only examples; Stage 3 does the same with harder reasoning tasks; Stage 4 applies Direct Preference Optimization, a method that pushes the model toward preferred responses, to the LLM layers only to improve fluency and alignment. Special tokens for grounding, OCR, text phrases, objects, bounding boxes, and multiple locations let the model emit structured outputs for referring and grounding in both languages. The argument is that staged unfreezing plus the periodic return to text-only data preserves the text backbone's knowledge, while the preference-optimization pass improves generation quality.

What would settle it

Have a fresh Korean-speaking panel, blind to the answer keys, re-answer a stratified sample of K-SEED and K-MMStar questions from the images, and replace the automatic judge for K-LLaVA-W with human raters; if a multilingual open VLM trained only on English multimodal data scores near chance on the Korean translations, or if human rankings disagree with the automatic judge's rankings, the reported Korean gains would not be reliable evidence of a Korean-specific capability.

Watch

Extended reading notes

Core claim

The paper's central claim is that a 14B open-weight VLM can be genuinely bilingual in Korean and English while remaining strong at text-only language tasks. The authors report that VARCO-VISION-14B tops all compared open models under 20B on K-MMBench (82.21), K-SEED (75.39), K-MMStar (57.33), and the new K-DTCBench (84.58), and reaches 84.74 on K-LLaVA-W, an open-ended generation benchmark scored by a large language model judge. On English closed-set tests it also beats the under-20B comparisons on MMBench, SEED, and MMStar, although not on MMMU, and it scores 820 on OCRBench, above every compared VLM including 72B models. Its text-only scores (8.69 on LogicKor, 8.39 on KoMT-Bench, 8.80 on MT-Bench) are the highest among compared VLMs and close to those of a dedicated bilingual language model. The authors attribute this balanced profile to the four-stage recipe and, in particular, to interspersing text-only data through Stages 2-4 and applying preference optimization in the final stage.

Load-bearing premise

The load-bearing premise is that the new Korean evaluation datasets measure real model ability rather than rewarding quirks of the translation, the question selection, or the judge model used to score free-form answers.

Editorial extensions

If this is right

  • A 14B open model can serve many Korean-English multimodal applications without relying on a proprietary API.
  • Researchers get five standardized Korean benchmarks for closed-set and open-ended evaluation, so future Korean VLM comparisons need not be built from scratch.
  • Training recipes that interleave text-only data can preserve or even improve a VLM's language scores while adding visual skills.
  • The same staged approach could be used to add OCR, grounding, and referring abilities to other language-specific VLMs.
  • The model's OCR and document/table/chart performance suggests it is usable for digitizing Korean documents and form-like images, not just answering image questions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same four-stage recipe with text-only data interspersed may transfer to other under-resourced languages; nothing in the paper ties the mechanism to Korean specifically.
  • Inference: the large gap between VARCO-VISION and other open models on K-DTCBench could indicate that cheaply generated synthetic Korean documents, tables, and charts are a high-leverage training signal for non-English document understanding.
  • Inference: an ablation that removes the text-only stages or the DPO stage would isolate exactly which component preserves the language scores; the paper does not report such an ablation.
  • Inference: because the same commercial API used for translation is also used for reference answers and judging, the K-LLaVA-W scores may partly reflect stylistic similarity to that API's writing rather than a separate notion of Korean quality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces VARCO-VISION-14B, an open-source Korean-English vision-language model built on Qwen-2.5-14B-Instruct and SigLIP, and trained in four stages (feature alignment pre-training, basic supervised fine-tuning, advanced supervised fine-tuning, and DPO) with text-only data interspersed. The authors also release five Korean evaluation benchmarks: K-MMBench, K-SEED, K-MMStar, K-DTCBench, and K-LLaVA-W. They report that the model outperforms similar-scale open-source models on Korean and English MCQA benchmarks, text-only benchmarks, and OCR tasks, and that it approaches proprietary model performance on several benchmarks.

Significance. If the results hold, the paper makes a valuable contribution by providing a strong open bilingual VLM and a set of Korean multimodal benchmarks that were previously lacking. The released model and datasets enable further research on Korean vision-language models. The finding that interleaving text-only data preserves language ability while adding vision skills is a practically useful training recipe, and the English benchmark results and OCR and grounding demonstrations support the effectiveness of the staged training. However, the validity of the newly constructed Korean benchmarks is not yet established, which tempers the significance of the cross-lingual comparisons.

major comments (4)
  1. [Section 3.1.1 (K-SEED)] The selection of the first 20 percent of SEED-Bench image questions is arbitrary and may not preserve the ability distribution of the full benchmark. The paper does not justify this subset or show that it measures the same dimensions with similar difficulty. To make the K-SEED scores interpretable, the authors should compare per-dimension performance of several baselines on the selected subset versus the full English SEED (or the full translated Korean set), and ideally provide the exact subset selection criterion.
  2. [Section 3.1.1 and Appendix A.1 (K-MMStar)] The modification or re-creation of MMStar items (Types 2 and 3) introduces a potential bias that is not quantified. No per-item diff or counts of Type 1/2/3 items are released, and the human post-editing process is not described with inter-annotator agreement. Because these edits were made by the authors, they could align with the model's training distribution. The authors should release the full list of changed items with original and Korean versions, report the distribution of the three types, and evaluate all baselines on the corresponding original English items to check whether the relative ranking changes between the English and Korean versions.
  3. [Section 3.1.1 (K-DTCBench)] K-DTCBench is a fully synthetic, author-constructed benchmark for document, table, and chart understanding, which are exactly the skills targeted in Stage 2 of training. The large margin over Qwen2-VL-7B (84.58 vs 75.00) could reflect overlap with the training data rather than generalizable capability. The authors should provide external validation, for example by evaluating on an independent Korean document/chart benchmark, or at least show per-image-type scores and demonstrate that the synthetic images are not similar to training examples. Without this, the headline superiority claim on K-DTCBench is not robust.
  4. [Tables 1 and 2 (and Table 3)] The headline comparisons lack any measure of uncertainty. For instance, K-DTCBench has only 240 questions, and several gaps such as K-SEED (75.39 vs 74.08 over Qwen2-VL-7B) are within expected sampling noise. The authors should report binomial standard errors or bootstrap confidence intervals for all MCQA scores and specify the number of questions per benchmark. This is particularly important because the paper's central claim of 'outstanding performance' rests on differences that may not be statistically distinguishable.
minor comments (6)
  1. [Section 1] The claim of being the 'first to release open-source Korean evaluation benchmarks' is too strong given that the paper itself cites existing Korean VQA and OCR datasets (references [12] and [13]). Consider rephrasing to 'first general-purpose Korean multimodal comprehension benchmarks of this scope.'
  2. [Section 2.1] The architecture description states that training focuses on single-image examples but does not explain why multi-image support is excluded; a short note on this design choice would help readers understand the scope and limitations.
  3. [Section 3.1.2] The K-LLaVA-W evaluation uses GPT-4o as both the judge and the baseline model. This coupling should be acknowledged as a potential source of bias in the ratio score, and the paper should report the raw JudgeLLM scores for both the target model and the baseline rather than only the ratio.
  4. [Table 3] The LogicKor and KoMT-Bench scores are based on LLM-as-a-judge, which is stochastic, but no standard deviations or multiple-run results are reported. Please provide error bars or a statement about the variance of the judge.
  5. [Appendix A.3.2] The JudgeLLM prompt is provided only in Korean; an English translation should be included for the accessibility of non-Korean readers and for reproducibility.
  6. [Figure 7] In the mathematical reasoning example, the sector area formula is written as 'π × r' and the squared exponent is missing in the displayed text, although the calculation uses r². Please correct the inconsistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: performance claims are empirical and benchmarked against external and translated benchmarks; author-constructed Korean benchmarks raise validity concerns but do not create a self-referential reduction.

full rationale

The paper's derivation chain is a training recipe followed by empirical evaluation, not a mathematical derivation from fitted parameters. No benchmark score is fed back as an input to the training procedure, and no equation defines the reported performance in terms of the authors' own construction choices. The English evaluations (MMBench, SEED, MMStar, MMMU, OCRBench) are externally established benchmarks, and the Korean closed-set benchmarks are translations of externally rooted English datasets (K-MMBench, K-SEED, K-MMStar, K-LLaVA-W), with K-DTCBench as the only fully synthetic, author-created instrument. The authors' modifications to K-MMStar and the synthetic design of K-DTCBench are legitimate threats to benchmark validity and generalization, but they are not circularity: the benchmark scores are reported measurements, not predictions derived from the benchmark definitions themselves. The K-LLaVA-W evaluation uses an external GPT judge and an external baseline, further separating the model's score from the authors' fitted inputs. Architectural adoption from LLaVA-OneVision is stated explicitly and is not used to smuggle in an ansatz or to import a uniqueness result. There are no load-bearing self-citations. The central claim of bilingual performance is supported by external and semi-external evaluations, so no circular step can be exhibited.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper contains no formal derivation, so the ledger records benchmark-design choices and evaluation assumptions that the performance claims depend on. The model has no fitted constants beyond ordinary learned weights; the empirical numbers rest on translation fidelity, question curation, and LLM judging. No new physical or conceptual entities are introduced.

free parameters (2)
  • K-SEED subset fraction = first 20% of image questions
    Section 3.1.1: only the first 20 percent of SEED-Bench image questions are included; the choice is arbitrary and not justified, and it can affect category coverage and difficulty.
  • K-DTCBench composition = 240 questions; 50% digital, 50% handwritten
    Section 3.1.1: the benchmark is synthesized by the authors with made-up values and an equal digital/handwritten split; this design choice is unvalidated externally.
assumptions (3)
  • domain assumption Translated benchmarks preserve the validity and difficulty of the original English benchmarks.
    Section 3.1.1: translations are produced with GPT-4o and human post-editing, but no validation study compares score distributions or difficulty against the English originals.
  • domain assumption Human curation of K-MMStar produces correct labels for modified and re-created questions.
    Appendix A.1: questions are modified or re-created when unanswerable or vague, with no inter-annotator agreement or independent label verification reported.
  • domain assumption GPT-4o as JudgeLLM is a valid measure of Korean generation quality relative to a GPT-4o baseline.
    Section 3.1.2 and Appendix A.3.2: K-LLaVA-W scores depend on GPT-4o judging helpfulness, relevance, accuracy, detail, and Korean generation; judge bias and prompt-order effects are not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VARCO-VISION: Expanding Frontiers in Korean Vision-Language Models." pith.science (2026). https://pith.science/paper/F4IT654T

@misc{pith2026241119103,
  author       = {Pith},
  title        = {Pith review of: VARCO-VISION: Expanding Frontiers in Korean Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F4IT654T}},
  note         = {Machine review of arXiv:2411.19103}
}
read the original abstract

In this paper, we introduce an open-source Korean-English vision-language model (VLM), VARCO-VISION. We incorporate a step-by-step training strategy that allows a model learn both linguistic and visual information while preserving the backbone model's knowledge. Our model demonstrates outstanding performance in diverse settings requiring bilingual image-text understanding and generation abilities compared to models of similar size. VARCO-VISION is also capable of grounding, referring, and OCR, expanding its usage and potential applications for real-world scenarios. In addition to the model, we release five Korean evaluation datasets, including four closed-set and one openset benchmarks. We anticipate that our milestone will broaden the opportunities for AI researchers aiming to train VLMs. VARCO-VISION is available at https://huggingface.co/NCSOFT/VARCO-VISION-14B.

Figures

Figures reproduced from arXiv: 2411.19103 by the authors.

Figure 1
Figure 1. VARCO-VISION Application Examples: Visual Question Answering (VQA), Optical Character Recognition (OCR), Referring, and Grounding. Our model excels at both Korean/English vision-text and text-only tasks. Please see B for more detailed examples. The Korean closed-set benchmarks, K-MMBench, K-SEED, and K-MMStar, are multiple-choice question answering (MCQA) tasks, which allow objective evaluation of MLLMs. K-LLaVA-W i… view at source ↗
Figure 2
Figure 2. K-MMStar Example K-MMStar has three different types of questions. We noticed that there are unanswerable or vague questions in the original MMStar, thus modified the question or created a new one [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. K-DTCBench Example A.3 K-LLaVA-W A.3.1 Example In K-LLaVA-W, we changed the English text into Korean text for images with texts. If an original LLaVA-W image did not contain any text, we left it unchanged to preserve its authenticity. In [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: K-LLaVA-W Example 13 [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: K-LLaVA-W Evaluation Prompt. We translated the LLaVA-W prompts and added specific [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Text Recognition and Analysis Example (English) [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Mathematical Reasoning Example (Korean) 16 [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Information Extraction and Calculation Example (Korean) [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Grounding Example (Korean) 18 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Grounding Example (English) 19 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Referring Example (Korean) 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: OCR Example 21 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: OCR Example 22 [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Summarization Example (English) 23 [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Text Recognition Example (Korean) 24 [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts

    cs.CV 2025-08 conditional novelty 6.0 of 10

    KRETA, a 2,577-item Korean text-rich VQA benchmark, shows vision-language models recognize Korean text well but lag in multi-step reasoning, especially in open-source models.

Reference graph

Works this paper leans on

53 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 , 2024

    arXiv 2024

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Pixtral 12b

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024

    arXiv

  4. [4]

    Claude 3.5 sonnet, 2024

    Anthropic. Claude 3.5 sonnet, 2024. URL https://www.anthropic.com/news/ claude-3-5-sonnet

    2024

  5. [5]

    Are we on the right way for evaluating large vision-language models? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=evP9mxNNxJ

    work page 2024

  6. [6]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821 , 2024

    arXiv 2024

  7. [7]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2024

    arXiv 2024

  8. [8]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models, 2024

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models, 2024. URL https://arxiv.org/abs/ 2407.11691

    arXiv 2024

Show all 53 references
  1. [9]

    Pororo: Platform of neural models for natural language processing

    Hoon Heo, Hyunwoong Ko, Soohwan Kim, Gunsoo Han, Jiwoo Park, and Kyubyong Park. Pororo: Platform of neural models for natural language processing. https://github.com/ kakaobrain/pororo, 2021. 8

    2021

  2. [10]

    Icdar 2013 robust reading competition

    Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Big- orda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. Icdar 2013 robust reading competition. In 2013 12th international conference ...

    2013

  3. [11]

    Icdar 2015 competition on robust reading

    Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. In 2015 13th international conference on docume...

    2015

  4. [12]

    Ocr-free document understanding transformer

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Conference on Computer Vision (ECCV), 2022

    2022

  5. [13]

    Korean Localization of Visual Question Answering for Blind People

    Jin-hwa Kim, Soohyun Lim, Jaesun Park, and Hansu Cho. Korean Localization of Visual Question Answering for Blind People. In AI for Social Good workshop at NeurIPS , 2019

    2019

  6. [14]

    Building and bet- ter understanding vision-language models: insights and future directions

    Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and bet- ter understanding vision-language models: insights and future directions. arXiv preprint arXiv:2408.12637, 2024

    2024 arXiv

  7. [15]

    Popeval: A character-level approach to end-to-end evaluation compatible with word-level benchmark dataset

    Hong-Seok Lee, Youngmin Yoon, Pil Hoon Jang, and Chankyu Choi. Popeval: A character-level approach to end-to-end evaluation compatible with word-level benchmark dataset. In 2019 International Conference on Document Analysis and Recognition (ICDAR) , pages 1207–1213. IEEE, 2019

    2019

  8. [16]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  9. [17]

    Seed-bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308, 2024

    2024

  10. [18]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024

    2024 arXiv

  11. [19]

    A survey on benchmarks of multimodal large language models, 2024

    Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, Ying Tai, Wankou Yang, Yabiao Wang, and Chengjie Wang. A survey on benchmarks of multimodal large language models, 2024. URL https://arxiv.org/abs/ 2408.08632

    2024 arXiv

  12. [20]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc., 2023. U...

    2023

  13. [21]

    Visual instruction tuning.Advances in neural information processing systems , 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems , 36, 2024

    2024

  14. [22]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2025

    2025

  15. [23]

    On the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895, 2023

    2023 arXiv

  16. [24]

    Deepseek-vl: towards real-world vision-language under- standing

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language under- standing. arXiv preprint arXiv:2403.05525, 2024. 9

    2024 arXiv

  17. [25]

    Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models, 2024

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models, 2024. URL https://arxiv.org/abs/2403.03003

    2024 arXiv

  18. [26]

    Cord: A consolidated receipt dataset for post-ocr parsing

    Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwal- suk Lee. Cord: A consolidated receipt dataset for post-ocr parsing. In Document Intelligence Workshop at Neural Information Processing Systems, 2019

    2019

  19. [27]

    Kosmos-2: Grounding multimodal large language models to the world, 2023

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world, 2023. URL https://arxiv.org/abs/2306.14824

    2023 arXiv

  20. [28]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024

    2024

  21. [29]

    Exaone 3.0 7.8 b instruction tuned language model

    LG Research, Soyoung An, Kyunghoon Bae, Eunbi Choi, Stanley Jungkyu Choi, Yemuk Choi, Seokhee Hong, Yeonjung Hong, Junwon Hwang, Hyojin Jeon, et al. Exaone 3.0 7.8 b instruction tuned language model. arXiv preprint arXiv:2408.03541, 2024

    2024

  22. [30]

    X-LLaV A: Optimizing bilingual large vision-language alignment

    DongJae Shin, HyeonSeok Lim, Inho Won, ChangSu Choi, Minjun Kim, SeungWoo Song, HanGyeol Yoo, SangMin Kim, and KyungTae Lim. X-LLaV A: Optimizing bilingual large vision-language alignment. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Findings of the Association for ...

    2024 doi

  23. [31]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  24. [32]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https:// qwenlm.github.io/blog/qwen2.5/

    2024

  25. [33]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  26. [34]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  27. [35]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on C...

    2024

  28. [36]

    Pangea: A fully open multilingual multimodal llm for 39 languages

    Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neu- big. Pangea: A fully open multilingual multimodal llm for 39 languages. arXiv preprint arXiv:2410.16153, 2024

    arXiv

  29. [37]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023

    2023

  30. [38]

    Lmms-eval: Reality check on the evaluation of large multimodal models, 2024

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024. URL https://arxiv.org/abs/2407.12772

    2024 arXiv

  31. [39]

    Swift:a scal- able lightweight infrastructure for fine-tuning, 2024

    Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swift:a scal- able lightweight infrastructure for fine-tuning, 2024. URL https://arxiv.org/abs/2408. 05517. 10

    2024

  32. [40]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36:46595–46623, 2023

    2023

  33. [41]

    축구” (soccer) and “미식축구

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 3: System...

    2024 arXiv

  34. [42]

    각 어시스턴트의 응답을 다음 기준으로 분석하세요: - `유용성`: 응답이 사용자의 질문을 얼마나 잘 해결하는가? - `관련성`: 응답이 사용자의 질문에 얼마나 적절한가? - `정확성`: 응답에서 제공한 정보가 얼마나 정확한가? - `세부 수준`: 응답이 과하지 않게 충분히 자세한가? - `한국어 생성능력`: 생성된 한국어 문장이 자연스럽고 문법적으로 올바른가?

  35. [43]

    분석에 기반하여 각 어시스턴트에게 1에서 10까지의 점수를 부여하세요

  36. [44]

    두 점수를 공백으로 구분하여 한 줄로 제공하세요

  37. [45]

    # 출력 형식 - 첫 번째 줄: `어시스턴트1_점수 어시스턴트2_점수` (예: `8 9`) - 두 번째 줄: `유용성`, `관련성`, `정확성`, `세부 수준`, `한국어 생성능력` 기준으로 점수를 설명 하는 자세한 문단을 제공합니다

    점수에 대한 이유를 강조하면서 포괄적인 평가를 제공하고, 편견을 피하며 응답의 순서가 판단에 영향을 미치지 않도록 하세요. # 출력 형식 - 첫 번째 줄: `어시스턴트1_점수 어시스턴트2_점수` (예: `8 9`) - 두 번째 줄: `유용성`, `관련성`, `정확성`, `세부 수준`, `한국어 생성능력` 기준으로 점수를 설명 하는 자세한 문단을 제공합니다. # 주의사항 - 평가 시 잠재적 편견을 방지하여 객관성을 확보하세요. - 분석과 설명에서 일관성과 명확성을 유지하세요. Figure 5:...

  38. [46]

    전체 도형을 분석합니다 중심각이 90°인 부채 반지름이 4cm인 원의 일부

  39. [47]

    부채꼴의 넓이를 계산합니다 부채꼴의 면적 공식: (중심각/360) × π × r 여기서 r = 4cm, 중심각 = 90 계산: (90/360) × π × 4² = 4π cm²

  40. [48]

    삼각형의 넓이를 계산합니다 밑변 = 4cm, 높이 = 4c 삼각형의 넓이 공식: (밑변 × 높이) / 계산: (4 × 4) / 2 = 8 cm²

  41. [49]

    색칠한 부분의 넓이를 구합니다 부채꼴 넓이 - 삼각형 넓 4π cm² - 8 cm²

  42. [50]

    최종 답을 계산합니다 4π - 8 ≈ 4 × 3.14 - 8 = 12.56 - 8 = 4.56 cm² 따라서 색칠한 부분의 넓이는 약 4.56 cm²입니다. To solve this problem, we will follow these steps Analyze the overall shape A sector with a central angle of 90 Part of a circle with a radius of 4 c Calculate the area of the sector Formul...

  43. [51]

    치즈버거의 가격 확인: - 치즈버거 (CHEESEBURGER) 가격: 14.9천원

  44. [52]

    탄산음료의 가격 확인: - 탄산음료 (SODA) 가격: 3.9천원

  45. [53]

    STARBUCKS

    두 메뉴의 가격 합산: - 14.9천원 + 3.9천원 = 18.8천원 따라서, 치즈버거 한 개와 탄산음료 한 개를 주문할 경우 총 18.8천원을 내야 합니다. Here's how to calculate the total amount Check the price of cheeseburger: - CHEESEBURGER price: 14.9K wo Check the price of soda: - SODA price: 3.9K wo Sum up the prices of both items: -14...

    2025

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.