{"id":"4fb25f49-3dbe-4aa4-9353-0778296cb016","arxiv_id":"2411.19103","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"VARCO-VISION-14B is a Korean-English vision-language model that reports strong results among similar-size open models and introduces five Korean multimodal benchmarks.","lead":"This paper releases VARCO-VISION-14B, an open Korean-English vision-language model trained in four stages, plus five Korean evaluation benchmarks. The authors report it outperforms similar-size open models on Korean and English image tasks and supports OCR, grounding, and referring.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Self-built Korean benchmarks (especially re-created K-MMStar and synthetic K-DTCBench) lack external validation; without independent confirmation, the claimed superiority over similar-scale models could be a curation artifact.","rationale":"The reader's weakest assumption correctly targets the validity of the author-constructed Korean benchmarks. I agree with the direction and sharpen it to one load-bearing concern: the model's comparative advantage is measured largely on benchmarks the authors built, and no external Korean benchmark confirms the ranking. The training recipe is internally coherent, the released weights and datasets are concrete contributions, and I do not see a fatal error in the reported numbers. However, the K-MMStar modifications and the small synthetic K-DTCBench create a real risk that the observed gaps partly reflect benchmark construction rather than model capability. The conditional verdict is therefore appropriate; the concrete external-benchmark check plus per-item audit would settle the concern. No change to the reader's verdict is needed.","tokens_in":17246,"tokens_out":6297,"duration_ms":60510,"concrete_test":"Run VARCO-VISION and the Table 1 baselines on an independently constructed Korean multimodal benchmark not used in training (e.g., a fresh Korean translation of MMMU or AI2D with independent human post-editing). If the model's margin over Qwen2-VL-7B and LLaVA-OneVision-7B does not persist, the headline superiority is likely a curation artifact. Additionally, publish a per-item diff of the modified/re-created MMStar items and recompute K-MMStar scores using only Type-1 (simple translation) items; if the Type-1 margin is near zero, the re-created items are the source of the claimed advantage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline comparison rests on five Korean benchmarks, four of which are constructed by the authors. Section 3.1.1 states that K-MMStar is not a simple translation: unanswerable cases were 'modify[ied] or re-created,' and Appendix A.1 documents both option replacement (Type 2) and entirely new question generation (Type 3). No per-item diff is released, so readers cannot tell whether the replacements or Korean wording track VARCO-VISION's training distribution, including the authors' own Korean post-editing style. K-SEED uses only the first 20% of SEED image questions, which may not preserve the original ability distribution, and K-DTCBench is a 240-question fully synthetic set written to test exactly the document/table/chart skills trained in Stage 2. None of these instruments has been validated against an independent Korean VLM benchmark, and Table 1 reports no confidence intervals. Curation effects are plausible: on English MMStar, VARCO-VISION is 2.0 points above LLaVA-OneVision-7B, while on K-MMStar the margin is 3.3 points; on K-DTCBench the spread among baselines is 52.9 to 84.6 with no error bars. This does not prove the model is weak; it means the central 'outstanding performance' claim needs an independent yardstick.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VARCO-VISION-14B, an open-source Korean-English vision-language model built on Qwen-2.5-14B-Instruct and SigLIP, and trained in four stages (feature alignment pre-training, basic supervised fine-tuning, advanced supervised fine-tuning, and DPO) with text-only data interspersed. The authors also release five Korean evaluation benchmarks: K-MMBench, K-SEED, K-MMStar, K-DTCBench, and K-LLaVA-W. They report that the model outperforms similar-scale open-source models on Korean and English MCQA benchmarks, text-only benchmarks, and OCR tasks, and that it approaches proprietary model performance on several benchmarks.","tokens_in":17464,"tokens_out":5003,"duration_ms":43969,"significance":"If the results hold, the paper makes a valuable contribution by providing a strong open bilingual VLM and a set of Korean multimodal benchmarks that were previously lacking. The released model and datasets enable further research on Korean vision-language models. The finding that interleaving text-only data preserves language ability while adding vision skills is a practically useful training recipe, and the English benchmark results and OCR and grounding demonstrations support the effectiveness of the staged training. However, the validity of the newly constructed Korean benchmarks is not yet established, which tempers the significance of the cross-lingual comparisons.","major_comments":[{"comment":"The selection of the first 20 percent of SEED-Bench image questions is arbitrary and may not preserve the ability distribution of the full benchmark. The paper does not justify this subset or show that it measures the same dimensions with similar difficulty. To make the K-SEED scores interpretable, the authors should compare per-dimension performance of several baselines on the selected subset versus the full English SEED (or the full translated Korean set), and ideally provide the exact subset selection criterion.","section":"Section 3.1.1 (K-SEED)"},{"comment":"The modification or re-creation of MMStar items (Types 2 and 3) introduces a potential bias that is not quantified. No per-item diff or counts of Type 1/2/3 items are released, and the human post-editing process is not described with inter-annotator agreement. Because these edits were made by the authors, they could align with the model's training distribution. The authors should release the full list of changed items with original and Korean versions, report the distribution of the three types, and evaluate all baselines on the corresponding original English items to check whether the relative ranking changes between the English and Korean versions.","section":"Section 3.1.1 and Appendix A.1 (K-MMStar)"},{"comment":"K-DTCBench is a fully synthetic, author-constructed benchmark for document, table, and chart understanding, which are exactly the skills targeted in Stage 2 of training. The large margin over Qwen2-VL-7B (84.58 vs 75.00) could reflect overlap with the training data rather than generalizable capability. The authors should provide external validation, for example by evaluating on an independent Korean document/chart benchmark, or at least show per-image-type scores and demonstrate that the synthetic images are not similar to training examples. Without this, the headline superiority claim on K-DTCBench is not robust.","section":"Section 3.1.1 (K-DTCBench)"},{"comment":"The headline comparisons lack any measure of uncertainty. For instance, K-DTCBench has only 240 questions, and several gaps such as K-SEED (75.39 vs 74.08 over Qwen2-VL-7B) are within expected sampling noise. The authors should report binomial standard errors or bootstrap confidence intervals for all MCQA scores and specify the number of questions per benchmark. This is particularly important because the paper's central claim of 'outstanding performance' rests on differences that may not be statistically distinguishable.","section":"Tables 1 and 2 (and Table 3)"}],"minor_comments":[{"comment":"The claim of being the 'first to release open-source Korean evaluation benchmarks' is too strong given that the paper itself cites existing Korean VQA and OCR datasets (references [12] and [13]). Consider rephrasing to 'first general-purpose Korean multimodal comprehension benchmarks of this scope.'","section":"Section 1"},{"comment":"The architecture description states that training focuses on single-image examples but does not explain why multi-image support is excluded; a short note on this design choice would help readers understand the scope and limitations.","section":"Section 2.1"},{"comment":"The K-LLaVA-W evaluation uses GPT-4o as both the judge and the baseline model. This coupling should be acknowledged as a potential source of bias in the ratio score, and the paper should report the raw JudgeLLM scores for both the target model and the baseline rather than only the ratio.","section":"Section 3.1.2"},{"comment":"The LogicKor and KoMT-Bench scores are based on LLM-as-a-judge, which is stochastic, but no standard deviations or multiple-run results are reported. Please provide error bars or a statement about the variance of the judge.","section":"Table 3"},{"comment":"The JudgeLLM prompt is provided only in Korean; an English translation should be included for the accessibility of non-Korean readers and for reproducibility.","section":"Appendix A.3.2"},{"comment":"In the mathematical reasoning example, the sector area formula is written as 'π × r' and the squared exponent is missing in the displayed text, although the calculation uses r². Please correct the inconsistency.","section":"Figure 7"}],"recommendation":"major_revision","confidential_remarks":"This is a technical-report-style paper, and the release of the model and five Korean benchmarks is a significant practical contribution. The main risk to the central claim is the validity of the self-constructed Korean benchmarks, particularly K-SEED, K-MMStar, and K-DTCBench. If the authors can provide external validation, per-item diffs, and confidence intervals, the paper could become acceptable. The 'first Korean benchmarks' wording should also be qualified. The paper fits a venue that accepts technical reports with released artifacts, but the current evidence is not yet sufficient to support the strong superiority claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. First, this is a release paper: the model weights and all five Korean benchmarks are on HuggingFace, and the four-stage training recipe is clearly spelled out. Second, the paper's central claim—superiority on Korean benchmarks over similar-scale models—rests almost entirely on benchmarks the authors constructed themselves, so treat those performance gaps as provisional until someone checks them externally.\n\nThe genuinely new pieces are the five Korean benchmarks: translated and human-post-edited K-MMBench, K-SEED, K-MMStar, K-LLaVA-W, plus the from-scratch synthetic K-DTCBench for documents, tables, and charts. That is a real contribution for Korean VLM research. The model itself is LLaVA-OneVision-style with Qwen-2.5-14B and SigLIP; the novelty is the staged training and the text-only data interspersed to preserve language ability. The paper shows sensible English-benchmark results too, which gives some confidence the model is not just tuned to the Korean test sets.\n\nNow the soft spots, in proportion. The stress-test concern is valid and it lands on the place the paper itself is most honest: Section 3.1.1 admits K-MMStar required modifying or re-creating questions because the original had unanswerable cases. That is disclosed, credit where due, but no per-item diff is released, so readers cannot check whether the rewrites accidentally track the training data. K-SEED uses only the first 20 percent of image questions, which may not preserve the ability distribution. K-DTCBench is 240 synthetic questions written to test exactly the skills trained in Stage 2. No confidence intervals or variances are reported anywhere. None of this proves the model is weak—the English results support that it is a decent VLM—but it does mean the 'outstanding performance' headline on Korean is not yet established by this paper alone. The LLM-judge evaluation for K-LLaVA-W uses GPT-4o as both judge and baseline, a known circularity, though it is disclosed and standard in the field.\n\nWho is this for? Researchers training bilingual or multilingual VLMs, and anyone who needs Korean evaluation material. It deserves a serious referee, because the artifacts are concrete and the benchmark construction is described in enough detail to reproduce. My recommendation: send it to review, but ask the authors for external validation of the Korean benchmarks—ideally a comparison on an independently constructed Korean VLM benchmark or at least per-item diff and a human agreement study on the translated sets.","headline":"Honest, useful resource-paper: releases a 14B Korean-English VLM and five Korean benchmarks, but the headline performance claims on the author-built Korean benchmarks still need an independent yardstick.","tokens_in":18035,"tokens_out":2496,"would_cite":true,"duration_ms":22810,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An open 14B Korean-English vision-language model claims to beat same-size open models on its own translated Korean benchmarks and to match much larger proprietary systems, while keeping its language scores intact.","keywords":["Korean vision-language model","bilingual multimodal","four-stage training","Direct Preference Optimization","OCR","grounding and referring","Korean multimodal benchmarks","open-source VLM"],"falsifier":"Have a fresh Korean-speaking panel, blind to the answer keys, re-answer a stratified sample of K-SEED and K-MMStar questions from the images, and replace the automatic judge for K-LLaVA-W with human raters; if a multilingual open VLM trained only on English multimodal data scores near chance on the Korean translations, or if human rankings disagree with the automatic judge's rankings, the reported Korean gains would not be reliable evidence of a Korean-specific capability.","tokens_in":17006,"feed_emoji":"🖼️","tokens_out":9629,"duration_ms":80836,"temperature":0.7,"pith_summary":"VARCO-VISION-14B is an open-weight bilingual vision-language model that the authors train in four stages so it can handle Korean and English images and text without losing the language ability it starts with. The paper argues that adding visual skills gradually — feature alignment first, then broad instruction tuning, then harder reasoning, then preference optimization — lets one 14B model do visual question answering, OCR, grounding, and referring in both languages. To support this claim, the authors translate three English multiple-choice benchmarks and one open-ended generation benchmark into Korean, human-edit the translations, and build a new document, table, and chart benchmark from scratch. On these released benchmarks the model reports the highest scores among open models of similar size, and on several closed-set tests it is competitive with much larger open models and with proprietary APIs. The paper also releases the five Korean evaluation datasets themselves, so future models can be compared on the same ground.","feed_headline":"14B Korean-English vision model tops same-size rivals in new tests","feed_subtitle":"Four-stage training preserves language skill while adding OCR, grounding, and referring in Korean and English.","key_machinery":"The load-bearing mechanism is a four-stage training schedule that progressively unfreezes the model. Stage 1 trains only the randomly initialized MLP projector on image-caption pairs; Stage 2 unfreezes all layers and fine-tunes on six task families including OCR, grounding, referring, captioning, document/table/chart/math, and text-only examples; Stage 3 does the same with harder reasoning tasks; Stage 4 applies Direct Preference Optimization, a method that pushes the model toward preferred responses, to the LLM layers only to improve fluency and alignment. Special tokens for grounding, OCR, text phrases, objects, bounding boxes, and multiple locations let the model emit structured outputs for referring and grounding in both languages. The argument is that staged unfreezing plus the periodic return to text-only data preserves the text backbone's knowledge, while the preference-optimization pass improves generation quality.","core_discovery":"The paper's central claim is that a 14B open-weight VLM can be genuinely bilingual in Korean and English while remaining strong at text-only language tasks. The authors report that VARCO-VISION-14B tops all compared open models under 20B on K-MMBench (82.21), K-SEED (75.39), K-MMStar (57.33), and the new K-DTCBench (84.58), and reaches 84.74 on K-LLaVA-W, an open-ended generation benchmark scored by a large language model judge. On English closed-set tests it also beats the under-20B comparisons on MMBench, SEED, and MMStar, although not on MMMU, and it scores 820 on OCRBench, above every compared VLM including 72B models. Its text-only scores (8.69 on LogicKor, 8.39 on KoMT-Bench, 8.80 on MT-Bench) are the highest among compared VLMs and close to those of a dedicated bilingual language model. The authors attribute this balanced profile to the four-stage recipe and, in particular, to interspersing text-only data through Stages 2-4 and applying preference optimization in the final stage.","pith_inferences":["Inference: the same four-stage recipe with text-only data interspersed may transfer to other under-resourced languages; nothing in the paper ties the mechanism to Korean specifically.","Inference: the large gap between VARCO-VISION and other open models on K-DTCBench could indicate that cheaply generated synthetic Korean documents, tables, and charts are a high-leverage training signal for non-English document understanding.","Inference: an ablation that removes the text-only stages or the DPO stage would isolate exactly which component preserves the language scores; the paper does not report such an ablation.","Inference: because the same commercial API used for translation is also used for reference answers and judging, the K-LLaVA-W scores may partly reflect stylistic similarity to that API's writing rather than a separate notion of Korean quality."],"forward_implications":["A 14B open model can serve many Korean-English multimodal applications without relying on a proprietary API.","Researchers get five standardized Korean benchmarks for closed-set and open-ended evaluation, so future Korean VLM comparisons need not be built from scratch.","Training recipes that interleave text-only data can preserve or even improve a VLM's language scores while adding visual skills.","The same staged approach could be used to add OCR, grounding, and referring abilities to other language-specific VLMs.","The model's OCR and document/table/chart performance suggests it is usable for digitizing Korean documents and form-like images, not just answering image questions."],"supporting_citations":[{"why":"Supplies the architecture and training recipe that this paper extends to Korean-English data.","marker":"[16]"},{"why":"Provides the 14B instruction-tuned text backbone whose linguistic knowledge the staged training is designed to preserve.","marker":"[32]"},{"why":"Vision encoder whose features the MLP projector aligns to the text model in the first training stage.","marker":"[37]"},{"why":"Algorithm for the final preference-optimization stage that the paper credits for improved readability and text-only scores.","marker":"[28]"},{"why":"Source of K-MMBench after translation and human post-editing; also used as an English closed-set comparison.","marker":"[22]"},{"why":"Source of K-SEED, constructed by taking the first 20 percent of SEED-Bench image questions.","marker":"[17]"},{"why":"Source of K-MMStar, with questions modified or re-created to be answerable from a single image.","marker":"[5]"},{"why":"Source of K-LLaVA-W, whose images are localized to Korean and whose judge prompts are translated.","marker":"[21]"},{"why":"The API used to translate benchmarks and to produce reference answers and judge scores for K-LLaVA-W.","marker":"[2]"}],"fun_headline_variants":["14B Korean-English VLM beats same-size rivals in bilingual tests","Open 14B VLM tops Korean benchmarks, keeps text skills","Korean-English 14B model leads under-20B vision-language field","VARCO-VISION: 14B bilingual VLM outruns peers on Korean tests","Bilingual VLM preserves language, adds OCR and grounding at 14B"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the new Korean evaluation datasets measure real model ability rather than rewarding quirks of the translation, the question selection, or the judge model used to score free-form answers.","fun_headline_variants_meta":{"raw":{"variants":["14B Korean-English VLM beats same-size rivals in bilingual tests","Open 14B VLM tops Korean benchmarks, keeps text skills","Korean-English 14B model leads under-20B vision-language field","VARCO-VISION: 14B bilingual VLM outruns peers on Korean tests","Bilingual VLM preserves language, adds OCR and grounding at 14B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000764,"raw_usage":{"total_tokens":3382,"prompt_tokens":933,"completion_tokens":2449,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":2350}},"tokens_in":549,"tokens_out":2449,"duration_ms":18552,"temperature":1.0,"reasoning_tokens":2350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:32:13.551975+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a fresh Korean-speaking panel, blind to the answer keys, re-answer a stratified sample of K-SEED and K-MMStar questions from the images, and replace the automatic judge for K-LLaVA-W with human raters; if a multilingual open VLM trained only on English multimodal data scores near chance on the Korean translations, or if human rankings disagree with the automatic judge's rankings, the reported Korean gains would not be reliable evidence of a Korean-specific capability.","supporting_citations":[{"cited_title":"Are we on the right way for evaluating large vision-language models? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024","cited_arxiv_id":null,"evidence_quote":"Source of K-MMStar, with questions modified or re-created to be answerable from a single image."}],"review_version":1}