Pith. sign in

REVIEW 4 major objections 6 minor 56 references

Decoupling VLM data mixtures into inter-class ratio search and intra-class convex allocation yields recipes that beat heuristic stacking and transfer from small proxies to larger scales.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 12:40 UTC pith:OKHOUVWK

load-bearing objection Solid VLM data-ops methods paper: beats stacking and gives a cleaner admission protocol, but “seamless transfer” is only proven against stacking, not against re-search at scale. the 4 major comments →

arxiv 2607.24516 v1 pith:OKHOUVWK submitted 2026-07-27 cs.CV cs.AI

DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes

Jiahao Xie , Zhongbin Guo , Qianle Wang , Ruiqi Lu , Dongling Xiao , Wanxuan Sun , Cheng Yang This is my paper
classification cs.CV cs.AI
keywords VLM data mixtureinter-class ratio searchintra-class convex allocationdataset-level quality and difficultyproxy-scale transfermultimodal pretraining recipesattributable dataset admissioncontinue-pretraining
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Public practice for building vision-language pretraining mixtures is still mostly guesswork: filter for quality, stack whatever remains, and set cross-domain ratios by intuition. This paper argues that the problem can be turned into a reproducible engineering loop by splitting it into two orthogonal pieces—how much of the token budget each capability category gets, and how to mix candidate datasets inside one category. Inter-class ratios are found with a cheap single-variable sweep on a small proxy model; within a class, each dataset is scored for quality and difficulty and the mix is solved as a constrained convex program that also rewards diversity. The resulting recipes consistently beat size-proportional stacking, and ratios found only at 2.5B tokens transfer without retuning to larger budgets and even to a 32B model. With 80B additional multimodal continue-pretraining tokens their 4B model matches a strong open baseline on average before instruction tuning, and the same fixed-ratio setup turns “should we admit this new dataset?” into a controlled, attributable experiment rather than a confounded retrain.

Core claim

DecoupleMix treats continue-pretraining data construction as two decoupled sub-problems: single-variable iterative search for inter-class capability ratios, and constrained convex allocation of intra-class weights that maximize a weighted quality–difficulty objective plus an entropy diversity term under token bounds. Optimal ratios discovered only at a 2.5B-token proxy transfer to larger data and model scales without retuning, consistently outperform heuristic stacking, and make new-dataset admission an attributable intervention under fixed budget and ratios.

What carries the argument

DecoupleMix—the hierarchical split into inter-class budget ratios (coordinate-style single-variable search on a proxy) and intra-class sampling weights solved as the convex program maximizing α·quality + β·difficulty + γ·entropy(w) on the simplex with box constraints. That split makes high-dimensional mixture search tractable, enables scale transfer, and renders dataset admission identifiable.

Load-bearing premise

That automated quality and difficulty scores at the whole-dataset level, from a frozen judge on 150 samples per dataset, plus a fixed hand-chosen capability taxonomy, are stable and complete enough that the mixtures they drive are truly better and transferable—not artifacts of the proxy, the multiple-choice eval suite, or the score weights.

What would settle it

If inter-class ratios searched only at 2.5B tokens failed to beat matched-budget size-proportional stacking when transferred unchanged to larger token budgets or a different model scale under the same unified evaluation, or if admitting one dataset under the fixed-ratio convex protocol produced no more localized, attributable gains than naive stacking or displacement, the central transfer and attribution claims would fall.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Mixture search can be paid once on a cheap proxy and the recipe reused at larger data budgets and model sizes without retuning.
  • New candidate datasets can be accepted or rejected under fixed total budget and inter-class ratios, so the measured change is attributable to that source.
  • Intra-class convex allocation with an entropy term beats both quantity-proportional and quality-only sampling by balancing difficulty and long-tail coverage.
  • A systematically built recipe lets a smaller VLM trained with 80B additional multimodal tokens match stronger open models that used larger multimodal budgets, before instruction tuning.
  • Capability gaps surfaced by the taxonomy give a concrete signal for what data to collect next.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same hierarchical split could be tried for fully omni-modal mixtures once a shared capability taxonomy across audio, video, and text exists.
  • Keeping datasets as atomic units leaves sample-level reweighting inside high-scoring sources on the table as a further gain the current method forgoes.
  • Pipelines that already shelve sources by capability folder could drop the convex allocator in place of size-proportional sampling without redesigning their taxonomy.
  • The attributable-admission protocol sketches a continuous data-ops loop—score, re-optimize weights, short proxy test, integrate or reject—closer to controlled A/B than full retrain ablations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes DecoupleMix, a hierarchical procedure for constructing VLM continue-pretraining data mixtures. Inter-class (capability-category) ratios are found by a coordinate-style single-variable search on a cheap 4B proxy model at 2.5B tokens, using MCQ-based validation; intra-class allocation within each category is formulated as a constrained convex program (Eq. 1) maximizing a linear Quality/Difficulty score (from an LLM-as-a-Judge rubric over 150 samples per dataset) plus an entropy diversity term, solved per category. A third component fixes inter-class ratios, budget, and allocator so that admitting a new dataset becomes a controlled, attributable intervention (Eq. 5). Experiments show gains over size-proportional stacking at 2.5B/5B/10B tokens (Table 3), transfer of the 2.5B-searched recipe to a 32B model without retuning (Table 4), improved locality of admission effects vs Stack/Displace (Table 2), and an end-to-end 4B model trained on 80B tokens that matches Qwen3-VL-4B-Instruct's 16-benchmark average (62.5) under a unified evaluation protocol before instruction tuning (Table 1).

Significance. If the results hold, the paper offers a genuinely useful engineering contribution to an understudied problem: public VLM pretraining recipes are heuristic, and a reproducible, low-cost procedure for setting mixture ratios has clear practical value. Specific strengths worth naming: the attributable single-dataset admission protocol (§2.3, Eq. 5) is a clean experimental design that fixes budget and inter-class ratios so admission effects are interpretable, and Table 2 shows it empirically reduces off-target drift relative to Stack/Displace; the evaluation is unified (all baselines re-run under one protocol, including fixed 8-frame video sampling); the ablations are controlled and complete (inter-only vs full in Table 5, quantity/quality/convex intra-class in Table 6); and full per-benchmark scores plus total compute (~3500 GPU hours) are disclosed. The hierarchical decoupling itself is a reasonable, clearly motivated tractability device rather than a claimed optimum, and the limitations section is candid. The main weaknesses are evidentiary rather than conceptual: the strongest claim (scale transfer without retuning) is never tested against the scale-native counterfactual, and all训练⁠

major comments (4)
  1. [§4.3, Tables 3-4] The headline claim that proxy-found inter-class ratios 'transfer seamlessly to larger scales without retuning' is not tested against the relevant counterfactual. At 5B, 10B, and 32B the transferred recipe is compared only to heuristic stacking, never to a recipe produced by re-running the single-variable search at that scale. The evidence therefore supports 'the 2.5B-searched recipe beats stacking at every scale' but not 'retuning is unnecessary.' The paper's own citation [27] (scaling laws for optimal mixtures) reports that mixture optima can drift with compute, so the possibility that a 10B-native search recovers different ratios and adds another point is not excluded. Since the amortized-search-cost argument ('the search cost is paid once') rests entirely on this claim, the authors should either (a) run the inter-class search at one larger budget (e.g., 10B, ~480 GPU hours per Appendi
  2. [§4.1, Tables 3-6] All results appear to be single training runs with no seed variance reported. The central quantitative findings are average gaps of +1.6/+2.0/+1.4 (Table 3) and +1.2 (Table 4) over 16 benchmarks, and the narrative leans on per-benchmark differences (e.g., PuzzleVQA 49.0 vs 43.0 in §4.1; Table 4, where the transferred recipe loses PuzzleVQA 56.8 vs 60.4 despite winning the average). Multimodal pretraining runs commonly exhibit per-benchmark fluctuations of this order across seeds. At minimum, the authors should quantify run-to-run variance on a representative subset (e.g., repeat the 2.5B or 1B-budget intra-class comparison of Table 6 with 2-3 seeds and report std), and state explicitly which reported gaps exceed run noise. Without this, 'consistent superiority' is asserted rather than demonstrated.
  3. [§2.1, Eq. (1), Appendix B.1] The Quality/Difficulty scores that drive the convex objective in Eq. (1) rest on a thin validation. Human alignment is measured on only 10 instances (Table 7); Spearman r=0.82/0.75 on n=10 has wide confidence intervals and does not establish dataset-level score stability, which is the quantity actually consumed by the optimizer (an aggregate over 150 judged samples per dataset). The authors should (a) report judge score variance across the 150 samples and across judge resamples, (b) expand the human-alignment study or provide an inter-annotator agreement for the three raters, and (c) ideally show robustness of the final recipe to perturbations of q_i/d_i (e.g., re-solving Eq. (1) with scores jittered within their estimated noise and checking allocation stability). As it stands, a load-bearing input to the method is the least-validated component.
  4. [§2.2, §3.2, Table 5] The inter-class search is driven entirely by MCQ-style validation on the same 16-benchmark suite used for final reporting (§2, §3.2). This creates a mild selection-on-the-test-set risk: ratios are chosen to maximize average performance on the evaluation suite, then gains are reported on that same suite. The authors acknowledge MCQ validation reduces dependence on instruction-following, but should discuss this coupling explicitly and, if possible, report a small held-out set (benchmarks excluded from the search objective) to show gains are not suite-specific. This is particularly relevant because the inter-only ablation gain over stacking is small (+0.5 avg, Table 5), so even modest selection bias could account for it.
minor comments (6)
  1. [Tables 2 and 8] The ΔAvg values in Table 2 do not match the absolute scores in Table 8: baseline avg is 57.8 and Ours avg is 57.9 (Δ=+0.1, reported +0.2); Displace avg 57.4 (Δ=-0.4, reported -0.3). Please reconcile or state the rounding convention.
  2. [Table 1, Abstract] Table 1 compares the authors' pre-instruction-tuning 4B model against Qwen3-VL-4B-**Instruct**. This is disclosed and the unified evaluation protocol is commendable, but the abstract's 'matches Qwen3-VL-4B-Instruct average (62.5)' should carry the pre-IT qualifier, since the comparison cuts both ways (their model lacks IT; the baseline had a far larger multimodal budget).
  3. [Abstract, §4.1] The claim that baselines were 'trained with substantially larger multimodal budgets' (abstract, §4.1) is not verifiable from cited sources; Qwen3-VL's multimodal token count is not public. Please soften or cite a source.
  4. [§2.1, Ref [13]] Reference [13] is the Seed1.5-VL technical report but is cited as the source for the 'Seed-1.6' judge. Please correct or clarify which model actually performed the judging.
  5. [Abstract, Fig. 1, Fig. 2, Table 5] Typos and figure text: 'asystematic' (abstract/intro), 'Combintorial' (Fig. 1), 'Table 5 sentence beginning "Our Inter-only strategy (Avg 57.1), compared with..."' is ungrammatical. Fig. 2 would benefit from axis tick values and a statement of how many datasets are plotted per panel.
  6. [§2.2, Eqs. (1)-(3)] In Eq. (1) and the surrounding text, L_i is described as 'token-level bounds' (plural) but only the lower-bound role is clear; please define L_i and M_i precisely (is L_i a minimum-allocation floor per dataset, and who sets it?). The values of α, β, γ used in all experiments should be reported, along with the warm-start r^(0) and the candidate sweep grid for the inter-class search, without which the recipe is not reproducible.

Circularity Check

0 steps flagged

No significant circularity: mixture weights are tuned on proxy runs and judged on held-out benchmarks, not defined equal to the reported metrics.

full rationale

DecoupleMix’s load-bearing chain is empirical, not definitional. Inter-class ratios are found by single-variable search that maximizes proxy validation performance and are then frozen and compared against size-proportional stacking at larger token and model scales (Tables 3–4). Intra-class weights solve a convex program (Eq. 1) that maximizes a linear combination of LLM-as-a-Judge Quality/Difficulty scores plus entropy; those scores are inputs to the allocator, not the capability metrics reported in the tables. End-to-end and ablation results are measured on a separate 16-benchmark suite under a fixed protocol. Nothing in the paper equates the convex objective, the judge scores, or the proxy search criterion with the claimed Avg / per-domain gains by construction. Author-overlapping citation [19] only supplies the MCQ evaluation protocol and is not used to force the mixture result. Gaps such as the missing scale-native re-search counterfactual for the “transfer without retuning” claim are evidence-strength issues, not circular reductions. Steps list is empty.

Axiom & Free-Parameter Ledger

5 free parameters · 7 axioms · 3 invented entities

The central claims rest on empirical optimization under several modeling choices: a fixed capability taxonomy, LLM-as-judge dataset scores, a weighted Q/D+entropy objective with free tradeoff weights, single-variable (not joint) inter-class search, MCQ proxy metrics, and the assumption that small-proxy optima transfer. No new physical entities; the 'invented' pieces are methodological constructs.

free parameters (5)
  • α, β, γ (quality, difficulty, entropy weights in Eq. 1) = unspecified (≥0)
    Relative contributions in the intra-class objective are free nonnegative knobs; paper does not report a search or sensitivity study for them.
  • Inter-class warm-start r^(0) and candidate sweep grid = average of reference recipes; grid not fully specified
    Search starts from average of reference recipes and sweeps one coordinate at a time over an unspecified candidate set until stabilization.
  • Quality/Difficulty dimension weights (Eqs. 6–7) = fixed in Appendix B (denominators 4.8 and 5.8)
    Hand-set coefficients (e.g., Accuracy 2, Hallucination 1.5; Cross-Modal 2, Prior Knowledge 1.5) define dataset scores that drive allocation.
  • Per-dataset allocation bounds L_i, M_i → ℓ_i, u_i = dataset-dependent; not tabulated
    Lower/upper token bounds constrain the convex program; choice affects whether sources get zero weight (screening).
  • Samples per dataset for judging (n=150) = 150
    Finite sample size for LLM-as-judge estimates of qi, di.
axioms (7)
  • domain assumption Capability categories specified in natural language can be assigned automatically at dataset level and form an adequate taxonomy for budget allocation.
    §2.1: taxonomy is extensible and LLM-classified; all inter-class search depends on this partition.
  • domain assumption Datasets are cohesive enough that whole-dataset Quality/Difficulty scores (not sample-level) suffice for mixture optimization.
    §2.1 explicitly chooses datasets as minimal units for industrial pipelines.
  • domain assumption LLM-as-a-Judge (Seed-1.6) scores correlate with human quality/difficulty judgments enough to drive optimization.
    Supported only by n=10 human study (Table 7); load-bearing for Eq. (1).
  • domain assumption Closed-task MCQ validation on proxy checkpoints is a valid surrogate for pretraining mixture quality without instruction tuning.
    §2 and §3.2; cites Closed-Task Validation workshop paper; all search loops use this.
  • ad hoc to paper Decoupling inter-class and intra-class optimization (vs joint high-dimensional search) is near-enough optimal for practical recipes.
    Core methodological bet; Limitations §6 admits richer joint search could improve recipes.
  • ad hoc to paper Single-variable coordinate-style search finds stable inter-class ratios that transfer across data and model scale.
    §2.2 Inter-class Strategy; empirical support in Tables 3–4 but assumed general.
  • standard math Convex program with entropy regularizer is an appropriate intra-class diversity mechanism (H concave ⇒ convex problem solvable by ECOS).
    Standard convex optimization fact used in Eq. (1)–(2).
invented entities (3)
  • DecoupleMix hierarchical recipe (inter-class single-variable search + intra-class convex allocation) no independent evidence
    purpose: Turn VLM mixture construction into a reproducible two-stage engineering procedure.
    Named framework combining known optimization tools in a specific VLM data-ops workflow; validated empirically inside the paper.
  • Attributable dataset-admission protocol (fix r, T, optimizer g; measure ΔP on single-class insertion) no independent evidence
    purpose: Make integrate-or-reject decisions identifiable without global confounding.
    Defined via first-order decomposition Eq. (4)–(5); evidence is internal ablation Table 2.
  • Automated multi-dimensional dataset-level Q/D assessment rubric no independent evidence
    purpose: Produce qi, di inputs to the convex allocator without sample-level filtering.
    Prompts and weighted formulas in Appendix B; only thin external human alignment.

pith-pipeline@v1.2.0-grok45-kimik3 · 22086 in / 4438 out tokens · 91446 ms · 2026-07-31T12:40:22.882749+00:00 · methodology

0 comments
read the original abstract

While data curation for Vision Language Models (VLMs) is increasingly active, public practice for constructing pretraining mixtures remains largely heuristic: practitioners stack datasets that pass quality filters, set cross-domain ratios by intuition, and lack a principled, attributable criterion for admitting new data, while frontier recipes remain undisclosed. We formulate data construction as a systematic mixture-optimization problem and turn it into a reproducible engineering discipline by decoupling the mixture into two orthogonal sub-problems: inter-class ratios across capabilities and intra-class ratios within a category. For inter-class allocation, we use a single-variable iterative search; for intra-class composition, we apply a multidimensional, dataset-level assessment scoring Quality and Difficulty, and formulate selection as a constrained convex optimization with a diversity objective. The DecoupleMix framework delivers two critical capabilities: guiding what data to collect next and rendering dataset validation a controlled, attributable experiment. Experiments show our approach consistently surpasses heuristic baselines. Moreover, optimal ratios discovered on small-scale proxies transfer seamlessly to larger scales without retuning. Using 80B additional multimodal continue-pretraining tokens, our VLM is competitive with strong open-source models trained with substantially larger multimodal budgets.

Figures

Figures reproduced from arXiv: 2607.24516 by Cheng Yang, Dongling Xiao, Jiahao Xie, Qianle Wang, Ruiqi Lu, Wanxuan Sun, Zhongbin Guo.

Figure 1
Figure 1. Figure 1: Overview of DecoupleMix. (a) Recipe search: annotated datasets are optimized along two decoupled axes—inter-class budget allocation and intra-class convex selection—then validated and scaled up on a cheap proxy model (2.5B→10B tokens). (b) Dynamic admission: a new dataset is evaluated under the fixed configuration, turning its acceptance into a controlled, attributable integrate-or-reject decision. • Over-… view at source ↗
Figure 2
Figure 2. Figure 2: visualizes token allocations across the Quality and Difficulty dimensions for each strategy. Quantity-proportional sampling assigns substantial mass to lower-quality, lower-difficulty regions, whereas quality-only sampling concentrates toward higher Quality scores without explicitly accounting for Difficulty. Our convex allocation shifts more token mass toward datasets that score highly on both dimensions … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 12 linked inside Pith

  1. [1]

    Gpt-5.6 preview system card, 2026

    OpenAI. Gpt-5.6 preview system card, 2026. URL https://deploymentsafety.openai.com/ gpt-5-6-preview/gpt-5-6-preview.pdf

    2026

  2. [2]

    Seed2.0 model card: Towards intelligence frontier for real-world complexity

    Bytedance Seed. Seed2.0 model card: Towards intelligence frontier for real-world complexity. 2026. URL https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/ seed2/0214/Seed2.0%20Model%20Card.pdf

    2026

  3. [3]

    System card: Claude fable 5 & claude mythos 5, 2026

    Anthropic. System card: Claude fable 5 & claude mythos 5, 2026. URL https://www-cdn.anthropic. com/d00db56fa754a1b115b6dd7cb2e3c342ee809620.pdf

    2026

  4. [4]

    Gemini 3.5 flash model card

    Google Deepmind. Gemini 3.5 flash model card. 2026. URL https://storage.googleapis.com/ deepmind-media/Model-Cards/Gemini-3-5-Flash-Model-Card.pdf

    2026

  5. [5]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  6. [6]

    Llava-onevision-1.5: Fully open framework for democratized multimodal training

    Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, et al. Llava-onevision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661, 2025. 10

    Pith/arXiv arXiv 2025

  7. [7]

    Qwen3-vl technical report, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

    Pith/arXiv arXiv 2025

  8. [8]

    Molmo2: Open weights and data for vision-language models with video understanding and grounding

    Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammadreza Salehi, Rohun Tripathi, Sangho Lee, Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, et al. Molmo2: Open weights and data for vision-language models with video understanding and grounding. arXiv preprint arXiv:2601.10611, 2026

    Pith/arXiv arXiv 2026

  9. [9]

    A survey of multimodal large language model from a data-centric perspective

    Tianyi Bai, Hao Liang, Binwang Wan, Ling Yang, Bozhou Li, Yifan Wang, Bin Cui, Conghui He, Binhang Yuan, and Wentao Zhang. A survey of multimodal large language model from a data-centric perspective. CoRR, abs/2405.16640, 2024. URLhttps://doi.org/10.48550/arXiv.2405.16640

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.16640 2024

  10. [10]

    Ailin Deng, Tri Cao, Zhirui Chen, and Bryan Hooi. Words or vision: Do vision-language models have blind faith in text? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3867–3876, June 2025

    2025

  11. [11]

    Quadmix: Quality-diversity balanced data selection for efficient llm pretraining

    Fengze Liu, Weidong Zhou, Binbin Liu, Zhimiao Yu, Yifan Zhang, Haobin Lin, Yifeng Yu, Bingni Zhang, Xiaohuan Zhou, Taifeng Wang, et al. Quadmix: Quality-diversity balanced data selection for efficient llm pretraining. arXiv preprint arXiv:2504.16511, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Quality over quantity? llm-based curation for a data-efficient audio-video foundation model

    Ali V osoughi, Dimitra Emmanouilidou, and Hannes Gamper. Quality over quantity? llm-based curation for a data-efficient audio-video foundation model. In 2025 33rd European Signal Processing Conference (EUSIPCO), pages 286–290. IEEE, 2025

    2025

  13. [13]

    Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Learning to see before seeing: Demystifying LLM visual priors from language pre-training

    Junlin Han, Shengbang Tong, David Fan, Yufan Ren, Koustuv Sinha, Philip Torr, and Filippos Kokkinos. Learning to see before seeing: Demystifying LLM visual priors from language pre-training. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview. net/forum?id=pfw176o1YJ

    2026

  15. [15]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://llava-vl.github.io/ blog/2024-01-30-llava-next/

    2024

  16. [16]

    Curriculum learning with quality-driven data selection

    Biao Wu and Ling Chen. Curriculum learning with quality-driven data selection. arXiv preprint arXiv:2407.00102, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Ecos: An socp solver for embedded systems

    Alexander Domahidi, Eric Chu, and Stephen Boyd. Ecos: An socp solver for embedded systems. In 2013 European control conference (ECC), pages 3071–3076. IEEE, 2013

    2013

  18. [18]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    ZHANG, Jiahao Xie, Ruiqi Lu, Boyan Zhou, and Cheng Yang

    Enci Zhang, Z.Q. ZHANG, Jiahao Xie, Ruiqi Lu, Boyan Zhou, and Cheng Yang. Closed-task validation: A more robust and efficient proxy for guiding VLM training. In1st Workshop on VLM4RWD @ NeurIPS 2025, 2025. URLhttps://openreview.net/forum?id=fNxm8jlEWD

    2025

  20. [20]

    Bee: A high-quality corpus and full-stack suite to unlock advanced fully open mllms

    Yi Zhang, Bolin Ni, Xin-Sheng Chen, Heng-Rui Zhang, Yongming Rao, Houwen Peng, Qinglin Lu, Han Hu, Meng-Hao Guo, and Shi-Min Hu. Bee: A high-quality corpus and full-stack suite to unlock advanced fully open mllms. arXiv preprint arXiv:2510.13795, 2025. 11

    arXiv 2025

  21. [21]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  22. [22]

    Scalable vision language model training via high quality data curation

    Hongyuan Dong, Zijian Kang, Weijie Yin, Xiao Liang, Chao Feng, and Jiao Ran. Scalable vision language model training via high quality data curation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V olume1: Long Papers), pages 3...

    doi:10.18653/v1/2025.acl-long 2025

  23. [23]

    Doremi: Optimizing data mixtures speeds up language model pretraining

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openreview.net/forum?id=lXuByUeHhd

    2023

  24. [24]

    Data mixing laws: Optimizing data mixtures by predicting language modeling performance

    Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum? id=jjCB27TMK3

    2025

  25. [25]

    Regmix: Data mixture as regression for language model pre-training

    Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. In International Conference on Learning Representations, volume 2025, pages 38305–38339, 2025

    2025

  26. [26]

    Regmix-d: Dynamic data mixing via proxy training trajectories

    Kaiyan Zhao, Zhongtao Miao, Akiko Aizawa, and Yoshimasa Tsuruoka. Regmix-d: Dynamic data mixing via proxy training trajectories. arXiv preprint arXiv:2606.18663, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    Scaling laws for optimal data mixtures

    Mustafa Shukor, Louis Béthune, Dan Busbridge, David Grangier, Enrico Fini, Alaaeldin El-Nouby, and Pierre Ablin. Scaling laws for optimal data mixtures. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URLhttps://openreview.net/forum?id=vVU1KTOsju

    2025

  28. [28]

    Mm1: methods, analysis and insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Anton Belyi, et al. Mm1: methods, analysis and insights from multimodal llm pre-training. In European Conference on Computer Vision, pages 304–323. Springer, 2024

    2024

  29. [29]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024

    2024

  30. [30]

    MME: A com- prehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. MME: A com- prehensive evaluation benchmark for multimodal large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Tra...

    2025

  31. [31]

    Are we on the right way for evaluating large vision-language models?Advances in Neural Information Processing Systems, 37:27056–27087, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?Advances in Neural Information Processing Systems, 37:27056–27087, 2024

    2024

  32. [32]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, 2024

    2024

  33. [33]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 148–166. Springer, 2024. 12

    2024

  34. [34]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pa...

    2024

  35. [35]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In European conference on computer vision, pages 235–251. Springer, 2016

    2016

  36. [36]

    Ocrbench: on the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):220102, 2024

    2024

  37. [37]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022

    2022

  38. [38]

    Charxiv: Charting gaps in realistic chart understanding in multimodal llms

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Advances in Neural Information Processing Systems, 37:113569–113697, 2024

    2024

  39. [39]

    Logicvista: Multimodal llm logical reasoning benchmark in visual contexts

    Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal llm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.04973, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models

    Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, et al. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. arXiv preprint arXiv:2504.15279, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Puzzlevqa: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns

    Yew Ken Chia, Vernon Toh, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. Puzzlevqa: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16259–16273, 2024

    2024

  42. [42]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24108–24118, 2025

    2025

  43. [43]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024

    2024

  44. [44]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  45. [45]

    Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, page...

    2022

  46. [47]

    •Grammar:Linguistic correctness & fluency of text

    Evaluation Dimensions Score each dimension on a 1-5 scale (1=lowest, 5=highest): •Accuracy:Factual correctness and verifiability of the assistant’s response... •Grammar:Linguistic correctness & fluency of text. •Correlation:Relevance of the question and answer to the image content... •Hallucination:Degree of hallucinated content, i.e., claims or inference...

  47. [48]

    Scoring Protocol - Base scores strictly on concrete, verifiable evidence from both text and image. - Score based on the percentage of errors: 5 points for completely correct, 4 points for errors within 10%, 3 points for errors within 20%, 2 points for errors within 35%, and 1 point for errors at 40% or above. - Flag hallucinations, misinformation, or unsu...

  48. [49]

    dimension_scores

    Output Format Respond with a JSON dictionary: { "dimension_scores": { "accuracy": <int>, "grammar": <int>, "correlation": <int>, "hallucination": <int> }, "flags": "...", "rationale": "...", "recommendation": "keep/review/discard" } B.3. Diversity Categorization Prompt To systematically measure the semantic diversity of candidate datasets, we employ an LL...

  49. [50]

    •OCR:Extracting and using text from images to solve tasks (text_detection, text_recognition, text_understanding, scene_text, document_ocr)

    Primary Categories & Subcategories •Caption:textual descriptions of single or tightly arranged multiple images (basic_caption, fine_grained_caption, scene_caption, comparative_caption, multilingual_caption). •OCR:Extracting and using text from images to solve tasks (text_detection, text_recognition, text_understanding, scene_text, document_ocr). •Groundin...

  50. [51]

    tags": {

    Output Format Respond strictly in JSON with: { "tags": { "primary_category": "<caption/.../text_only>", "subcategory": "<corresponding subcategory>" }, "flags": "<comma-separated data features>", "rationale": "<detailed explanation linking data to category>" } B.4. Difficulty Assessment Formulation The Difficulty score D identifies samples requiring advan...

  51. [52]

    Use 2 or 4 for intermediate difficulty: a

    Core Evaluation Dimensions Rate 6 key difficulty dimensions on a 1-5 scale. Use 2 or 4 for intermediate difficulty: a. Image Complexity:Measures the difficulty of understanding the image on its own. (1: Simple object; 3: Multiple interacting objects; 5: Abstract/hyper-detailed content) b. Text Complexity:Measures the difficulty of understanding the answer...

  52. [53]

    dimension_scores

    Output Format Respond strictly in JSON with: { "dimension_scores": { "image_complexity": <int>, "text_complexity": <int>, "cross_modal_synthesis": <int>, "prior_knowledge_demand": <int>, "visual_cue_sensitivity": <int>, "task_ambiguity": <int> }, "flags": "...", "rationale": "..." } C. Detailed Hardware and Hyperparameters For Stage 2, we use a global bat...

  53. [54]

    penalty,

    Accuracy: The response incorrectly claims the player is celebrating a “penalty,” which has no basis in the image. While the OCR text is accurate, the penalty assertion is a significant factual error ( ∼30%), justifying a score of 2

  54. [55]

    soccer league , match

    Grammar: Misplaced comma (“soccer league , match”) and vague “person” instead of “soccer player” lead to awkward phrasing—score 3

  55. [56]

    Bramble Hall

    Correlation: Relevant elements (celebration, OCR) are present but an irrelevant penalty claim detracts— score 3. 4.Hallucination: The “penalty” claim is a severe hallucination with no visual evidence—score 1. 19 Case 2: High-Quality General Sample Question: <image> Considering the security measures in place at the entrance, what might be the level of secu...

  56. [1595]

    URLhttps://aclanthology.org/2025.acl-long.1595/

    2025