Pith. sign in

REVIEW 3 major objections 5 minor 61 references

PRISM: Priority-aware Rubric Internalization via Structured Multimodal Data Synthesis

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that rubric comprehension — executing a typed, prioritized checklist of rules on an image — can be taught with just 10K synthetically generated samples, lifting Qwen3-VL-4B from 9.5% to 30.1% Strict accuracy on PRISM-Eval…

desk verdict PRISM is a genuinely new executor-side rubric-comprehension formulation with a strong ablation study, but its only in-domain benchmark shares generators with its training data, so the headline gains may be partly distribution matching. read the letter →

arxiv 2608.05249 v1 pith:GS455CME submitted 2026-08-05 cs.LG cs.AI

Xiaomin He , Dongling Xiao , Jiahao Xie , Ruiqi Lu , Qianle Wang , Zhongbin Guo , Wanxuan Sun This is my paper · ORCID
classification cs.LGcs.AI
keywords rubriccomprehensionmultimodalinstructionfollowingdatasynthesispriority-awarereasoningstructuredsupervisionvision-languagemodelsPRISM-Eval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current multimodal instruction data miss a common real-world need: instructions that bundle several requirements of unequal importance. It defines rubric comprehension as the task of executing a typed, prioritized checklist of rules on an image and producing both per-rule judgments and an overall verdict. To create supervision for this task, PRISM synthesizes persona–task pairs, rule sets, quality-filtered rubrics, and structured verification traces from roughly 250K images, yielding a 110K training pool and a 1K evaluation set. The central result is that 10K of these samples lift Qwen3-VL-4B from 9.5% to 30.1% Strict accuracy on PRISM-Eval while preserving general-benchmark performance, with the gain transferring to four other open-source multimodal models. The paper also acknowledges that the evaluation labels come from the same frozen generator that produced the training traces, with independent human validation on 100 of the 1,000 samples.

What carries the argument

The central object is the structured rubric: a typed, prioritized rule set where each rule has a type (Perceptual, Reasoning, Content, Format, Linguistic) and a priority (Critical, Important, Optional), together with the supervision trace $\tau = (g, \{(j_i, e_i)\}, y)$ that the model must emit. The trace couples visual grounding, per-rule verification, and a priority-aware verdict (any Critical failure vetoes; otherwise Important rules weight 3 and Optional rules weight 1), and the aggregation formula is deliberately hidden at inference so the model must internalize it. The PRISM pipeline produces these tuples through four stages — persona–task discovery, prefix-guided rule completion, two-tier rule quality filtering, and structured response generation — and the evaluation protocol scores Loose (overall verdict) and Strict (overall plus every rule) deterministically against fixed labels.

What would settle it

Re-label a large random subset of PRISM-Eval with labels from an independent source — full human annotation or a different generator — and re-run the Baseline vs. PRISM comparison. If the 9.5% to 30.1% Strict gain shrinks markedly when training and evaluation labels no longer share the same generator, the central claim would be falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that placing rubrics on the model-input side and supervising a structured verification trace turns rubric comprehension from a scaling problem into a data-kind problem. Each rule in the rubric carries a type (Perceptual, Reasoning, Content, Format, Linguistic) and a priority (Critical, Important, Optional), and the supervision trace is $\tau = (g, \{(j_i, e_i)\}, y)$: visual grounding, per-rule Pass/Fail judgments with justifications, and an overall verdict computed by a fixed priority-aware policy that is withheld at inference. The paper reports that 10K PRISM samples lift Qwen3-VL-4B from 9.5% to 30.1% Strict accuracy on PRISM-Eval while leaving the general-benchmark average essentially unchanged, and that the same injection raises Strict accuracy by 9.1–20.6 percentage points across four additional open-source backbones spanning dense and MoE architectures. Even GPT-5.4 reaches only 42.1% Strict on the same evaluation, which the paper reads as evidence that the bottleneck is the kind of supervision rather than parameter count.

Load-bearing premise

The load-bearing premise is that the PRISM-Eval labels produced by the frozen Seed2.0 Lite generator are accurate enough that a model trained on the same generator's traces is genuinely learning rubric comprehension rather than imitating the generator's output patterns; only 100 of the 1,000 evaluation samples received independent human adjudication.

Editorial extensions

If this is right

  • A 10K-sample injection of PRISM data is enough to move a 4B model from near-floor Strict accuracy (9.5%) to 30.1%, and the same recipe yields 9.1–20.6 percentage-point Strict gains on four other open-source backbones spanning dense and MoE designs from 9B to 35B.
  • Because removing the Priority field causes the largest measured drop (Strict −6.5, Loose −4.3) while shuffling rule order does not, the paper concludes that priority semantics, not surface ordering, is what the models internalize.
  • The scaling curves imply that rubric fidelity keeps improving with more synthesized supervision (Strict rises monotonically from 10K to 80K samples), while general-benchmark performance peaks at moderate pool sizes.
  • Inference-time strategies such as two-shot trace prompting or per-rule sequential queries recover only part of the gain (21.0% vs. 30.1% Strict on the 4B learner) and cost up to 10,253 calls per sample, so the paper frames training-time internalization as both more accurate and more efficient.
  • If the gains hold, the same synthesis procedure can be applied to new image pools without retraining the pipeline, although the paper notes that independent validation is needed when the source distribution changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the paper's logic: if the priority-aware aggregation policy is truly internalized rather than memorized, then training on rubrics whose gold verdicts are generated by a different, independent policy should still transfer to PRISM-Eval, whereas training on the same generator's policy might not.
  • The typed taxonomy suggests a natural next probe: replacing the five fixed rule types with a larger open type vocabulary, or removing type tags entirely, should show whether the gains come from semantic differentiation or just from having any structured tag.
  • Because the paper's evaluation is restricted to single-image static verification, an untested implication is that the same rubric-comprehension framing could extend to constrained generation (e.g., advertising copy, document compliance) where the model must produce compliant text rather than a verdict; the current discriminative proxy would need re-validation for generative outputs.
  • The 100-sample human validation leaves a cheap, high-value experiment: fully re-labeling all 1,000 PRISM-Eval samples to measure how much of the headline gain survives when every evaluation label is independently grounded.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper defines "rubric comprehension" as an executor-side multimodal task in which a model receives an image and a typed, prioritized rubric and must output per-rule Pass/Fail judgments plus an overall verdict. It proposes PRISM, a four-stage data synthesis pipeline (persona–task discovery, prefix-guided rule completion, quality filtering, and structured trace generation) that produces a 110K training pool and a 1,000-sample benchmark, PRISM-Eval, with deterministic Loose/Strict metrics. On Qwen3-VL-4B, adding 10K PRISM samples raises Strict accuracy from 9.5% to 30.1% with no drop in general-benchmark average, and the gain transfers to four additional open-source MLLMs. Extensive ablations isolate the contributions of quality filtering, structured CoT, and priority semantics; the paper also reports scaling curves and representation-robustness checks.

Significance. If the central claim holds, the paper offers a new task formulation, a credible synthesis recipe, and constructive evidence that structured rubric supervision is a scalable complement to generic visual instruction data. The paper is unusually thorough on internal validity: the pipeline prompts are reproduced in the appendix, the evaluation metrics are deterministic and judge-free, the ablations are well designed, the scaling study covers 10K–80K samples, and the gains replicate across dense and MoE backbones. The principal weakness is that the only in-domain evaluation instrument, PRISM-Eval, is generated by the same pipeline and the same frozen generators as the training supervision; the 100-sample human validation is a partial check on label accuracy but does not establish independence of the task/rule distribution or the label-error patterns. The paper's headline claim is therefore plausible but not yet externally grounded.

major comments (3)
  1. [§3.2 (Stage 4) and §3.3] The supervision traces in Stage 4 and the gold labels in PRISM-Eval are both produced by the same frozen Seed2.0 Lite model, and the rule content is generated by Qwen3-VL-30B in Stage 2; PRISM-Eval is a strict-threshold subset of the same candidate pool that feeds the training data. The headline Strict improvement (9.5% to 30.1%) therefore conflates rubric comprehension with matching the generator's own label and rule distribution. The human validation in §3.3 covers only 100 of 1,000 samples (967 rule labels) and reports a 1.45% label-correction rate, leaving open systematic label biases in the remaining 900 samples; it also validates labels on already-generated rules, not the representativeness of the rule/task distribution. The Limitations section addresses SFT, video, and rubric size but does not mention this overlap. I request an evaluation on independently authored rubrics (e.g., human-written rules or rules from a different generator), or full human adjudication of PRISM-Eval, and at minimum an explicit analysis of whether training and eval label-error patterns coincide.
  2. [§4.6 / Figure 7] The text states that Gen. Avg follows a "shallow inverted-U, peaking at a moderate scale before retreating," but the plotted Gen. Avg values increase from 60.30 at 0K to 61.95 at 80K, with the highest value at the largest scale and only a small dip at 60K. If the 80K point is a typo, it should be corrected; if it is not, the inverted-U conclusion in §4.6 and the "scalable path" wording in the abstract need to be revised.
  3. [§3.2 (aggregation policy) and §4.5] The aggregation formula uses manually specified weights (Important=3, Optional=1) and a failure threshold (s<0.5), and this same formula is used both to construct the supervision target and to derive the gold overall verdicts in PRISM-Eval. The w/o Priority ablation in Table 4 demonstrates that priority labels matter, but it does not validate that this particular weighting reflects human priority semantics or that the learned behavior generalizes to other reasonable aggregation policies. Please add a sensitivity analysis over the weights/threshold, or a small human preference study on overall verdicts under conflicting rule configurations, to support the priority-aware generalization claim.
minor comments (5)
  1. [Table 3] The column header "ChXivd/r" should read "CharXiv d/r" to match the CharXiv benchmark.
  2. [§4.2 / References] The model listed as "LLaVA-1.5 7B" cites reference [60], which is titled "LLaVA-OneVision" in the bibliography; please reconcile the model name and the citation.
  3. [Figure 3] The caption reports 98.55% rule-label agreement between original and adjudicated labels, 99.76% original-adjudicated agreement, and a 1.45% correction rate; the relationship among these numbers should be spelled out in one sentence to avoid confusion.
  4. [§4.4] The claim that the Shuffled-rules variant is "statistically indistinguishable" from the full PRISM model is not supported by any significance test or confidence interval; please either add one or temper the wording.
  5. [Abstract and §4.3] The abstract says PRISM "preserves" average general-benchmark performance, but the main result shows a Gen. Avg increase of +0.57 on Qwen3-VL-4B and positive deltas on most other backbones; "without degrading" or "with slight average gains" would be more accurate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PRISM-Eval shares its generator with the training pool, but as a disjoint held-out benchmark the measured gains are not forced by construction.

full rationale

The paper's claimed derivation is an empirical data-synthesis and fine-tuning pipeline rather than a formal derivation from first principles, and its in-domain evaluation uses a disjoint held-out benchmark, PRISM-Eval, constructed by the same four-stage procedure as the PRISM-110K training pool. The reader's main concern is therefore accurate as a construct-validity limitation: Section 3.2 Stage 4 states that Seed2.0 Lite synthesizes the supervision traces, and Section 3.3 states that 'PRISM-110K and PRISM-Eval are disjoint and share only the generation procedure.' However, under the strict criteria for circularity, this is not a derivation loop: no parameter is fitted to the PRISM-Eval labels; the evaluation samples are disjoint from training samples; the paper explicitly disclaims label infallibility, noting that 'Judge-free refers specifically to scoring' and 'does not assume that synthesized labels are intrinsically error-free' (Section 3.1); and Section 3.3 provides an independent, albeit small, human anchor with 91.31% rule-level annotator agreement and 98.55% original-versus-adjudicated rule-label agreement. The headline result is also not true by construction: an all-Pass predictor reaches only 13.7% Strict accuracy, and the generic data-synthesis baselines in Table 3 cluster near the 9.5% Baseline rather than near the 30.1% PRISM result. The transfer of gains to four additional backbones, the preservation of general-benchmark performance, and the representation-robustness variants in Section 4.7 are externally falsifiable outcomes that do not reduce to the training objective. The only self-citation, the ByteDance Seed2.0 Lite model card, is used as a data generator and leaderboard reference, not as an authority, uniqueness argument, or justification for the method's correctness. Accordingly, no equation in the paper is equivalent to its inputs by construction, and no fitted parameter is relabeled as a prediction; the same-generator evaluation loop should be noted as a limitation for external generalization but is not circularity under the definitions used here.

Assumptions & free parameters 2 free parameters · 4 assumptions · 2 invented entities

The method's effectiveness depends on design choices and assumptions that are not derived from external data: hand-set aggregation weights, hand-set quality thresholds, the assumed validity of the synthetic labels, and the representativeness of Cambrian images. The human validation provides partial support for label accuracy but does not independently validate the benchmark distribution.

free parameters (2)
  • Aggregation weights (Important=3, Optional=1) and fail threshold (s<0.5) = 3, 1, 0.5
    Used in Stage 4 to construct supervision targets and eval labels; hand-set without external grounding.
  • Stage 3 quality thresholds (relaxed: min score>=4 and at least two 5s; strict: all 5s) = relaxed min 4 / two 5s; strict 5/5/5/5
    Hand-chosen admission criteria for train vs eval pools; determine PRISM-110K and PRISM-Eval composition.
assumptions (4)
  • domain assumption The rubric comprehension operationalization (typed rules, three priorities, per-rule verification, priority-weighted aggregation) captures real-world multi-rule instruction following.
    The paper defines the task itself with no external validation task; all evaluations use this operationalization.
  • domain assumption Seed2.0 Lite and Qwen3-VL-30B generate high-quality rubrics, labels, and traces with acceptable error rates.
    Used for stages 1, 3, and 4 (Seed2.0 Lite) and stage 2 (Qwen3-VL-30B); label accuracy checked only on a 100-sample human subset.
  • domain assumption The Cambrian-7M image pool provides representative coverage of real-world multimodal instruction scenarios.
    All source images come from Cambrian; no independent evidence that this distribution corresponds to real user rubric requests.
  • domain assumption The 100-example human validation subset is representative of the full 1,000-example PRISM-Eval.
    The paper states a stratified 100-example subset was validated, but does not show that the remaining 900 have similar quality.
invented entities (2)
  • Rubric comprehension task (executor-side)
    purpose: Defines the training objective and evaluation protocol for multi-rule, priority-aware instruction following.
    The task is introduced by this paper; no prior benchmark exists to calibrate it.
  • PRISM-Eval benchmark
    purpose: Provides Loose and Strict accuracy measures for trained models.
    Human adjudication on 100 samples gives partial label-quality evidence, but the benchmark itself is generated by the same PRISM pipeline and is not an external test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PRISM: Priority-aware Rubric Internalization via Structured Multimodal Data Synthesis." pith.science (2026). https://pith.science/paper/GS455CME

@misc{pith2026260805249,
  author       = {Pith},
  title        = {Pith review of: PRISM: Priority-aware Rubric Internalization via Structured Multimodal Data Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GS455CME}},
  note         = {Machine review of arXiv:2608.05249}
}
read the original abstract

Real-world multimodal instructions often bundle multiple requirements with unequal importance, yet most multimodal training data still reduce instruction following to answering one self-contained question. We study this gap through \textbf{rubric comprehension}, which casts the model not as a generator measured against rubrics but as an \textbf{executor} that follows them: given an image and a typed, prioritized rubric, the model must verify each rule before producing an overall judgment. To support this setting, we propose \textbf{PRISM}, a four-stage data synthesis framework that produces persona--task pairs, prefix-guided rule sets, quality-filtered rubrics, and structured verification traces. We further introduce \textbf{PRISM-Eval}, whose Loose and Strict metrics use deterministic matching against fixed labels and therefore require no inference-time judge model. With only 10K synthesized samples, PRISM lifts Qwen3-VL-4B from 9.5\% to 30.1\% Strict accuracy on PRISM-Eval while preserving average performance on general benchmarks, and the gains transfer to four additional open-source MLLMs across dense and MoE architectures, suggesting that structured rubric supervision is a scalable path toward multi-rule, priority-aware multimodal instruction following.

Figures

Figures reproduced from arXiv: 2608.05249 by the authors.

Figure 1
Figure 1. From standard VQA to rubric comprehen￾sion. For instance, when asked to write a poster slogan, the user may have multiple requirements. Each such requirement is a rule ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the PRISM data synthesis pipeline. Part (a) shows the four stages of PRISM: (1) Persona– Task Discovery, (2) Prefix-Guided Rule Completion, (3) Rule Quality Assessment, and (4) Structured Response Generation, which produces the supervision trace τ=(g, {(ji , ei)}, y). PRISM is employed to generate PRISM￾110K for SFT and PRISM-Eval for evaluation, as shown in part (b). Stage 1: Persona–Task Discovery. Giv… view at source ↗
Figure 3
Figure 3. Independent human validation of PRISM-Eval. (a) Agreement between two independent an￾notators. (b) Original and adjudicated human rule labels agree on 98.55% of 967 rules. (c) Critical rules show particularly high reliability: 409 of 410 original labels are retained. Using the adjudicated labels as an alternative gold view yields the same conclusion as the full benchmark. On Qwen3-VL-4B, PRISM raises per-rule accura… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Diversity of PRISM-110K. (a) The Sankey diagram connects image domain, rule type, and priority. (b) The adjacent table reports exact-duplicate and semantic near-neighbor rates (%); contextualized rules include their persona–task context. Diversity & composition [PITH_…
Figure 5
Figure 5. Figure 5: PRISM-110K data composition. Left: Domain-level distribution of PRISM-110K. The inner ring follows the verified domain proportions (General 43.1%, OCR 35.5%, Counting 8.6%, Math 6.5%, Science 6.3%), while the outer ring preserves the corresponding child-dataset proport…
Figure 6
Figure 6. Figure 6: Error breakdown on PRISM-Eval. Baseline denotes Qwen3-VL-4B fine-tuned on LLaVA-NeXT-100K, and PRISM denotes the same backbone fine-tuned on LLaVA-NeXT-100K + 10K PRISM samples; GPT-5.4 is the strongest closed-source reference. We report per-rule accuracy stratified by…
Figure 7
Figure 7. Figure 7: Scaling curves of PRISM on Qwen3-VL-4B. We investigate the effect of training-set size (from 10K to 80K synthesized samples) on Qwen3-VL-4B performance. The two curves diverge in shape. Gen. Avg follows a shal￾low inverted-U, peaking at a moderate scale before retreati…
Figure 8
Figure 8. Figure 8: Training-free inference strategies on PRISM-Eval. Two-shot trace prompting and Sequential rule-wise querying recover part of the capability, but PRISM achieves the highest Loose and Strict accuracy with a single joint call. Hatching denotes Strict accuracy. 13 [PITH_F…
Figure 9
Figure 9. Figure 9: Prompt used in Stage 1 to jointly extract image intent, derive a domain-specific persona, and propose a multi-criteria judgment task. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Prompt used in Stage 3 rubric quality assessment. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Prompt used in Stage 4 to elicit the structured discriminative trace τ = (g, {(ji , ei)}, y). Type tags steer the per-rule reasoning mode, while priority tags impose a Critical-first verification order and an aggregation rule under priority precedence. 21 [PITH_FULL_…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 22 canonical work pages

  1. [1]

    Introducing GPT-5.4

    OpenAI. Introducing GPT-5.4. https://openai.com/index/introducing-gpt-5-4/ , 2025. Blog post

    work page 2025

  2. [2]

    Qwen3-VL Technical Report, November 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

    arXiv 2025

  3. [3]

    MMEvol: Empowering multimodal large language models with evol-instruct.arXiv preprint arXiv:2409.05840, 2024

    Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Yongbin Li, Minzheng Wang, Pengpeng Zeng, Lianli Gao, Heng Tao Shen, Yunshui Li, Xiaobo Xia, Fei Huang, and Jingkuan Song. MMEvol: Empowering multimodal large language models with evol-instruct.arXiv preprint arXiv:2409.05840, 2024. URLhttps://arxiv.org/abs/2409.05840

    arXiv 2024

  4. [4]

    Mulberry: Empowering MLLM with o1-like reasoning and reflection via collective monte carlo tree search

    Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Feng, Yibo Wan, Shunyu Xiao, Shuai Liu, Yijie Guo, Weihao Zhang, et al. Mulberry: Empowering MLLM with o1-like reasoning and reflection via collective monte carlo tree search. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. Spotlight

    work page 2025

  5. [5]

    LLaVA-CoT: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440, 2024

    Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. LLaVA-CoT: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440, 2024. doi: 10.48550/arXiv.2411.10440. URL https://arxiv.org/abs/2411.10440

    arXiv doi:10.48550/arxiv.2411.10440 2024

  6. [6]

    Oasis: One image is all you need for 14 multimodal instruction data synthesis.arXiv preprint arXiv:2503.08741, 2025

    Letian Zhang, Quan Cui, Bingchen Zhao, and Cheng Yang. Oasis: One image is all you need for 14 multimodal instruction data synthesis.arXiv preprint arXiv:2503.08741, 2025. URL https://arxiv. org/abs/2503.08741

    arXiv 2025

  7. [8]

    Yun He, Wenzhe Li, Hejia Zhang, Songlin Li, Karishma Mandyam, Sopan Khosla, Yuanhao Xiong, Nanshu Wang, Selina Peng, Beibin Li, Shengjie Bi, Shishir G. Patil, Qi Qi, Shengyu Feng, Julian Katz-Samuels, Richard Yuanzhe Pang, Sujan Gonugondla, Hunter Lang, Yue Yu, Yundi Qian, Maryam Fazel-Zarandi, Licheng Yu, Amine Benhalloum, Hany Awadalla, and Manaal Faruq...

    arXiv 2025

  8. [9]

    Rubrics as rewards: Reinforcement learning beyond verifiable domains.arXiv preprint arXiv:2507.17746, 2025

    Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains.arXiv preprint arXiv:2507.17746, 2025. doi: 10.48550/arXiv.2507.17746. URLhttps://arxiv.org/abs/2507.17746

    arXiv doi:10.48550/arxiv.2507.17746 2025

Show all 61 references
  1. [10]

    Alternating reinforcement learning for rubric-based reward modeling in non-verifiable LLM post-training.arXiv preprint arXiv:2602.01511, 2026

    Ran Xu, Tianci Liu, Zihan Dong, Tony Yu, Ilgee Hong, Carl Yang, Linjun Zhang, Tuo Zhao, and Haoyu Wang. Alternating reinforcement learning for rubric-based reward modeling in non-verifiable LLM post-training.arXiv preprint arXiv:2602.01511, 2026. URLhttps://arxiv.org/abs/2602.01511

    2026

  2. [11]

    Magpie: Alignment data synthesis from scratch by prompting aligned LLMs with nothing.arXiv preprint arXiv:2406.08464, 2024

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned LLMs with nothing.arXiv preprint arXiv:2406.08464, 2024. URLhttps://arxiv.org/abs/2406.08464

    2024 arXiv

  3. [12]

    InFoBench: Evaluating instruction following ability in large language models

    Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. InFoBench: Evaluating instruction following ability in large language models. InFindings of the Association for Computational Linguistics: ACL 2024, p...

    2024 doi

  4. [13]

    Breaking the exploration bottleneck: Rubric-scaffolded reinforcement learning for general LLM reasoning.arXiv preprint arXiv:2508.16949, 2025

    Yang Zhou, Sunzhu Li, Shunyu Liu, Wenkai Fang, Kongcheng Zhang, Jiale Zhao, Jingwen Yang, Yihe Zhou, Jianwei Lv, Tongya Zheng, Hengtong Lu, Wei Chen, Yan Xie, and Mingli Song. Breaking the exploration bottleneck: Rubric-scaffolded reinforcement learning for general LLM reasoni...

    2025 arXiv

  5. [14]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information Processing Systems, 37:87310–87356, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai C Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information Processing Systems, 37:87310–...

    2024

  6. [15]

    Seed 2.0 Model Card

    ByteDance. Seed 2.0 Model Card. https://lf3-static.bytednsdoc.com/obj/eden-cn/ lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf, 2026. Model card

    2026

  7. [16]

    Allava: Harnessing gpt4v-synthesized data for a lite vision-language model.arXiv preprint arXiv:2402.11684, 2024

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model.arXiv preprint arXiv:2402.11684, 2024

    2024 arXiv

  8. [17]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InECCV, 2014

    2014

  9. [18]

    Q-instruct: Improving low-level visual abilities for multi-modality foundation models.arXiv preprint arXiv:2311.06783, 2023

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Kaixin Xu, Chunyi Li, Jingwen Hou, Guangtao Zhai, et al. Q-instruct: Improving low-level visual abilities for multi-modality foundation models.arXiv preprint arXiv:2311.06783, 2023

    2023 arXiv

  10. [19]

    Jordi Pont-Tuset, Jasper R. R. Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. InECCV, 2020

    2020

  11. [20]

    How many unicorns are in this image? a safety evaluation benchmark for vision llms.arXiv preprint arXiv:2311.16101, 2023

    Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, 15 Huaxiu Yao, and Cihang Xie. How many unicorns are in this image? a safety evaluation benchmark for vision llms.arXiv preprint arXiv:2311.16101, 2023

    2023 arXiv

  12. [21]

    Fine-tuning large vision-language models as decision-making agents via reinforcement learning.arXiv preprint arXiv:2405.10292, 2024

    Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning.arXiv preprint arXiv:2405.10292, 2024

    2024 arXiv

  13. [22]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. InCVPR, 2018

    2018

  14. [23]

    Visually dehallucinative instruction generation: Know what you don’t know.arXiv preprint arXiv:2402.09717, 2024

    Sungguk Cha, Jusung Lee, Younghyun Lee, and Cheoljong Yang. Visually dehallucinative instruction generation: Know what you don’t know.arXiv preprint arXiv:2402.09717, 2024

    2024 arXiv

  15. [24]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations.IJCV, 2016

    2016

  16. [25]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InCVPR, 2019

    2019

  17. [26]

    Dvqa: Understanding data visualizations via question answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. InCVPR, 2018

    2018

  18. [27]

    Donut: Document understanding transformer without ocr

    Geewook Kim, Teakgyu Hong, Moonbin Yim, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Donut: Document understanding transformer without ocr. InECCV, 2022

    2022

  19. [28]

    Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models.arXiv preprint arXiv:2403.00231, 2024

    Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models.arXiv preprint arXiv:2403.00231, 2024

    2024 arXiv

  20. [29]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. InEuropean conference on computer vision, pages 235–251. Springer, 2016

    2016

  21. [30]

    Screenqa: Large-scale question-answer pairs over mobile app screenshots.arXiv preprint arXiv:2209.08199, 2022

    Yu-Chung Hsiao, Fedir Zubach, Maria Wang, et al. Screenqa: Large-scale question-answer pairs over mobile app screenshots.arXiv preprint arXiv:2209.08199, 2022

    2022 arXiv

  22. [31]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding.arXiv preprint arXiv:2306.17107, 2023

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding.arXiv preprint arXiv:2306.17107, 2023

    2023 arXiv

  23. [32]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InWACV, 2021

    2021

  24. [33]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. InACL, 2022

    2022

  25. [34]

    Seq2sql: Generating structured queries from natural language using reinforcement learning.arXiv preprint arXiv:1709.00103, 2017

    Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning.arXiv preprint arXiv:1709.00103, 2017

    2017 arXiv

  26. [35]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. InInternational Conference on Document Analysis and Recognition, 2019

    2019

  27. [36]

    Compositional semantic parsing on semi-structured tables

    Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. InACL, 2015

    2015

  28. [37]

    Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning

    Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song- Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. InNeurIPS, 2021

    2021

  29. [38]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross 16 Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017

    2017

  30. [39]

    Tallyqa: Answering complex counting questions

    Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. InAAAI, 2019

    2019

  31. [40]

    G-llava: Solving geometric problem with multi-modal large language model.arXiv preprint arXiv:2312.11370, 2023

    Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. G-llava: Solving geometric problem with multi-modal large language model.arXiv preprint arXiv:2312.11370, 2023

    2023 arXiv

  32. [41]

    Measuring multimodal mathematical reasoning with math-vision dataset.Advances in Neural Information Processing Systems, 37:95095–95169, 2024

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset.Advances in Neural Information Processing Systems, 37:95095–95169, 2024

    2024

  33. [42]

    Raven: A dataset for relational and analogical visual reasoning

    Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. InCVPR, 2019

    2019

  34. [43]

    Xing, and Pengtao Xie

    Xuehai He, Yichen Zhang, Luntian Mou, Eric P. Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering.CoRR, abs/2003.10286, 2020

    2003 arXiv

  35. [44]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. InNeurIPS, 2022

    2022

  36. [45]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

    2025 arXiv

  37. [46]

    Qwen3.5.https://qwen.ai/blog?id=qwen3.5, 2026

    Qwen Team. Qwen3.5.https://qwen.ai/blog?id=qwen3.5, 2026. Blog post

    2026

  38. [47]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024

    2024 arXiv

  39. [48]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024

    2024

  40. [49]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.Advances in Neural Information Processing Systems, 38, 2026

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.Advances in Neural Information Processing Systems, 38, 2026

    2026

  41. [50]

    Are we on the right way for evaluating large vision-language models?Advances in Neural Information Processing Systems, 37:27056–27087, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?Advances in Neural Information Processing Systems, 37:27056–27087, 2024

    2024

  42. [51]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF conference on co...

    2024

  43. [52]

    Ocrbench: on the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):220102, 2024

    2024

  44. [53]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022

    2022

  45. [54]

    Charxiv: Charting gaps in realistic chart understanding in multimodal llms.Advances in Neural Information Processing Systems, 37:113569–113697, 2024

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms.Advances in Neural Information Processing Systems, 37:113569–113697, 2024. 17

    2024

  46. [55]

    Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models

    Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. In International Conference on Learning Representations, volume 2025, pages 48337–48383, 2025

    2025

  47. [56]

    Logicvista: Multimodal llm logical reasoning benchmark in visual contexts.arXiv preprint arXiv:2407.04973, 2024

    Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal llm logical reasoning benchmark in visual contexts.arXiv preprint arXiv:2407.04973, 2024

    2024 arXiv

  48. [57]

    Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models.arXiv preprint arXiv:2504.15279, 2025

    Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, et al. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models.arXiv preprint arXiv:2504.15279, 2025

    2025 arXiv

  49. [58]

    Teaching CLIP to Count to Ten.arXiv preprint arXiv:2302.12066, 2023

    Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching CLIP to Count to Ten.arXiv preprint arXiv:2302.12066, 2023

    2023 arXiv

  50. [59]

    Gemini 3 Flash: Frontier intelligence built for speed

    Google. Gemini 3 Flash: Frontier intelligence built for speed. https://blog.google/products/ gemini/gemini-3-flash/, 2025. Blog post

    2025

  51. [60]

    usage intent

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024. 18 Prompt Used for Persona–Task Generation [SYSTEM PROMPT] You are a m...

    2024 arXiv

  52. [772]

    URLhttps://aclanthology.org/2024.findings-acl.772/

    2024

  53. [2023]

    URLhttps://arxiv.org/abs/2311.07911

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.