Pith. sign in

REVIEW 4 major objections 7 minor 3 cited by

Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An external critic model trained on synthetic corrupted answers can catch and fix VLM reasoning errors, pushing Qwen2-VL-7B past GPT-4V on five of eight multimodal benchmarks.

desk verdict A likely-useful external-critic method for VLMs, but the unnamed training sources and missing decontamination leave the headline gains unverified. read the letter →

arxiv 2411.18203 v5 pith:YLI236GC submitted 2024-11-27 cs.CV cs.CL

Di Zhang , Junxian Li , Jingdi Lei , Xunzhi Wang , Yujie Liu , Zonglin Yang , Jiatong Li , Weida Wang
show 5 more authors
Suorong Yang Jianbo Wu Peng Ye Wanli Ouyang Dongzhan Zhou
This is my paper · ORCID
classification cs.CVcs.CL
keywords multimodalreasoningvision-languagemodelscriticmodeldirectpreferenceoptimizationself-correctionhallucinationdetectionrule-basedrewardVisualQAbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a separate, trained critic model can catch and correct errors made by a vision-language model, improving multimodal reasoning more than letting the model critique itself. It does this by generating 29,012 examples where a strong model inserts fake details into true answers, having several VLMs write critiques of the tampered answers, ranking those critiques, and training a critic with direct preference optimization. At inference, the critic's natural-language feedback is appended to the reasoner's prompt in a loop. The authors report that a 7B model with this critic beats the proprietary GPT-4V on five of eight benchmarks, with the largest gains on visual math tasks.

What carries the argument

The load-bearing mechanism is the Reasoner-Critic loop, framed as in-context reinforcement learning in which the critique itself acts as the gradient signal. The training data comes from the Vision Error Insertion Technique (VEST): GPT-4o adds one to five false statements to a correct VQA answer, three different VLMs write critiques identifying the false statements, and a rule-based reward scores each critique by $J(G,C)=|G\cap C|/|G\cup C|$ plus a GPT-based regularization term. The Critic is then trained with DPO to prefer high-scoring critiques, and at inference the Reasoner's prompt is updated by concatenating the Critic's feedback, so the text prompt itself plays the role of a policy that evolves.

What would settle it

A reader could settle the claim by taking the released critique-VQA dataset, checking its questions against the questions in MathVista, MathVerse, MMBench, SEEDBench, MMStar, MMT-Bench, ScienceQA, and RealWorldQA, and re-running the evaluation only on the non-overlapping subset; if the gains shrink to near the Self-Refine baseline, the conclusion that the trained critic generalizes would not hold.

Watch

Extended reading notes

Core claim

The claim is that VLM reasoning errors can be reduced by decoupling reasoning from evaluation: an independent Critic, trained to spot injected errors, gives natural-language feedback that the Reasoner folds into its text prompt, iterating until the Critic is satisfied. The Critic is a Qwen2-VL-7B fine-tuned with DPO on a dataset of 29,012 critique pairs, where the preferred critique is selected by a rule-based reward that combines a Jaccard overlap between injected and detected errors with a GPT-4o quality score. When the loop is applied to Qwen2-VL-7B and DeepSeek-VL-7B, the paper reports gains on 23 of 24 benchmark/model comparisons and top scores on five of eight benchmarks, including +11.8 on MathVista and +7.1 on MathVerse for Qwen2-VL-7B.

Load-bearing premise

The training pairs come from question-image sets that are not named, so the reported gains assume none of those questions overlap with the eight evaluation benchmarks.

Editorial extensions

If this is right

  • A single DPO-trained critic can be bolted onto at least three different 7B reasoners, Qwen2-VL-7B, DeepSeek-VL-7B, and LLaVA-v1.5-7B, and improve accuracy across most benchmarks without retraining the reasoner.
  • The largest gains occur on math-heavy benchmarks, suggesting the loop is most valuable when reasoning errors, rather than pure perception errors, dominate the failure cases.
  • The critic adds only a few dozen extra tokens per question, so its accuracy benefit comes with modest additional inference cost.
  • DPO training of the critic is the key ingredient: a Self-Refine baseline without the trained critic improves far less and even hurts on one benchmark.
  • Because the framework formalizes critique as a text-prompt policy update, the same loop can inject constraints or knowledge into reasoning, not just corrections of factual errors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: since the critic is trained only on injected textual errors, its benefit may concentrate on answer-level verification rather than deeper plan-level reasoning; a test that asks the critic to flag flawed intermediate reasoning steps, not just false facts, would separate the two.
  • Beyond the paper: the synthetic-corruption and preference-ranking recipe could be applied to train critics for video, audio, or embodied action sequences, where ground-truth answers are harder to obtain but injected errors are easy to define.
  • Beyond the paper: if the unnamed VQA sources behind the 29,012 training pairs are disclosed, a direct overlap check against the eight evaluation benchmarks would reveal whether part of the reported gain is memorization rather than generalized criticism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Critic-V, an actor-critic style framework for vision-language models in which a separate Critic VLM provides natural-language feedback to a Reasoner VLM. The Critic is trained with DPO on a preference dataset of critiques built by VEST: GPT-4o inserts one to five fake details into ground-truth VQA answers, several VLMs produce critiques, and a rule-based reward combining a Jaccard index with a GPT-4o evaluation score ranks the critiques. At inference, the Reasoner generates an answer, the Critic critiques it, and the Reasoner revises its answer with the critique appended to the prompt, iterating until a stopping condition. Experiments on eight multimodal benchmarks show consistent improvements over the base models (Qwen2-VL-7B, DeepSeek-VL-7B, LLaVA-v1.5-7B), with Qwen2-VL-7B+Critic-V reported to outperform GPT-4V on five of the benchmarks.

Significance. If the empirical claims hold, Critic-V offers a practical, model-agnostic recipe for improving VLM reasoning by adding an external, preference-optimized critic, and the released 29,012-pair critique dataset is a useful resource. The paper's strengths include a clean framework, a large constructed dataset, consistent improvements across many benchmarks, and ablations separating the contribution of DPO training and of the evaluation prompt. The main significance caveat is that the transferability claim -- that the critic improves reasoning generally rather than memorizing question-specific hints -- depends on training-data provenance and on how well GPT-4o-inserted errors match real VLM errors; both are currently unverified.

major comments (4)
  1. [Section 2.1, Section 6 (Algorithm 3), Appendix 10] The training data provenance is not disclosed. The paper states that VEST is applied to 'question-image pairs from VQA datasets' (Section 2.2) and Figure 3 says only 'Several VQA Datasets', while Table 7 reports only token statistics, not the source datasets or splits. The evaluation suite (RealWorldQA, MathVista, MathVerse, ScienceQA, MMT-Bench, MMBench, MMStar, SEEDBench) itself consists of VQA-style benchmark datasets. If any of these benchmark questions, images, or near-duplicates appear in the critique-VQA training set, the critic could memorize question-specific facts from the GPT-4o-generated fake answers and the VLM critiques, and then supply targeted hints on exactly those benchmark questions, inflating gains such as MathVista +11.8 and MathVerse +7.1 without improving general reasoning. The paper contains no decontamination statement and no overlap analysis. Because the framework's practical value rests on the critic's transferable feedback, the authors should name the contributing datasets and splits and report a formal overlap check against all eight evaluation benchmarks.
  2. [Section 2.2, Eq. (8), Appendix 9] The theoretical framing does not match the implemented algorithm. Equations (2)-(5) describe a reinforcement-learning/TextGrad-style prompt update with gradients and a learning-rate parameter eta, and Equation (6) describes a policy-gradient update for the Critic. However, Algorithm 3 implements a simple loop where the critique is concatenated to the prompt (P_reasoner <- P_reasoner + delta P_reasoner), and Appendix 10 states eta=1.0, i.e., full concatenation. TextGrad is not used in the evaluation loop, the Critic is not updated by policy gradient, and the 'satisfactory' stopping condition in Algorithm 3 is not supported by any mechanism described in the Critic's training or inference prompt. The claim that the framework is 'theoretically driven by a reinforcement learning framework' (Abstract) is therefore not substantiated by the presented algorithm. Please either align the theoretical development with the actual mechanism or remove/replace the RL/TextGrad claims.
  3. [Table 1 and Section 3.2] The critique-quality signal is self-referential in a way that may limit transfer. The fake errors inserted into the ground-truth answers are generated by GPT-4o (Section 2.2, Algorithm 1), and the RBR score itself includes a GPT-4o-based evaluation term (Eq. (8), with alpha=0.1). The Critic is therefore trained to detect the particular error distribution that GPT-4o produces when asked to fabricate details, and the preference labels are partly assigned by the same model family. The paper assumes, but does not demonstrate, that this distribution simulates the errors made by Qwen2-VL-7B, DeepSeek-VL-7B, and LLaVA-v1.5-7B at inference time. Since the central claim is that the trained critic generalizes to real VLM errors, the authors should provide evidence for this match: for example, compare the distribution of GPT-4o-inserted bugs with actual VLM error patterns on a held-out sample, or evaluate the critic's error-detection accuracy against human-annotated VLM errors.
  4. [Section 3.2] The empirical comparison lacks variance or significance information. Table 1 reports single percentage numbers per benchmark, and the text describes improvements such as '+4.8', '+2.1', '+1.8', '+1.6', and '+0.4' as significant, but no standard deviations, confidence intervals, or repeated-evaluation results are provided. Even with temperature near zero, evaluation randomness can arise from benchmark subsets, decoding settings, or the Critic's stopping decisions. For the smaller deltas, the claim that Critic-V outperforms baselines on those benchmarks is not statistically supported. Please add multiple runs with variance estimates or a bootstrap analysis over the evaluation questions, particularly for the benchmarks with gains below about two points.
minor comments (7)
  1. [Figure 3 caption] The heading 'Result ans Analysis' contains a typo; it should read 'Results and Analysis'.
  2. [Section 2.1, Eq. (4)] The caption uses 'VisualQA (critique-VQA)', but the dataset is called 'critique-VQA' elsewhere; unify the name.
  3. [Table 1] Equation (4) is mathematically unclear: it writes pi_theta_critic(delta P | P) = E[pi_theta_critic(delta P | P, s, a)] without specifying the distribution over which the expectation is taken. Please provide a precise definition or remove the equation.
  4. [References] In Table 1, ScienceQA is marked as '(mm-only)' for some models but not for others; the text and benchmark description in Section 3.1 do not explain this difference. Clarify which splits are used for each model.
  5. [Appendix 9] Reference [53] is written as 'X. Grok-1.5 vision preview'; this is not a proper citation for RealWorldQA. The dataset is from xAI and should be cited with its official title and authors.
  6. [Section 2.3 and Algorithm 3] The DPO hyperparameter section states 'preference loss is set to sigmoid' and 'preference parameter beta of 1.0'; it is helpful to state explicitly that this is the standard DPO loss from Eq. (9) and whether the reference model is the frozen base Qwen2-VL-7B.
  7. [Section 2.2] Algorithm 3's stopping condition 'If Critic determines that critique is satisfactory' is not operationalized. It is unclear whether the Critic is given a binary satisfaction task or whether the loop always runs to 'max iterations'; please specify the stopping decision used in the experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline benchmark comparisons are external measurements, and the DPO training signal includes an independent Jaccard ground-truth term.

full rationale

The paper's central empirical claim is that adding the Critic-V critic improves reasoner accuracy on eight external benchmarks. That claim is not forced by construction: the critique-VQA training set is generated by inserting GPT-4o fake details into VQA answers and having several VLMs critique them, and the preference score used for DPO is Score(i) = Jaccard(i) + alpha * GPT(i), where the Jaccard index independently compares the set of errors detected by the critique against the set of errors actually planted (Eq. 7--8). The GPT term is a regularization component, not the sole ground truth. No parameter fitted to one benchmark is later renamed as a prediction on that benchmark, and the reasoner's policy-gradient equations (Eqs. 1--6) are formal descriptions rather than fitted quantities that reappear as results. The paper's self-citations ([60], [61]) appear only in related work and are not load-bearing for the method or evaluation. The reviewer concern about unnamed training VQA sources and possible benchmark overlap is a data-contamination and generalization risk, not a demonstrated circular reduction: the paper provides no equation or construction that identifies the training data with the evaluation data. Under the rule that circularity must be exhibited by a specific reduction in the text, no such reduction is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the synthetic-error distribution matching real VLM errors, the RBR ranker being a valid critique-quality measurement, and no overlap between the unnamed training VQA sources and evaluation benchmarks. None of these are established with evidence, which is why the soundness score is a 4. The free parameters are α, η, and standard DPO hyperparameters; they are set by inspection rather than fitted to the evaluation results.

free parameters (3)
  • α (RBR weighting) = 0.1
    Weight combining Jaccard and GPT-4o score in Eq. 8; no sensitivity analysis is reported.
  • η (prompt update learning rate) = 1.0
    Appendix 10 sets η=1.0, meaning the critique is fully concatenated to the prompt; the "learning rate" framing is not used in any learning step.
  • DPO training hyperparameters = β=1.0, LoRA rank=8, α=16, lr=5e-6, 3 epochs
    Chosen without ablations reported; these affect Critic quality but are standard tuning, not central fitted values.
assumptions (4)
  • ad hoc to paper GPT-4o-inserted fake details simulate the real error distribution of VLMs at inference time.
    VEST is used to build the training signal; the paper provides no validation that synthetic errors resemble reasoner errors.
  • domain assumption The RBR score (Jaccard plus GPT-4o judgment) correctly ranks critique quality.
    The DPO preference pairs are built from this ranking; if the ranker is biased, the Critic is trained to mimic a biased notion of quality.
  • domain assumption The training VQA datasets do not overlap with the evaluation benchmarks.
    The paper names neither training datasets nor an overlap-exclusion step; this is load-bearing for the validity of the benchmark gains.
  • domain assumption A two-round greedy conversation with temperature near zero is a fair, stable evaluation protocol.
    All results are single deterministic runs without variance estimates; conclusions about significance rest on this protocol.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning." pith.science (2026). https://pith.science/paper/YLI236GC

@misc{pith2026241118203,
  author       = {Pith},
  title        = {Pith review of: Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YLI236GC}},
  note         = {Machine review of arXiv:2411.18203}
}
read the original abstract

Vision-language models (VLMs) have shown remarkable advancements in multimodal reasoning tasks. However, they still often generate inaccurate or irrelevant responses due to issues like hallucinated image understandings or unrefined reasoning paths. To address these challenges, we introduce Critic-V, a novel framework inspired by the Actor-Critic paradigm to boost the reasoning capability of VLMs. This framework decouples the reasoning process and critic process by integrating two independent components: the Reasoner, which generates reasoning paths based on visual and textual inputs, and the Critic, which provides constructive critique to refine these paths. In this approach, the Reasoner generates reasoning responses according to text prompts, which can evolve iteratively as a policy based on feedback from the Critic. This interaction process was theoretically driven by a reinforcement learning framework where the Critic offers natural language critiques instead of scalar rewards, enabling more nuanced feedback to boost the Reasoner's capability on complex reasoning tasks. The Critic model is trained using Direct Preference Optimization (DPO), leveraging a preference dataset of critiques ranked by Rule-based Reward~(RBR) to enhance its critic capabilities. Evaluation results show that the Critic-V framework significantly outperforms existing methods, including GPT-4V, on 5 out of 8 benchmarks, especially regarding reasoning accuracy and efficiency. Combining a dynamic text-based policy for the Reasoner and constructive feedback from the preference-optimized Critic enables a more reliable and context-sensitive multimodal reasoning process. Our approach provides a promising solution to enhance the reliability of VLMs, improving their performance in real-world reasoning-heavy multimodal applications such as autonomous driving and embodied intelligence.

Figures

Figures reproduced from arXiv: 2411.18203 by the authors.

Figure 1
Figure 1. Offline training of critic model and response supervision [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The scoring method combines GPT’s evaluation with several predefined rules and the Jaccard index. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The annotation framework for our critique on the VisualQA (critique-VQA) dataset. We collect questions and images from [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The comparison between GPT-4V and Qwen2-VL [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Case studies on evaluation samples from ScienceQA (left) and SEEDBench (right). Our Critic-V accurately identifies Salem as [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 8
Figure 8. Figure 8: Learning rate vs. training steps. Question: As shown in the figure, click C on the ray BM, CF is the flat line of ∠ACM, and CF∥AB, ∠ACB = 50 °, then the degree of ∠B is (), Answer: This question examines the nature of the parallel line, the definition of the angle divi…
Figure 6
Figure 6. Figure 6: Training loss vs. training steps. 0 100 200 300 400 Current Step 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Accuracy vs Current Step [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Training accuracy vs. training steps. 13. Our critique-VQA Dataset Example In this section, we show three examples in [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 10
Figure 10. Figure 10: A real-world example of public market signage. Fake [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: A driving car example. Fake Answer indicates the [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model

    cs.CV 2025-01 conditional novelty 6.0 of 10

    IXC-2.5-Reward is an open-source multimodal reward model that achieves 70.0% macro accuracy on VL-RewardBench and improves LVLM chat via PPO.

  2. VERDICT: Training-Free Step-Wise Verification of Multimodal Reasoning via Disagreement-Aware Consensus

    cs.AI 2026-08 conditional novelty 5.0 of 10

    VERDICT verifies each reasoning step of a multimodal LLM by combining scores from three frozen judges through a closed-form consensus, reporting up to +5.95 accuracy points across six benchmarks.

  3. Quo Vadis, World Modeling?

    cs.CV 2026-08 conditional novelty 5.0 of 10

    An agent-centric reframing of world modeling, replacing physical state prediction with 'information transitions' organized into six proxy functions and three empowerment levels.

Reference graph

Works this paper leans on

79 extracted references · 30 canonical work pages · cited by 3 Pith papers

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  2. [2]

    Openflamingo: An open- source framework for training large autoregressive vision- language models

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision- language models. arXiv preprint arXiv:2308.01390 , 2023. 1

    arXiv 2023

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 8

    arXiv 2023

  4. [4]

    A multitask, multilingual, multi- modal evaluation of chatgpt on reasoning, hallucination, and interactivity

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multi- modal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023, 2023. 2

    arXiv 2023

  5. [5]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 5

    arXiv 2023

  6. [6]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,

    arXiv

  7. [7]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 1, 5, 8

    work page 2024

  8. [8]

    CAST: Cross-modal Alignment Similarity Test for Vision Language Models

    Gautier Dagan, Olga Loginova, and Anil Batra. Cast: Cross- modal alignment similarity test for vision language models. arXiv preprint arXiv:2409.11007, 2024. 2

    work page Pith review arXiv 2024

Show all 79 references
  1. [9]

    Gemini-1.5-pro, 2024

    Google DeepMind. Gemini-1.5-pro, 2024. Accessed: 2024- 11-6. 1, 6

    2024

  2. [10]

    Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model. arXiv preprint arXiv:2401.1...

    2024 arXiv

  3. [11]

    Palm- e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm- e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. 1

    2023 arXiv

  4. [12]

    Chatglm: A family of large language mod- els from glm-130b to glm-4 all tools, 2024

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie H...

    2024

  5. [13]

    Roscoe: A suite of metrics for scoring step-by- step reasoning

    Olga Golovneva, Moya Chen, Spencer Poff, Martin Corre- dor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Ce- likyilmaz. Roscoe: A suite of metrics for scoring step-by- step reasoning. arXiv preprint arXiv:2212.07919, 2022. 4

    2022 arXiv

  6. [14]

    Self-correction is more than refinement: A learning frame- work for visual and language reasoning tasks

    Jiayi He, Hehai Lin, Qingyun Wang, Yi Fung, and Heng Ji. Self-correction is more than refinement: A learning frame- work for visual and language reasoning tasks. arXiv preprint arXiv:2410.04055, 2024. 2, 6, 7, 8

    2024 arXiv

  7. [15]

    V- star: Training verifiers for self-taught reasoners

    Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V- star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457, 2024. 8

    2024 arXiv

  8. [16]

    St-p3: End-to-end vision-based au- tonomous driving via spatial-temporal feature learning

    Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision-based au- tonomous driving via spatial-temporal feature learning. In European Conference on Computer Vision, pages 533–549. Springer, 2022. 1

    2022

  9. [17]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Confer...

    2024

  10. [18]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- 9 hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 8

    2024 arXiv

  11. [19]

    Vad: Vectorized scene representa- tion for efficient autonomous driving

    Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representa- tion for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision ,...

    2023

  12. [20]

    Vima: General robot manipulation with multimodal prompts

    Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anand- kumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094, 2(3):6, 2022. 1

    arXiv

  13. [21]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information pro- cessing systems, 35:22199–22213, 2022. 8

    2022

  14. [22]

    In-context reinforcement learning with algorithm distillation

    Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, et al. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2210.14215, 2022. 2

    arXiv

  15. [23]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 5, 6, 7

    2023 arXiv

  16. [24]

    Silkie: Preference distillation for large visual lan- guage models

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual lan- guage models. arXiv preprint arXiv:2312.10665, 2023. 2, 8

    2023 arXiv

  17. [25]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 1

    2023 arXiv

  18. [26]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Ed- wards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023. 2, 8

    2023 arXiv

  19. [27]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representa- tions, 2023. 1

    2023

  20. [28]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 5

    2024

  21. [29]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 8

    2024

  22. [30]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2025. 5, 6, 7

    2025

  23. [31]

    Deepseek-vl: towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision- language understanding. arXiv preprint arXiv:2403.05525,

    arXiv

  24. [32]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems ...

    2022

  25. [33]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts. InIn- ternational Conference on Learning Represen...

  26. [34]

    Self-refine: It- erative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hal- linan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: It- erative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024. 2, 8

    2024

  27. [35]

    Llm critics help catch llm bugs

    Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm critics help catch llm bugs. arXiv preprint arXiv:2407.00215, 2024. 2, 5

    2024 arXiv

  28. [36]

    Llama-3.2-11b-vision, 2024

    Meta. Llama-3.2-11b-vision, 2024. Accessed: 2024-10-28. 1, 5, 6

    2024

  29. [37]

    Rule based rewards for lan- guage model safety

    Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for lan- guage model safety. arXiv preprint arXiv:2411.01111, 2024. 2, 4

    2024 arXiv

  30. [38]

    Gpt-4v(ision) system card, 2023

    OpenAI. Gpt-4v(ision) system card, 2023. Accessed: 2024- 11-6. 1, 6, 8

    2023

  31. [39]

    Hello GPT-4o

    OpenAI. Hello GPT-4o. https : / / openai . com / index/hello-gpt-4o/ , 2024. Accessed: 2024-05-26. 2

    2024

  32. [40]

    Gpt-4o mini: advancing cost-efficient intelligence,

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence,

  33. [41]

    Refiner: Reasoning feedback on intermediate representa- tions

    Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback on intermediate representa- tions. arXiv preprint arXiv:2304.01904, 2023. 8

    2023 arXiv

  34. [42]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. 2, 8

    2024

  35. [43]

    Deepspeed: System optimizations enable train- ing deep learning models with over 100 billion parame- ters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable train- ing deep learning models with over 100 billion parame- ters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pag...

    2020

  36. [44]

    Learning to summarize with human feed- back

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feed- back. Advances in Neural Information Processing Systems , 33:3008–3021, 2020. 2, 8 10

    2020

  37. [45]

    Aligning large multi- modal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multi- modal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023. 8

    2023 arXiv

  38. [46]

    Policy gradient methods for reinforcement learning with function approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999. 2

    1999

  39. [47]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 5

    2023 arXiv

  40. [48]

    Llms cannot find reasoning errors, but can correct them given the error location

    Gladys Tyen, Hassan Mansoor, Peter Chen, Tony Mak, and Victor C ˘arbune. Llms cannot find reasoning errors, but can correct them given the error location. arXiv preprint arXiv:2311.08516, 2023. 8

    2023 arXiv

  41. [49]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  42. [50]

    Enhancing visual- language modality alignment in large vision language mod- els via self-improvement

    Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, et al. Enhancing visual- language modality alignment in large vision language mod- els via self-improvement. arXiv preprint arXiv:2405.15973,

    arXiv

  43. [51]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in neural information processing systems, 35:24824–24837, 2022. 8

    2022

  44. [52]

    Don’t miss the forest for the trees: Atten- tional vision calibration for large vision language models

    Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, and Changick Kim. Don’t miss the forest for the trees: Atten- tional vision calibration for large vision language models. arXiv preprint arXiv:2405.17820, 2024. 2

    2024 arXiv

  45. [53]

    Grok-1.5 vision preview, 2024

    X. Grok-1.5 vision preview, 2024. Accessed: 2024-11-06. 5, 6, 7

    2024

  46. [54]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision)

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9(1):1, 2023. 5

    2023 arXiv

  47. [55]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Sys- tems, 36, 2024. 8

    2024

  48. [56]

    Learning from correctness without prompting makes llm ef- ficient reasoner

    Yuxuan Yao, Han Wu, Zhijiang Guo, Biyan Zhou, Jiahui Gao, Sichun Luo, Hanxu Hou, Xiaojin Fu, and Linqi Song. Learning from correctness without prompting makes llm ef- ficient reasoner. arXiv preprint arXiv:2403.19094, 2024. 8

    2024 arXiv

  49. [57]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 1, 4, 5

    2024 arXiv

  50. [58]

    Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi, 2024

    Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Ren- rui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmt-bench: A com...

    2024

  51. [59]

    Textgrad: Automatic” differentiation” via text

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic” differentiation” via text. arXiv preprint arXiv:2406.07496, 2024. 3

    2024 arXiv

  52. [60]

    Accessing gpt-4 level mathemat- ical olympiad solutions via monte carlo tree self-refine with llama-3 8b: A technical report

    Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing gpt-4 level mathemat- ical olympiad solutions via monte carlo tree self-refine with llama-3 8b: A technical report. arXiv preprint arXiv:2406.07394, 2024. 8

    2024 arXiv

  53. [61]

    Llama-berry: Pairwise optimization for o1- like olympiad-level mathematical reasoning

    Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, et al. Llama-berry: Pairwise optimization for o1- like olympiad-level mathematical reasoning. arXiv preprint arXiv:2410.02884, 2024. 8

    arXiv

  54. [62]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, J...

    2024 arXiv

  55. [63]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14624, 2024

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14624, 2024. 5, 6

    2024 arXiv

  56. [64]

    Spa-vl: A comprehensive safety preference alignment dataset for vision language model

    Yongting Zhang, Lu Chen, Guodong Zheng, Yifeng Gao, Rui Zheng, Jinlan Fu, Zhenfei Yin, Senjie Jin, Yu Qiao, Xuanjing Huang, et al. Spa-vl: A comprehensive safety preference alignment dataset for vision language model. arXiv preprint arXiv:2406.12030, 2024. 2

    2024 arXiv

  57. [65]

    Automatic chain of thought prompting in large language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022. 8

    2022 arXiv

  58. [66]

    Aligning modalities in vision large lan- guage models via preference fine-tuning

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large lan- guage models via preference fine-tuning. arXiv preprint arXiv:2402.11411, 2024. 6, 7, 8

    2024 arXiv

  59. [67]

    Calibrated self-rewarding vision language models

    Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622 , 2024. 6, 7, 8

    2024 arXiv

  60. [68]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1 11

    2023 arXiv

  61. [69]

    VGA: Vision GUI assis- tant - minimizing hallucinations through image-centric fine- tuning

    Meng Ziyang, Yu Dai, Zezheng Gong, Shaoxiong Guo, Min- glong Tang, and Tongquan Wei. VGA: Vision GUI assis- tant - minimizing hallucinations through image-centric fine- tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 1261–1279. Associat...

    2024

  62. [71]

    Pseudo-code for Main Algorithms Algorithm 1 Bug Insertion and Rule-based Reward for Preference Data Collection 1: Input: True answer Atrue, Question-Image pair (Q(i), I(i)) 2: Output: Critique score score 3: Step 1: Generate a fake answer with inserted bugs 4: Afake ← Atrue 5:...

  63. [72]

    Prompt Template For multiple-choice questions (MCQ), the template of prompt is designed as follows, Hint: {hints} Question: {question} Options: {options} Please select the correct answer from the options above. As well as open-ended visual question-answering (VQA) tasks, Algor...

  64. [73]

    The GPT-4o Evaluation Rules In this section, we provided a detailed description of the evaluation criteria for erroneous detected by the VLMs as shown in Table 5

  65. [74]

    For preference-aligned fine-tuning, we utilize Direct Preference Optimization (DPO) on 29,012 samples from the critique- VQA dataset

    Hyperparameters of Critic Model’s Training We adopt Qwen2-VL-7B as our base model due to its strong performance in vision-language understanding. For preference-aligned fine-tuning, we utilize Direct Preference Optimization (DPO) on 29,012 samples from the critique- VQA datase...

  66. [75]

    In this section, we will list out the hyperparameters we choose for evaluation

    Evaluation Hyperparameters for experi- ments. In this section, we will list out the hyperparameters we choose for evaluation. For the Qwen2-VL-7B and DeepSeek-VL-7B, we set the generation parameters as follows: max new tokens to 1024, top p to 0.001, top k to 1, temperature to...

  67. [76]

    Token Consumption We explore the token consumption of Critic-V across dif- ferent benchmarks as shown in Table 6

  68. [77]

    You can find them in Figure 6, Figure 7 and Figure 8

    Visualization of Training Process In this section, we show the entire training process by sev- eral visual aids. You can find them in Figure 6, Figure 7 and Figure 8. We can obviously discover that our method convergence well experimentally. /uni00000013 /uni00000014/uni000000...

  69. [78]

    Our critique-VQA Dataset Example In this section, we show three examples in Figure 9, Fig- ure 10 and Figure 11 sampled from critique-VQA dataset

  70. [79]

    Details of Training data and Benchmarks for Evaluation In this section, we list some details of our training data and benchmarks for evaluation, as Table 7 and Table 8 shows. /uni00000013 /uni00000014/uni00000013/uni00000013/uni00000015/uni00000013/uni00000013/uni00000016/uni0...

  71. [2024]

    Accessed: 2024-11-7. 4

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.