Pith. sign in

REVIEW 4 major objections 5 minor 31 references

Mordal: Automated Pretrained Model Selection for Vision Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An automated search picks the best vision encoder and language model for a given task at roughly one-tenth the GPU cost of exhaustively trying every pair.

desk verdict A practical, well-measured engineering contribution to VLM component selection, with a real but not fatal caveat around its CKA-based pruning assumption and a few overclaims in the abstract. read the letter →

arxiv 2502.00241 v2 pith:B7ODG3AN submitted 2025-02-01 cs.LG cs.AIcs.CLcs.CV

Shiqi He , Insu Jang , Mosharaf Chowdhury This is my paper · ORCID
classification cs.LGcs.AIcs.CLcs.CV
keywords visionlanguagemodelspretrainedmodelselectionsearchrepresentationsimilaritycenteredkernelalignmentobservationalscalinglawearlystoppingmultimodal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models are built by pairing a pretrained vision encoder with a pretrained large language model and training a small projector to align them, but no single pair wins on every task and training every pair costs over a thousand GPU hours. Mordal is an automated search framework that attacks both costs at once: it groups similar candidate pairs into clusters using representation similarity, tests one representative per cluster to discard unpromising groups, and predicts each remaining candidate's full-data accuracy from short training runs using an observed log-linear scaling law. The paper reports that this finds the best VLM for a given task with 8.9x–11.6x lower GPU hours than grid search, while keeping near-optimal selection quality and ranking candidates better than existing model-selection methods. If correct, the result makes per-task VLM selection practical instead of relying on human cherry-picking or exhaustive search.

What carries the argument

The load-bearing mechanism is a two-stage search with three parts. First, candidate clustering: Mordal computes CKA similarity between vision encoders on the task's images, clusters them, then clusters LLMs within each vision-encoder cluster using a fixed representation from the cluster medoid, forming candidate clusters as all encoder-LLM combinations within each pair of clusters. Second, inter-cluster evaluation: one medoid candidate per cluster is trained with a Successive Halving early-stopping schedule, evaluating on increasing data budgets and keeping the top 1/eta, to discard poorly performing clusters. Third, intra-cluster evaluation: surviving candidates are trained on shrinking sampled data ratios while Mordal fits a linear regression on log(data ratio) versus log(error), the observational scaling law, and uses the fitted line at ratio 1 to predict full-data accuracy, selecting the best predicted candidate.

What would settle it

For a new target task and model zoo, train every candidate to completion and compare within-cluster accuracy spreads from CKA clustering: if any cluster contains two candidates whose full-data accuracies differ by more than the gap between the cluster medians that drove inter-cluster elimination, then Mordal can be led to return a suboptimal model; the failure would be observable as a task where Mordal's top-1 accuracy is below the grid-search top-1 because the best candidate's cluster was pruned.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that pretrained model selection for VLMs can be solved as a resource-constrained search rather than a manual design choice. Given an alignment dataset and a target task, Mordal searches over vision encoders and LLMs, clusters candidates by centered kernel alignment (CKA) similarity computed on task data, eliminates whole clusters after evaluating a medoid candidate, and uses early stopping plus an observational scaling law to estimate each surviving candidate's performance with a fraction of the training data. Across six visual QA, document QA, and knowledge tasks with 49 candidate pairs, Mordal identifies the top-1 candidate for five of six tasks, matches grid search's top-1 accuracy in those cases, and does so in 469–607 GPU hours versus 5439 GPU hours for exhaustive search. The paper also reports that the best pair is task-dependent and that several found combinations outperform the LLaVA-1.5-7B architecture on which the baseline is built.

Load-bearing premise

The whole search rests on the assumption that candidates placed in the same cluster by representation similarity will have similar downstream accuracy after alignment, so that testing one representative per cluster and discarding the rest never throws away the true best model.

Editorial extensions

If this is right

  • A developer can search a zoo of pretrained vision encoders and LLMs for a specific task at about one tenth of the GPU cost of training every pair, making per-task VLM construction affordable in resource-constrained settings.
  • The best vision encoder and LLM combination is task-dependent, so off-the-shelf VLMs are not a reliable substitute for a targeted search; several Mordal-discovered pairs beat the LLaVA-1.5-7B architecture on the same alignment data.
  • Because clusters are eliminated before full training, the search cost grows sublinearly with the number of candidates, so expanding model zoos with new encoders or LLMs does not force a proportional increase in evaluation cost.
  • Mordal's candidate ranking, measured by weighted Kendall's tau, is strong enough that the search output can be used not just to pick one model but to compare a shortlist of good candidates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The same cluster-by-representation-similarity and predict-from-short-runs recipe should transfer to other compound-model selections, for example retriever-plus-LLM question answering or audio-encoder-plus-LLM speech models, because neither assumption is specific to vision encoders.
  • Inference: The reported speedup assumes a shared alignment dataset across tasks; if users bring their own alignment data, the cluster structure and scaling-law fit may change, so a natural test is to rerun the pipeline with several different alignment sets and measure how often the top-1 selection changes.
  • Inference: The paper's discussion of similar user requests hints at a caching amortization that Mordal does not implement; grouping contemporaneous tasks with overlapping candidate pools could cut redundant training further in a multi-tenant setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Mordal, a pipeline for selecting pretrained vision encoders (VEs) and language models (LLMs) to construct a vision-language model (VLM) for a downstream task. Mordal first clusters candidate VE/LLM pairs using CKA-based representation similarity, then performs inter- and intra-cluster evaluation with a successive halving early-stopping mechanism, and finally uses an observational scaling law to extrapolate performance from partial alignment data. On six benchmarks with a 7x7 model zoo, Mordal is reported to identify the grid-search-best VLM on five of six tasks with 8.9x-11.6x lower GPU hours than exhaustive grid search. The abstract additionally claims about 69% higher weighted Kendall's tau than a state-of-the-art model selection method, but no such baseline is implemented in the evaluation.

Significance. If the reported speedups hold, this is a practically useful contribution: it makes automated VLM construction feasible at roughly one tenth of the GPU cost of exhaustive search and provides concrete evidence that the optimal VE/LLM pair is task-dependent. The paper's main strength is the direct comparison against a full 49-candidate grid search on six public benchmarks, with reported GPU-hour totals and speedups. However, the central claim is weakened by the ChartQA top-1 miss, the unsupported state-of-the-art baseline claim, and the lack of validation of the clustering assumption at the full candidate-zoo level. The work is promising but needs revision before the claimed near-optimality can be accepted.

major comments (4)
  1. [Abstract; §4.2 (Table 2)] The abstract states that Mordal 'can find the best VLM for a given problem using 8.9x-11.6x lower GPU hours than grid search.' This is not supported by the paper's own results: in Table 2, ChartQA's grid-search top-1 is CLIP-Qwen at 20.2, while Mordal returns DFN5B-Qwen at 18.6. The accurate claim is that Mordal identifies the grid-search top-1 on five of six tasks. Please revise the abstract and Section 4.2 accordingly and add a diagnosis of where the ChartQA miss originates (cluster pruning, early stopping, or scaling prediction).
  2. [Abstract; §4.1-4.2] The abstract also claims about 69% higher weighted Kendall's tau than the state-of-the-art model selection method, but the evaluation compares only to grid search. No LogME or other state-of-the-art baseline is implemented or reported. Either add the baseline comparison or remove the claim; as published, the claim is unverifiable.
  3. [§3.1; Appendix C.2 (Table 5)] The core pruning step assumes that candidates in the same CKA cluster have similar post-alignment performance, and inter-cluster evaluation irreversibly discards whole clusters after testing one medoid. The validation in Figure 8 and Table 3 only covers four vision encoders with a single LLM on two tasks, not the 7x7 two-factor candidate zoo used for the main results. More importantly, Table 5 shows that changing tve from 0.7 to 0.5 drops weighted Kendall's tau on GQA from 0.81 to 0.52, showing that the ranking quality is sensitive to the clustering threshold. Without a direct test that CKA cluster membership tracks aligned accuracy across the full candidate zoo and tasks, the near-optimality claim is not established; at minimum, the authors should report cluster-level accuracy deltas and the cluster assignment of the missed ChartQA best candidate.
  4. [§3.2; Figure 9; Algorithm 1] The scaling prediction is a load-bearing speedup component, but it is validated only on five vision encoders with one LLM on two tasks (Figure 9), and the curves are presented without error bars or fit-quality statistics. Algorithm 1 fits a line through p=3 points and extrapolates to r=1, yet the paper notes that log-linear scaling appears only after a certain number of training samples. The authors should provide per-candidate fit diagnostics (residuals, R^2, extrapolation error) on the main six-task evaluation to rule out that the reported speedups are driven by optimistic extrapolation.
minor comments (5)
  1. [Table 2 and §4.2] The column labeled 'Time' in Table 2 is ambiguous; the text says grid search requires 5439 GPU hours, but the table lists values 483-607 h without stating that these are Mordal times. Add explicit column labels (Grid-Search Time, Mordal Time) and list the 5439 h grid-search total in the table.
  2. [Eq. (4) and §4.1] Equation (4) defines the unweighted Kendall's tau, while the text says the authors use the weighted coefficient from You et al. Provide the weighted definition or clarify that Eq. (4) is only illustrative.
  3. [Appendix B, Listing 1] Listing 1 has a missing comma after the early_stopping dictionary, and the paper does not provide a code repository URL despite stating that Mordal provides a flexible interface.
  4. [Throughout] There are several typos and formatting inconsistencies, e.g., 'LLaV A' in Figure 1 and Table 1, 'fo VLM' in Appendix D, and the incomplete phrase 'Kendall tau i' in Section 4.1.
  5. [§3.2 and Appendix C.2] The sentence in §3.2 about starting evaluation of r/u from existing intermediate checkpoints appears to describe decreasing data ratios and needs rewording for clarity; the sensitivity discussion in Appendix C.2 also deserves a brief interpretation of why the tve=0.5 setting specifically harms ranking quality.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: Mordal's speedup and near-optimality are empirical claims validated against exhaustive grid search, not derivations that presuppose their own conclusions.

full rationale

Mordal's derivation chain consists of three efficiency mechanisms: CKA-based candidate clustering, SHA early stopping, and per-candidate log-linear scaling prediction. None of these makes the target result an input. In the scaling prediction (Algorithm 1), the algorithm collects performance points P = {(log r, log Err)} at data ratios r < R and fits a linear regression f_c, then evaluates f_c(1); full-data accuracy is never used to fit f_c, so the 'prediction' is an extrapolation rather than a fitted input renamed as a prediction. The CKA clustering step uses representation similarity and the empirical prior that similar models tend to have similar performance; although this assumption is validated only on four vision encoders and two tasks and the sensitivity analysis shows fragility (tau drops from 0.81 to 0.52 on GQA when tve=0.5), this is a testable empirical premise, not a definitional equivalence. The paper also reports a concrete miss (ChartQA top-1 18.6 vs grid-search 20.2), which makes the near-optimality claim falsifiable rather than forced. The only self-citation is Lai et al. 2023 (ModelKeeper) among three citations supporting the similarity-performance premise; it is not load-bearing because the paper provides its own validation and the central speedup claim rests on comparison to grid search. Therefore no circular step is established; the main risks are soundness risks about CKA transfer and scaling-law extrapolation, not circularity.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central speedup depends on two empirical bets: representation similarity predicts task performance, and partial-training error extrapolates log-linearly. Both are validated on small slices of the search space with no error bars. No new physical or conceptual entities are introduced; the paper's invented objects are all hyperparameters and fitted regression lines.

free parameters (7)
  • t_ve = 0.7
    Clustering threshold for vision encoders; hand-chosen and controls cluster granularity. Sensitivity analysis shows it affects tau and search time.
  • t_llm = 0.8
    Clustering threshold for language models; hand-chosen and affects cluster composition and pruning aggressiveness.
  • top_k_inter and top_k_intra = 3 and 3
    Number of clusters kept after inter-cluster evaluation and number of candidates kept for intra-cluster scaling prediction; hand-set and tested only on GQA.
  • SHA budget parameters (R, b, eta) = R=0.125, b=0.03, eta=2
    Successive halving parameters for early stopping; chosen without ablation across all tasks.
  • Scaling prediction parameters (R, u, p, delta) = R=0.125, u=2, p=3, delta=5e-5 or 0.01
    Controls how many sampled data ratios are used and when the log-linear fit is trusted. There is an inconsistency between Listing 1 and Appendix C in delta.
  • warmup_rounds = 10
    Number of projector warmup steps before LLM clustering CKA; set in Appendix C without sensitivity analysis.
  • Per-candidate scaling regression slope and intercept = not reported
    A linear regression is fit for each candidate on log error versus log sample ratio and extrapolated to full data. The fitted values are the actual selection signal.
assumptions (6)
  • domain assumption Similar models tend to have similar performance after alignment.
    Stated in Section 3.1 and validated only on four vision encoders and two datasets in Section 4.3. If false, inter-cluster pruning can remove the best candidate.
  • domain assumption CKA on chosen representations captures functionally relevant similarity across different pretrained architectures.
    Used in Section 3.1 and Appendix A with MinibatchCKA and last hidden states from the LLM. No systematic comparison against other similarity measures is provided.
  • domain assumption An observational log-linear scaling law holds for VLM alignment error versus number of training samples after a warmup.
    The entire scaling prediction component rests on this. Figure 9 shows the pattern on five VLMs and two tasks, with p=3 points per fit and no confidence intervals.
  • domain assumption The LLaVA-1.5-Instruction alignment dataset is sufficient to determine which component pair is best for each target task.
    All 49 candidates are aligned on this one dataset; a different alignment distribution could change the ranking of component pairs.
  • domain assumption Grid search top-1 is the ground truth for the best VLM.
    The evaluation defines 'best' by the grid search ranking, which itself depends on one training run per candidate with a fixed seed configuration and decoding setting.
  • domain assumption The seven vision encoders and seven LLMs selected by popularity and performance are a representative model zoo.
    The paper claims generality beyond this zoo but only tests this hand-picked 7x7 set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mordal: Automated Pretrained Model Selection for Vision Language Models." pith.science (2026). https://pith.science/paper/B7ODG3AN

@misc{pith2026250200241,
  author       = {Pith},
  title        = {Pith review of: Mordal: Automated Pretrained Model Selection for Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B7ODG3AN}},
  note         = {Machine review of arXiv:2502.00241}
}
abstract

Incorporating multiple modalities into large language models (LLMs) is a powerful way to enhance their understanding of non-textual data, enabling them to perform multimodal tasks. Vision language models (VLMs) form the fastest growing category of multimodal models because of their many practical use cases, including in healthcare, robotics, and accessibility. Unfortunately, even though different VLMs in the literature demonstrate impressive visual capabilities in different benchmarks, they are handcrafted by human experts; there is no automated framework to create task-specific multimodal models. We introduce Mordal, an automated multimodal model search framework that efficiently finds the best VLM for a user-defined task without manual intervention. Mordal achieves this both by reducing the number of candidates to consider during the search process and by minimizing the time required to evaluate each remaining candidate. Our evaluation shows that Mordal can find the best VLM for a given problem using $8.9\times$--$11.6\times$ lower GPU hours than grid search. We have also discovered that Mordal achieves about 69\% higher weighted Kendall's $\tau$ on average than the state-of-the-art model selection method across diverse tasks.

Figures

Figures reproduced from arXiv: 2502.00241 by the authors.

Figure 1
Figure 1. Benchmark performance of five latest open-source VLMs on six multimodal tasks. detailed natural language descriptions. Although the techni￾cal details behind GPT-4V remain undisclosed, researchers have proposed a number of publicly available VLMs (e.g., MiniGPT-4 (Chen et al., 2023), LLaVA (Liu et al., 2023a) and Qwen-VL (Bai et al., 2023)) that aim to match GPT￾4’s capabilities. Many of these open-source VLMs share… view at source ↗
Figure 2
Figure 2. An overview figure for Mordal. Gray circles and blocks represent pretrained models and VLM candidates, respectively. White blocks represent inactive or eliminated candidates. Mordal first groups similar candidates into clusters, where each candidate consists of one VE and one LLM. During efficient evaluation, every cluster picks one candidate (i.e., inter-cluster) and Mordal evaluates them. Poor-performed clusters a… view at source ↗
Figure 3
Figure 3. An example showing language model clustering process with four VEs and two LLMs. Different VE clusters will lead to different LLM clusters. CKA. A distance matrix Distve is then constructed based on the dissimilarity values. The clustering func￾tion will take an input threshold tve and output the vision encoder clusters Cve. • Language model clustering. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Applying early stopping mechanism and scaling predic￾tion to inter- and intra-cluster evaluation. the budget to b × η per candidate. This repeats until rep￾resentative candidates are converged or Top-K candidates are determined. In cases where the number of remaining c…
Figure 6
Figure 6. Figure 6: Top-5 candidates from grid search and Mordal on GQA. τ coefficient defined as: τ = 2 M(M − 1) X 1≤i<j≤M sgn(Ti−Tj )sgn(Si−Sj ) (4) where M is the total number of candidates and sgn() is the sign function. A perfect ranking match results in τ = 1. To further focus on to…
Figure 9
Figure 9. Figure 9: Observational scaling law validation for five VLMs with different vision encoders and same language model Qwen2-7B. The results are on GQA and AI2D. GQA AI2D 0 500 1000 1500 2000 GPU hours (a) Training Time GQA AI2D 0.5 0.6 0.7 0.8 0.9 1.0 V a l u e (b) τ Value Method …
Figure 8
Figure 8. Figure 8: Similarity scores between four vision encoders on Sci￾enceQA and VizWiz. Higher is better. clustering, early stopping and scaling prediction. Model similarity validation. To validate the effectiveness of CKA, we train four VLM candidates with four vision encoders and t…
Figure 11
Figure 11. Figure 11: Total evaluation time breakdown for Mordal on six datasets. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Alternative alignment approaches for VLM instruction tunning. scratch using visual alignment datasets like VQA (Antol et al., 2015) . During the alignment process, pretrained components may remain frozen or be further finetuned during alignment training (Liu et al., 2…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 5 canonical work pages

  1. [1]

    A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al

    Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable lan- guage model locally on your phone. arXiv preprint arXiv:2404.14219,

    arXiv

  2. [6]

    URL https://lmsys.org/blog/ 2023-03-30-vicuna/ . Dao, T. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08691,

    arXiv 2023

  3. [7]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  4. [12]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    arXiv 2001

  5. [13]

    Prismatic vlms: Investigating the design space of visually-conditioned language models

    9 Mordal: Automated Pretrained Model Selection for Vision Language Models Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., and Sadigh, D. Prismatic vlms: Investigating the design space of visually-conditioned language models. arXiv preprint arXiv:2402.07865,

    arXiv

  6. [15]

    Selecting large language model to fine-tune via rectified scaling law.arXiv preprint arXiv:2402.02314,

    Lin, H., Huang, B., Ye, H., Chen, Q., Wang, Z., Li, S., Ma, J., Wan, X., Zou, J., and Liang, Y . Selecting large language model to fine-tune via rectified scaling law.arXiv preprint arXiv:2402.02314,

    arXiv

  7. [16]

    Liu, H., Li, C., Li, Y ., and Lee, Y . J. Improved base- lines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023a. Liu, H., Li, C., Wu, Q., and Lee, Y . J. Visual instruction tuning, 2023b. Liu, Y ., Duan, H., Zhang, Y ., Li, B., Zhang, S., Zhao, W., Yuan, Y ., Wang, J., He, C., Liu, Z., Chen, K., and Lin, D. Mmbench: Is your multi-mod...

    arXiv

  8. [17]

    X., Tan, J

    Masry, A., Long, D. X., Tan, J. Q., Joty, S., and Hoque, E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244,

    arXiv

Show all 31 references
  1. [19]

    Dinov2: Learning robust visual features without supervision

    Oquab, M., Darcet, T., Moutakanni, T., V o, H., Szafraniec, M., Khalidov, V ., Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193,

    arXiv

  2. [20]

    J., and Hashimoto, T

    Ruan, Y ., Maddison, C. J., and Hashimoto, T. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405.10938,

    arXiv

  3. [21]

    Eva- clip: Improved training techniques for clip at scale

    Sun, Q., Fang, Y ., Wu, L., Wang, X., and Cao, Y . Eva- clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389,

    arXiv

  4. [22]

    S., Love, J., et al

    Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi`ere, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295,

    arXiv

  5. [23]

    C., Yang, J., Yang, S., Iyer, A., Pan, X., et al

    Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860,

    arXiv

  6. [24]

    Llama: Open and efficient foundation lan- guage models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971, 2023a. Touvron, H., Martin, L., Stone, K., Albert, P...

    arXiv

  7. [25]

    Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke, A., Maji, S., and Iyyer, M

    doi: 10.1038/s41592-019-0686-2. Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke, A., Maji, S., and Iyyer, M. Exploring and predicting transferability across nlp tasks. arXiv preprint arXiv:2005.00770,

    2005 arXiv

  8. [26]

    Lvlm-ehub: A comprehensive evaluation benchmark for large vision- language models

    Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y ., and Luo, P. Lvlm-ehub: A comprehensive evaluation benchmark for large vision- language models. arXiv preprint arXiv:2306.09265 ,

    arXiv

  9. [27]

    Qwen2 technical report

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M....

    arXiv

  10. [28]

    A., Hu, K., Liu, S., Zhang, Y ., Yang, J., Li, C., et al

    Zhang, K., Li, B., Zhang, P., Pu, F., Cahyono, J. A., Hu, K., Liu, S., Zhang, Y ., Yang, J., Li, C., et al. Lmms- eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772,

    arXiv

  11. [29]

    In LLM clustering, we use the last hidden state from LLM as the sentence representation for CKA computation since it produces the best clustering performance

    and later used in (Raghu et al., 2021). In LLM clustering, we use the last hidden state from LLM as the sentence representation for CKA computation since it produces the best clustering performance. We leverage the hierarchical clustering from scipy.cluster.hierarchy library i...

    2021

  12. [30]

    For ConvNeXt, we interpolate the output embeddings to 16x16 patches following Cambrian-1 (Tong et al., 2024)

    while EV A-CLIP and ConvNeXt are supported bytimm library (Wightman, 2019). For ConvNeXt, we interpolate the output embeddings to 16x16 patches following Cambrian-1 (Tong et al., 2024). To make a fair comparison with LLaV A-1.5-7B equivalent structure, we train an MLP projecto...

    2019

  13. [31]

    Recently, some proprietary models have employed end-to-end training without using any pretrained models (Bai et al., 2023), but it is not common due to the excessive training cost

    During the alignment process, pretrained components may remain frozen or be further finetuned during alignment training (Liu et al., 2023a). Recently, some proprietary models have employed end-to-end training without using any pretrained models (Bai et al., 2023), but it is no...

    2023

  14. [2015]

    Qwen-vl: A frontier large vision- language model with versatile abilities

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308.12966,

    arXiv

  15. [2016]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  16. [2018]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556,

    arXiv

  17. [2019]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 1877–1901,

    1901

  18. [2020]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krish- namoorthi, R., Chandra, V ., Xiong, Y ., and Elhoseiny, M. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478,

    arXiv

  19. [2021]

    Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth

    Nguyen, T., Raghu, M., and Kornblith, S. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint arXiv:2010.15327,

    2010 arXiv

  20. [2022]

    J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

    arXiv

  21. [2023]

    Plato: Pre- trained dialogue generation model with discrete latent variable

    Bao, S., He, H., Wang, F., Wu, H., and Wang, H. Plato: Pre- trained dialogue generation model with discrete latent variable. arXiv preprint arXiv:1910.07931,

    1910 arXiv

  22. [2024]

    A diagram is worth a dozen images

    Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pp. 235–251. Springer,

    2016

  23. [2025]

    Accessed: 2025-01-30

    URL https:// huggingface.co/models?sort=downloads. Accessed: 2025-01-30. Fang, A., Jose, A. M., Jain, A., Schmidt, L., Toshev, A. T., and Shankar, V . Data filtering networks. InInternational Conference on Learning Representations,

    2025

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.