Vision Generalist Model: A Survey

Hongmin Liu; Jie Zhou; Jiwen Lu; Shuofeng Sun; Xinrun Liu; Xumin Yu; Yanbo Wang; Yi Wei; Yongming Rao; Ziyi Wang

REVIEW 3 major objections 6 minor 222 references

Vision Generalist Model: A Survey

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that vision generalist models form a coherent field organized by two architectural families, and supports the claim with a ten-model, twenty-two-benchmark comparison.

desk verdict Useful early survey with a sensible taxonomy, but the auditability of its headline comparison and a scope contradiction keep it from being a clean reference yet. read the letter →

arxiv 2506.09954 v1 pith:G7FORIMV submitted 2025-06-11 cs.CV cs.AI

Ziyi Wang , Yongming Rao , Shuofeng Sun , Xinrun Liu , Yi Wei , Xumin Yu , Zuyan Liu , Yanbo Wang

show 3 more authors

Hongmin Liu Jie Zhou Jiwen Lu

This is my paper

classification cs.CVcs.AI

keywords visiongeneralistmodelmultimodallearningunifiedtransformersequence-to-sequencemulti-taskPerceiverfamilybenchmarkcomparisonopenvocabulary

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey tries to establish that 'vision generalist model' names a coherent research area and that the area can be understood through a two-way split: models that unify input representations (encoding-based) and models that unify input and output as token sequences (sequence-to-sequence). It backs that organization with a review of datasets, tasks, and benchmarks, a technique layer, a ten-model/twenty-two-benchmark comparison, and a discussion of applications, open challenges, and future directions. A reader who accepts the survey is meant to come away with an orientation to the main model families, the reported performance leaders per task group, and the field's central unsolved problem, a general-purpose input/output format. The paper's contribution is organizational: if the taxonomy and the comparison are faithful, it gives newcomers a reliable map of a fast-moving area.

What carries the argument

The load-bearing device is the two-family taxonomy, with the template equations that instantiate each family. Encoding-based models are captured by formulas such as $O_X = D_X(TD(TE(E_X(I_X))))$ for unified Transformers, by cross-attention over fixed-size latent queries for the Perceiver family, and by maximum-likelihood matching $P(x,y)\propto\exp(\cos(f(x),f(y))/\tau)$ for the Uni-Perceiver family. Sequence-to-sequence models are captured by the template $O_X = D_X(\mathrm{split}_X(TD(TE(\mathrm{cat}_i(T_{X_i})))))$, in which every modality is tokenized, concatenated, and decoded as one token stream. The taxonomy does the organizational work of the survey; the equations show that models inside each family share a common unification strategy.

What would settle it

Apply Section 5.2's exclusion rule to Section 3.2.1 and Table 2: under the stated rule, Kosmos-1, Kosmos-2, MiniGPTv2, VisionLLM, PixelLM, GiT, and Chameleon should leave the survey's scope, so checking whether the Figure 3 comparison and model counts survive that removal settles whether the scope is coherent. Separately, re-running Figure 3 with zero-shot and fine-tuned entries separated would test whether the per-task-group leaders persist.

Watch

Extended reading notes

Core claim

The paper's central claim is that the many recent models that handle multiple vision tasks in one trained system can be treated as one field, and that the field's architecture space is captured by two families. Encoding-based frameworks (unified Transformers, the Perceiver family, and the Uni-Perceiver family) seek a shared representation space and then decode with task-specific heads or queries; sequence-to-sequence frameworks (prefix language modeling, masked language modeling, and generalized sequence generation) reduce every task to token prediction over a unified vocabulary. Drawing on its comparison of ten models across twenty-two benchmarks, the paper concludes that no single generalist model is best on every task and that the reported leaders differ by task group. On that reading, the open problem that defines the next stage of the field is a general-purpose input/output format for vision.

Load-bearing premise

The load-bearing premise is that 'vision generalist model' is a coherent category and that the survey applies its own inclusion rule consistently; Section 5.2 says models whose outputs are primarily language-based are excluded, while Section 3.2.1 and Table 2 include several such models.

Editorial extensions

If this is right

A newcomer can place almost any recent vision generalist model into one of two families, encoding-based or sequence-to-sequence, and infer its basic design commitments from that placement.
The comparison implies that generalists are task-group specialists in practice: One-Peace and Uni-Perceiver v2 lead image-text retrieval, Uni-Perceiver-MoE leads video-text retrieval, Florence leads vision-only perception, and PALI and PALI-X lead visual question answering.
Because no single model dominates all twenty-two benchmarks, the survey supports treating composite multi-benchmark evaluation as the standard way to measure a generalist.
The open problems it names, especially the lack of a general-purpose I/O format and the cost of labeled visual data, define the near-term research agenda.
The technique layer implies that Transformers, diffusion models, and mixture-of-experts are complementary building blocks for future generalists.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Taking Section 5.2's exclusion rule literally would remove Kosmos-1, Kosmos-2, MiniGPTv2, VisionLLM, PixelLM, GiT, and Chameleon from Table 2, which would change the model inventory and the Figure 3 comparison; the paper does not reconcile this with their inclusion in Section 3.2.1.
Figure 3 is built from heterogeneous reports that mix zero-shot and fine-tuned numbers, so the stable conclusion is the qualitative one that no model wins everywhere; the exact per-task rankings could shift under a unified protocol.
If a general-purpose I/O format does emerge, the encoding-based/sequence-to-sequence divide should blur, since models such as InstructDiffusion already frame perception and generation as one conditional generation process.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper is a legitimate early survey of vision generalist models, organized around an encoding-based vs sequence-to-sequence split. That split is a real organizing contribution, and the coverage is broad and mostly faithful to the primary papers. The benchmark discussion (Section 3.3) and Figure 3 are its quantitative backbone, and that is where the problems are. The figure mixes zero-shot and fine-tuned entries, different metric types, and heterogeneous protocols with no per-bar provenance. The authors themselves concede that a comprehensive comparison is not feasible. Given that, the specific leader claims (One-Peace for image-text retrieval, Uni-Perceiver-MoE for video-text, Florence for perception, PALI-X for VQA) are not auditable. This isn't fatal to the survey's taxonomic value, but it does undercut the one place where the paper claims quantitative support. Second real issue is a scope contradiction. Section 5.2 says LLM-integrated models that primarily produce language outputs are excluded from 'generalist vision models.' Section 3.2.1 and Table 2 include Kosmos-1/2, MiniGPTv2, VisionLLM, PixelLM, GiT, and Chameleon, all of which are LLM-integrated. You can defend including them if the criterion is handling vision tasks beyond language, but as written the paper excludes them. That needs to be fixed. Smaller stuff: duplicate citation keys for the same works (Uni-Perceiver, PaLI-X, Unified-IO appear under multiple keys), and the Section 6.1 claims about Tesla and Meta's product capabilities have no sources. These are minor but reduce trust. What the paper does well: the taxonomy genuinely helps navigate a crowded field, the background section on datasets/tasks/benchmarks is a useful entry point, and the authors are honest about the comparison problem. There is no new model or data here, but a survey doesn't need that. My take: this deserves peer review. It's not ready as-is; the authors should add a provenance table for Figure 3, resolve the scope inconsistency, clean up the references, and source the industry claims. For a newcomer to the area, the survey is already useful, and with those fixes it would be a solid reference. I'd send it to a serious referee.

Referee Report

3 major / 6 minor

Summary. This survey offers a broad literature review of vision generalist models (VGMs). It reviews datasets, tasks, and benchmarks (§2); proposes a two-family taxonomy of encoding-based frameworks (unified Transformers, Perceiver, Uni-Perceiver) and sequence-to-sequence frameworks (prefix language modeling, masked language modeling, generalized sequence generation) (§3); surveys techniques for input unification, model design, and output decoding (§4); relates VGMs to multi-task learning, vision-language learning, and open-vocabulary learning (§5); and closes with applications, challenges, and future directions (§6). The empirical core is Figure 3, a 10-model/22-benchmark comparison from which §3.3 draws qualitative performance-leader conclusions. The paper explicitly acknowledges in §3.3 that a full cross-benchmark comparison is infeasible due to varying experimental settings.

Significance. If the scope and comparison issues are fixed, the survey would be a useful entry point: it covers a wide set of representative models and techniques, organizes them in a memorable two-family taxonomy, and provides a compact per-method summary in Table 2. I credit the authors for stating the limits of the comparison in §3.3 and for compiling a dense Table 2. Since this is a literature survey, there are no code or machine-checked proofs to verify; the manuscript's credibility rests on faithful citation, consistent application of its own scope rules, and auditable quantitative claims. The main value, however, depends on whether the inclusion rule for 'vision generalist model' is applied consistently and whether Figure 3 can be read as evidence for the stated leaders.

major comments (3)

[§5.2 vs. §3.2.1/Table 2] The scope rule stated in Section 5.2—that LLM-integrated methods 'primarily produce language-based outputs, we exclude them from the scope of generalist vision models, which emphasize the handling of vision-specific tasks'—is not applied consistently elsewhere. Section 3.2.1 and Table 2 include Kosmos-1, Kosmos-2, MiniGPTv2, VisionLLM, VisionLLMv2, PixelLM, GiT, and Chameleon, which are LLM-integrated and are not obviously distinguished by the stated exclusion, and Section 5.2 itself later discusses several LLM-based models (Frozen, LLaVA, Qwen-VL, Video-LLaVA, PointLLM) without applying a systematic criterion. This makes the survey's model inventory and the Figure 3 comparison ill-defined. Please state an operational inclusion criterion (for example, whether the defining property is the unlabeled output space, the joint training objective, or the absence of a language interface) and apply it to every row of Table 2, or move the affected models to Section 5.2 as related work.
[§3.3 / Figure 3] Figure 3 mixes bars labeled '(zero-shot)' with unlabeled bars whose evaluation protocol is not documented, and it plots different metric families (retrieval recall, CIDEr/BLEU, classification accuracy, VQA accuracy) without a per-bar source or protocol column. Section 3.3's conclusions that One-Peace and Uni-Perceiver-v2 lead image-text retrieval, Uni-Perceiver-MoE leads video-text retrieval, Florence stands out in vision-only perception, and PALI/PALI-X deliver exceptional VQA results therefore cannot be audited: a reader cannot tell whether these differences reflect model capability or evaluation protocol. Since these conclusions are the quantitative backbone of the survey's orientation value, please add for every bar the evaluation mode (zero-shot or fine-tuned), metric, split, and original source, and restrict qualitative leader claims to bars that are directly comparable.
[Table 2] Table 2, which is the survey's compact model inventory, contains several inaccuracies that need correction in a revision. For instance, KOSMOS-2's Inputs are listed as 'I' although the model accepts image–text pairs; HiP's venue is misspelled as 'Arvix'; the Meta-Transformer entry uses the citation label 2023b while the text and bibliography use 2023c; and duplicate bibliographic entries for PALI-X (2023c and 2023d) and Unified-IO (2022b and 2022c) make it difficult to trace which source a Table 2 row refers to. Please audit Table 2 against the primary papers and resolve the citation duplicates.

minor comments (6)

[§3.3] The text states that the 22 benchmarks are grouped into five categories, but only three category descriptions are given (multimodal retrieval, vision-only perception, and other vision-language reasoning); please enumerate all five categories or remove the number.
[§3.1.3 / Eq. (4)] Equation (4) is difficult to parse: the denominator contains 'P i≠j' without defining the index sets over which i and j range, and it is not clear whether the sum is over negative targets within a batch or over candidate targets generally; please rewrite the formula with explicit index notation.
[§4.3.1] The sentence 'In the preceding section, we will delve into the specifics' appears immediately before the specifics are given; 'preceding' should be 'following' or the sentence should be reworded.
[References] The bibliography contains duplicated entries (e.g., Chen et al. 2020a/2020b and 2022b/2022c, Lu et al. 2022b/2022c, Zhang et al. 2023b/2023c, Vaswani et al. 2017a/2017b, Rombach et al. 2022a/2022b), which obscures which entry is cited in a given claim and should be merged.
[Table 2] The table's formatting is inconsistent: model names appear variously as 'KOSMOS-1', 'KOSMOS-2', 'LAVENDER', 'Meta-Transformer', and 'Meta-Transformer'; please normalize naming and venue strings across all rows.
[§6.1] The application claims about Tesla's Full Self-Driving stack and Meta's Orion AR glasses are stated without references; please add citable sources for these proprietary systems.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey compiles external results and its taxonomy and comparison do not reduce to its own definitions, equations, or self-citations.

full rationale

This paper is a literature survey, not a derivation, so the circularity failure modes largely do not apply. It compiles externally published models, datasets, benchmarks, and performance numbers; it does not fit any parameter to a subset of data and then rename that fit as a prediction, and it does not invoke a uniqueness theorem or define a quantity in terms of its own output. The only self-citations are DenseCLIP (Rao et al. 2022), Dynamic Spatial Sparsification (Rao et al. 2023), and the text-to-image diffusion perception work (Zhao et al. 2023); these appear as technique references in Sections 4.1, 5.1, and 5.2 and are not load-bearing for the survey's central taxonomy, its Section 3 framework classification, or the Figure 3 comparison. The scope tension between Section 5.2's statement that LLM-integrated models are excluded 'as these methods primarily produce language-based outputs' and Section 3.2.1's inclusion of Kosmos-1, Kosmos-2, MiniGPTv2, VisionLLM, PixelLM, GiT, and Chameleon is a real consistency and audibility concern but not a circularity: the survey's qualitative conclusions are not entailed by its own inclusion rule. Similarly, Figure 3 mixes zero-shot and fine-tuned entries from heterogeneous protocols without a per-bar provenance table, which weakens the evidentiary support for the Section 3.3 leader claims, but this is an evidence-quality/correctness issue rather than a self-referential reduction. No step in the paper reduces by construction to its own inputs, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The survey has no free parameters and invents no entities, forces, or quantities; 'vision generalist model' is a label inherited from the surveyed literature. The load-bearing inputs are the fidelity of the taxonomy and the comparability of the compiled benchmark numbers, both only partially secured by the text.

assumptions (2)

domain assumption Vision generalist model is a coherent category whose membership can be consistently determined across the survey.
Section 1 defines VGM as unified frameworks handling diverse vision tasks, but the scope rule is applied inconsistently: Section 5.2 excludes LLM-based models with language outputs while Section 3.2.1 and Table 2 include Kosmos-1, Kosmos-2, MiniGPTv2, VisionLLM, PixelLM, GiT, and Chameleon.
domain assumption Reported benchmark numbers from heterogeneous sources can be placed in a single comparison chart.
Section 3.3 and Figure 3 mix zero-shot and fine-tuned results from 10 models across 22 benchmarks without a shared protocol; the paper itself concedes that a comprehensive comparison is not feasible.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Vision Generalist Model: A Survey." pith.science (2026). https://pith.science/paper/G7FORIMV

@misc{pith2026250609954,
  author       = {Pith},
  title        = {Pith review of: Vision Generalist Model: A Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G7FORIMV}},
  note         = {Machine review of arXiv:2506.09954}
}

read the original abstract

Recently, we have witnessed the great success of the generalist model in natural language processing. The generalist model is a general framework trained with massive data and is able to process various downstream tasks simultaneously. Encouraged by their impressive performance, an increasing number of researchers are venturing into the realm of applying these models to computer vision tasks. However, the inputs and outputs of vision tasks are more diverse, and it is difficult to summarize them as a unified representation. In this paper, we provide a comprehensive overview of the vision generalist models, delving into their characteristics and capabilities within the field. First, we review the background, including the datasets, tasks, and benchmarks. Then, we dig into the design of frameworks that have been proposed in existing research, while also introducing the techniques employed to enhance their performance. To better help the researchers comprehend the area, we take a brief excursion into related domains, shedding light on their interconnections and potential synergies. To conclude, we provide some real-world application scenarios, undertake a thorough examination of the persistent challenges, and offer insights into possible directions for future research endeavors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

222 extracted references · 4 canonical work pages

[1]

Gpt-4 technical report

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023
[2]

Deep learning using rectified linear units (relu)

AF Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375, 2018

arXiv 2018
[3]

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022

2022
[4]

Bottom-up and top-down attention for image captioning and visual question answering

Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077--6086, 2018

2018
[5]

Neural module networks

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39--48, 2016

2016
[6]

Vqa: Visual question answering

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425--2433, 2015

2015
[7]

Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens

Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.03413, 2024

arXiv 2024
[8]

Foundational models defining a new era in vision: A survey and outlook

Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundational models defining a new era in vision: A survey and outlook. arXiv preprint arXiv:2307.13721, 2023

arXiv 2023

Show all 222 references

[9]

Layer normalization

Jimmy Lei Ba. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

2016 arXiv
[10]

Multimae: Multi-modal multi-task masked autoencoders

Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi-task masked autoencoders. In ECCV, 2022

2022
[11]

Graph perceiver io: A general architecture for graph structured data

Seyun Bae, Hoyoon Byun, Changdae Oh, Yoon-Sik Cho, and Kyungwoo Song. Graph perceiver io: A general architecture for graph structured data. arXiv preprint arXiv:2209.06418, 2022

2022 arXiv
[12]

Data2vec: A general framework for self-supervised learning in speech, vision and language

Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, pages 1298--1312. PMLR, 2022

2022
[13]

Ofasys: A multi-modal multi-task learning system for building generalist models

Jinze Bai, Rui Men, Hao Yang, Xuancheng Ren, Kai Dang, Yichang Zhang, Xiaohuan Zhou, Peng Wang, Sinan Tan, An Yang, et al. Ofasys: A multi-modal multi-task learning system for building generalist models. arXiv preprint arXiv:2212.04408, 2022

2022 arXiv
[14]

Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023

2023 arXiv
[15]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

2025 arXiv
[16]

Sequential modeling enables scalable learning for large vision models

Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. In CVPR, 2024

2024
[17]

Visual prompting via image inpainting

Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 35: 0 25005--25017, 2022

2022
[18]

Estimating or propagating gradients through stochastic neurons for conditional computation

Yoshua Bengio, Nicholas L \'e onard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013

2013 arXiv
[19]

Mult: An end-to-end multitask learning transformer

Deblina Bhattacharjee, Tong Zhang, Sabine S \"u sstrunk, and Mathieu Salzmann. Mult: An end-to-end multitask learning transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12031--12041, 2022

2022
[20]

Rt-2: Vision-language-action models transfer web knowledge to robotic control

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

2023 arXiv
[21]

What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test

Patricia A Carpenter, Marcel A Just, and Peter Shell. What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test. Psychological review, 97 0 (3): 0 404, 1990

1990
[22]

Hip: Hierarchical perceiver

Joao Carreira, Skanda Koppula, Daniel Zoran, Adria Recasens, Catalin Ionescu, Olivier Henaff, Evan Shelhamer, Relja Arandjelovic, Matt Botvinick, Oriol Vinyals, et al. Hip: Hierarchical perceiver. arXiv preprint arXiv:2202.10890, 2022

2022 arXiv
[23]

Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558--3568, 2021

2021
[24]

Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310...

arXiv
[25]

Shikra: Unleashing multimodal llm's referential dialogue magic

Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023 b

2023 arXiv
[26]

Autoformer: Searching transformers for visual recognition

Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12270--12280, 2021

2021
[27]

A generalist framework for panoptic segmentation of images and videos

Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David J Fleet. A generalist framework for panoptic segmentation of images and videos. arXiv preprint arXiv:2210.06366, 2022 a

2022 arXiv
[28]

A unified sequence interface for vision tasks

Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J Fleet, and Geoffrey E Hinton. A unified sequence interface for vision tasks. Advances in Neural Information Processing Systems, 35: 0 31333--31346, 2022 b

2022
[29]

Pali: A jointly-scaled multilingual language-image model

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022 c

2022 arXiv
[31]

Pali-x: On scaling up a multilingual vision and language model

Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565, 2023 d

2023 arXiv
[32]

Learning a low-level vision generalist via visual task prompt

Xiangyu Chen, Yihao Liu, Yuandong Pu, Wenlong Zhang, Jiantao Zhou, Yu Qiao, and Chao Dong. Learning a low-level vision generalist via visual task prompt. In ACM MM, 2024

2024
[33]

Microsoft coco captions: Data collection and evaluation server

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015

2015 arXiv
[34]

Uniter: Universal image-text representation learning

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104--120. Springer, 2020 a

2020
[35]

Uniter: Universal image-text representation learning

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104--120. Springer, 2020 b

2020
[36]

Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks

Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794--803. PMLR, 2018

2018
[37]

Per-pixel classification is not all you need for semantic segmentation

Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34: 0 17864--17875, 2021

2021
[38]

Masked-attention mask transformer for universal image segmentation

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290--1299, 2022

2022
[39]

Unifying vision-and-language tasks via text generation

Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. In ICML, 2021

2021
[40]

Conditional positional encodings for vision transformers

Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102.10882, 2021

2021 arXiv
[41]

Instance-aware semantic segmentation via multi-task network cascades

Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3150--3158, 2016

2016
[42]

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

2018 arXiv
[43]

Decoupling zero-shot semantic segmentation

Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11583--11592, 2022

2022
[44]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

2010 arXiv
[45]

Palm-e: An embodied multimodal language model

Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023

2023 arXiv
[46]

Glam: Efficient scaling of language models with mixture-of-experts

Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547--5569. PMLR, 2022

2022
[47]

Multi-modal alignment using representation codebook

Jiali Duan, Liqun Chen, Son Tran, Jinyu Yang, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Multi-modal alignment using representation codebook. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15651--15660, 2022

2022
[48]

Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser

Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser. In Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th international joint conferen...

2015
[49]

Taming transformers for high-resolution image synthesis

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021

2021
[50]

Mmbench-video: A long-form multi-shot benchmark for holistic video understanding

Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. NeurIPS, 2024

2024
[51]

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022

2022
[52]

Mme: A comprehensive evaluation benchmark for multimodal large language models

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023

2023 arXiv
[53]

Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024 a

2024 arXiv
[54]

Mme-survey: A comprehensive survey on evaluation of multimodal llms

Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, et al. Mme-survey: A comprehensive survey on evaluation of multimodal llms. arXiv preprint arXiv:2411.15296, 2024 b

2024 arXiv
[55]

Multimodal compact bilinear pooling for visual question answering and visual grounding

Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606.01847, 2016

2016 arXiv
[56]

Llama-adapter v2: Parameter-efficient visual instruction model

Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023

2023 arXiv
[57]

Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning

Yuan Gao, Haoping Bai, Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu. Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11543--11552, 2020

2020
[58]

Instructdiffusion: A generalist modeling interface for vision tasks

Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Houqiang Li, Han Hu, et al. Instructdiffusion: A generalist modeling interface for vision tasks. In CVPR, 2024

2024
[59]

Scaling open-vocabulary image segmentation with image-level labels

Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision, pages 540--557. Springer, 2022

2022
[60]

Omnivore: A single model for many visual modalities

Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16102--16112, 2022

2022
[61]

Mixture of experts models

Isobel Claire Gormley and Sylvia Fr \"u hwirth-Schnatter. Mixture of experts models. Handbook of mixture analysis, pages 271--307, 2019

2019
[62]

3d semantic segmentation with submanifold sparse convolutional networks

Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In CVPR, pages 9224--9232, 2018

2018
[63]

Zero-shot detection via vision and language knowledge distillation

Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Zero-shot detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921, 2 0 (3): 0 4, 2021

2021 arXiv
[64]

Deepfake video detection using recurrent neural networks

David G \"u era and Edward J Delp. Deepfake video detection using recurrent neural networks. In AVSS, pages 1--6. IEEE, 2018

2018
[65]

Dynamic task prioritization for multitask learning

Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In Proceedings of the European conference on computer vision (ECCV), pages 270--287, 2018

2018
[66]

Grit: General robust image task benchmark

Tanmay Gupta, Ryan Marten, Aniruddha Kembhavi, and Derek Hoiem. Grit: General robust image task benchmark. arXiv preprint arXiv:2204.13653, 2022

2022 arXiv
[67]

Onellm: One framework to align all modalities with language

Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In CVPR, 2024

2024
[68]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770--778, 2016 a

2016
[69]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016 b

2016
[70]

Mask r-cnn

Kaiming He, Georgia Gkioxari, Piotr Doll \'a r, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961--2969, 2017

2017
[71]

Distilling the knowledge in a neural network

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

2015 arXiv
[72]

Classifier-free diffusion guidance

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022

2022 arXiv
[73]

Denoising diffusion probabilistic models

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33: 0 6840--6851, 2020

2020
[74]

Long short-term memory

Sepp Hochreiter and J \"u rgen Schmidhuber. Long short-term memory. Neural computation, 9 0 (8): 0 1735--1780, 1997

1997
[75]

Unit: Multimodal multitask learning with a unified transformer

Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified transformer. In ICCV, 2021

2021
[76]

An embodied generalist agent in 3d world

Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023 a

2023 arXiv
[77]

Language is not all you need: Aligning perception with language models

Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045, 2023 b

2023 arXiv
[78]

Perceiver io: A general architecture for structured inputs & outputs

Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021 a

2021 arXiv
[79]

Perceiver: General perception with iterative attention

Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In ICML, 2021 b

2021
[80]

Oneformer: One transformer to rule universal image segmentation

Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. In CVPR, 2023

2023
[81]

Vima: General robot manipulation with multimodal prompts

Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094, 2 0 (3): 0 6, 2022

arXiv
[82]

Unified language-vision pretraining with dynamic discrete visual tokenization

Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, et al. Unified language-vision pretraining with dynamic discrete visual tokenization. arXiv preprint arXiv:2309.04669, 2023

2023
[83]

Deep visual-semantic alignments for generating image descriptions

Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128--3137, 2015

2015
[84]

Multi-task learning using uncertainty to weigh losses for scene geometry and semantics

Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482--7491, 2018

2018
[85]

Segment anything

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, 2023 a

2023
[86]

Segment anything

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023 b

2023 arXiv
[87]

Uvim: A unified modeling approach for vision with learned guiding codes

Alexander Kolesnikov, Andr \'e Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Uvim: A unified modeling approach for vision with learned guiding codes. Advances in Neural Information Processing Systems, 35: 0 26295--26308, 2022

2022
[88]

Tabddpm: Modelling tabular data with diffusion models

Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Modelling tabular data with diffusion models. In International Conference on Machine Learning, pages 17564--17579. PMLR, 2023

2023
[89]

Visual genome: Connecting language and vision using crowdsourced dense image annotations

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer ...

2017
[90]

Imagenet classification with deep convolutional neural networks

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012

2012
[91]

Babytalk: Understanding and generating simple image descriptions

Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. Babytalk: Understanding and generating simple image descriptions. IEEE transactions on pattern analysis and machine intelligence, 35 0 (12): 0 2891--2903, 2013

2013
[92]

Base layers: Simplifying training of large, sparse models

Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, pages 6265--6274. PMLR, 2021

2021
[93]

Seed-bench: Benchmarking multimodal llms with generative comprehension

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023 a

2023 arXiv
[94]

Multimodal foundation models: From specialists to general-purpose assistants

Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao. Multimodal foundation models: From specialists to general-purpose assistants. arXiv preprint arXiv:2309.10020, 2023 b

2023 arXiv
[95]

Mask dino: Towards a unified transformer-based framework for object detection and segmentation

Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung-Yeung Shum. Mask dino: Towards a unified transformer-based framework for object detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

2023
[96]

Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks

Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hongsheng Li, Chun Yuan, Xiaohua Wang, Yu Qiao, Xiaogang Wang, Wenhai Wang, et al. Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks. In CVPR, 2023 d

2023
[97]

Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023 e

2023
[98]

Lavender: Unifying video-language understanding as masked language modeling

Linjie Li, Zhe Gan, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Ce Liu, and Lijuan Wang. Lavender: Unifying video-language understanding as masked language modeling. In CVPR, 2023 f

2023
[99]

Visualbert: A simple and performant baseline for vision and language

Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019

1908 arXiv
[100]

Grounded language-image pre-training

Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages ...

2022
[101]

Omg-seg: Is one model good enough for all segmentation? In CVPR, 2024

Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. Omg-seg: Is one model good enough for all segmentation? In CVPR, 2024

2024
[102]

Open-vocabulary semantic segmentation with mask-adapted clip

Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

2023
[103]

Evolutionary architecture search for deep multitask networks

Jason Liang, Elliot Meyerson, and Risto Miikkulainen. Evolutionary architecture search for deep multitask networks. In Proceedings of the genetic and evolutionary computation conference, pages 466--473, 2018

2018
[104]

Video-llava: Learning united visual representation by alignment before projection

Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023

2023 arXiv
[105]

Microsoft coco: Common objects in context

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Procee...

2014
[106]

Improved baselines with visual instruction tuning

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024 a

2024
[107]

Visual instruction tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2024 b

2024
[108]

End-to-end multi-task learning with attention

Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1871--1880, 2019

2019
[109]

Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216--233

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216--233. Springer, 2024 c

2024
[110]

Swin transformer: Hierarchical vision transformer using shifted windows

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021

2021
[111]

Video swin transformer

Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022

2022
[112]

Interngpt: Solving vision-centric tasks by interacting with chatgpt beyond language

Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Zeqiang Lai, Yang Yang, Qingyun Li, et al. Interngpt: Solving vision-centric tasks by interacting with chatgpt beyond language. arXiv preprint arXiv:2305.05662, 2023

2023 arXiv
[113]

Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS, 2022 a

2022
[114]

12-in-1: Multi-task vision and language representation learning

Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10437--10446, 2020

2020
[116]

Unified-io: A unified model for vision, language, and multi-modal tasks

Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916, 2022 c

2022 arXiv
[117]

Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In CVPR, 2024

2024
[118]

Open-vocabulary 3d detection via image-level class and debiased cross-modal contrastive learning

Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary 3d detection via image-level class and debiased cross-modal contrastive learning. arXiv preprint arXiv:2207.01987, 2022 d

2022 arXiv
[119]

Video-chatgpt: Towards detailed video understanding via large vision and language models

Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023

2023 arXiv
[120]

Egoschema: A diagnostic benchmark for very long-form video language understanding

Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. NeurIPS, 2023

2023
[121]

Cross-stitch networks for multi-task learning

Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3994--4003, 2016

2016
[122]

Embodiedgpt: Vision-language pre-training via embodied chain of thought

Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advances in Neural Information Processing Systems, 36, 2024

2024
[123]

Multi-task learning of hierarchical vision-language representation

Duy-Kien Nguyen and Takayuki Okatani. Multi-task learning of hierarchical vision-language representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10492--10501, 2019

2019
[124]

Glide: Towards photorealistic image generation and editing with text-guided diffusion models

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021

2021 arXiv
[125]

Improved denoising diffusion probabilistic models

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021

2021
[126]

All in tokens: Unifying output space of visual tasks via soft token

Jia Ning, Chen Li, Zheng Zhang, Zigang Geng, Qi Dai, Kun He, and Han Hu. All in tokens: Unifying output space of visual tasks via soft token. arXiv preprint arXiv:2301.02229, 2023

2023 arXiv
[127]

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurIPS, 35: 0 27730--27744, 2022

2022
[128]

Actor-mimic: Deep multitask and transfer reinforcement learning

Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06342, 2015

2015 arXiv
[129]

Kosmos-2: Grounding multimodal large language models to the world

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023

2023 arXiv
[130]

Deep contextualized word representations

ME Peters, M Neumann, M Iyyer, M Gardner, C Clark, K Lee, and L Zettlemoyer. Deep contextualized word representations. In NAACL, 2018

2018
[131]

Model compression via distillation and quantization

Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. arXiv preprint arXiv:1802.05668, 2018

2018 arXiv
[132]

Shapellm: Universal 3d object understanding for embodied interaction

Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understanding for embodied interaction. In European Conference on Computer Vision, pages 214--238. Springer, 2024

2024
[133]

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

2021
[134]

Scaling language models: Methods, analysis & insights from training gopher

Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021

2021 arXiv
[135]

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020

2020
[136]

a uble, Francesco Locatello, Alexandre Lacoste, Yoshua Bengio, Chris Pal, Li Erran Li, and Bernhard Sch \

Nasim Rahaman, Martin Weiss, Frederik Tr \"a uble, Francesco Locatello, Alexandre Lacoste, Yoshua Bengio, Chris Pal, Li Erran Li, and Bernhard Sch \"o lkopf. A general purpose neural architecture for geospatial systems. arXiv preprint arXiv:2211.02348, 2022

2022 arXiv
[137]

Zero-shot text-to-image generation

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821--8831. PMLR, 2021

2021
[138]

Denseclip: Language-guided dense prediction with context-aware prompting

Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18082--...

2022
[139]

Dynamic spatial sparsification for efficient vision transformers and convolutional neural networks

Yongming Rao, Zuyan Liu, Wenliang Zhao, Jie Zhou, and Jiwen Lu. Dynamic spatial sparsification for efficient vision transformers and convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

2023
[140]

Raven progressive matrices

Jean Raven. Raven progressive matrices. In Handbook of nonverbal assessment, pages 223--237. Springer, 2003

2003
[141]

A generalist agent

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022

2022 arXiv
[142]

Pixellm: Pixel reasoning with large multimodal model

Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In CVPR, 2024

2024
[143]

Scaling vision with sparse mixture of experts

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andr \'e Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34: 0 8583--8595, 2021

2021
[144]

High-resolution image synthesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684--10695, 2022 a

2022
[145]

High-resolution image synthesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022 b

2022
[146]

Routing networks: Adaptive selection of non-linear functions for multi-task learning

Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017

2017 arXiv
[147]

Latent multi-task architecture learning

Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders S gaard. Latent multi-task architecture learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4822--4829, 2019

2019
[148]

Learning internal representations by error propagation, 1985

David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, et al. Learning internal representations by error propagation, 1985

1985
[149]

Imagenet large scale visual recognition challenge

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 0 211--252, 2015

2015
[150]

Policy distillation

Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint arXiv:1511.06295, 2015

2015 arXiv
[151]

Laion-5b: An open large-scale dataset for training next generation image-text models

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Informa...

2022
[152]

Multi-task learning as multi-objective optimization

Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. NeurIPS, 31, 2018

2018
[153]

Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

2018
[154]

Learning to multi-task by active sampling

Sahil Sharma, Ashutosh Jha, Parikshit Hegde, and Balaraman Ravindran. Learning to multi-task by active sampling. arXiv preprint arXiv:1702.06053, 2017

2017 arXiv
[155]

Where to look: Focus regions for visual question answering

Kevin J Shih, Saurabh Singh, and Derek Hoiem. Where to look: Focus regions for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4613--4621, 2016

2016
[156]

Unival: Unified model for image, video, audio and language tasks

Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. Unival: Unified model for image, video, audio and language tasks. arXiv preprint arXiv:2307.16184, 2023

2023 arXiv
[157]

Very deep convolutional networks for large-scale image recognition

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

2014 arXiv
[158]

Geodesic multi-modal mixup for robust fine-tuning

Junhyuk So, Changdae Oh, Yongtaek Lim, Hoyoon Byun, Minchul Shin, and Kyungwoo Song. Geodesic multi-modal mixup for robust fine-tuning. arXiv preprint arXiv:2203.03897, 2022

2022 arXiv
[159]

Videobert: A joint model for video and language representation learning

Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7464--7473, 2019

2019
[160]

Sequence to sequence learning with neural networks

I Sutskever. Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215, 2014

2014 arXiv
[161]

Chameleon: Mixed-modal early-fusion foundation models

Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024

2024 arXiv
[162]

Training data-efficient image transformers & distillation through attention

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv \'e J \'e gou. Training data-efficient image transformers & distillation through attention. In ICML, 2021

2021
[163]

Llama: Open and efficient foundation language models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

2023 arXiv
[164]

Multimodal few-shot learning with frozen language models

Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. NeruIPS, 2021

2021
[165]

Neural discrete representation learning

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017

2017
[166]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017 a

2017
[167]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017 b

2017
[168]

Git: Towards generalist vision transformer through universal language interface

Haiyang Wang, Hao Tang, Li Jiang, Shaoshuai Shi, Muhammad Ferjad Naeem, Hongsheng Li, Bernt Schiele, and Liwei Wang. Git: Towards generalist vision transformer through universal language interface. In ECCV, 2024 a

2024
[169]

Git: A generative image-to-text transformer for vision and language

Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022 a

2022 arXiv
[170]

Omnivl: One foundation model for image-language and video-language tasks

Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. Omnivl: One foundation model for image-language and video-language tasks. NeurIPS, 2022 b

2022
[171]

Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML, 2022 c

2022
[172]

One-peace: Exploring one general representation model toward unlimited modalities

Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. One-peace: Exploring one general representation model toward unlimited modalities. arXiv preprint arXiv:2305.11172, 2023 a

2023 arXiv
[173]

Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024 b

2024 arXiv
[174]

Lvbench: An extreme long video understanding benchmark

Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024 c

2024 arXiv
[175]

Visionllm: Large language model is also an open-ended decoder for vision-centric tasks

Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. NeurIPS, 2024 d

2024
[176]

Images speak in images: A generalist painter for in-context visual learning

Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6830--6839, 2023 b

2023
[177]

Internvideo: General video foundation models via generative and discriminative learning

Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022 d

2022 arXiv
[178]

Simvlm: Simple visual language model pretraining with weak supervision

Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904, 2021

2021 arXiv
[179]

The emergence of deepfake technology: A review

Mika Westerlund. The emergence of deepfake technology: A review. Technology innovation management review, 9 0 (11), 2019

2019
[180]

Diffusion models for medical anomaly detection

Julia Wolleb, Florentin Bieder, Robin Sandk \"u hler, and Philippe C Cattin. Diffusion models for medical anomaly detection. In International Conference on Medical image computing and computer-assisted intervention, pages 35--45. Springer, 2022

2022
[181]

Longvideobench: A benchmark for long-context interleaved video-language understanding

Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. NeurIPS, 2024 a

2024
[182]

Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks

Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Wenhai Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. arXiv preprint arXiv:2406.08394, 2024 b

2024 arXiv
[183]

Rethinking and improving relative position encoding for vision transformer

Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In ICCV, 2021

2021
[184]

Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise

Julian Wyatt, Adam Leach, Sebastian M Schmon, and Chris G Willcocks. Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 650--656, 2022

2022
[185]

Groupvit: Semantic segmentation emerges from text supervision

Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134--18144, 2022

2022
[186]

Show, attend and tell: Neural image caption generation with visual attention

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048--2057. PMLR, 2015

2015
[187]

Pointllm: Empowering large language models to understand point clouds

Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. In ECCV, 2024

2024
[188]

Universal instance perception as object discovery and retrieval

Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In CVPR, 2023

2023
[189]

Videococa: Video-text modeling with zero-shot transfer from contrastive captioners

Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Videococa: Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:2212.04979, 2022

2022 arXiv
[190]

Second: Sparsely embedded convolutional detection

Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18 0 (10): 0 3337, 2018

2018
[191]

Trace norm regularised deep multi-task learning

Yongxin Yang and Timothy M Hospedales. Trace norm regularised deep multi-task learning. arXiv preprint arXiv:1606.04038, 2016

2016 arXiv
[192]

A unified query-based paradigm for point cloud understanding

Zetong Yang, Li Jiang, Yanan Sun, Bernt Schiele, and Jiaya Jia. A unified query-based paradigm for point cloud understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8541--8551, 2022

2022
[193]

Stacked attention networks for image question answering

Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21--29, 2016

2016
[194]

Diffusion models and semi-supervised learners benefit mutually with few labels

Zebin You, Yong Zhong, Fan Bao, Jiacheng Sun, Chongxuan Li, and Jun Zhu. Diffusion models and semi-supervised learners benefit mutually with few labels. Advances in Neural Information Processing Systems, 36, 2024

2024
[195]

Gradient surgery for multi-task learning

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. NeurIPS, 33: 0 5824--5836, 2020

2020
[196]

Mm-vet: Evaluating large multimodal models for integrated capabilities

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023

2023 arXiv
[197]

Florence: A new foundation model for computer vision

Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021

2021 arXiv
[198]

Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on C...

2024
[199]

Open-vocabulary object detection using captions

Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14393--14402, 2021

2021
[200]

Adding conditional control to text-to-image diffusion models

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023 a

2023
[201]

Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention

Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In ICLR, 2024 a

2024
[202]

Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257, 2024 b

Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2...

2024 arXiv
[204]

Meta-transformer: A unified framework for multimodal learning

Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta-transformer: A unified framework for multimodal learning. arXiv preprint arXiv:2307.10802, 2023 c

2023 arXiv
[205]

A survey on multi-task learning

Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 34 0 (12): 0 5586--5609, 2021

2021
[206]

Facial landmark detection by deep multi-task learning

Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 94--108. Springer, 2014

2014
[207]

Ufc-bert: Unifying multi-modal controls for conditional image synthesis

Zhu Zhang, Jianxin Ma, Chang Zhou, Rui Men, Zhikang Li, Ming Ding, Jie Tang, Jingren Zhou, and Hongxia Yang. Ufc-bert: Unifying multi-modal controls for conditional image synthesis. Advances in Neural Information Processing Systems, 34: 0 27196--27208, 2021

2021
[208]

Unleashing text-to-image diffusion models for visual perception

Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. arXiv preprint arXiv:2303.02153, 2023

2023 arXiv
[209]

A modulation module for multi-task learning with applications in image retrieval

Xiangyun Zhao, Haoxiang Li, Xiaohui Shen, Xiaodan Liang, and Ying Wu. A modulation module for multi-task learning with applications in image retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), pages 401--416, 2018

2018
[210]

Diffusion models for missing value imputation in tabular data

Shuhan Zheng and Nontawat Charoenphakdee. Diffusion models for missing value imputation in tabular data. arXiv preprint arXiv:2210.17128, 2022

2022 arXiv
[211]

Regionclip: Region-based language-image pretraining

Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, ...

2022
[212]

Learning procedure-aware video representation from instructional videos and their narrations

Yiwu Zhong, Licheng Yu, Yang Bai, Shangwen Li, Xueting Yan, and Yin Li. Learning procedure-aware video representation from instructional videos and their narrations. In CVPR, 2023

2023
[213]

Extract free dense labels from clip

Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pages 696--712. Springer, 2022 a

2022
[214]

Unified vision-language pre-training for image captioning and vqa

Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13041--13049, 2020

2020
[215]

a henb \

Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Kr \"a henb \"u hl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European Conference on Computer Vision, pages 350--368. Springer, 2022 b

2022
[216]

Minigpt-4: Enhancing vision-language understanding with advanced large language models

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023

2023 arXiv
[217]

Uni-perceiver-moe: Learning sparse generalist models with conditional moes

Jinguo Zhu, Xizhou Zhu, Wenhai Wang, Xiaohua Wang, Hongsheng Li, Xiaogang Wang, and Jifeng Dai. Uni-perceiver-moe: Learning sparse generalist models with conditional moes. Advances in Neural Information Processing Systems, 35: 0 2664--2678, 2022 a

2022
[218]

Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks

Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

2022
[219]

Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks

Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In CVPR, 2022 c

2022
[220]

, " * write output.state after.block = add.period write newline

ENTRY address archive author booktitle chapter doi edition editor eid eprint howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mi...
[221]

write newline

" write newline "" before.all 'output.state := FUNCTION add.period duplicate empty 'skip "." * add.blank if FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap dupl...
[222]

, " * write output.state after.block = add.period write newline

ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...
[223]

write newline

" write newline "" before.all 'output.state := FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap duplicate "7" = swap duplicate "8" = swap "9" = or or or or or or...
[224]

, " * write output.state after.block = add.period write newline

ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...
[225]

write newline

" write newline "" before.all 'output.state := FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap duplicate "7" = swap duplicate "8" = swap "9" = or or or or or or...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Gpt-4 technical report

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023

[2] [2]

Deep learning using rectified linear units (relu)

AF Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375, 2018

arXiv 2018

[3] [3]

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022

2022

[4] [4]

Bottom-up and top-down attention for image captioning and visual question answering

Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077--6086, 2018

2018

[5] [5]

Neural module networks

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39--48, 2016

2016

[6] [6]

Vqa: Visual question answering

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425--2433, 2015

2015

[7] [7]

Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens

Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.03413, 2024

arXiv 2024

[8] [8]

Foundational models defining a new era in vision: A survey and outlook

Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundational models defining a new era in vision: A survey and outlook. arXiv preprint arXiv:2307.13721, 2023

arXiv 2023