Pith. sign in

REVIEW 3 major objections 6 minor 74 references

FREE: Fast and Robust Vision Language Models with Early Exits

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FREE claims that adversarially training intermediate transformer layers to mimic a frozen vision-language model's final layer lets early-exit inference run more than 1.5x faster with comparable accuracy, using only a small labeled set, or…

desk verdict Interesting adversarial early-exit method for frozen VLMs, but the headline 1.51x speedup is a parameter-count proxy that overstates real latency gains; needs wall-clock measurements before the quantitative claims hold. read the letter →

arxiv 2506.06884 v1 pith:X2JH3KM7 submitted 2025-06-07 cs.LG cs.CV

Divya Jyoti Bajpai , Manjesh Kumar Hanawal This is my paper
classification cs.LGcs.CV
keywords earlyexitinferencevision-languagemodelsadversarialtrainingGAN-basedfeaturealignmentinput-adaptiveaccelerationoverthinkingmitigationknowledgedistillation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models that couple a frozen image encoder with a large frozen language model are accurate but slow: every generated token passes through the full decoder even when the answer is obvious. This paper proposes FREE, a way to attach early-exit branches to such frozen backbones so that easy tokens are predicted after a handful of layers. The key move is to train each exit's extra transformer layer, in a GAN-style setup, to produce hidden representations indistinguishable from the final layer's, which lets the frozen final-layer classifier be reused at every exit instead of training new classifiers. The authors report inference speedups above 1.51x with accuracy matching (and sometimes exceeding) the vanilla model across captioning, VQA, and visual dialogue, while also showing the exits make the model more tolerant to noisy images. The claim matters because it offers a path to cheaper deployment of frozen-backbone VLMs with minimal labeled data.

What carries the argument

The central mechanism is a GAN-based alignment between each exit and the final layer: the exit transformer (a replica of one decoder layer, trainable) acts as generator, and a per-exit feature classifier acts as discriminator that scores whether a hidden state comes from the final layer or from the exit. Adversarial loss pushes exit representations toward final-layer representations, so the frozen final-layer classifier (LM head) can be reused as the exit classifier for all exits; this design choice is what cuts trainable parameters. To keep GAN training stable, the method adds either cross-entropy on small labeled sets, KL divergence from final-layer soft labels, or CapFilt-generated synthetic captions when no labels exist. At inference, a confidence threshold on the reused classifier decides, token by token, which layer emits the next word; the reported speedup is the ratio $\frac{\text{total backbone parameters}}{\text{average parameters used}}$.

What would settle it

A direct check: run the released code on the VQAv2 validation set on a single GPU and compare wall-clock time per generated token for FREE versus vanilla BLIP-2 at the same batch size and beam size; if the measured speedup is well below the reported 1.45x to 1.77x (for instance under 1.2x), the central speed claim fails even if accuracy matches.

Watch

Extended reading notes

Core claim

The paper's central claim is that input-adaptive early exiting can be made to work in autoregressive vision-language models without training large exit classifiers, by exploiting the frozen model's own final-layer classifier. FREE attaches, at chosen decoder layers, an exit consisting of one trainable exit-transformer layer plus a feature classifier; the feature classifier is trained to tell exit representations from final-layer representations, and the exit transformer is trained to fool it, so that exit features converge toward final-layer features. Because exit features mimic the final layer, the frozen final-layer classifier (the LM head) can serve as the exit classifier at every exit, cutting trainable parameters roughly in half relative to classical early-exit training. At inference, a token is emitted as soon as the max-softmax confidence at any exit clears a threshold, so easy tokens skip most of the decoder. The paper argues this simultaneously mitigates two failure modes it identifies in frozen-backbone VLMs: overthinking, where deeper layers corrupt already-correct predictions, and the mid-crisis, where intermediate layers lose information that deeper layers must reacquire. On COCO, NoCaps, VQAv2, OK-VQA, GQA, VizWiz, and VisDial, the method reports accuracy comparable to vanilla BLIP-2 with parameter-based speedups of 1.45x to 1.77x, and matching behavior on MiniGPT and InstructBLIP.

Load-bearing premise

The weakest link is the speedup metric: the paper counts parameters saved, not time saved, and assumes a model that uses half the layers will run about twice as fast, which is often false for autoregressive decoding where reading the weights and key-value cache costs time regardless of how few layers are used.

Editorial extensions

If this is right

  • On the reported benchmarks, FREE matches or beats vanilla BLIP-2 accuracy on captioning and VQA while using a fraction of the decoder layers, implying overthinking is a real cost in frozen-backbone VLMs and can be removed without retraining the backbone.
  • Because only one extra transformer layer per exit is trained and the final classifier is frozen, the method needs roughly 52% fewer trainable parameters than classical early exits for OPT-2.7B, lowering the barrier to attaching exits to large frozen decoders.
  • Unsupervised variants using knowledge distillation or CapFilt synthetic captions reach accuracy comparable to the vanilla model, so the speedup is available even when no labeled task data exists.
  • Noise-injected images degrade FREE less than they degrade vanilla BLIP-2, because confident early exits avoid corrupted deeper-layer representations.
  • The same exit recipe applied to MiniGPT and InstructBLIP yields roughly 1.5x to 1.7x parameter-based speedups with small accuracy costs, suggesting the approach transfers across frozen-backbone architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not in the paper: a single-GPU wall-clock latency profile across the reported tasks would test whether the parameter-based speedup holds, since autoregressive decoding pays fixed per-token costs that do not shrink when fewer layers run.
  • A natural follow-up the paper leaves open is data-driven exit placement: its own mid-crisis curves suggest putting exits where intermediate accuracy dips most, and optimising placement under a parameter budget is an explicit open problem in the limitations section.
  • Because the exit classifier is the frozen final LM head, the recipe should port to larger frozen decoders and to decoders with tied input-output embeddings, where the per-exit parameter savings are largest; extending to OPT-6.7B would test that.
  • The noise-robustness result hints at a sharper test the paper does not run: if early exits avoid corrupted deep layers, FREE may weather adversarial image perturbations better than vanilla inference, which could be checked with standard attack budgets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes FREE, an early-exit method for frozen-backbone vision-language models. Exits consisting of one exit transformer layer and the frozen final-layer classifier are attached to intermediate decoder layers. The exit transformers are trained adversarially, with a per-exit feature classifier as discriminator, to produce representations similar to the final layer; inference exits per token when the frozen classifier's max-softmax confidence exceeds a threshold. The method is evaluated on BLIP-2 (OPT-2.7B and FlanT5-XL), MiniGPT, and InstructBLIP across captioning (COCO, NoCaps), VQA (VQAv2, OK-VQA, GQA, VizWiz), and visual dialogue (VisDial), with claims of 'inference speed >1.51x' and comparable or better accuracy, plus an appendix on robustness to Gaussian noise. The paper also introduces the term 'mid-crisis' to describe an intermediate-layer accuracy dip in vanilla early-exit VLMs.

Significance. The direction is practically motivated and timely: accelerating frozen-backbone VLMs without large labeled datasets is a useful goal, and the adversarial feature-mimicry formulation with a shared frozen final classifier is a reasonable way to reduce exit-parameter overhead. The breadth of backbones and tasks is a strength, as are the qualitative token-exit visualization and the explicit robustness experiment. If the speed claims were backed by end-to-end wall-clock measurements, the contribution would be clearly valuable. However, the headline speed result currently rests on a parameter-count proxy rather than measured latency, and the one wall-clock measurement reported disagrees with the proxy in the expected direction; the significance is therefore conditional on additional measurement.

major comments (3)
  1. [Section 4 (Speedup formula); Appendix D] The speedup metric is a parameter-count proxy, not a latency measurement, and it systematically inflates end-to-end speed. The denominator counts only decoder transformer layers via the term (i+k)p, while the numerator is the whole-model parameter count; fixed components such as the ViT-g image encoder, Q-Former/projection, token embeddings, and the LM head are included in the numerator but never in the denominator. As written, the formula would give speedup 1.0x to any model without exits, yet Table 3 reports 1.07x for vanilla BLIP-2-OPT2.7B and 0.05x for Flamingo80B, so the metric is also not applied as stated (it is actually relative to FlanT5XL's total parameter count). The only wall-clock measurement in Appendix D (about 7 minutes for FREE vs 11 minutes for vanilla on COCO Karpathy) yields about 1.57x, while Table 4 reports 1.75x for that same setting, a discrepancy in the direction predicted by the omitted fixed overhead. Since no wall-clock data are reported for VQAv2, OK-VQA, GQA, or VisDial, the abstract's claim of 'inference speed >1.51x' is not established for those tasks.
  2. [Section 4, Training/Inference; Table 2] The evaluation protocol for VQA and VisDial is underspecified. The text says exits are trained on the validation split (80% train, 20% development) and that results are reported on the test dataset, but Table 2 reports both 'VQAv2 train' and 'VQAv2 test' without identifying which official splits these are (e.g., test-dev, test-std, or a held-out portion of validation). It is also not stated explicitly whether the confidence threshold alpha is selected on the 20% development split and then frozen for the reported test evaluation. This ambiguity creates a risk of threshold overfitting or accidental overlap between the split used for exit training/threshold selection and the split used for the reported numbers; the authors should specify the exact split provenance for each table.
  3. [Tables 1-4, 6-8] All accuracy and speed results are single-run point estimates with no variance information. Several key comparisons are within one point or less (e.g., Table 3: FREE FlanT5XL VQAv2 test 62.1 vs BLIP-2 62.5; Table 4: FREE-V-O BLEU-4 41.9 vs BLIP-2-V-O 41.7). Because the threshold is tuned on a development split and no seeds or confidence intervals are reported, the 'comparable accuracy' claim cannot be distinguished from run-to-run noise. At minimum, the authors should report multiple seeds for the main tables or justify why a single run is representative for these large models.
minor comments (6)
  1. [References; Section 1] The in-text citation 'Bajpai and Hanawal' for the BEEM work has no year, and the corresponding reference entry also lacks a year; please resolve the citation.
  2. [Abstract and full text] The code URL appears as 'available at/gtb' in the full text; use the URL provided in the abstract (github.com/Div290/FREE) consistently.
  3. [Tables 1 and 3] The column headers '#Train Params' and '#Total params' should be reconciled and defined, since the speedup formula's numerator depends on which parameter count is used.
  4. [Section 3.1] 'Mid-crisis' is used before being defined; give a formal definition and state explicitly how it is measured and how it differs from overthinking beyond the qualitative description.
  5. [Appendix D] The paper reports that FREE training takes about 26 hours vs 18 hours for vanilla BLIP-2; this training-time overhead should be mentioned in the main text because it is part of the cost/benefit trade-off.
  6. [Figure 4a] The legend and markers in Figure 4a are very small and hard to distinguish; enlarge the figure in the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported results are empirical evaluations, and the speedup figure is a defined proxy metric rather than a prediction derived from fitted inputs.

full rationale

The paper's central claims are empirical: exit transformer layers are trained with the stated losses (cross-entropy or KL plus adversarial generator/discriminator losses, Eqs. 1-3 in Sections 3.2-3.4) and then evaluated on COCO, NoCaps, VQAv2, OK-VQA, GQA, and VisDial. The claimed speedup is not obtained by re-inserting a fitted constant into the same formula that generated it; it is a defined metric, Speedup = Total parameters / Average number of parameters used (Section 4), measured from exit-exit distributions reported in Table 11. Even if this parameter-count proxy overstates wall-clock speedup relative to the single reported COCO timing, that is a measurement-validity concern, not circularity. The self-citations to CapEEN, DAdEE, and the line of early-exit work are architectural precedents and implementation details, not load-bearing inputs to the result; the statement 'Similar to Bajpai and Hanawal (2024c), we use a feature classifier...' concerns a hyperparameter choice. The acknowledged limitation that optimal exit placement remains unexplored (Section 7) is a genuine open problem and does not reintroduce the central result as an input. No equation in the paper reduces to its own assumptions, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No physically invented entities are introduced. The method relies on two domain assumptions (adversarial alignment transfers and confidence calibration) and one tuned threshold. The parameter-based speedup metric is an additional measurement assumption, listed in weakest_assumption.

free parameters (1)
  • exit confidence threshold alpha = 0.8 (chosen from {0.5, 0.6, 0.7, 0.8, 0.9, 1.0} on the held-out validation split)
    Controls the speed/accuracy tradeoff; all reported speedups are at the best threshold on the development split, so the headline speedup numbers are conditional on this choice.
assumptions (3)
  • domain assumption Adversarial feature matching between exit transformers and the final layer produces representations that transfer to the frozen final-layer classifier.
    Core mechanism of FREE; no convergence or transfer guarantee is proven, only empirical evidence.
  • domain assumption Maximum softmax probability of the frozen final-layer classifier is a valid confidence measure for early exiting.
    Inference policy uses S_i = max_v C_N(E_i(h_i_t))(v) > alpha; if confidence is miscalibrated, exits will misroute tokens.
  • domain assumption The final layer of the frozen backbone is a better prediction target than intermediate layers for all tasks.
    Exits are trained to mimic final-layer features; if final-layer features are suboptimal for some tokens, exits inherit that limitation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FREE: Fast and Robust Vision Language Models with Early Exits." pith.science (2026). https://pith.science/paper/X2JH3KM7

@misc{pith2026250606884,
  author       = {Pith},
  title        = {Pith review of: FREE: Fast and Robust Vision Language Models with Early Exits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X2JH3KM7}},
  note         = {Machine review of arXiv:2506.06884}
}
read the original abstract

In recent years, Vision-Language Models (VLMs) have shown remarkable performance improvements in Vision-Language tasks. However, their large size poses challenges for real-world applications where inference latency is a concern. To tackle this issue, we propose employing Early Exit (EE) strategies in VLMs. However, training exit classifiers in VLMs is challenging, particularly with limited labeled training data. To address this, we introduce FREE, an adversarial training approach within a GAN-based framework. Here, each exit consists of a transformer layer and a classifier. The transformer layer is adversarially trained to produce feature representations similar to the final layer, while a feature classifier serves as the discriminator. Our method focuses on performing input-adaptive inference that increases inference speed with minimal drop in performance. Experimental results demonstrate the effectiveness of our approach in enhancing accuracy and model robustness by mitigating overthinking and the phenomenon of mid-crisis that we highlight. We experimentally validate that our method speeds up the inference process by more than 1.51x while retaining comparable performance. The source code is available at https://github.com/Div290/FREE.

Figures

Figures reproduced from arXiv: 2506.06884 by the authors.

Figure 1
Figure 1. This figure provides some example outputs of FREE ViT-g OPT [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. This figure illustrates our model’s architecture. The Q-Former output and previous tokens pass through [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Left: VQA accuracy on the VQAv2 dataset with BLIP-2-ViT-g-FlanT5 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Left: BLEU-1 score for COCO with model BLIP-2-ViT-L-OPT [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

74 extracted references · 30 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2019. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948--8957

    work page 2019

  4. [4]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716--23736

    2022

  5. [5]

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382--398. Springer

    2016

  6. [6]

    Bajpai, Aastha Jaiswal, and Manjesh K

    Divya J. Bajpai, Aastha Jaiswal, and Manjesh K. Hanawal. 2024 a . https://doi.org/10.1109/ICC51166.2024.10622954 I-splitee: Image classification in split computing dnns with early exits . In ICC 2024 - IEEE International Conference on Communications, pages 2658--2663

    arXiv 2024

  7. [7]

    Divya Jyoti Bajpai and Manjesh Hanawal. 2024 a . https://doi.org/10.18653/v1/2024.findings-acl.101 C ee BERT : Cross-domain inference in early exit BERT . In Findings of the Association for Computational Linguistics: ACL 2024, pages 1736--1748, Bangkok, Thailand. Association for Computational Linguistics

    work page doi:10.18653/v1/2024.findings-acl.101 2024

  8. [8]

    Beem: Boosting performance of early exit dnns using multi-exit classifiers as experts

    Divya Jyoti Bajpai and Manjesh Kumar Hanawal. Beem: Boosting performance of early exit dnns using multi-exit classifiers as experts. In The Thirteenth International Conference on Learning Representations

    work page

Show all 74 references
  1. [9]

    Divya Jyoti Bajpai and Manjesh Kumar Hanawal. 2024 b . https://doi.org/10.18653/v1/2024.findings-emnlp.376 C ap EEN : Image captioning with early exits and knowledge distillation . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6458--6472, Miam...

    2024 doi

  2. [10]

    Divya Jyoti Bajpai and Manjesh Kumar Hanawal. 2024 c . https://doi.org/10.18653/v1/2024.findings-emnlp.371 DA d EE : Unsupervised domain adaptation in early exit PLM s . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6389--6400, Miami, Florida,...

    2024 doi

  3. [11]

    Divya Jyoti Bajpai and Manjesh Kumar Hanawal. 2024 d . Distributed inference on mobile edge and cloud: An early exit based clustering approach. arXiv preprint arXiv:2410.05338

    2024 arXiv

  4. [12]

    Divya Jyoti Bajpai, Vivek Kumar Trivedi, Sohan L Yadav, and Manjesh Kumar Hanawal. 2024 b . https://doi.org/10.1145/3639856.3639873 Splitee: Early exit in deep neural networks with split computing . In Proceedings of the Third International Conference on AI-ML Systems, AIMLSys...

    2024

  5. [13]

    Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72

    2005

  6. [14]

    Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, and Furu Wei. 2021. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. arXiv preprint arXiv:2111.02358

    2021 arXiv

  7. [15]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  8. [16]

    Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. 2022 a . Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18030--18040

    2022

  9. [17]

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. 2022 b . Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794

    2022 arXiv

  10. [18]

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104--120. Springer

    2020

  11. [19]

    Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning, pages 1931--1942. PMLR

    2021

  12. [20]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1--53

    2024

  13. [21]

    Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. 2018. Generative adversarial networks: An overview. IEEE signal processing magazine, 35(1):53--65

    2018

  14. [22]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2024. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36

    2024

  15. [23]

    Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jos \'e MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 326--335

    2017

  16. [24]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint...

    2020 arXiv

  17. [25]

    Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. 2024. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710

    2024 arXiv

  18. [26]

    Zhengcong Fei, Xu Yan, Shuhui Wang, and Qi Tian. 2022. Deecap: Dynamic early exiting for efficient image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12216--12226

    2022

  19. [27]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904--6913

    2017

  20. [28]

    Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven CH Hoi. 2022. From images to textual prompts: Zero-shot vqa with frozen large language models. arXiv preprint arXiv:2212.10846

    2022 arXiv

  21. [29]

    Yizeng Han, Dongchen Han, Zeyu Liu, Yulin Wang, Xuran Pan, Yifan Pu, Chao Deng, Junlan Feng, Shiji Song, and Gao Huang. 2023. Dynamic perceiver for efficient visual recognition. arXiv preprint arXiv:2306.11248

    2023 arXiv

  22. [30]

    Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv:1703.09844

    2017 arXiv

  23. [31]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700--6709

    2019

  24. [32]

    Yixin Ji, Jikai Wang, Juntao Li, Qiang Chen, Wenliang Chen, and Min Zhang. 2023. Early exit with disentangled representation and equiangular tight frame. In Findings of the Association for Computational Linguistics: ACL 2023, pages 14128--14142

    2023

  25. [33]

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904...

    2021

  26. [34]

    Weiyu Ju, Wei Bao, Dong Yuan, Liming Ge, and Bing Bing Zhou. 2021. Learning early exit for deep neural network inference on mobile devices through multi-armed bandits. In 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid), pages 11--20. IEEE

    2021

  27. [35]

    Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. 2019. Shallow-deep networks: Understanding and mitigating network overthinking. In International conference on machine learning, pages 3301--3310. PMLR

    2019

  28. [36]

    Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  29. [37]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of...

    2017

  30. [38]

    Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven CH Hoi. 2022 a . Lavis: A library for language-vision intelligence. arXiv preprint arXiv:2209.09019

    2022 arXiv

  31. [39]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR

    2023

  32. [40]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022 b . Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888--12900. PMLR

    2022

  33. [41]

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694--9705

    2021

  34. [42]

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August...

    2020

  35. [43]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, ...

    2014

  36. [44]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  37. [45]

    Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju. 2020. Fastbert: a self-distilling bert with adaptive inference time. arXiv preprint arXiv:2004.02178

    2020 arXiv

  38. [46]

    Xiangyang Liu, Tianxiang Sun, Junliang He, Lingling Wu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, and Xipeng Qiu. 2021. Towards efficient nlp: A standard evaluation and a strong baseline. arXiv preprint arXiv:2110.07038

    2021 arXiv

  39. [47]

    Oscar Ma \ n as, Pau Rodriguez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, and Aishwarya Agrawal. 2022. Mapl: Parameter-efficient adaptation of unimodal pre-trained models for vision-language few-shot prompting. arXiv preprint arXiv:2210.07179

    2022 arXiv

  40. [48]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195--3204

    2019

  41. [49]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  42. [50]

    Mary Phuong and Christoph H Lampert. 2019. Distillation-based training for multi-exit architectures. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1355--1364

    2019

  43. [51]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  44. [52]

    Minho Ryu, Geonseok Lee, and Kichun Lee. 2022. Knowledge distillation for bert unsupervised domain adaptation. Knowledge and Information Systems, 64(11):3113--3128

    2022

  45. [53]

    Shengkun Tang, Yaqing Wang, Zhenglun Kong, Tianchi Zhang, Yao Li, Caiwen Ding, Yanzhi Wang, Yi Liang, and Dongkuan Xu. 2023. You need multiple exiting: Dynamic early exiting for accelerating unified vision language model. In Proceedings of the IEEE/CVF Conference on Computer V...

    2023

  46. [54]

    Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. 2016. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2464--2469. IEEE

    2016

  47. [55]

    Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi. 2022. Plug-and-play vqa: Zero-shot vqa by conjoining large pretrained models with zero training. arXiv preprint arXiv:2210.08773

    2022 arXiv

  48. [56]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  49. [57]

    Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200--212

    2021

  50. [58]

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566--4575

    2015

  51. [59]

    Meiqi Wang, Jianqiao Mo, Jun Lin, Zhongfeng Wang, and Li Du. 2019. Dynexit: A dynamic early-exit strategy for deep residual networks. In 2019 IEEE International Workshop on Signal Processing Systems (SiPS), pages 178--183. IEEE

    2019

  52. [60]

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022 a . Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine...

    2022

  53. [61]

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2022 b . Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442

    2022 arXiv

  54. [62]

    Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904

    2021 arXiv

  55. [63]

    Zizhao Wang, Wei Bao, Dong Yuan, Liming Ge, Nguyen H Tran, and Albert Zomaya. 2020. Accelerating on-device dnn inference during service outage through scheduling early exit. Computer Communications, 162:69--82

    2020

  56. [64]

    Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. Deebert: Dynamic early exiting for accelerating bert inference. arXiv preprint arXiv:2004.12993

    2020 arXiv

  57. [65]

    Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang. 2020. Resolution adaptive networks for efficient inference. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2369--2378

    2020

  58. [66]

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. 2021. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783

    2021 arXiv

  59. [67]

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917

    2022 arXiv

  60. [68]

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. 2022. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123--18133

    2022

  61. [69]

    Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5579--5588

    2021

  62. [70]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022 a . Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  63. [71]

    Zhen Zhang, Wei Zhu, Jinfan Zhang, Peng Wang, Rize Jin, and Tae-Sun Chung. 2022 b . Pcee-bert: accelerating bert inference via patient and confident early exiting. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 327--338

    2022

  64. [72]

    Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33:18330--18341

    2020

  65. [73]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

  66. [74]

    Wei Zhu. 2021. Leebert: Learned early exit for bert with cross-level optimization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pag...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.