pith. machine review for the scientific record. sign in

arxiv: 2304.10592 · v2 · submitted 2023-04-20 · 💻 cs.CV

Recognition: 2 theorem links

· Lean Theorem

MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

Deyao Zhu, Jun Chen, Mohamed Elhoseiny, Xiang Li, Xiaoqian Shen

Pith reviewed 2026-05-10 20:32 UTC · model grok-4.3

classification 💻 cs.CV
keywords vision-language modellarge language modelmultimodal alignmentprojection layerimage captioningemergent abilitiestwo-stage training
0
0 comments X

The pith

Aligning a frozen visual encoder to Vicuna via one projection layer and two-stage training produces GPT-4-like multimodal abilities such as sketch-to-website generation.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that advanced vision-language skills demonstrated by GPT-4 arise when visual features from a frozen encoder are aligned to a frozen large language model using only a single projection layer. Training occurs in two stages: first on basic image-caption pairs, then on a curated set of detailed image descriptions to improve output naturalness and reduce repetition. A sympathetic reader would care because the result suggests these capabilities can appear without jointly training an entire multimodal system from scratch.

Core claim

By aligning a frozen visual encoder with the frozen Vicuna LLM using one projection layer, trained first on short image captions and then on detailed image descriptions, MiniGPT-4 acquires numerous advanced multi-modal abilities including generating detailed image descriptions, creating websites from hand-drawn drafts, writing stories and poems inspired by images, teaching cooking from food photos, and other emerging capabilities similar to those in GPT-4.

What carries the argument

The single projection layer that maps outputs from the frozen visual encoder into the input space of the frozen Vicuna language model, enabling the LLM to interpret and respond to visual information after two-stage training.

If this is right

  • The model generates detailed and natural image descriptions without repetition or fragmentation.
  • Websites can be created directly from hand-drawn drafts or sketches.
  • Stories and poems can be written based on input images.
  • Cooking instructions can be provided from photos of food.
  • Additional emergent abilities such as identifying humorous elements in images appear.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This indicates that freezing both the vision encoder and the LLM while training only the connector suffices for advanced multimodal performance.
  • Similar alignment could be tested with other base LLMs to determine if the choice of Vicuna is necessary for these specific capabilities.
  • The emphasis on a second-stage dataset of detailed descriptions implies that data curation may be as important as the alignment architecture itself for usable outputs.

Load-bearing premise

The observed advanced abilities result primarily from alignment with the advanced LLM rather than from the choice of training data or the projection layer memorizing patterns in the detailed description dataset.

What would settle it

Train the identical projection layer on the same data but align it to a weaker language model instead of Vicuna, then test whether capabilities such as website creation from hand-drawn drafts disappear.

read the original abstract

The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text and identifying humorous elements within images. These features are rarely observed in previous vision-language models. However, the technical details behind GPT-4 continue to remain undisclosed. We believe that the enhanced multi-modal generation capabilities of GPT-4 stem from the utilization of sophisticated large language models (LLM). To examine this phenomenon, we present MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer. Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, such as detailed image description generation and website creation from hand-drawn drafts. Furthermore, we also observe other emerging capabilities in MiniGPT-4, including writing stories and poems inspired by given images, teaching users how to cook based on food photos, and so on. In our experiment, we found that the model trained on short image caption pairs could produce unnatural language outputs (e.g., repetition and fragmentation). To address this problem, we curate a detailed image description dataset in the second stage to finetune the model, which consequently improves the model's generation reliability and overall usability. Our code, pre-trained model, and collected dataset are available at https://minigpt-4.github.io/.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript presents MiniGPT-4, which aligns a frozen visual encoder with the frozen Vicuna LLM using a single projection layer. It employs two-stage training—first on short image-caption pairs, then fine-tuning on a curated dataset of detailed image descriptions—to achieve advanced vision-language capabilities similar to GPT-4, including detailed image descriptions, website generation from hand-drawn sketches, story/poem writing from images, and instructional responses from visual inputs. The authors support these claims with qualitative examples and release the code, pre-trained weights, and dataset.

Significance. If the central claim holds, the work is significant because it shows that sophisticated multi-modal generation abilities can be obtained by aligning visual features with an advanced frozen LLM without retraining the language model itself. The public release of code, weights, and the detailed-description dataset is a clear strength that enables reproducibility and community follow-up. The primarily qualitative evaluation and lack of isolating experiments, however, limit the strength of the attribution to the LLM choice.

major comments (2)
  1. [Experiments] Experiments section: The assertion that the second-stage fine-tuning on the curated detailed-description dataset resolves unnatural outputs (repetitions and fragmentation) is supported solely by anecdotal before-and-after examples; no quantitative metrics (e.g., human preference scores, perplexity on held-out captions, or automated coherence measures) are reported to document the magnitude or reliability of the improvement.
  2. [Method and Experiments] Method and Experiments sections: The central claim that advanced capabilities arise from alignment with an advanced LLM (Vicuna) is not isolated from confounding factors; the manuscript contains no ablations comparing the same pipeline with a weaker LLM (e.g., base LLaMA), with the second-stage dataset removed, or with a higher-capacity projection layer, leaving open the possibility that observed fluency and task performance derive primarily from the high-quality second-stage data or projection-layer memorization.
minor comments (2)
  1. [Abstract] Abstract: The phrase 'for the first time' overstates novelty given prior alignment work (e.g., BLIP-2); rephrase to highlight the specific combination of Vicuna and the two-stage detailed-description stage.
  2. [Qualitative results] Qualitative results: The presented examples would be strengthened by inclusion of failure cases or a broader range of out-of-distribution images to give readers a balanced view of model limitations.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for their constructive comments and positive assessment of the work's significance and reproducibility. We address the major comments point-by-point below and outline proposed revisions.

read point-by-point responses

  1. Referee: Experiments section: The assertion that the second-stage fine-tuning on the curated detailed-description dataset resolves unnatural outputs (repetitions and fragmentation) is supported solely by anecdotal before-and-after examples; no quantitative metrics (e.g., human preference scores, perplexity on held-out captions, or automated coherence measures) are reported to document the magnitude or reliability of the improvement.

    Authors: We thank the referee for this observation. Our current evidence for the benefits of the second-stage fine-tuning is qualitative. We agree that quantitative support would be valuable. In the revised manuscript, we will include results from a human study where evaluators compare first-stage and second-stage outputs on naturalness and coherence for a held-out set of images, reporting win rates or preference percentages. revision: yes

  2. Referee: Method and Experiments sections: The central claim that advanced capabilities arise from alignment with an advanced LLM (Vicuna) is not isolated from confounding factors; the manuscript contains no ablations comparing the same pipeline with a weaker LLM (e.g., base LLaMA), with the second-stage dataset removed, or with a higher-capacity projection layer, leaving open the possibility that observed fluency and task performance derive primarily from the high-quality second-stage data or projection-layer memorization.

    Authors: We concur that isolating the contribution of the advanced LLM through ablations would strengthen the attribution. However, we did not perform training with base LLaMA due to the substantial computational cost and time required. The manuscript already notes that the first-stage model (without second-stage fine-tuning) produces unnatural outputs, indicating the second stage's role in improving language quality. We used a minimal projection layer to show that advanced capabilities do not require complex alignment modules. In revision, we will add a dedicated limitations paragraph discussing these points and the potential role of the second-stage data, while maintaining that the simple alignment to Vicuna enables the observed GPT-4-like behaviors as evidenced by the qualitative demonstrations. revision: partial

standing simulated objections not resolved
  • We cannot conduct the full set of ablation experiments with base LLaMA or additional quantitative isolation studies within the scope of this work due to resource limitations.

Circularity Check

0 steps flagged

No circularity; empirical alignment demonstration with no derivation chain or self-referential predictions.

full rationale

The paper makes no mathematical or first-principles claims. It describes an empirical two-stage training procedure (short captions then curated detailed descriptions) to align a frozen visual encoder to a frozen Vicuna LLM via a single projection layer, then reports qualitative capabilities. No equations define a quantity in terms of itself, no fitted parameters are relabeled as predictions, and no load-bearing steps reduce to self-citations or ansatzes imported from prior author work. The central observation—that alignment yields GPT-4-like behaviors—is presented as an empirical finding supported by released model outputs and a public dataset, not as a derivation that collapses to its inputs by construction.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on the empirical observation that alignment plus a second-stage dataset produces the listed capabilities. No new physical or mathematical axioms are introduced.

free parameters (1)
  • projection layer weights
    The only trainable parameters; fitted on image-text pairs in stage 1 and detailed descriptions in stage 2.
axioms (1)
  • domain assumption Frozen visual encoder and frozen Vicuna LLM preserve their pre-trained capabilities when only the projection is trained.
    Invoked in the method description to justify not updating the large models.

pith-pipeline@v0.9.0 · 5562 in / 1273 out tokens · 28031 ms · 2026-05-10T20:32:24.701364+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

    cs.CL 2024-09 accept novelty 8.0

    MMMU-Pro is a stricter multimodal benchmark that removes text-only solvable questions, augments options, and requires reading text from images, yielding substantially lower model scores of 16.8-26.9%.

  2. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

    cs.AI 2024-04 accept novelty 8.0

    OSWorld provides the first unified real-computer benchmark for open-ended multimodal agent tasks, exposing large performance gaps between humans and state-of-the-art LLM/VLM agents.

  3. DistractMIA: Black-Box Membership Inference on Vision-Language Models via Semantic Distraction

    cs.CV 2026-05 unverdicted novelty 7.0

    DistractMIA performs output-only black-box membership inference on vision-language models by inserting semantic distractors and measuring shifts in generated text responses.

  4. Chronicles-OCR: A Cross-Temporal Perception Benchmark for the Evolutionary Trajectory of Chinese Characters

    cs.CV 2026-05 unverdicted novelty 7.0

    Chronicles-OCR is the first benchmark with 2,800 images across the complete evolutionary trajectory of Chinese characters, defining four tasks to evaluate VLLMs' cross-temporal visual perception.

  5. OZ-TAL: Online Zero-Shot Temporal Action Localization

    cs.CV 2026-05 unverdicted novelty 7.0

    Defines OZ-TAL task and presents a training-free VLM-based method that outperforms prior approaches for online and offline zero-shot temporal action localization on THUMOS14 and ActivityNet-1.3.

  6. CATS: Curvature Aware Temporal Selection for efficient long video understanding

    cs.CV 2026-05 unverdicted novelty 7.0

    CATS uses temporal curvature of query-frame relevance to select informative frames, achieving 93-95% of heavy multi-stage accuracy at 3-4% of the preprocessing cost on long-video benchmarks.

  7. UniShield: Unified Face Attack Detection via KG-Informed Multimodal Reasoning

    cs.CV 2026-05 unverdicted novelty 7.0

    UniShield introduces a knowledge-graph-informed multimodal framework that improves unified detection of physical and digital face attacks through instruction tuning and consistency-optimized reasoning.

  8. PolarVLM: Bridging the Semantic-Physical Gap in Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 7.0

    PolarVLM is the first VLM framework to integrate polarimetric physical parameters via dual-stream architecture and progressive training, delivering 25.4% gains over RGB baselines on reflection and transparency tasks w...

  9. PolarVLM: Bridging the Semantic-Physical Gap in Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 7.0

    PolarVLM integrates polarimetric physical parameters into VLMs via dual-stream architecture and progressive training, outperforming RGB baselines by 25.4% on a new 75K-pair polarization-aware VQA benchmark.

  10. Masks Can Talk: Extracting Structured Text Information from Single-Modal Images for Remote Sensing Change Detection

    cs.CV 2026-05 unverdicted novelty 7.0

    S2M extracts structured text quadruples from change masks to provide noise-free multimodal supervision, achieving 17.80% Sek and 66.14% F_scd on the new Gaza-Change-v2 dataset and outperforming LLM-based multimodal methods.

  11. ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Models

    cs.AI 2026-05 unverdicted novelty 7.0

    ICU-Bench is a new continual unlearning benchmark for MLLMs using 1000 privacy profiles, 9500 images, and 100 forget tasks, showing existing methods fail to balance forgetting, utility, and scalability.

  12. VoxAfford: Multi-Scale Voxel-Token Fusion for Open-Vocabulary 3D Affordance Detection

    cs.CV 2026-05 unverdicted novelty 7.0

    VoxAfford fuses multi-scale voxel features into MLLM output tokens using cross-attention with a learned compatibility gate to achieve SOTA open-vocabulary 3D affordance detection with ~8% mIoU gain and zero-shot robot...

  13. LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models

    cs.CV 2026-04 unverdicted novelty 7.0

    LearnPruner prunes vision tokens to 5.5% of the original count while retaining about 95% of VLM performance and delivering 3.2 times faster inference by fixing attention sink in encoders and using unbiased middle-laye...

  14. ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety

    cs.CR 2026-04 unverdicted novelty 7.0

    ProjLens shows that backdoor parameters in MLLMs are encoded in low-rank subspaces of the projector and that embeddings shift toward the target direction with magnitude linear in input norm, activating only on poisone...

  15. AnchorSeg: Language Grounded Query Banks for Reasoning Segmentation

    cs.CV 2026-04 unverdicted novelty 7.0

    AnchorSeg uses ordered query banks of latent reasoning tokens plus a spatial anchor token and a Token-Mask Cycle Consistency loss to achieve 67.7% gIoU and 68.1% cIoU on the ReasonSeg benchmark.

  16. Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contexts

    cs.CL 2026-04 unverdicted novelty 7.0

    Introduces culture-aware humorous captioning task and staged alignment framework that improves contextual fit and balances image relevance with humor in multimodal LLMs.

  17. SLQ: Bridging Modalities via Shared Latent Queries for Retrieval with Frozen MLLMs

    cs.CV 2026-04 unverdicted novelty 7.0

    SLQ turns frozen MLLMs into retrievers via shared latent queries appended to inputs, outperforming fine-tuning on COCO and Flickr30K while introducing KARR-Bench for knowledge-aware evaluation.

  18. Semantic-Geometric Dual Compression: Training-Free Visual Token Reduction for Ultra-High-Resolution Remote Sensing Understanding

    cs.CV 2026-04 unverdicted novelty 7.0

    DualComp uses a lightweight router to split visual token compression into a semantic stream with size-adaptive clustering and a geometric stream with path-tracing recovery, enabling low-cost high-fidelity UHR remote s...

  19. Skill-Conditioned Visual Geolocation for Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 7.0

    GeoSkill uses an evolving Skill-Graph initialized from expert trajectories and grown via autonomous analysis of successful and failed reasoning rollouts to boost geolocation accuracy, faithfulness, and generalization ...

  20. SVAgent: Storyline-Guided Long Video Understanding via Cross-Modal Multi-Agent Collaboration

    cs.CV 2026-04 unverdicted novelty 7.0

    SVAgent improves long video question answering by constructing storylines via multi-agent collaboration and aligning cross-modal predictions for more robust, human-like reasoning.

  21. Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models

    cs.LG 2026-04 unverdicted novelty 7.0

    RL post-training on hallucination-forced multimodal data improves reasoning performance and can outperform standard training.

  22. 3D-VLA: A 3D Vision-Language-Action Generative World Model

    cs.CV 2024-03 unverdicted novelty 7.0

    3D-VLA is a new embodied foundation model that uses a 3D LLM plus aligned diffusion models to generate future images and point clouds for improved reasoning and action planning in 3D environments.

  23. Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

    cs.CV 2023-10 accept novelty 7.0

    Set-of-Mark prompting marks segmented image regions with alphanumerics and masks to let GPT-4V achieve state-of-the-art zero-shot results on referring expression comprehension and segmentation benchmarks like RefCOCOg.

  24. SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension

    cs.CL 2023-07 unverdicted novelty 7.0

    SEED-Bench is a new benchmark of 19K multiple-choice questions for evaluating generative comprehension in multimodal LLMs across 12 image and video dimensions.

  25. Evaluating Object Hallucination in Large Vision-Language Models

    cs.CV 2023-05 accept novelty 7.0

    Large vision-language models exhibit severe object hallucination that varies with training instructions, and the proposed POPE polling method evaluates it more stably and flexibly than prior approaches.

  26. VideoChat: Chat-Centric Video Understanding

    cs.CV 2023-05 conditional novelty 7.0

    VideoChat integrates video models and LLMs via a learnable interface for chat-based spatiotemporal and causal video reasoning, trained on a new video-centric instruction dataset.

  27. WizardLM: Empowering large pre-trained language models to follow complex instructions

    cs.CL 2023-04 conditional novelty 7.0

    WizardLM uses LLM-driven iterative rewriting to generate complex instruction data and fine-tunes LLaMA to reach over 90% of ChatGPT capacity on 17 of 29 evaluated skills.

  28. GRIP-VLM: Group-Relative Importance Pruning for Efficient Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 6.0

    GRIP-VLM applies group-relative policy optimization via reinforcement learning to prune visual tokens in VLMs, yielding up to 15% inference speedup at matched accuracy over prior methods.

  29. Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination

    cs.MM 2026-05 unverdicted novelty 6.0

    LVLMs show vocabulary hijacking by inert tokens that decode to hijacking anchors; HABI locates them, NHAR finds resilient heads, and HAVAE boosts those heads to cut hallucinations.

  30. Guaranteed Jailbreaking Defense via Disrupt-and-Rectify Smoothing

    cs.CR 2026-05 unverdicted novelty 6.0

    DR-Smoothing introduces a disrupt-then-rectify prompt processing scheme into smoothing defenses, delivering tight theoretical bounds on success probability against both token- and prompt-level jailbreaks.

  31. LLaVA-UHD v4: What Makes Efficient Visual Encoding in MLLMs?

    cs.CV 2026-05 unverdicted novelty 6.0

    LLaVA-UHD v4 reduces visual-encoding FLOPs by 55.8% for high-resolution images in MLLMs via slice-based encoding plus intra-ViT early compression while matching or exceeding baseline performance on document, OCR, and ...

  32. A Multimodal Dataset for Visually Grounded Ambiguity in Machine Translation

    cs.CL 2026-05 unverdicted novelty 6.0

    VIDA provides 2,500 visually-dependent ambiguous MT instances and LLM-judge metrics; chain-of-thought SFT improves disambiguation accuracy over standard SFT, especially out-of-distribution.

  33. Chain of Evidence: Pixel-Level Visual Attribution for Iterative Retrieval-Augmented Generation

    cs.CV 2026-05 unverdicted novelty 6.0

    CoE applies vision-language models directly to document screenshots to deliver pixel-level bounding-box attribution for evidence in iterative retrieval-augmented generation, outperforming text baselines on visual-layo...

  34. Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs

    cs.CV 2026-05 unverdicted novelty 6.0

    PVM adds a parallel branch to LVLMs that directly supplies visual embeddings to prevent attention decay over long generated sequences, yielding accuracy gains on reasoning tasks with minimal overhead.

  35. Online Self-Calibration Against Hallucination in Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 6.0

    OSCAR exploits the generative-discriminative gap in LVLMs to build online preference data with MCTS and dual-granularity rewards for DPO-based calibration, claiming SOTA hallucination reduction and improved multimodal...

  36. State Beyond Appearance: Diagnosing and Improving State Consistency in Dial-Based Measurement Reading

    cs.CV 2026-04 unverdicted novelty 6.0

    MLLMs ignore dial state geometry and cluster by appearance, causing inconsistency under variations; TriSCA's state-distance alignment, metadata supervision, and objective alignment improve robustness on clock and gaug...

  37. See Further, Think Deeper: Advancing VLM's Reasoning Ability with Low-level Visual Cues and Reflection

    cs.CV 2026-04 unverdicted novelty 6.0

    ForeSight lets VLMs use low-level visual cues and mask-based visual feedback within an RL loop to reason more accurately, with the 7B model beating same-scale peers and some closed-source SOTA on a new benchmark.

  38. GazeVLA: Learning Human Intention for Robotic Manipulation

    cs.RO 2026-04 unverdicted novelty 6.0

    GazeVLA pretrains on large human egocentric datasets to capture gaze-based intention, then finetunes on limited robot data with chain-of-thought reasoning to achieve better robotic manipulation performance than baselines.

  39. ChangeQuery: Advancing Remote Sensing Change Analysis for Natural and Human-Induced Disasters from Visual Detection to Semantic Understanding

    cs.CV 2026-04 unverdicted novelty 6.0

    ChangeQuery is a new multimodal framework for semantic disaster change analysis that combines optical and SAR data with a custom dataset and annotation pipeline to support interactive damage assessment.

  40. Latent Denoising Improves Visual Alignment in Large Multimodal Models

    cs.CV 2026-04 unverdicted novelty 6.0

    A latent denoising objective with saliency-aware corruption and contrastive distillation improves visual alignment and corruption robustness in large multimodal models.

  41. V-tableR1: Process-Supervised Multimodal Table Reasoning with Critic-Guided Policy Optimization

    cs.AI 2026-04 unverdicted novelty 6.0

    V-tableR1 uses a critic VLM for dense step-level feedback and a new PGPO algorithm to shift multimodal table reasoning from pattern matching to verifiable logical steps, achieving SOTA accuracy with a 4B open-source model.

  42. SSL-R1: Self-Supervised Visual Reinforcement Post-Training for Multimodal Large Language Models

    cs.CV 2026-04 unverdicted novelty 6.0

    SSL-R1 reformulates visual SSL tasks into verifiable puzzles to supply rewards for RL post-training of MLLMs, yielding gains on multimodal benchmarks without external supervision.

  43. R-CoV: Region-Aware Chain-of-Verification for Alleviating Object Hallucinations in LVLMs

    cs.CV 2026-04 conditional novelty 6.0

    R-CoV is a six-step region-aware chain-of-verification technique that elicits coordinate and description outputs from LVLMs themselves to detect and reduce object hallucinations without external models or retraining.

  44. Weakly-Supervised Referring Video Object Segmentation through Text Supervision

    cs.CV 2026-04 unverdicted novelty 6.0

    WSRVOS enables referring video object segmentation with text-only supervision by combining MLLM-based expression augmentation, multimodal feature interaction, pseudo-mask fusion, and temporal ranking constraints.

  45. PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging

    cs.CV 2026-04 unverdicted novelty 6.0

    PivotMerge merges heterogeneous multimodal pre-trained models via shared-space decomposition to filter conflicts and layer-wise weights based on alignment contributions, outperforming baselines on multimodal benchmarks.

  46. G-MIXER: Geodesic Mixup-based Implicit Semantic Expansion and Explicit Semantic Re-ranking for Zero-Shot Composed Image Retrieval

    cs.CV 2026-04 unverdicted novelty 6.0

    G-MIXER achieves state-of-the-art zero-shot composed image retrieval by using geodesic mixup to build diverse implicit candidates and MLLM-derived explicit semantics for re-ranking.

  47. SLQ: Bridging Modalities via Shared Latent Queries for Retrieval with Frozen MLLMs

    cs.CV 2026-04 conditional novelty 6.0

    SLQ adapts frozen MLLMs for multimodal retrieval by appending shared latent queries to text and image tokens and introduces KARR-Bench to test knowledge-aware reasoning retrieval.

  48. UHR-BAT: Budget-Aware Token Compression Vision-Language model for Ultra-High-Resolution Remote Sensing

    cs.CV 2026-04 unverdicted novelty 6.0

    UHR-BAT is a budget-aware framework that uses text-guided multi-scale importance estimation plus region-wise preserve and merge strategies to compress visual tokens in ultra-high-resolution remote sensing vision-langu...

  49. Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models

    cs.CV 2026-04 unverdicted novelty 6.0

    CoM-PT trains vision foundation models in ascending size order using inverse knowledge transfer, allowing larger models to achieve superior performance with significantly reduced overall computational cost compared to...

  50. ReflectCAP: Detailed Image Captioning with Reflective Memory

    cs.AI 2026-04 unverdicted novelty 6.0

    ReflectCAP distills model-specific hallucination and oversight patterns into Structured Reflection Notes that steer LVLMs toward more factual and complete image captions, reaching the Pareto frontier on factuality-cov...

  51. Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs

    cs.LG 2026-04 unverdicted novelty 6.0

    DACO curates a 15,000-concept dictionary from 400K image-caption pairs and uses it to initialize an SAE that enables granular, concept-specific steering of MLLM activations, raising safety scores on MM-SafetyBench and...

  52. LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving

    cs.CV 2026-04 unverdicted novelty 6.0

    LMGenDrive unifies LLM-based multimodal understanding with generative world models to output both future driving videos and control signals for end-to-end closed-loop autonomous driving.

  53. Phantasia: Context-Adaptive Backdoors in Vision Language Models

    cs.CV 2026-04 unverdicted novelty 6.0

    Phantasia is a new backdoor attack on VLMs that dynamically aligns malicious outputs with input context to achieve higher stealth and state-of-the-art success rates compared to static-pattern attacks.

  54. SMART: When is it Actually Worth Expanding a Speculative Tree?

    cs.DC 2026-04 unverdicted novelty 6.0

    SMART uses marginal benefit-cost analysis to dynamically build efficient speculative trees, achieving 15-20% additional speedup in LLM and MLLM inference.

  55. Small Vision-Language Models are Smart Compressors for Long Video Understanding

    cs.CV 2026-04 unverdicted novelty 6.0

    Tempo uses a 6B SVLM as a local temporal compressor with training-free adaptive token allocation to achieve SOTA long-video understanding at 0.5-16 tokens per frame, scoring 52.3 on 4101s LVBench under 8K budget.

  56. CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning

    cs.CV 2026-04 unverdicted novelty 6.0

    CoME-VL fuses contrastive and self-supervised vision encoders via entropy-guided multi-layer aggregation and RoPE cross-attention to improve vision-language model performance on benchmarks.

  57. LLaVA-Video: Video Instruction Tuning With Synthetic Data

    cs.CV 2024-10 unverdicted novelty 6.0

    LLaVA-Video-178K is a new synthetic video instruction dataset that, when combined with existing data to train LLaVA-Video, produces strong results on video understanding benchmarks.

  58. Emu3: Next-Token Prediction is All You Need

    cs.CV 2024-09 unverdicted novelty 6.0

    Emu3 shows that next-token prediction on a unified discrete token space for text, images, and video lets a single transformer outperform task-specific models such as SDXL and LLaVA-1.6 in multimodal generation and perception.

  59. Are We on the Right Way for Evaluating Large Vision-Language Models?

    cs.CV 2024-03 conditional novelty 6.0

    Current LVLM benchmarks overestimate capabilities because many questions can be answered without images due to design flaws or data leakage; MMStar is a human-curated set of 1,500 vision-indispensable samples across 6...

  60. DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models

    cs.CV 2024-02 unverdicted novelty 6.0

    DriveVLM adds vision-language models with scene description, analysis, and hierarchical planning modules to autonomous driving, paired with a hybrid DriveVLM-Dual system tested on nuScenes and SUP-AD datasets and depl...

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages · cited by 90 Pith papers · 11 internal anchors

  1. [1]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901,

    work page 1901

  2. [2]

    Video chatcaptioner: Towards the enriched spatiotemporal descriptions

    Jun Chen, Deyao Zhu, Kilichbek Haydarov, Xiang Li, and Mohamed Elhoseiny. Video chatcaptioner: Towards the enriched spatiotemporal descriptions. arXiv preprint arXiv:2304.04227,

    work page arXiv

  3. [3]

    PaLM: Scaling Language Modeling with Pathways

    URL https: //vicuna.lmsys.org. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311,

    work page internal anchor Pith review arXiv

  4. [4]

    Scaling Instruction-Finetuned Language Models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416,

    work page internal anchor Pith review arXiv

  5. [5]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    PaLM-E: An Embodied Multimodal Language Model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378,

    work page internal anchor Pith review arXiv

  7. [7]

    Eva: Exploring the limits of masked visual represen- tation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636,

    work page arXiv

  8. [8]

    Training Compute-Optimal Large Language Models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Language is not all you need: Aligning perception with language models

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045,

    work page arXiv

  11. [11]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pp. 787–798,

    work page 2014

  12. [12]

    BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597,

    work page internal anchor Pith review arXiv

  13. [13]

    Connecting vision and language with localized narratives

    Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16 , pp. 647–664. Springer,

    work page 2020

  14. [14]

    Object Hallucination in Image Captioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156,

    work page Pith review arXiv

  15. [15]

    BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´c, Daniel Hesslow, Roman Castagn´e, Alexandra Sasha Luccioni, Fran c ¸ois Yvon, Matthias Gall ´e, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022a. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´c,...

    work page internal anchor Pith review arXiv

  16. [16]

    Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model, 2022

    Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990,

    work page arXiv

  17. [17]

    arXiv preprint arXiv:2303.08128 , year=

    D´ıdac Sur´ıs, Sachit Menon, and Carl V ondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128,

    work page arXiv

  18. [18]

    Plug-and-play vqa: Zero- shot vqa by conjoining large pretrained models with zero training.arXiv preprint arXiv:2210.08773, 2022

    Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi. Plug-and- play vqa: Zero-shot vqa by conjoining large pretrained models with zero training. arXiv preprint arXiv:2210.08773,

    work page arXiv

  19. [19]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

    ISSN 2835-8856. URL https://openreview.net/forum?id=yzkSU5zdwD. Survey Certification. Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Vi- sual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671,

    work page internal anchor Pith review arXiv

  21. [21]

    Zero-shot video question answering via frozen bidirectional language models

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. arXiv preprint arXiv:2206.08155,

    work page arXiv

  22. [22]

    OPT: Open Pre-trained Transformer Language Models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068,

    work page internal anchor Pith review arXiv

  23. [23]

    Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions

    Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elho- seiny. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint arXiv:2303.06594,

    work page arXiv

  24. [24]

    mondayjustmonday,

    12 Preprint A A PPENDIX A.1 M ORE QUALITATIVE RESULTS Explain why this meme is funny. Thismemeisfunnybecauseitshowsadoglayingonthefloorwithitsheadrestingonitspaws,asifitistiredorsleepy.Thecaption,"mondayjustmonday,"addstothehumorbysuggestingthatthedogisfeelingthesamewayasmanypeopledoonMondays,whichareoftenconsideredtobethemostdreadeddayoftheweek.Thememeis...

    work page 1969

  25. [25]

    Additionally, to showcase the potential of MiniGPT-4 with traditional benchmarks, we conduct a straightforward ablation study

    and GQA (Hudson & Manning, 2019). Additionally, to showcase the potential of MiniGPT-4 with traditional benchmarks, we conduct a straightforward ablation study. Here, we simply unfreeze the LLM using LoRA (Hu et al.,

    work page 2019