{"id":"58743d93-11e9-4522-ab75-853264b7da6b","arxiv_id":"2412.04424","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using Florence-2's multi-depth, multi-prompt visual features in an MLLM yields consistent benchmark gains over CLIP-based counterparts, especially on text-heavy tasks.","lead":"Florence-VL replaces the usual CLIP-style image encoder in multimodal language models with Florence-2, a generative vision model. It fuses visual features taken from different network depths and different task prompts, and reports gains on OCR, chart, and general visual question answering benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's 'matched training' comparison changes the pretraining recipe, not just the vision encoder, so the Florence-2 advantage is not cleanly isolated.","rationale":"The reader's weakest-assumption analysis focused on whether the alignment-loss metric in Section 4 is a reliable proxy for downstream MLLM quality. That is a valid concern, but it is secondary to a more direct threat to the central claim: Table 3, which is the paper's main controlled evidence for Florence-2's advantage, is not a clean ablation. Supplementary Section 9 explicitly says Florence-VL is trained with full-model end-to-end pretraining, while LLaVA-1.5 only tunes the projection layer. The paper says it 'maintains the same training data and backbone LLMs,' but training data and backbone are not the only variables; the optimization protocol differs. This confound means the benchmark improvements could stem from fine-tuning the vision encoder and LLM rather than from the generative nature of Florence-2 or from DBFusion. The Section 4 alignment-loss plots are intended to isolate the encoder, but the paper provides no evidence that the loss correlates with the Table 3 deltas. For example, Florence-2's alignment-loss advantage might reflect the ease of fitting a trainable projection to its features without implying better answer accuracy after full-model training. My recommendation keeps the reader's CONDITIONAL verdict: the paper's claims are plausible and the controlled LLaVA-1.5-style data comparison is commendable, but the missing recipe-matched control is a concrete gap that should be closed before attributing the gains to the vision encoder. I would not escalate to REJECT because the alignment-loss analysis and the open-sourced recipe provide substantial partial support; I would not accept outright because the central causal claim currently rests on a confounded comparison.","tokens_in":14406,"tokens_out":3419,"duration_ms":34867,"concrete_test":"Re-run the Table 3 comparison under a matched pretraining recipe: train LLaVA-1.5 with the full end-to-end pretraining protocol described in Supplementary Section 9 while keeping its CLIP encoder, and train Florence-VL with LLaVA-1.5's projection-only pretraining protocol while keeping Florence-2 frozen; evaluate both on the same benchmarks. If the Florence-VL advantage over LLaVA-1.5 largely disappears when training recipes are matched, the reported gains are due to full-model fine-tuning rather than the generative vision encoder. If the gaps persist under both recipe-matching directions, the encoder attribution is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6 presents Table 3 as a controlled comparison in which 'the key difference lies in the vision encoders used (CLIP for LLaVA vs. Florence-2 for our model), while we maintain the same training data and backbone LLMs.' However, Supplementary Section 9 states that during pretraining, 'unlike LLaVA 1.5 which only tunes the projection layer, we fine-tune the entire model, including the vision backbone Florence-2, projection layer, and language model.' This is a second, substantial difference: LLaVA-1.5 freezes the vision encoder and LLM during pretraining, while Florence-VL updates both. End-to-end tuning of the vision backbone and LLM is known to change MLLM performance independently of the encoder choice, so the benchmark gaps in Table 3 cannot be attributed solely to Florence-2 features or DBFusion. The paper's own alignment-loss analysis (Section 4) attempts to isolate encoder quality, but that metric is optimized with a trainable projection and is never validated against downstream benchmark deltas; Figures 4 and 5 therefore do not establish that lower alignment loss causes the Table 3 gains. The load-bearing condition for the central claim is that Table 3 isolates the encoder change, and that condition is not met as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Florence-VL replaces CLIP-style contrastive vision encoders with Florence-2, a generative vision foundation model, and proposes Depth-Breadth Fusion (DBFusion) to combine features extracted from different encoder depths and from three task prompts (detailed caption, OCR, dense region caption). The fused features are channel-concatenated and projected into a pretrained LLM (Phi-3.5 or Llama-3). The model is trained with an end-to-end pretraining stage followed by partial fine-tuning on open-source data, and is evaluated on 25 benchmarks covering general VQA, vision-centric tasks, knowledge-based tasks, and OCR/Chart tasks. The paper also introduces an alignment-loss measurement to compare vision encoders and provides ablations on the depth and breadth components.","tokens_in":14820,"tokens_out":5171,"duration_ms":52097,"significance":"If the main empirical claims were cleanly established, this paper would be a useful contribution: it shows that a single generative vision encoder with prompt-conditioned features can rival or surpass multi-encoder systems such as Cambrian-8B, and it proposes a simple fusion method that does not increase the visual token count. The paper is also commendable for open-sourcing models and the full training recipe, and for attempting a matched-data comparison in Table 3. However, the central controlled comparison is confounded by a difference in the training recipe, and the alignment-loss analysis that motivates the mechanism is not validated against downstream task performance. The breadth and depth ablations show differences that are within noise. With revisions that address these issues, the contribution could be significant for the MLLM community.","major_comments":[{"comment":"The caption of Table 3 claims 'The key difference between them lies in the vision encoders used (CLIP for LLaVA vs. Florence-2 for our model), while we maintain the same training data and backbone LLMs for both.' This is contradicted by Supplementary Section 9, which states that during pretraining 'unlike LLaVA 1.5 which only tunes the projection layer, we fine-tune the entire model, including the vision backbone Florence-2, projection layer, and language model.' Since the authors themselves note that tuning the entire model yields better performance than freezing, the Table 3 comparison confounds the choice of vision encoder with a change in the pretraining recipe. The benchmark gains in Table 3 therefore cannot be attributed solely to Florence-2 features or DBFusion as claimed.","section":"Section 6, Table 3 (vs. Supplementary Section 9)"},{"comment":"The alignment-loss metric is computed by optimizing a trainable projection P to minimize a cross-entropy loss on the LLaVA-1.5 pretraining captions, but the paper never validates that this metric predicts downstream MLLM quality. No experiment shows that the ordering of encoders by alignment loss matches their ordering after full fine-tuning on standard benchmarks. Consequently, Figures 4 and 5 demonstrate only that Florence-2 features can be brought closer to frozen LLM text embeddings on the caption training distribution; they do not establish that lower alignment loss causes the benchmark improvements in Table 3, nor that the depth and breadth features are responsible for those improvements.","section":"Section 4, Figures 4 and 5"},{"comment":"The ablation study for breadth features reports average differences of less than one point across the benchmark suite, and several individual benchmarks improve when a feature is removed (e.g., removing the grounding feature improves GQA, VizWiz, MME-C, and Seed-image relative to the full model). Without error bars or significance tests, the claim that 'all high-level features are essential for maintaining optimal performance' is not supported. The same concern applies to the depth ablation in Table 4, where the full model outperforms [V] on most benchmarks but the margins are modest and unquantified.","section":"Table 5 (and Table 4)"}],"minor_comments":[{"comment":"The paper consistently misspells 'breadth' as 'breath' (e.g., 'depth-breath fusion' in the title and Section 3.2), and contains typos such as 'LLama-3-8B-Instrcut', 'Cambrain', and 'Abalation' in Table 5 of the supplement.","section":"Throughout"},{"comment":"The definition of the alignment loss is unclear: the indicator matrix I(i,j)_n is not defined, the notation for the concatenated matrices F^v_n and F^t_n is inconsistent with the earlier per-pair notation f^v_n and f^t_n, and it is not stated whether the LLM is frozen when optimizing P. Please clarify the indexing and the optimization procedure.","section":"Section 4, alignment loss equation"},{"comment":"After average pooling, the feature matrices F^v_n and F^t_n are said to be in R^{N×d}, but the earlier definitions have vision tokens as R^{r_n×d'} and text tokens as R^{s_n×d}. The pooling and concatenation steps should be stated precisely so the dimensions are consistent.","section":"Section 4"},{"comment":"HallusionBench and MMStar are evaluated but not cited in the reference list; please add the corresponding benchmark papers.","section":"References"},{"comment":"The main results and the matched-data comparison are reported without error bars or significance tests, so the repeated use of 'significant' in the abstract and Section 5 is not statistically substantiated. Reporting variances or a small number of repeated runs would strengthen the claims.","section":"Tables 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The paper comes from Microsoft Research authors who are also co-authors of Florence-2. This relationship is not a problem per se, but the paper should explicitly disclose it and should be careful not to overclaim the independence of the encoder comparison. The central issue is fixable: a properly controlled comparison would either freeze the vision backbone and LLM during pretraining for both LLaVA-1.5 and Florence-VL, or add a variant of Florence-VL that only tunes the projection layer, so that the encoder choice is the only remaining difference."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's genuine contribution is showing that Florence-2, a generative vision model, can replace CLIP as the sole vision backbone in an MLLM and produce competitive or better results across 25 benchmarks, including OCR and chart tasks where text-heavy features matter. The DBFusion idea—concatenating features from different depths and three prompts (caption, OCR, dense region caption) along the channel dimension—is simple but effective, and the practical advantage of one encoder over multi-encoder systems like Cambrian is real. They also ship their model and training recipe, which is exactly what the subfield needs. The main soft spot is the claim in Section 6 that Table 3 'disentangles' the effects of architecture by using the same data and backbone as LLaVA-1.5. The supplementary material clearly states that during pretraining they fine-tune the entire model, whereas LLaVA-1.5 only trains the projection layer. So the benchmark gains in Table 3 could come from end-to-end tuning as much as from Florence-2 features. That means the \"key difference lies in the vision encoders\" sentence is simply wrong. The alignment-loss analysis in Section 4 tries to isolate encoder quality, but the metric is never shown to correlate with downstream performance, and the trainable projection weakens it as a pure encoder measure. The ablations (Tables 4 and 5) support the depth and breadth contributions, but the differences are small—often within 1–2 points—and there are no error bars or significance tests anywhere. Calling every feature 'essential' overstates what the data show. For the main benchmark table (Table 2), the comparison is against published numbers with different training data, so the 'significant improvements' language needs a statistical or at least variance-based qualifier. Still, the core idea is sensible and the empirical gains on OCR and chart tasks are consistent with the mechanism they propose. This is a practical advance, not a paradigm shift. It deserves a serious referee, but the authors should be pushed to fix the Table 3 framing, add uncertainty estimates, and tone down the 'essential' and 'significant' claims. I'd bring it to a reading group as an example of how generative encoders can work in MLLMs, but I'd also use it as a case study in how 'controlled' comparisons can quietly introduce a second variable.","headline":"Useful practical result: a single generative encoder (Florence-2) with multi-prompt, multi-depth features can beat CLIP and even multi-encoder systems, but the paper's 'controlled' comparison changes more than the encoder and should be read with that caveat.","tokens_in":740,"tokens_out":2192,"would_cite":true,"duration_ms":37010,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A generative vision encoder with prompt-conditioned depth-breadth fusion outperforms CLIP-based encoders for multimodal LLMs on 25 benchmarks.","keywords":["multimodal large language models","vision encoder","generative pretraining","Florence-2","depth-breadth fusion","vision-language alignment","OCR","instruction tuning"],"falsifier":"Compute the alignment loss for several encoders and also train each encoder in the full Florence-VL recipe on a fixed data budget; the paper's mechanism predicts that the ranking by alignment loss matches the ranking by final benchmark scores, so a single encoder with higher alignment loss that consistently outscores a lower-loss encoder would falsify the alignment-based explanation.","tokens_in":14230,"feed_emoji":"🖼️","tokens_out":8615,"duration_ms":69686,"temperature":0.7,"pith_summary":"The paper tries to show that multimodal large language models are better built on a generative vision encoder than on the contrastive CLIP encoders that are currently standard. It uses Florence-2, which can be prompted to extract different kinds of visual information from the same image, and fuses those prompt-conditioned features with a lower-level visual stream using a channel-concatenation step called Depth-Breadth Fusion. The resulting models beat established baselines across 25 benchmarks, and a same-data comparison with LLaVA 1.5 isolates the encoder as the source of the gain. If the claim holds, a single multi-task generative encoder can replace the current practice of stitching together several specialized encoders, simplifying MLLM design while improving OCR, chart, and fine-grained perception performance.","feed_headline":"Generative vision encoder beats CLIP in multimodal LLMs","feed_subtitle":"A single encoder with depth-breadth fusion outdoes multi-encoder designs on 25 benchmarks.","key_machinery":"The central mechanism is Depth-Breadth Fusion (DBFusion): Florence-2 is asked to encode the same image under three task prompts (detailed caption, OCR, and dense region caption), producing three high-level token sets, and the raw DaViT encoder features are kept as a low-level stream; all four are concatenated along the channel dimension and passed through an MLP projection into the LLM. Channel concatenation keeps the token count at 576 while enriching each token, and the alignment-loss experiments show this fused representation aligns to LLM text embeddings better than CLIP, SigLIP, DINOv2, or Stable Diffusion features.","core_discovery":"The paper claims that the standard choice of CLIP-style contrastive encoders is suboptimal for multimodal LLMs: a generative vision model that can be prompted to produce task-specific features yields representations that align better with the LLM's text space. Using Florence-2 as the single vision encoder, with DBFusion to combine low-level DaViT features and high-level caption, OCR, and grounding features, Florence-VL outperforms established MLLMs, including models that fuse multiple separate encoders, across 25 benchmarks spanning general VQA, perception, hallucination, OCR, charts, and knowledge tasks, while using only 576 visual tokens. Under matched training data (the LLaVA 1.5 recipe), replacing CLIP with Florence-2 and DBFusion consistently raises scores, with the largest gains on OCR and chart/document understanding.","pith_inferences":["The alignment-loss metric could be reused as a cheap, full-training-free screen for future vision encoders; if it is reliable, developers could rank candidate encoders in hours rather than weeks.","The experiments do not isolate whether Florence-2's advantage comes from its generative pretraining objective, its prompt-conditioned task diversity, or its DaViT architecture; a controlled study varying one factor at a time would separate these causes.","Since the features are selected by textual prompts, a routing layer that picks the OCR or grounding prompt on the fly based on the input image, or lets the LLM request a new visual feature mid-generation, is a natural extension the paper leaves open.","The OCR and chart gains suggest the same depth-breadth recipe could be pushed further by combining it with high-resolution tiling, which the paper notes is compatible."],"forward_implications":["Replacing a contrastive encoder with a generative, promptable encoder plus channel-concatenation fusion improves alignment and benchmark results under identical training data, so the encoder choice is a first-order factor in MLLM quality.","A single Florence-2 encoder can match or beat systems that mix multiple specialized encoders, saving the memory and latency of running several backbones.","The OCR-prompted feature is the main contributor to gains on text-heavy document and chart tasks; removing it degrades OCRBench, ChartQA, DocVQA, and InfoVQA.","Fusing the low-level DaViT feature with the high-level prompt-conditioned features is necessary; using only low-level or only high-level features raises alignment loss and lowers benchmark averages.","End-to-end pretraining of vision backbone, projector, and LLM, followed by finetuning of projector and LLM, is the recipe behind these results and is released in full."],"supporting_citations":[{"why":"Supplies Florence-2, the generative vision backbone whose prompt-conditioned, multi-depth features are the core of the architecture.","marker":"[45]"},{"why":"Provides the LLaVA 1.5 training data and architecture used for the matched-data comparison that isolates the encoder's contribution.","marker":"[26]"},{"why":"The CLIP contrastive encoder replaced by Florence-2; it is the baseline in the alignment-loss experiment and the default encoder in LLaVA-style models.","marker":"[35]"},{"why":"The Cambrian baseline, an advanced multi-encoder MLLM that Florence-VL outperforms with a single encoder; also a source of instruction-tuning data.","marker":"[41]"},{"why":"One of the vision encoders compared in the alignment-loss plot, representing diffusion-pretrained features.","marker":"[36]"},{"why":"One of the vision encoders compared in the alignment-loss plot, representing self-supervised features.","marker":"[34]"},{"why":"Cited as prior evidence that cross-modal alignment can be measured without full supervised fine-tuning; motivates the alignment-loss metric.","marker":"[15]"},{"why":"Cited alongside [15] for evaluating modality alignment; supports the choice of the alignment-loss objective.","marker":"[43]"}],"fun_headline_variants":["Florence-VL: generative vision encoder outdoes CLIP","Single generative encoder beats multi-encoder LLMs","Florence-VL: one encoder does all, beats CLIP and more","Depth-breadth fusion: the secret to better vision-language models","Generative encoder + DBFusion lifts MLLM scores on 25 benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Section 4 alignment loss, a cross-entropy score between projected image features and text features, actually predicts downstream MLLM quality; if that proxy is unreliable, the paper's explanation of why Florence-2 works best is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["Florence-VL: generative vision encoder outdoes CLIP","Single generative encoder beats multi-encoder LLMs","Florence-VL: one encoder does all, beats CLIP and more","Depth-breadth fusion: the secret to better vision-language models","Generative encoder + DBFusion lifts MLLM scores on 25 benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000765,"raw_usage":{"total_tokens":3420,"prompt_tokens":996,"completion_tokens":2424,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":2331}},"tokens_in":612,"tokens_out":2424,"duration_ms":19390,"temperature":1.0,"reasoning_tokens":2331,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:23:06.524902+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the alignment loss for several encoders and also train each encoder in the full Florence-VL recipe on a fixed data budget; the paper's mechanism predicts that the ranking by alignment loss matches the ranking by final benchmark scores, so a single encoder with higher alignment loss that consistently outscores a lower-loss encoder would falsify the alignment-based explanation.","supporting_citations":[{"cited_title":"Florence-2: Advancing a unified representation for a variety of vision tasks","cited_arxiv_id":null,"evidence_quote":"Supplies Florence-2, the generative vision backbone whose prompt-conditioned, multi-depth features are the core of the architecture."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides the LLaVA 1.5 training data and architecture used for the matched-data comparison that isolates the encoder's contribution."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"One of the vision encoders compared in the alignment-loss plot, representing diffusion-pretrained features."}],"review_version":1}