{"id":"2c6a1c18-928d-413d-9930-33ee71457070","arxiv_id":"2411.14402","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AIMV2 pre-trains vision encoders by autoregressively predicting both image patches and text tokens, beating CLIP and SigLIP on many recognition and multimodal benchmarks.","lead":"This paper introduces AIMV2, a way to pre-train vision models by predicting both image patches and text captions during training. The resulting encoders match or beat popular contrastive models like CLIP and SigLIP on many vision and multimodal benchmarks, including 89.5% ImageNet top-1 accuracy with a frozen trunk.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline margins over SigLIP/CLIP are confounded with data: AIMV2's 12B mixture includes proprietary HQITP and ~4.2B synthetic captions, and the controlled same-data comparisons in Table 9b do not cover the deployed baselines; the method claim survives, but 'consistently outperforms SOTA' is not…","rationale":"I agree with the reader's weakest_assumption. The strongest evidence for the method is Table 9b, which controls for data and architecture and shows AIMV2 beating CLIP and CapPa, so I would not reject the paper. But the headline 'consistently outperforms SOTA contrastive models' is a cross-model claim, and those models are not trained on AIMV2's mixture. The composition difference is large: roughly 4.2B synthetic-caption samples and 565M HQITP pairs; both are the kind of data advantage that could explain gains. The reader's conditional verdict is exactly right; my independent test is to retrain a contrastive model on the same mixture. If data alone closes the gap, the contribution should be reframed as 'autoregressive objective with a better data pipeline' rather than as a method-level advantage over SigLIP/CLIP. The paper's own text notes the LiT zero-shot case where SigLIP is ahead, so the abstract's 'consistently' is too strong. No issues with internal math or architecture consistency were found; this is an attribution concern, not a correctness defect.","tokens_in":28002,"tokens_out":8893,"duration_ms":90612,"concrete_test":"Retrain a contrastive baseline (SigLIP or OAI CLIP) on the exact AIMV2 data recipe—same 12B-sample mixture from Table 2, including synthetic captions from the same captioner, same encoder size, optimizer, and batch size—and evaluate on Table 7 and Table 3. If the retrained baseline matches AIMV2 within 1 point on TextVQA, VQAv2, and IN-1k, the headline advantage is attributable to data, not the objective. As a feasible public-subset proxy, train both AIMV2 and CLIP on DFN-2B + COYO with synthetic captions at 12B scale; if the Table 9b margin shrinks materially, the same confound is present.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central method claim—that the multimodal autoregressive objective is a good recipe for vision encoders—is supported by the controlled ablation in Table 9b, where AIMV2, CLIP, and CapPa are trained with identical architecture and data, and AIMV2 wins on TextVQA by 11–13 points. The load-bearing gap is the paper's broader claim that AIMV2 'consistently outperforms state-of-the-art contrastive models (e.g., CLIP, SigLIP)' in multimodal understanding and recognition. Those headline comparisons (Tables 3 and 7) use AIMV2's full 12B mixture, which is 35% synthetic captions and roughly 8% private HQITP pairs (Table 2), while the SigLIP/CLIP checkpoints were trained on different private corpora. A contrastive model trained on the same caption quality could plausibly close most of the gap, since synthetic captions and HQITP are exactly the kind of data improvements that have historically moved contrastive models. The abstract also ignores the zero-shot LiT result in Table 5, where SigLIP beats AIMV2 by 3.4 points. These issues do not falsify the method, but they mean the 'consistently outperforms SOTA' claim is not fully supported by the current experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AIMV2, a family of ViT-based vision encoders pre-trained with a multimodal autoregressive objective. The vision encoder uses prefix attention, and a causal multimodal decoder predicts both raw image patches (ℓ2 loss) and caption tokens (cross-entropy) on a 12B image-text mixture (DFN, COYO, proprietary HQITP, and synthetic captions). The authors evaluate the resulting encoders on frozen-trunk recognition, open-vocabulary detection and grounding, multimodal instruction tuning, in-context learning, zero-shot LiT, and native-resolution adaptation, reporting strong results including 89.5% ImageNet-1k top-1 with a frozen trunk. The central methodological claim is that joint autoregressive prediction of patches and text yields a simple, scalable, and generalist vision encoder that matches or outperforms contrastive pre-training.","tokens_in":28251,"tokens_out":3836,"duration_ms":38615,"significance":"If the central claim holds, this is a significant result: it demonstrates that a straightforward autoregressive multimodal objective can compete with contrastive objectives for vision-encoder pre-training, with the added benefits of dense supervision, modest batch sizes, and natural compatibility with LLM-based multimodal pipelines. The paper's strengths include a carefully controlled ablation in Table 9b comparing AIMV2 against CLIP and CapPa under identical architecture and data, a scaling analysis in Figure 2 that mirrors Hoffmann-style compute-optimal behavior, and a public code release. These elements make the core method claim credible. However, the broader advertised claim that AIMV2 'consistently outperforms state-of-the-art contrastive models' is only partially supported: the headline comparisons in Tables 3 and 7 mix method and data differences, and the paper's own results in Table 5 and Appendix D.3 show cases where baselines outperform AIMV2. The contribution is valuable but the presentation needs to separate method advantage from data advantage.","major_comments":[{"comment":"The headline gains over SigLIP and CLIP are confounded with pre-training data. AIMV2 is trained on a 12B mixture that includes 3.8B synthetic DFN captions, 431.5M synthetic HQITP captions, and 564.6M proprietary HQITP alt-text pairs (Table 2), while the SigLIP and CLIP checkpoints used in Tables 3 and 7 were trained on different private corpora. The controlled same-data comparison in Table 9b covers only CLIP and CapPa at 2B pairs, not the deployed SigLIP models. A contrastive model trained on the same caption quality could plausibly close a substantial portion of the reported margins, since data filtering and synthetic captions are known to improve contrastive models as well. This does not falsify the method claim, but it means the abstract's 'consistently outperforms state-of-the-art contrastive models' is not fully supported by the evidence as presented. I recommend either adding a same-data SigLIP-style baseline or explicitly qualifying the claim as holding for AIMV2's data mixture.","section":"§2.3, Table 2 vs. Tables 3 and 7; §5, Table 9b"},{"comment":"The claim of 'consistently outperforms' is contradicted by results within the paper itself. Table 5 shows SigLIP ViT-So400m at 80.4 zero-shot ImageNet top-1 versus 77.0 for AIMV2-3B, and Table D3 shows DINOv2 outperforming AIMV2 on COCO detection/segmentation (55.5 vs. 54.0 AP). The text in Section 4.1 and Appendix D.2 acknowledges these cases, but the abstract and conclusion do not carry the same qualification. Please temper the claims to 'outperforms or matches' with explicit exceptions, or restrict the claim to the specific settings where the controlled evidence supports it.","section":"Abstract and Conclusion vs. Table 5 and Table D3"},{"comment":"Several design choices are recommended on the basis of differences that are likely within training noise. For example, Table 9c reports TextVQA 37.5 for α=0.4 versus 37.4 for α=0.2 and 0.6; Table 9e shows decoder width 512 at 35.9 versus 1536 at 36.9; and Table 9f shows depth 12 versus 16 at 37.5 versus 36.6. None of these experiments report multiple seeds or error bars. I am not asking for a full seed study, but the text should avoid presenting these differences as conclusive evidence for a particular hyperparameter or architecture choice, or the authors should add at least a few repeated runs for the key ablations.","section":"§5, Table 9c–9f"}],"minor_comments":[{"comment":"The term 'synthetic' captions is used without specifying the captioner model or its filtering procedure, despite citing Lai et al. [63]. A sentence describing the captioning pipeline and any quality filtering would help reproducibility.","section":"§2.3 and Table 2"},{"comment":"The in-context learning comparison reports only results for OAI CLIP and DFN-CLIP as quoted from McKinzie et al. [85], without the MM1 ViT-L baseline under identical pre-training data. At minimum, clarify whether the ICL comparison holds the instruction-tuning data fixed.","section":"§4.3.2, Table 8"},{"comment":"There are numerous typos and grammatical slips: 'factorizatized' (§2.1), 'task' for 'tasks' (§2.4), 'hyperaparmeters' in Tables A1, A2, C1, and 'the model’s predicted patch ˆxi(θ)' with mismatched parentheses in §2.1. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The caption reads 'AIMV2 vs. CLIP..' but the table also includes CapPa; the caption should list all three methods. Also, the CapPa row is trained at batch size 8k only, while CLIP is given at 8k and 16k; note in the text why CapPa was not run at 16k.","section":"§5, Table 9b caption"},{"comment":"The comparison of AIMV2-3B at 448px against baselines at 224px is apples-to-oranges. The table caption notes the resolution, but the text should explicitly state that the 89.5% result uses a higher-resolution fine-tuned model, not the base pre-training resolution.","section":"§4.1, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong empirical contribution, and the controlled ablation in Table 9b is exactly the kind of evidence that should anchor the method claim. The main concern is scope of the advertised claim: the abstract and conclusion overstate what the current experiments show, because the headline comparisons are data-confounded and the paper itself contains counterexamples. I would like to see the authors either add a same-data SigLIP-style baseline (even at reduced scale) or carefully reword the claims to separate method from data. The self-citation pattern is notable but not inappropriate given the direct lineage from AIM; the overlapping data-filtering and synthetic-caption infrastructure with DFN and Lai et al. should be acknowledged explicitly when interpreting results. The single-run ablations are a secondary concern; they do not threaten the central method claim but they do limit the strength of the design-choice recommendations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core here is genuinely new and the paper earns its keep. Combining pixel-level autoregression with captioning in a single joint decoder is a natural extension of the authors' own AIM work and CapPa, and the controlled ablation in Table 9b is exactly what you want to see: same architecture, same data, and AIMv2 beats CLIP and CapPa by a wide margin on TextVQA (37.5 vs. 24.6/30.6) while matching on IN-1k. The pixel objective also adds a small but consistent gain over captioning-only across model and data scales (Figure 4). That is a real result, and the paper is honest enough to show the captioning-only baseline losing steam when scaled, which makes the pixel signal look like more than a fixed scalar. The broad evaluation is also a plus: recognition, detection, grounding, instruction tuning, in-context learning—not just one or two benchmarks. The ICL numbers are copied from MM1, which is fine for a like-for-like comparison, and the appendix gives enough hyperparameter detail to reimplement the training recipe if you have the compute and the data. Now the soft spots, and they are mostly about the packaging rather than the method. The headline claim that AIMv2 consistently outperforms state-of-the-art contrastive models does not survive contact with the data. The cross-model tables (3, 7) compare AIMv2 trained on 12B pairs that include the proprietary HQITP set and roughly 35% synthetic captions against SigLIP/CLIP checkpoints trained on different private corpora. Same-data ablations in Table 9b only cover CLIP and CapPa, not SigLIP. So the abstract's consistent is too strong, especially because the paper itself reports that SigLIP beats AIMv2 by 3.4 points on zero-shot IN-1k after LiT (Table 5). That is not fatal, but it should be in the abstract or at least in the conclusions. The other issues are mechanical: no error bars or seeds (common for this scale, but it does mean the small margins on CIFAR-10 and PCAM are within noise), and the linked code is just the AIM repo with no commit hash or pretrained weights released at the time of writing. None of this undercuts the main finding that the objective works. The paper is a serious, empirical contribution, not a solving of an open theoretical problem, and it should get a proper peer review rather than a desk reject. I would not cite it for the consistent outperformance claim, but I would cite it for the objective and the controlled ablation.","headline":"AIMv2 is a clean, well-ablated empirical paper: the method—adding captioning to autoregressive image modeling—clearly works, but the headline claims overstate the evidence because the strongest comparisons are confounded with data.","tokens_in":809,"tokens_out":928,"would_cite":true,"duration_ms":20701,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Autoregressive prediction of both image patches and text produces generalist vision encoders that outperform contrastive models such as CLIP and SigLIP.","keywords":["vision encoder pre-training","autoregressive modeling","multimodal learning","image-text pre-training","vision transformer","generative pre-training","representation learning","scaling laws"],"falsifier":"Retrain a CLIP or SigLIP model on exactly the AIMV2 12B mixture, including HQITP and the synthetic captions, with matched compute, and compare frozen-trunk and instruction-tuned benchmarks; if the margins vanish or reverse, the claim that multimodal autoregressive pre-training causes the gains is falsified. A second, cleaner test is to remove the proprietary HQITP and synthetic caption subsets from AIMV2's training mix and check whether its advantage persists.","tokens_in":27771,"feed_emoji":"🖼️","tokens_out":6374,"duration_ms":54656,"temperature":0.7,"pith_summary":"This paper tries to show that generative autoregressive pre-training, applied jointly to image patches and text tokens, can produce general-purpose vision encoders that rival and often beat contrastive models such as CLIP and SigLIP. The proposed AIMV2 family couples a prefix-attention vision transformer with a causal multimodal decoder that predicts the next image patch and the next text token, drawing a training signal from every input. The authors argue that this simple recipe scales like large language models, is easy to implement without huge batch sizes, and yields strong frozen-trunk performance across recognition, detection, grounding, and multimodal question answering. If true, it would offer a simpler alternative to contrastive pre-training for building generalist vision backbones.","feed_headline":"Predicting image patches and text beats CLIP-style pretraining","feed_subtitle":"AIMV2's frozen encoder hits 89.5% ImageNet-1k top-1 and wins most multimodal QA benchmarks.","key_machinery":"The load-bearing mechanism is the unified autoregressive pre-training objective over a concatenated sequence of image patches and text tokens. A vision transformer encodes patches under a randomly sampled prefix attention mask, and a causal multimodal decoder predicts the shifted sequence: image patches are regressed with a normalized $\\ell^2$ pixel loss (following He et al. [48]) and text tokens with cross-entropy, combined as $L = L_{\\text{text}} + \\alpha\\,L_{\\text{img}}$ with $\\alpha \\approx 0.4$. The prefix attention lets the encoder later switch to bidirectional attention without additional tuning, while the decoder provides dense supervision from every patch and token.","core_discovery":"The paper's central claim is that multimodal autoregressive modeling—factorizing the joint sequence of image patches and caption text as $P(S) = \\prod_j P(S_j \\mid S_{<j})$ and training with a pixel MSE loss plus a text cross-entropy loss—is an effective objective for pre-training large vision encoders. With this objective, AIMV2-3B reaches 89.5% ImageNet-1k top-1 accuracy under attentive probing with a frozen trunk, and AIMV2 encoders outperform CLIP, SigLIP, and DINOv2 on most multimodal instruction-tuning benchmarks while remaining competitive on recognition, detection, and referring-expression comprehension. The paper further claims that the image-level objective adds signal beyond captioning alone, that the method scales consistently with data and parameters, and that it achieves these results while seeing fewer training samples than the contrastive baselines.","pith_inferences":["What the paper leaves open is whether the margin over CLIP and SigLIP is an objective effect or a data effect; the inference that the objective alone drives the gains would be confirmed by swapping in matched data for the baselines.","The dense patch-level supervision suggests an untested prediction: AIMV2 should degrade less than captioning-only models on tasks needing fine spatial detail, such as small-object detection, which the paper's detection results roughly support but do not isolate.","One could extend the recipe to video or audio by treating frame or spectrogram patches as additional sequence tokens in the same factorization; the paper does not report such experiments.","The prefix attention trick implies that the encoder is trained to produce useful representations from partial images, which may explain the robustness to cropping and tiling seen in the high-resolution evaluations."],"forward_implications":["If correct, generative multimodal autoregression is a viable drop-in pre-training objective for generalist vision encoders, reducing the need for the large batch sizes and careful data filtering that contrastive methods require.","AIMV2-3B's 89.5% ImageNet-1k accuracy with a frozen trunk implies that representation quality comparable to the best discriminative models can come from image-plus-text next-token prediction.","The reported scaling behavior (performance improves with model size and sample count, while the optimal size grows with compute) suggests the recipe will keep improving as models and data grow, in line with LLM-style scaling.","Consistent gains on text-rich benchmarks such as TextVQA, DocVQA, and ChartQA indicate that the multimodal objective is especially useful when downstream tasks require fine-grained reading and localization."],"supporting_citations":[{"why":"Supplies the autoregressive image-modeling framework and prefix attention that AIMV2 extends from vision-only to multimodal.","marker":"[33]"},{"why":"Provides the masked autoencoder patch-normalization approach reused for the ℓ2 pixel regression loss.","marker":"[48]"},{"why":"Contributes the DFN-2B dataset and the Data Filtering Network pipeline used in pre-training, plus the DFN-CLIP baseline.","marker":"[35]"},{"why":"Defines the contrastive CLIP objective and baseline model that AIMV2 is compared against throughout.","marker":"[94]"},{"why":"Provides the SigLIP baseline, the main contrastive rival, and the observation that zero weight decay is needed for high-resolution adaptation.","marker":"[133]"},{"why":"Defines the CapPa captioning-only baseline used in controlled ablations with matched data and architecture.","marker":"[118]"},{"why":"Gives the LLM scaling-law methodology used to analyze AIMV2's data and compute scaling.","marker":"[50]"},{"why":"Supplies the synthetic captioning approach used to generate part of AIMV2's 12B training pairs.","marker":"[63]"},{"why":"Supplies the DINOv2 self-supervised baseline compared on recognition, detection, and grounding.","marker":"[87]"}],"fun_headline_variants":["Autoregressive multimodal pretraining beats CLIP and SigLIP","Frozen AIMV2 encoder hits 89.5% ImageNet-1k, beats CLIP","Predicting patches and text beats contrastive pretraining","AIMV2: multimodal autoregression outperforms CLIP-style models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that AIMV2's advantage over CLIP and SigLIP comes from its training objective rather than from its particular 12B image-text mixture, which includes a proprietary high-quality set and synthetic captions, because the headline cross-model comparisons are not matched on data.","fun_headline_variants_meta":{"raw":{"variants":["Autoregressive multimodal pretraining beats CLIP and SigLIP","Frozen AIMV2 encoder hits 89.5% ImageNet-1k, beats CLIP","Predicting patches and text beats contrastive pretraining","AIMV2: multimodal autoregression outperforms CLIP-style models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001395,"raw_usage":{"total_tokens":5615,"prompt_tokens":892,"completion_tokens":4723,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":4651}},"tokens_in":508,"tokens_out":4723,"duration_ms":35082,"temperature":1.0,"reasoning_tokens":4651,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:12:58.824171+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain a CLIP or SigLIP model on exactly the AIMV2 12B mixture, including HQITP and the synthetic captions, with matched compute, and compare frozen-trunk and instruction-tuned benchmarks; if the margins vanish or reverse, the claim that multimodal autoregressive pre-training causes the gains is falsified. A second, cleaner test is to remove the proprietary HQITP and synthetic caption subsets from AIMV2's training mix and check whether its advantage persists.","supporting_citations":[{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Defines the contrastive CLIP objective and baseline model that AIMV2 is compared against throughout."},{"cited_title":"Sigmoid loss for language image pre-training","cited_arxiv_id":null,"evidence_quote":"Provides the SigLIP baseline, the main contrastive rival, and the observation that zero weight decay is needed for high-resolution adaptation."},{"cited_title":"Image captioners are scalable vision learners too","cited_arxiv_id":null,"evidence_quote":"Defines the CapPa captioning-only baseline used in controlled ablations with matched data and architecture."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DINOv2 self-supervised baseline compared on recognition, detection, and grounding."}],"review_version":1}