{"id":"32f72264-1422-42c5-9df9-a357ba62f632","arxiv_id":"2412.13303","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A five-stage hybrid encoder that downsamples images 64x produces 4x fewer tokens than prior hybrid designs and 16x fewer than a ViT, giving about a 3x faster first-token time at similar VLM accuracy in Apple's benchmarks.","lead":"Apple's FastVLM pairs a new hybrid vision encoder, FastViTHD, with a language model to read high-resolution images using far fewer visual tokens and much less time to the first answer token. In the paper's on-device benchmarks it is roughly 3x faster than prior vision-language pipelines while keeping accuracy at a similar level.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Latency benchmark is Apple-specific and selectively reported; without neutral-platform validation, the claimed accuracy-latency frontier may not transfer.","rationale":"We agree with the reader that the weakest point is the on-device latency benchmarking. The paper is otherwise strong: the controlled ablations in Sec. 3.2.1 compare FastViTHD vs FastViT with matched parameter counts, report variance (<0.5) for the Avg-5 metric, and release checkpoints. The specific load-bearing assumption is that CoreML/MLX latencies on one M1 Max are a faithful proxy for the architectural efficiency of the compared encoders. That assumption is undermined by the disclosed selection rule (only 'format favorable to MLX' models are reported) and by the fact that the 85x headline uses a 7290-token dynamic-resolution model vs a 256-token static model. If a neutral-platform benchmark shows that SigLIP or ConvNeXt are not actually 2-3x slower when fairly converted (or that the excluded models would reverse the ordering), the Pareto-optimality claim loses its empirical basis. The accuracy numbers themselves are internally consistent and the released checkpoints would allow an independent check, so the concern is addressable and does not require a rejection. We recommend keeping the CONDITIONAL verdict until the latency measurements are reproduced on a standard accelerator or at least include the excluded models.","tokens_in":31304,"tokens_out":11062,"duration_ms":95141,"concrete_test":"Run the released FastVLM R4 checkpoint and the official SigLIP-SO400M, ConvNeXt-L/XXL, and ViT-L/14 baselines in the same LLaVA-1.5 instruction-tuned configuration on an NVIDIA A100 with FP16/TensorRT, measuring vision-encoder latency + LLM prefill for matched (resolution, token-count) pairs; also benchmark the models marked '-' in App. Tab. 10. If the FastViTHD speedup over SigLIP/ConvNeXT falls below ~1.5x, or the excluded models have latencies within 2x of their included peers, the claimed frontier is an Apple-specific artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central efficiency claim rests on TTFT measured only on one M1 Max MacBook Pro, with vision encoders converted to CoreML and LLMs to MLX FP16. The paper states it reports latency only for models that are 'publicly available and in a format favorable to MLX' (Sec. 4), and App. Tab. 10 marks several baselines (e.g., MM1/ViT-H) as '-' due to export difficulty. This is a selection bias: models with poor CoreML/MLX conversion are excluded from the latency comparison, so the reported 3.2x (vs SigLIP-SO400M) and 2.3x (vs ConvNeXT) speedups, and the Pareto curve in Fig. 4, are computed on a favorable subset. Additionally, the 85x headline vs LLaVA-OneVision mixes a 256-token static model with a 7,290-token dynamic-resolution model: LLM prefill is 11.4s vs 50ms, so the ratio is dominated by token count, not by the encoder. No neutral-platform (e.g., A100/TensorRT) validation is provided, so it is unknown whether the Apple-specific conversion artifacts or the architecture drive the gains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FastViTHD, a five-stage hybrid vision encoder with an overall 64x downsampling, and FastVLM, a vision-language model that uses it. The central claim is that FastViTHD provides a better accuracy-latency frontier than ViT, SigLIP, and ConvNeXt encoders in the LLaVA-1.5 setup, with reported speedups of 3.2x vs SigLIP-SO400M, 2.3x vs ConvNeXt, and 85x vs LLaVA-OneVision, while being smaller or comparable in accuracy. The paper also presents an analysis of the interplay between resolution, visual token count, LLM size, and TTFT, and argues that static high-resolution scaling outperforms tile-based dynamic resolution except at extreme resolutions. The authors release code and multiple checkpoints, and report variance over three training runs for key ablations.","tokens_in":31479,"tokens_out":4587,"duration_ms":43401,"significance":"If the efficiency claims hold, this is a useful contribution: it demonstrates that a hierarchical hybrid encoder with aggressive downsampling can substantially reduce visual tokens and encoding latency without sacrificing VLM accuracy, and it provides a systematic empirical mapping of the resolution-LLM-token trade-off. The released checkpoints and code make the accuracy claims checkable, and the variance reporting (Sec. D.3) is a strength. However, the efficiency claims rest entirely on measurements on one Apple M1 Max machine using CoreML and MLX conversions, with latency reported only for models that convert favorably to those frameworks. This limits the generality of the headline speedups and of the Pareto-frontier comparisons, and the 85x figure conflates token-count reduction with encoder efficiency.","major_comments":[{"comment":"The latency comparison is restricted to models that are 'publicly available and in a format favorable to MLX' (Sec. 4), and Table 10 marks several baselines (e.g., MM1, ViT-H) as '-' due to export difficulty. This excludes models that convert poorly and biases the Pareto frontier in Fig. 4 and the headline speedups in favor of FastViTHD. Without latency measurements on a neutral platform (e.g., A100 with TensorRT or a standard PyTorch benchmark), or at least a report of failed conversions and their latencies, the central accuracy-latency claim is not established beyond Apple-specific conversion artifacts.","section":"Sec. 4 (Benchmarking) and Table 10"},{"comment":"The 85x TTFT comparison between FastVLM (R4) and LLaVA-OneVision (R2) is dominated by LLM prefilling, not the vision encoder: Table 10 shows prefill of 11,402.4 ms vs 50.5 ms and encoder latency of 2,721.4 ms vs 116.3 ms. The prefill difference is due to 7,290 vs 256 visual tokens, and the claimed 85x ratio therefore conflates token-count reduction with encoder efficiency. The paper should report encoder-only latency separately, or compare at matched token budgets, to support the claim that the architecture, rather than token count, drives the speedup.","section":"Sec. 4.1, Table 6 (R2 vs R4)"},{"comment":"The claim that the Pareto-optimal curve for FastViTHD is 'significantly better' with 'an improvement of over 2.5 points on the Average-5 metric' is presented without the underlying per-configuration table. Since the Avg-5 metric is defined in this paper and the figure uses log-scale axes with overlapping points, the reader cannot verify the 2.5-point improvement quantitatively. Please provide the (resolution, LLM, Avg-5, TTFT) values for all points in Fig. 4, or a table, so the Pareto claim is checkable.","section":"Sec. 3.2.1, Fig. 4"}],"minor_comments":[{"comment":"The headline '85x faster TTFT' should be qualified as 'on Apple silicon' or 'in our benchmarking setup' to avoid over-generalization to other hardware.","section":"Abstract"},{"comment":"The definition of TTFT as vision encoder latency plus LLM prefill excludes the first decode step; clarify that this is a prefill-only metric rather than the standard time to first generated token.","section":"Sec. 4 (Benchmarking)"},{"comment":"The footnote about 'format favorable to MLX' is easy to miss; consider moving the selection-bias caveat to the main text of Sec. 4.","section":"Table 6 caption"},{"comment":"The marker for FastViT at resolution 2048^2 appears to be missing the '2048' label in the figure; please check.","section":"Fig. 4"},{"comment":"Rows R3 and R3* are not distinguished by an asterisk in the table; fix the formatting.","section":"Table 10"},{"comment":"The sentence 'FastVLM (R40) outperforms Cambrian-1 (R44) ... while being 7.9x faster' relies on TTFT values from Table 10 that are only available for some models; ensure the 7.9x ratio is computed consistently and note any missing entries.","section":"Sec. 4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is from Apple and the efficiency evaluation is entirely on Apple silicon with proprietary conversion tools. This is not a problem per se, but the authors should be encouraged to include at least one open-comparison benchmark or to reframe the claims as Apple-specific. Also note that the paper builds on the authors' own FastViT and MobileCLIP; the novelty is incremental but acceptable. The released checkpoints and code are a strong point that increases confidence in the accuracy claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The real contribution is FastViTHD, a five-stage hybrid encoder that downsamples 64x and emits 16x fewer tokens than ViT-L/14 at the same resolution. That token reduction is a genuine architectural idea, not a resampler bolted on, and the paper backs it with a systematic Pareto analysis across resolution and LLM size. The accuracy tables look internally consistent, variance is reported for the ablation set, and the released checkpoints make the numbers checkable. This is careful empirical engineering, and the comparisons against token-pruning methods are a useful addition.\n\nThe soft spot is the latency story. All TTFT numbers come from one M1 Max MacBook Pro, with encoders converted to CoreML and LLMs to MLX FP16. The paper discloses that it reports latency only for models in a format favorable to MLX, so baselines that convert poorly are absent from the latency comparison. That's a selection bias, and the 3.2x and 2.3x speedups could shrink or invert on a neutral platform. The 85x claim against LLaVA-OneVision is also partly a token-count artifact: 256 tokens versus 7,290, where long-context prefill dominates the LLM time. The token reduction is real, but the headline ratio conflates it with MLX's prefill behavior.\n\nNone of this is fatal. The accuracy numbers stand on their own, and the token reduction is measurable regardless of platform. The missing piece is neutral-platform validation or latency numbers for the excluded baselines. This is for anyone building small or on-device VLMs and choosing a vision backbone for high-resolution, text-rich tasks. I'd send it to peer review and ask for those additions. The citation pattern is fine: the paper builds on the authors' own published FastViT and MobileCLIP, which is appropriate given they are the prior state of the art here.","headline":"FastViTHD is a genuinely useful token-efficient encoder with a real Pareto story, but the headline latency ratios rest on Apple-only, selectively reported benchmarks.","tokens_in":32125,"tokens_out":2950,"would_cite":true,"duration_ms":25648,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single hybrid vision encoder, FastViTHD, that downsamples images 64x before self-attention produces far fewer visual tokens and encodes high-resolution images several times faster than ViT, SigLIP, and ConvNeXt encoders, while matching…","keywords":["vision language models","hybrid vision encoder","time-to-first-token","high-resolution image understanding","visual token efficiency","FastViTHD","LLM prefilling","on-device inference"],"falsifier":"Run the exact same controlled comparison (same LLaVA-1.5 training, same LLM) on a different hardware platform, e.g., an NVIDIA GPU with standard PyTorch or TensorRT, benchmarking all encoders including those excluded from the MLX comparison, and compare TTFT at matched token counts; if FastViTHD's time-to-first-token advantage falls below roughly 2x, the central claim fails.","tokens_in":31069,"feed_emoji":"⚡","tokens_out":3476,"duration_ms":32452,"temperature":0.7,"pith_summary":"FastVLM claims that the main bottleneck for high-resolution vision-language models is not just the vision encoder's own latency but the flood of visual tokens it sends to the language model. The paper introduces FastViTHD, a hybrid convolutional-transformer encoder with an extra downsampling stage that yields 4x fewer tokens than FastViT and 16x fewer than ViT-L/14 at the same input resolution. In controlled comparisons, FastViTHD achieves 3.2x faster time-to-first-token and 3.6x smaller size than SigLIP-SO400M, and 2.3x faster and 1.7x smaller than ConvNeXt, with equal or better benchmark accuracy. If correct, this means high-resolution, text-rich VLM inference could run on-device at a fraction of the current latency without sacrificing accuracy, and without needing token pruning heuristics.","feed_headline":"A 64x-downsampled encoder makes high-res VLMs 3x faster","feed_subtitle":"FastViTHD cuts visual tokens to a fraction of ViT's while matching accuracy on text-rich benchmarks like DocVQA and TextVQA.","key_machinery":"FastViTHD is a five-stage hybrid encoder: the first three stages use RepMixer convolutional blocks, and the last two stages use self-attention, with an added patch-embedding layer that downsamples the input by a total factor of 64. This makes self-attention operate on a small tensor (16x16 for a 1024x1024 image), cutting both encoder latency and the number of tokens passed to the LLM, which is what reduces LLM prefilling time and thus time-to-first-token. Multi-scale features pooled from earlier stages with depthwise convolutions add a small accuracy boost. The architecture is pretrained with MobileCLIP's reinforced image-text pipeline and then fine-tuned end-to-end in the LLaVA-1.5 two-stage recipe.","core_discovery":"The paper's central claim is that a hybrid vision backbone with a 64x downsampled final stage dominates isotropic ViTs and pure convolutional encoders on the accuracy-latency frontier for vision-language models. FastViTHD runs self-attention only on a heavily downsampled feature map, so it encodes a 1024x1024 image in 235 ms on an M1 MacBook Pro while producing only 256 visual tokens, versus ViT-L/14's 576 tokens from a 336x336 image. Across LLMs of 0.5B, 1.5B, and 7B parameters, the Pareto-optimal curve of FastViTHD is over 2.5 points better on the Average-5 metric than the best FastViT curve, and it reaches a target VLM performance up to 3x faster. The paper further shows that scaling input resolution directly beats tiling (AnyRes) except at extreme resolutions, and that a hierarchical encoder with few tokens beats token-pruning methods applied to ViTs.","pith_inferences":["The headline 85x TTFT improvement versus LLaVA-OneVision mixes several factors at once: it compares a 256-token static-resolution model to a 7,290-token dynamic-resolution configuration with a different LLM, so the speedup is not solely attributable to the vision encoder.","The benchmark stack (CoreML for encoders, MLX for LLMs, on Apple silicon) may favor hybrid convolutional encoders that convert cleanly; on other hardware with different conversion overheads, the relative gap to ViT and SigLIP could shrink.","The finding that a 64x-downsampled self-attention stage is enough for competitive VLM accuracy suggests a design rule that could extend to video or multi-image inputs, where token budgets explode even faster.","A direct test of the mechanism would be to pretrain FastViTHD at even higher downsampling (e.g., 128x) or to scale resolution beyond 1024 in the same setup; if accuracy plateaus while latency keeps falling, the current 64x choice is close to optimal."],"forward_implications":["High-resolution VLMs for text-rich images could run at roughly one-third the time-to-first-token of prior ViT-based models, making on-device document and chart understanding practical.","The need for token-pruning and resampling modules disappears for hierarchical encoders: simply training at lower input resolution yields token counts as low as 16 while outperforming prune-then-feed ViT methods.","Because the encoder produces far fewer tokens, smaller LLMs (e.g., 0.5B) can handle high-resolution inputs better, and the Pareto analysis shows which (resolution, LLM size) pair is optimal for a given latency budget.","Scaling visual instruction-tuning data further improves FastVLM, suggesting that the efficient token representation transfers to larger datasets and stronger benchmarks.","Dynamic-resolution tiling is largely unnecessary with FastViTHD: static resolution scaling is the better accuracy-latency trade-off except at extreme resolutions like 1536x1536."],"supporting_citations":[{"why":"Provides the LLaVA-1.5 two-stage training recipe and evaluation benchmarks that all controlled comparisons use.","marker":"[53]"},{"why":"Supplies the FastViT hybrid architecture, RepMixer blocks, and ConvFFN layers that FastViTHD builds on.","marker":"[82]"},{"why":"Provides the MobileCLIP pretraining pipeline and the DataCompDR dataset used to pretrain FastViTHD.","marker":"[83]"},{"why":"Defines the SigLIP-SO400M baseline that FastViTHD is compared against for size and latency.","marker":"[94]"},{"why":"Defines the ConvNeXt-based convolutional baseline (ConvLLaVA) that FastViTHD is compared against.","marker":"[28]"},{"why":"Defines the ViTamin hybrid transformer baseline that FastViTHD outperforms in size and retrieval performance.","marker":"[12]"},{"why":"Supplies the Qwen2 LLM family used in the Pareto-optimal accuracy-latency analysis and final comparisons.","marker":"[86]"},{"why":"Provides the MLX framework used to benchmark LLM prefilling latency and TTFT.","marker":"[31]"},{"why":"Defines the LLaVA-OneVision baseline at highest dynamic resolution that FastVLM compares against for the 85x TTFT claim.","marker":"[45]"},{"why":"Defines the Cambrian-1 multi-encoder baseline that FastVLM compares against using a single encoder.","marker":"[78]"}],"fun_headline_variants":["FastViTHD: 64x downsampling cuts VLM latency and tokens","FastViTHD: 256 tokens beat ViT's 576, 3.4x smaller, 85x faster","FastViTHD: Hierarchical encoder gives VLMs 3.2x faster TTFT","FastViTHD: Pareto-optimal VLM encoding with fewer tokens","FastViTHD: Scaling resolution beats tiling for high-res VLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed speedups are measured only on Apple-silicon devices using CoreML for encoders and MLX for the LLM, and only for models that convert cleanly to those formats, so the latency comparisons could be artifacts of the conversion stack rather than of the architectures themselves.","fun_headline_variants_meta":{"raw":{"variants":["FastViTHD: 64x downsampling cuts VLM latency and tokens","FastViTHD: 256 tokens beat ViT's 576, 3.4x smaller, 85x faster","FastViTHD: Hierarchical encoder gives VLMs 3.2x faster TTFT","FastViTHD: Pareto-optimal VLM encoding with fewer tokens","FastViTHD: Scaling resolution beats tiling for high-res VLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000389,"raw_usage":{"total_tokens":2125,"prompt_tokens":1093,"completion_tokens":1032,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":709,"completion_tokens_details":{"reasoning_tokens":915}},"tokens_in":709,"tokens_out":1032,"duration_ms":9097,"temperature":1.0,"reasoning_tokens":915,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:15:29.040951+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact same controlled comparison (same LLaVA-1.5 training, same LLM) on a different hardware platform, e.g., an NVIDIA GPU with standard PyTorch or TensorRT, benchmarking all encoders including those excluded from the MLX comparison, and compare TTFT at matched token counts; if FastViTHD's time-to-first-token advantage falls below roughly 2x, the central claim fails.","supporting_citations":[{"cited_title":"Improved baselines with visual instruction tuning, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the LLaVA-1.5 two-stage training recipe and evaluation benchmarks that all controlled comparisons use."},{"cited_title":"Fastvit: A fast hybrid vision transformer using structural reparameterization","cited_arxiv_id":null,"evidence_quote":"Supplies the FastViT hybrid architecture, RepMixer blocks, and ConvFFN layers that FastViTHD builds on."},{"cited_title":"Mobile- clip: Fast image-text models through multi-modal reinforced training","cited_arxiv_id":null,"evidence_quote":"Provides the MobileCLIP pretraining pipeline and the DataCompDR dataset used to pretrain FastViTHD."},{"cited_title":"Sigmoid loss for language image pre-training","cited_arxiv_id":null,"evidence_quote":"Defines the SigLIP-SO400M baseline that FastViTHD is compared against for size and latency."},{"cited_title":"Cambrian-1: A fully open, vision-centric exploration of multimodal llms,","cited_arxiv_id":null,"evidence_quote":"Defines the Cambrian-1 multi-encoder baseline that FastVLM compares against using a single encoder."}],"review_version":1}