{"id":"b58b6eff-8b13-4159-a56d-37f77e678dbd","arxiv_id":"2504.14032","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A coordinate-based cross-attention transformer, trained with mask-refined and self-distilled pseudo-groundtruth, upsamples VFM features to full resolution and improves downstream task performance.","lead":"LoftUp is a new upsampler that turns the low-resolution feature maps of vision foundation models into sharp, full-resolution maps by letting each output pixel attend to the whole low-res feature grid and to the original image. Trained with SAM-mask guidance and self-distillation, it beats prior upsamplers on six dense prediction tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pseudo-GT foundation is under-specified and bootstrapped: Eq. (1) is self-cancelling as printed, and Stage 2 distills an EMA teacher initialized from the same model, so the high-resolution target is not independent; the transfer claim needs a targeted ablation.","rationale":"The reader's weakest assumption is exactly where I would put the stress: the pseudo-GT is the only task-agnostic supervision, and if it is not a reliable proxy for full-resolution features, the reported downstream gains are not evidence for the stated method. I keep the reader's CONDITIONAL verdict because the paper does provide some support (Tables 2 and 5, code release, cross-task consistency), but the condition should be made explicit: the authors need to show that the specific pseudo-GT construction, and not the SAM-mask boundary prior or the EMA bootstrap, is responsible for the gains. I do not elevate to REJECT because the empirical comparisons are systematic and the code is public, so the concern is checkable. I did not make the LiFT 14x issue the primary concern because it affects only the magnitude of the 'SoTA' comparison, whereas the pseudo-GT issue goes to the mechanism and generalizability of the central claim.","tokens_in":16470,"tokens_out":14179,"duration_ms":133783,"concrete_test":"Recover the actual Stage-1 loss from the released code; if it differs from the literal Eq. (1), correct the equation and run an ablation in which the mask-mean term is replaced by the identity (no mask smoothing), keeping all other training choices and downstream evaluations identical. If COCO/Cityscapes mIoU and DAVIS J&F do not drop materially without mask smoothing, the pseudo-GT is not the causal driver of the reported gains; if they drop, the mask boundary prior is the mechanism and the transfer claim still needs to be separated from true high-resolution feature recovery.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of improved downstream performance rests on the pseudo-GT pipeline, but that pipeline is not precisely specified and may be bootstrapped. In Eq. (1), the mask-refined pseudo-GT is printed as FMask-Bicubic[m] = α FBicubic[m] + (1−α) FBicubic[m]; both terms are identical, so α cancels and the 'mask refinement' has no effect. The surrounding text says the mean feature inside the mask is blended with the original features, so the equation is almost certainly missing a mask-mean symbol; as written, Stage 1's training target is undefined. More substantively, both stages construct the target from the same low-resolution VFM features: Stage 1 bicubic-upsamples F and smooths with SAM masks; Stage 2 uses an EMA teacher initialized from that Stage 1 model, so the 'ground truth' is a function of the student's own upsampler. The only external information is the class-agnostic mask boundary prior. Downstream tasks are segmentation, depth, normal, and interactive/object segmentation, all of which reward boundary sharpness, so the observed gains could reflect the mask prior rather than the recovery of true high-resolution features. If the pseudo-GT is not a valid surrogate, the claim that LoftUp 'captures fine-grained details' beyond previous methods is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LoftUp, a coordinate-based cross-attention transformer that upsamples vision foundation model (VFM) features to full input resolution. The training pipeline has two stages: Stage 1 constructs pseudo-groundtruth features by bicubic upsampling and then refining them with class-agnostic SAM masks; Stage 2 uses self-distillation, where an EMA teacher upsampler, initialized from the Stage 1 model, processes high-resolution crops and supervises the student. The authors evaluate LoftUp on semantic segmentation, depth and normal estimation, video object segmentation, open-vocabulary segmentation, and interactive segmentation using DINOv2, with additional CLIP and RADIO experiments in the supplement. They report consistent improvements over bilinear upsampling, FeatUp, and LiFT, and argue that the coordinate-based architecture supports arbitrary upsampling scales at low computational overhead. Code is released.","tokens_in":16786,"tokens_out":4392,"duration_ms":42771,"significance":"If the central claims hold, LoftUp would be a practical, task-agnostic module for improving dense prediction from frozen VFMs, combining a cross-attention upsampler with a pseudo-GT training objective that avoids per-image optimization. The paper's strengths are its breadth of downstream evaluation, the architecture ablations in Table B.3, the inclusion of multiple VFM backbones in the supplement, and the release of code. The consistency of the reported gains across diverse tasks and backbones makes the core direction credible. However, the load-bearing pseudo-GT construction is under-specified as printed (Eq. (1) is self-cancelling), the Stage 2 target is bootstrapped from the student's own upsampler rather than an independent source of high-resolution feature evidence, and the LiFT comparison is conducted at an upsampling factor far outside LiFT's design range. These issues do not invalidate the approach, but they must be addressed before the paper's central claims can be considered established.","major_comments":[{"comment":"Equation (1) as printed defines FMask-Bicubic[m] as alpha * FBicubic[m] + (1-alpha) * FBicubic[m]; since both terms are identical, alpha cancels and the mask refinement has no effect. The surrounding text describes blending the mean feature inside each mask with the original features, so the equation is evidently missing a mean symbol or an analogous correction. This is a load-bearing issue because Eq. (1) defines the Stage 1 training target, which in turn initializes the Stage 2 teacher and student. Please replace the formula with the intended expression (e.g., alpha * mean(FBicubic[m]) + (1-alpha) * FBicubic) and update the text in Sec. 4.1 and the supplementary details accordingly.","section":"Sec. 4.1, Eq. (1)"},{"comment":"The Stage 2 target is not independent: the teacher is initialized from the Stage 1 student, updated as an EMA of the same student, and consumes the same frozen VFM, so the only external information in the entire pseudo-GT pipeline is the class-agnostic SAM mask prior. Consequently, the claim that LoftUp 'captures fine-grained details' beyond what mask-based boundary sharpening provides is not yet supported. I request targeted experiments: (i) compare against an oracle target obtained from genuinely high-resolution VFM features (e.g., features of a larger input image) on a holdout set; (ii) measure whether the learned upsampler improves similarity to such oracle features relative to bilinear and FeatUp; (iii) ablate Stage 2 by fixing the teacher after Stage 1 instead of EMA-updating it, and by turning the mask refinement on the teacher output on and off. Without such evidence, the observed downstream gains could be attributed to the mask prior and smoothing rather than to recovery of true high-frequency VFM features.","section":"Sec. 4.2, Stage 2 self-distillation"},{"comment":"LiFT is described in Sec. 4 as a method whose training objective uses 2x larger inputs, and previous work reports LiFT for 2x upsampling. Yet Tables 3 and 4 evaluate LiFT at 14x upsampling (and 8x for open-vocabulary segmentation). This is an unfair comparison when claiming 'previous SoTA' performance: a method trained for a 2x target cannot be expected to work well at 14x. Please either train LiFT for the target scale, evaluate it at its native 2x scale and discuss the mismatch, or clearly state that the comparison is out-of-distribution for LiFT. Additionally, Tables 3 and 4 report single runs without error bars or multiple seeds; given the small differences in some rows (e.g., depth and normal estimation in Table 3), please report mean and standard deviation over at least three seeds for the main tables.","section":"Sec. 5.1, Tables 3 and 4"}],"minor_comments":[{"comment":"The Introduction contains the stray token 'schannen2025siglip2' and references [54] and [55] are duplicated for SigLIP 2; the reference list and in-text citations should be cleaned up.","section":"Introduction and References"},{"comment":"The sentence describing the self-distillation loss states that the teacher output is downsampled to match crop(fstudent(I)) in R^{H/t x W/t}, which is dimensionally confusing because fstudent(I) is in R^{H x W}. Please define explicitly that the student output is spatially cropped to the region corresponding to the teacher's crop before the comparison.","section":"Sec. 4.2, notation"},{"comment":"The parameter counts in Table 7 are reported in a way that is hard to interpret: bilinear is listed with 22.1M parameters, and FeatUp-Implicit's inference time is written as 54.302 without an explicit unit. Please state whether the VFM backbone parameters are included and format the timing column consistently.","section":"Table 7"},{"comment":"There are typographical errors such as 'illsustrated' (Sec. 3), 'segementation' (Table 4 caption), and 'pseduo-GT' (supplement); a careful proofread is needed.","section":"Throughout"},{"comment":"The caption states 'We set alpha = 0.8 (in Eq. (1))', but Eq. (1) as printed has no effect of alpha; this cross-reference should be updated once Eq. (1) is corrected.","section":"Fig. 5 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper has a broad and seemingly consistent empirical story, and the code release is a plus. My main concern is that the two-stage pseudo-GT pipeline is the paper's central contribution, yet the printed equation for Stage 1 is degenerate and the Stage 2 target is bootstrapped from the student itself. If the authors can provide the oracle comparison and ablations requested in the major comments, and fix the LiFT evaluation protocol, I would be willing to support acceptance. The current version is not ready for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on LoftUp. The paper is a solid, useful advance. The new parts are the coordinate-based cross-attention upsampler and the two-stage pseudo-GT training with SAM masks plus self-distillation. The architecture is simple and the training objective is a clear improvement over FeatUp's per-image optimization and LiFT's 2x-only supervision. The experiments are broad: six downstream tasks, consistent wins over bilinear and FeatUp, and an efficiency table that shows it's practically free. The code release is real; I can run this.\n\nWhat needs work, in increasing order of seriousness. First, Eq. (1) as printed is self-cancelling: both terms are FBicubic[m], so alpha does nothing. The intended mean-feature blend is obvious from the text, but the typo should be fixed before publication. Second, the 'nearly 50%' VOS claim in the introduction is not supported by Table 4; the J&F mean is 60.25 vs. 44.30 for the best baseline, which is roughly 36% relative improvement. Maybe they're quoting F Mean, but as written it's an overstatement. Third, the LiFT comparison is unfair. LiFT's training objective is built around 2x upsampling; evaluating it at 14x seems to push it outside its design regime, and its catastrophic interactive-segmentation scores (29–41 IoU) suggest exactly that. The authors should either adapt LiFT to 14x or present it as a 2x comparison.\n\nThe biggest conceptual soft spot is the pseudo-GT bootstrap. Stage 1 builds the target from bicubic-upsampled features smoothed by SAM masks; Stage 2 uses an EMA teacher initialized from Stage 1. So the target is never independent of the upsampler's own output. That is self-distillation, not a circular proof — downstream benchmarks are external and never used in training — but it does mean the paper hasn't shown whether the gains come from recovering true high-res features or from injecting a boundary prior. Since semantic segmentation, depth, normals, and VOS all reward sharp boundaries, the SAM mask prior could explain a large chunk of the improvement. A simple ablation training the same LoftUp architecture on plain bicubic pseudo-GT (no masks, no self-distillation) would isolate this. That's a missing experiment, not a fatal flaw.\n\nOverall, the central claim is credible. The wins over bilinear and FeatUp are consistent and the architecture contribution is cleanly isolated in Table 5. I'd send this to peer review. A good referee will ask for the Eq. (1) fix, a fair LiFT comparison, error bars or multiple seeds, and the pseudo-GT ablation. If those land, this is a keep.","headline":"A solid, useful feature upsampler with a clear typo and an under-specified pseudo-GT; deserves review, needs a few targeted fixes.","tokens_in":17277,"tokens_out":4107,"would_cite":true,"duration_ms":37920,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A coordinate-based cross-attention upsampler trained with self-distilled pseudo-groundtruth recovers full-resolution features from vision foundation models, yielding 10–20% gains across dense prediction tasks and nearly 50% on video…","keywords":["feature upsampling","vision foundation models","DINOv2","coordinate-based transformer","cross-attention","self-distillation","pseudo-groundtruth","dense prediction"],"falsifier":"Run LoftUp on a benchmark where full-resolution features can be obtained as oracle ground truth, such as a vision model that natively outputs dense features at input resolution, and compare linear-probe segmentation or depth mIoU of the oracle features versus LoftUp's upsampled features under identical decoders. If LoftUp does not approach the oracle's performance, or if a LoftUp variant trained directly on oracle features substantially outperforms the pseudo-GT-trained version, then the pseudo-GT training objective is the bottleneck rather than the upsampler architecture.","tokens_in":16290,"feed_emoji":"🖼️","tokens_out":5437,"duration_ms":47352,"temperature":0.7,"pith_summary":"This paper argues that the coarse feature maps produced by vision foundation models can be upgraded to full-resolution, pixel-level features by a task-agnostic upsampler, and that two design choices matter most: the upsampler architecture and the training signal. The proposed upsampler, LoftUp, is a coordinate-based cross-attention transformer: each high-resolution pixel queries the low-resolution VFM feature map using its coordinates and RGB value, so details are synthesized with global semantic context rather than local kernels. Because no full-resolution feature ground truth exists, the paper constructs pseudo-groundtruth at image resolution by refining bicubic-upsampled features with class-agnostic masks and then self-distilling from a teacher upsampler that sees larger crops. Across semantic segmentation, depth and normal estimation, video object segmentation, and open-vocabulary and interactive segmentation, LoftUp reports consistent gains over prior upsamplers, with the largest on video object segmentation. If correct, this makes high-resolution dense prediction from frozen vision foundation models cheaper and more accurate than feeding larger images or training task-specific decoders.","feed_headline":"New upsampler lifts vision-model features, gains 10–20% on dense tasks","feed_subtitle":"A cross-attention transformer trained on self-distilled targets sharpens DINOv2 and CLIP features to full image resolution.","key_machinery":"The load-bearing object is the coordinate-based cross-attention transformer. High-resolution pixel coordinates are encoded with sinusoidal positional encodings, concatenated with RGB values, projected by a 3×3 convolution, and used as queries in cross-attention blocks whose keys and values are the low-resolution VFM features, replacing fixed or locally predicted kernels with global content-aware attention. The training machinery is a two-stage pseudo-groundtruth construction: Stage 1 blends bicubic-upsampled features with per-mask mean features computed from class-agnostic SAM masks and trains with an L2 loss; Stage 2 runs a teacher upsampler on larger image crops, refines those features with the same mask blending, and distills them into the student with an affinity-matrix loss, with the teacher updated by exponential moving average. Together these ingredients supply full-resolution supervision that previous upsamplers lacked.","core_discovery":"The paper's central claim is that the resolution bottleneck of vision foundation models is not fixed by the backbone but can be removed by a lightweight, task-agnostic upsampler, provided the upsampler uses global coordinate-based cross-attention and is trained against full-resolution pseudo-groundtruth features. Previous task-agnostic upsamplers either optimized per image at test time or supervised only at low resolution; LoftUp instead learns a direct coordinate-to-feature mapping and supervises it at image resolution, using SAM masks to enforce sharp boundaries and a self-distillation stage where a teacher upsampler processing high-resolution crops produces targets for the student. On the DINOv2-S/14 backbone with linear or lightweight decoders, the paper reports relative gains of 10–20% over the best previous upsamplers on most of six tasks, and nearly 50% on video object segmentation, along with consistent gains when the backbone is replaced by CLIP or RADIO.","pith_inferences":["Because the pseudo-groundtruth is constructed without task labels, the same Stage 1 plus Stage 2 recipe could plausibly be applied to other dense feature targets, such as distilling finer features from a stronger vision model or from multi-frame video, without changing the upsampler.","The cross-attention query design suggests a natural extension to non-image modalities with coordinate-like structure, such as point clouds or voxel grids, where low-resolution semantic tokens are queried by high-resolution geometric coordinates.","The large video-object-segmentation gain hints that much of the improvement comes from boundary sharpness rather than global semantics; a testable prediction is that LoftUp's advantage over baselines will correlate with the boundary density of the evaluation dataset.","If the 10–20% gains hold broadly, they imply that frozen VFM features encode far more spatial detail than their native resolution exposes, meaning the sparse token grid is a representational bottleneck that can be decoded with the right point-wise queries."],"forward_implications":["Task-agnostic feature upsampling becomes a plug-and-play enhancement: any frozen vision foundation model (DINOv2, CLIP, RADIO) can be paired with LoftUp and a lightweight decoder instead of retraining a task-specific head on high-resolution inputs.","Because LoftUp maps coordinates directly, one trained model can produce features at arbitrary upsampling scales, and the paper shows it can beat a VFM baseline that processes 2× larger images at roughly four times the computational cost.","Boundary-sensitive tasks benefit most: video object segmentation, where contours matter, shows the largest gain, with the J&F score rising from 36.66 to 60.25.","The pseudo-groundtruth itself is reusable: the paper shows it improves other upsampler architectures such as resize-conv and FeatUp-JBU, not just LoftUp.","With less than 20% parameter overhead over the backbone and inference time comparable to bilinear upsampling, the method is practical for deployment."],"supporting_citations":[{"why":"FeatUp is the model-agnostic feature upsampler baseline that LoftUp supersedes, and its per-image implicit coordinate-based approach defines the test-time optimization cost LoftUp avoids.","marker":"[12]"},{"why":"LiFT is the other task-agnostic upsampler baseline, using 2× larger input features as pseudo-GT at low resolution; LoftUp's objective is contrasted with it.","marker":"[52]"},{"why":"SAM supplies the class-agnostic masks and the SA1B training dataset used to construct pseudo-groundtruth and train the upsamplers.","marker":"[21]"},{"why":"DINOv2 is the primary vision foundation model backbone whose low-resolution features LoftUp upsamples in most experiments.","marker":"[37]"},{"why":"CLIP serves as an additional backbone to demonstrate that LoftUp transfers across different vision foundation models.","marker":"[41]"},{"why":"RADIO is the third backbone tested, further supporting the generality of the upsampling approach.","marker":"[43]"},{"why":"LIIF is the coordinate-based local implicit function baseline that LoftUp extends with global cross-attention, and it appears in the architecture comparisons.","marker":"[2]"}],"fun_headline_variants":["Coordinate-based upsampler sharpens VFM features for dense tasks","LoftUp: 10-20% gains on dense tasks with learned feature upsampler","New upsampler boosts DINOv2 and CLIP features to full resolution","Cross-attention upsampler lifts vision features up to 50% on video tasks","Learning a coordinate upsampler: sharp features for pixel-level understanding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pseudo-groundtruth features built from SAM-mask-smoothed bicubic features and from the teacher upsampler's high-resolution-crop outputs are accurate enough proxies for true full-resolution VFM features that training against them transfers to downstream tasks; if that surrogate is wrong, the reported downstream gains would not reflect genuine recovery of fine detail.","fun_headline_variants_meta":{"raw":{"variants":["Coordinate-based upsampler sharpens VFM features for dense tasks","LoftUp: 10-20% gains on dense tasks with learned feature upsampler","New upsampler boosts DINOv2 and CLIP features to full resolution","Cross-attention upsampler lifts vision features up to 50% on video tasks","Learning a coordinate upsampler: sharp features for pixel-level understanding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2729,"prompt_tokens":925,"completion_tokens":1804,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":1700}},"tokens_in":541,"tokens_out":1804,"duration_ms":12213,"temperature":1.0,"reasoning_tokens":1700,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:57:55.333327+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LoftUp on a benchmark where full-resolution features can be obtained as oracle ground truth, such as a vision model that natively outputs dense features at input resolution, and compare linear-probe segmentation or depth mIoU of the oracle features versus LoftUp's upsampled features under identical decoders. If LoftUp does not approach the oracle's performance, or if a LoftUp variant trained directly on oracle features substantially outperforms the pseudo-GT-trained version, then the pseudo-GT training objective is the bottleneck rather than the upsampler architecture.","supporting_citations":[{"cited_title":"Brandt, Axel Feld- mann, Zhoutong Zhang, and William T","cited_arxiv_id":null,"evidence_quote":"FeatUp is the model-agnostic feature upsampler baseline that LoftUp supersedes, and its per-image implicit coordinate-based approach defines the test-time optimization cost LoftUp avoids."},{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"SAM supplies the class-agnostic masks and the SA1B training dataset used to construct pseudo-groundtruth and train the upsamplers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DINOv2 is the primary vision foundation model backbone whose low-resolution features LoftUp upsamples in most experiments."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"CLIP serves as an additional backbone to demonstrate that LoftUp transfers across different vision foundation models."},{"cited_title":"Am-radio: Agglomerative vision foundation model reduce all domains into one","cited_arxiv_id":null,"evidence_quote":"RADIO is the third backbone tested, further supporting the generality of the upsampling approach."}],"review_version":1}