{"id":"4c4da53a-642e-48ee-a2ef-4cf6dfc22038","arxiv_id":"2502.02919","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding the position embedding to the final layer normalization, called MPVG, improves vision transformer top-1 accuracy by 0.27 to 1.37 percentage points across several GAP-based models and tasks.","lead":"The paper proposes a small architectural change, adding the position embedding to the final normalization layer of a vision transformer, which improves accuracy when the model uses global average pooling instead of a class token. This tweak yields consistent but modest gains on image classification, detection, and segmentation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MPVG's gains may not be attributable to position-embedding counterbalancing; a learnable constant-bias control is missing, and Eq. (11) is analytically redundant with the head bias.","rationale":"The reader's verdict is CONDITIONAL, and I agree, but for a slightly more specific reason. The reader identified the correlational basis of the counterbalancing claim; my concern sharpens that into a missing control that directly challenges the causal attribution. The empirical result of consistent top-1 gains is plausible, but the paper's headline mechanistic claim is not established because Eq. (11) could be replicated by a generic learnable bias. This is load-bearing because the stated novelty is the counterbalancing insight, not just the accuracy improvement. The proposed control experiment would settle the question cleanly. I do not think the paper should be rejected outright—the modification is simple, reproducible across models, and the gains, while small, are directionally consistent. But acceptance should be conditioned on this control, or on the authors explicitly reframing the contribution as a training trick rather than a PE-specific mechanism. My agreement_with_reader is 'partial' because the reader focuses on the correlational evidence for the counterbalancing role; I focus on the absence of a bias-control ablation and the redundancy with the head bias, which is a stronger and more precise test of the causal claim.","tokens_in":14237,"tokens_out":4714,"duration_ms":47694,"concrete_test":"Train DeiT-Ti under the PVG/MPVG setup with a modified MPVG where Eq. (11) is replaced by y = LN(x_{L+1}) + b, with b a learnable per-dimension vector initialized to zero (or sampled from the same initialization as LN'(pos0)) and otherwise identical hyperparameters. Run this control alongside MPVG and report both accuracies. If the control matches or exceeds MPVG, the improvement is not specific to position embedding. Additionally, freeze or remove the classification head bias and retrain MPVG to check whether the gain persists; if it disappears, Eq. (11) was merely acting as a redundant bias term.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that PE performs a counterbalancing role and that preserving this directionality in the Last LN is what improves accuracy (Eq. 11, Table 1). The load-bearing assumption is that the specific identity of the added vector matters, not merely the extra learnable parameters or a generic bias. However, the paper never tests a control where LN'(pos0) is replaced by an arbitrary learnable vector (e.g., a constant, random, or zero-initialized bias) of the same shape. Without this, the observed gains could be due to adding a learnable bias term, not to the position embedding's counterbalancing content. Furthermore, Eq. (11) adds a vector that is constant across tokens. After GAP, this is equivalent to adding a learnable bias to the final feature vector before the classification head. Since the head is typically a linear layer with its own bias, the modification is functionally redundant in representational capacity: any effective bias shift can be absorbed into the head bias. Thus, any improvement must come from optimization dynamics, not from the 'counterbalancing directionality' story. The paper's own Fig. 5 shows that the addition is harmful in a non-Layer-wise structure, but that is a comparison across architectures, not a control for the bias-vs-PE distinction. The missing ablation is decisive: if a random learnable vector yields the same accuracy as LN'(pos0), the counterbalancing mechanism is unsupported and the contribution reduces to a training trick. The paper also selects pos0 (Table 5) after trying several PE variants on the same test set, which inflates the reported gain of the specific choice, though all tested variants do beat PVG.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the combination of global average pooling (GAP) with layer-wise position embedding (PE) delivery in vision transformers. It observes that while GAP and layer-wise PE each help individually, combining them hurts accuracy. The authors propose MPVG, which adds the initial position embedding pos0 to the 'Last LN' before the classification head (Eq. 11), and PVG, a variant without this addition. They report consistent top-1 accuracy gains over the default and over LaPE across DeiT, Swin, CeiT, and T2T on ImageNet-1K, on CIFAR-100, and on COCO/ADE20K dense-prediction tasks. The paper interprets these gains as evidence that PE counterbalances token embeddings in deep layers and that maintaining this counterbalancing directionality after the final layer is beneficial.","tokens_in":14577,"tokens_out":4371,"duration_ms":38116,"significance":"If the central claim holds, this is a simple, broadly applicable modification to vision transformers that use GAP, with a suggestive new interpretation of PE beyond positional encoding. The empirical results are internally consistent and cover multiple architectures and tasks, which is a clear strength of the paper. The correlation analyses in Figs. 2 and 4 are interesting and could motivate further study. However, the paper's mechanistic claim—that the identity of the added PE, rather than a generic learned bias, drives the improvement—is not supported by the provided experiments. The paper also selects architectural details and the PE source on the test set without a validation split and reports single runs. These issues are central to the contribution as stated.","major_comments":[{"comment":"The paper does not include a control experiment that isolates whether the content of the added vector matters. After GAP, LN'(pos0) in Eq. (11) is constant across tokens, so it is functionally equivalent to adding a learnable bias to the pooled feature vector. Since the linear classification head already has a bias, any constant offset can be absorbed, and the observed gains could stem from the extra parameter, from optimization dynamics, or from the counterbalancing directionality claimed by the authors. To support the paper's central claim that PE performs a counterbalancing role, the authors should compare Eq. (11) against a same-shaped learnable vector, a random fixed vector, or a zero-initialized bias. Without such a control, the reported gains do not discriminate between the proposed mechanism and a generic bias-shift effect.","section":"Eq. (11) and 'Maximizing the Position Embedding with GAP'"},{"comment":"The architecture choices (excluding layer 0, hierarchical PE, adding PE to x before layer 0) and the choice of pos0 as the Last LN input are selected based on test-set top-1 accuracy on ImageNet-1K. For example, Table 6 shows differences of 0.2–0.3% between configurations, and Table 5 selects pos0 over pos5/pos8/pos11 by a margin of 0.1–0.2%. With single runs and no held-out validation split, these differences are within typical training noise, and the 'optimal' configuration may overfit the test set. The authors should either use a separate validation set for these selections or report multiple seeds with means and variances for all reported values.","section":"Tables 5 and 6, 'Ablation Study'"},{"comment":"The correlation analyses are purely descriptive and do not establish a causal counterbalancing role. The negative correlation between token and position embeddings after LayerNorm could arise trivially from the centering and scaling properties of LayerNorm, or from optimization constraints, rather than from a functional mechanism the model seeks to preserve. The paper's own appendix shows that the Last LN's beta parameter has high variance in PVG and low variance in MPVG, which is equally consistent with a simple bias-shift explanation. To support the counterbalancing hypothesis, the authors should provide a control that decorrelates the added PE from the counterbalancing direction (for example, adding a randomly sampled or sign-permuted PE) while keeping the same distributional statistics, and show that the gain disappears.","section":"Figs. 2, 4, 8 and the 'Analysis' section"}],"minor_comments":[{"comment":"The caption defines posN as LN'_N(pos_{N-1}), but Eq. (7) defines pos_l = LN'_{l-1}(pos_{l-1}). The subscript is off by one and should be corrected to avoid confusion.","section":"Table 5 caption and Eq. (7)"},{"comment":"The phrase 'the average value of token embedding' is ambiguous; it should clarify whether the heatmaps are averaged over tokens, batch samples, or both, and over which forward pass (training or validation).","section":"Figure 2 caption"},{"comment":"The paper reports only single-run top-1 accuracies without standard deviations or number of seeds. For a fair comparison of changes as small as 0.1%, at least three seeds should be reported, or the authors should state that the differences are beyond their evaluation noise.","section":"Tables throughout"},{"comment":"The typo 'MVPG' in the Table 2 caption should read 'MPVG'.","section":"Table 2 caption"},{"comment":"The header 'Prei-train' should be 'Pretrain' (or 'Pre-train').","section":"Appendix Tables 8 and 9"},{"comment":"The claim that MPVG is 'generally effective for vision transformers' is supported only for a limited set of architectures and training settings; the paper should soften this to 'effective across the evaluated architectures and tasks'.","section":"Introduction, contribution 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical pattern is consistent and the modification is simple enough to be of practical interest. However, the central mechanistic claim is not yet falsifiable from the reported experiments because the missing bias control leaves the alternative 'added learnable bias' explanation open. The test-set-driven architecture selection is also a methodological concern that should be addressed with a validation split or multi-seed reporting. These issues are fixable within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical core here is real and worth knowing: adding the initial position embedding to the Last LN in a layer-wise PE transformer that uses GAP gives a modest but consistent bump across six ImageNet models, plus gains on COCO and ADE20K. The fix is a one-line change and the effect appears in more than one architecture family. That is a useful result for people working on ViT tweaks.\n\nWhat I find genuinely new is the observation that GAP and layer-wise PE conflict with each other, and that adding PE to the Last LN resolves the conflict. The Fig. 5 control in the non-Layer-wise setting is a good sanity check: the same addition hurts without the layer-wise structure, so the effect is specific to that setting. That goes some way toward ruling out a trivial always-add-PE story.\n\nThe soft spot is the mechanism claim. The paper says PE 'counterbalances' token embeddings and that maintaining this directionality is what matters. But the evidence is correlational and post-hoc. The missing experiment is the obvious one: replace LN'(pos0) in Eq. (11) with a random learnable vector of the same shape, or with a zero-initialized bias, and train. Since the added vector is constant across tokens, after GAP it is functionally a learnable bias on the final feature vector, and the linear head already has its own bias. So any improvement could come from the extra capacity or from optimization dynamics rather than from the position-specific content. The paper's own Table 5 shows all tested PE variants beat PVG, which suggests the content matters less than the fact of adding a learnable vector. Without the random-bias control, the counterbalancing story is not supported.\n\nAlso worth noting: the paper selects pos0 on the test set (Table 5) with no validation split, and reports single runs with no variance. The gains are a few tenths of a percent, so seed noise alone could shift some of them. That does not kill the empirical result, but it means the paper should present the gains as a tendency, not as exact numbers.\n\nFor peer review: yes, send it out. The empirical finding is concise and reproducible, and a good reviewer can ask for the control and a softened mechanism claim. The paper as written overreaches, but the underlying result is worth a serious look.","headline":"A simple GAP fix gives small, consistent accuracy gains in ViTs, but the paper's counterbalancing mechanism is not actually supported until a random-bias control rules out a generic learnable-bias effect.","tokens_in":15117,"tokens_out":1833,"would_cite":false,"duration_ms":20381,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Position embeddings counterbalance token embeddings in vision transformers, and preserving that counterbalance through the final LayerNorm improves accuracy.","keywords":["position embedding","vision transformer","global average pooling","layer-wise position embedding","counterbalancing","Layer Normalization","MPVG","image classification"],"falsifier":"A concrete test: train a GAP vision transformer with layer-wise PE where the initial PE is added to the Last LN but with its sign flipped or with a learnable scalar multiplier; if the counterbalancing direction alone matters, the flipped sign should hurt, while a learnable scale should learn a positive value close to 1. If instead any additive vector to the Last LN helps (e.g., a random fixed vector), the benefit is not specific to the counterbalancing direction. Alternatively, measure the correlation between token and PE after the Last LN in MPVG: if the claim is right, the correlation should remain strongly negative after the Last LN, and models with higher post-LN negative correlation should show higher accuracy.","tokens_in":14033,"feed_emoji":"📈","tokens_out":4461,"duration_ms":35901,"temperature":0.7,"pith_summary":"This paper addresses why global average pooling (GAP) and layer-wise position embedding (PE) hurt each other when combined in vision transformers, even though each helps alone. It claims that in a layer-wise structure the PE increasingly counterbalances the token embedding values as layers deepen, but after the last layer the token embedding retains a directional imbalance that no longer gets compensated. The proposed fix, MPVG, is a one-line change: add the initial position embedding to the LayerNorm just before the classification head. On ImageNet-1K this yields consistent gains over the default model, ranging from +0.27 percentage points on Swin-Ti to +1.37 on DeiT-Ti, and also improves CIFAR-100, COCO object detection, and ADE20K segmentation. The paper's central contribution is the diagnosis of the counterbalancing role of PE and the demonstration that maintaining it matters.","feed_headline":"Position-embedding fix lifts vision transformers by up to 1.37%","feed_subtitle":"Adding the initial position embedding to the final LayerNorm keeps its counterbalancing role, improving six ImageNet-1K models.","key_machinery":"The central object is the Layer-wise position-embedding structure (introduced by LaPE) combined with GAP, and the one-line modification in Eq. (11). In that structure each layer applies separate Layer Normalizations to token embedding and PE, with PE delivered hierarchically through the layers. MPVG adds the initial PE (after its own LayerNorm) to the Last LN's output before the classification head. The paper's diagnostic tool is the correlation coefficient between token and position embeddings at each layer, which grows more negative with depth (e.g., from -0.37 at layer 4 to -0.95 at layer 11 for DeiT-Ti), indicating the counterbalancing role. The Last-LN addition preserves that directionality after the final layer, and the paper argues this preserves features that would otherwise be lost when the Last LN alone removes high-value dimensions.","core_discovery":"The central claim is that position embedding performs a counterbalancing role in layer-wise vision transformers: as the layers deepen, the correlation between token embedding and PE becomes increasingly negative, and this negative correlation is what keeps the token embedding values in a usable range. The paper further claims that this counterbalancing is incomplete after the final layer, so the model should add the initial PE to the Last LayerNorm (Eq. 11, $y = \\mathrm{LN}(x_{L+1}) + \\mathrm{LN}'(pos_0)$). When this is done, MPVG outperforms the default architecture, LaPE, and an intermediate PVG variant across image classification, object detection, and semantic segmentation.","pith_inferences":["The same counterbalancing argument might apply to class-token models if the class token undergoes similar distributional shifts, although the paper reports incompatibility with class tokens; a test would be to add a learned or transformed PE to the Last LN in a class-token model.","The correlation diagnostic could serve as a training-time monitor: if the negative correlation flattens or reverses, the model may be losing positional or counterbalancing information.","The mechanism suggests that LayerNorm's beta parameter is a limited substitute for PE; architectures that reduce reliance on LN statistics might see larger or smaller effects.","A testable extension: apply the same Last-LN PE addition in language transformers or other sequence models that use GAP-like pooling, to see if the counterbalancing role generalizes beyond vision."],"forward_implications":["Any vision transformer using GAP and layer-wise PE can be improved by a single addition of the initial PE to the final LayerNorm, with no extra parameters beyond the existing PE and LN.","The counterbalancing role of PE is not just an artifact but a functional mechanism that should be maintained after the last layer.","The inconsistency between GAP and layer-wise structures is explained and resolved, making the two compatible.","The gains transfer to dense prediction tasks (detection and segmentation) when the pretrained backbone uses MPVG.","The choice of which PE to add matters little: pos0 works best, but other layer-wise PE values still beat the baseline."],"supporting_citations":[{"why":"Introduces the Layer-wise PE structure (LaPE) with independent Layer Normalizations that this paper modifies and extends.","marker":"(Yu et al. 2023)"},{"why":"Provides the DeiT models used as the primary test bed for MPVG across classification, detection, and segmentation.","marker":"(Touvron et al. 2021)"},{"why":"Establishes that GAP outperforms the class token in vision transformers, motivating the GAP setting studied here.","marker":"(Chu et al. 2021b)"},{"why":"Defines the original ViT architecture with absolute position embedding and the class token baseline that this work contrasts with GAP.","marker":"(Dosovitskiy et al. 2020)"},{"why":"Provides the Swin Transformer, one of the models tested with MPVG on ImageNet-1K.","marker":"(Liu et al. 2021)"},{"why":"Provides the CeiT model, another ImageNet-1K test bed for MPVG.","marker":"(Yuan et al. 2021a)"},{"why":"Provides the T2T-ViT model, tested on both ImageNet-1K and CIFAR-100.","marker":"(Yuan et al. 2021b)"},{"why":"Supplies the ViT-Adapter backbone used to evaluate MPVG on COCO detection and ADE20K segmentation.","marker":"(Chen et al. 2022)"}],"fun_headline_variants":["MPVG: Add initial PE to final norm for stronger ViTs","Keep PE counterbalancing: MPVG lifts ViT performance","Adding first PE to final LayerNorm improves ViTs","Vision transformers boost by reusing first position embedding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the increasingly negative correlation between token embedding and position embedding in deeper layers is evidence of a counterbalancing function that should be preserved after the final layer; this is inferred from post-hoc correlation plots and ablations, not from an experiment that isolates the causal direction or rules out other explanations such as LayerNorm statistics.","fun_headline_variants_meta":{"raw":{"variants":["MPVG: Add initial PE to final norm for stronger ViTs","Keep PE counterbalancing: MPVG lifts ViT performance","Adding first PE to final LayerNorm improves ViTs","Vision transformers boost by reusing first position embedding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001318,"raw_usage":{"total_tokens":5350,"prompt_tokens":908,"completion_tokens":4442,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":4375}},"tokens_in":524,"tokens_out":4442,"duration_ms":58822,"temperature":1.0,"reasoning_tokens":4375,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:36:49.993539+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: train a GAP vision transformer with layer-wise PE where the initial PE is added to the Last LN but with its sign flipped or with a learnable scalar multiplier; if the counterbalancing direction alone matters, the flipped sign should hurt, while a learnable scale should learn a positive value close to 1. If instead any additive vector to the Last LN helps (e.g., a random fixed vector), the benefit is not specific to the counterbalancing direction. Alternatively, measure the correlation between token and PE after the Last LN in MPVG: if the claim is right, the correlation should remain strongly negative after the Last LN, and models with higher post-LN negative correlation should show higher accuracy.","supporting_citations":[],"review_version":1}