{"id":"eeb204c5-57a8-4025-b4bf-11aa6ef5c1c3","arxiv_id":"2501.03005","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A single-encoder, two-decoder model that jointly predicts pixels, latent features, and a [CLS] token outperforms pixel-only and latent-only masked image modeling on most high-level and low-level vision benchmarks.","lead":"PiLaMIM trains a vision transformer to reconstruct both raw pixels and a teacher model's hidden features at once, and also predicts a global [CLS] token. On linear probing benchmarks, it beats MAE, I-JEPA, and BootMAE on most classification, counting, and depth tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Clevr/Dist token choice was selected post hoc after seeing a performance gap; without reporting the alternative token results for I-JEPA and BootMAE, the claimed low-level gains are not yet established.","rationale":"The reader's conditional verdict identifies the post-hoc Clevr/Dist token choice as the weakest assumption, and I agree that this is the load-bearing issue. The paper's headline claim is explicitly about outperforming MAE, I-JEPA, and BootMAE 'in most cases,' and the low-level task results are the key evidence that the method captures both levels of visual information. The protocol description in Section 3 says I-JEPA and BootMAE used average pooled patch tokens while MAE and PiLaMIM used [CLS], and that the authors then switched to [CLS] for Clevr/Dist after observing a performance gap. That is exactly the kind of post-hoc selection that can invalidate a comparison unless the alternative numbers are reported and the conclusion is robust to the choice. The paper does not provide those numbers, so the central low-level claim is not fully verified. I also note the appendix's admission that BootMAE was taken from a public checkpoint, which adds a second, smaller confound; however, the primary issue remains the token-selection protocol. The method itself is simple and the loss definitions are internally consistent; my concern is with the experimental evidence, not the construction. Therefore the appropriate verdict remains conditional, matching the reader's assessment.","tokens_in":8414,"tokens_out":4248,"duration_ms":39583,"concrete_test":"Run linear probing on Clevr/Dist (and, for completeness, Clevr/Count) for all four checkpoints under both token protocols — [CLS] token and mean-pooled patch token — holding batch size, epochs, optimizer, and augmentation fixed. Report all numbers in a 4×2×2 table. If I-JEPA or BootMAE improves under its original mean-pooled protocol enough to surpass PiLaMIM's [CLS] score, the paper's low-level-improvement claim fails; if PiLaMIM remains best under both protocols, the concern is resolved. For full rigor, repeat with a locally retrained BootMAE to remove checkpoint provenance as a confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that PiLaMIM extracts richer low-level representations rests mainly on the Clevr/Count and Clevr/Dist rows of Table 1. Section 3 states that I-JEPA and BootMAE were evaluated with the average pooled patch token (following their original protocols) while MAE and PiLaMIM used the [CLS] token; then, after observing a large gap between these token choices on Clevr/Dist, the authors 'standardized' to [CLS]. This is a post-hoc protocol selection: if the choice was made using the test numbers, it is not a neutral evaluation, and the paper does not report the discarded numbers. For I-JEPA (62.0) and BootMAE (68.7), using their intended mean-pooled token could plausibly raise their Clevr/Dist scores and erase or reverse PiLaMIM's reported advantage (67.8). A secondary confound is acknowledged in A.1: BootMAE was not retrained in the authors' environment; an official checkpoint with different pretraining details was used. The low-level half of the 'richer visual representations' claim therefore depends on an unverified evaluation choice. This is not an accusation of intent; it is a missing control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PiLaMIM, a self-supervised pretraining method for Vision Transformers that combines pixel-level masked reconstruction (as in MAE) with latent-representation prediction (as in I-JEPA). A shared context encoder feeds two separate decoders, one for raw pixels and one for target-encoder latent features, and a third loss is imposed on the [CLS] token. The method is evaluated by linear probing on high-level tasks (ImageNet-1K, CIFAR-10/100, iNaturalist2021, Places365) and low-level tasks (Clevr/Count, Clevr/Dist), reporting that PiLaMIM outperforms MAE, I-JEPA, and BootMAE in most settings. The paper also includes an ablation of the [CLS] loss, a t-SNE visualization, and RankMe scores. Code is provided.","tokens_in":8711,"tokens_out":3380,"duration_ms":30742,"significance":"If the empirical claims hold, PiLaMIM offers a conceptually simple and reproducible way to obtain representations that are strong on both semantic and geometric low-level tasks, and the [CLS]-token loss is a small but interesting addition. The method is clearly specified: the losses in Eq. (4)-(5) are concrete, the architecture is described in sufficient detail, and the code repository is provided. However, the evidence supporting the central claim is weakened by a post-hoc change in the evaluation token on Clevr/Dist, the lack of multiple seeds or variance estimates, and the use of an external BootMAE checkpoint. These issues need to be addressed before the 'richer visual representations' claim is firmly established.","major_comments":[{"comment":"The evaluation protocol for Clevr/Dist is changed after observing results, which undermines the low-level comparison. The text states that I-JEPA and BootMAE were evaluated with average pooled patch tokens while MAE and PiLaMIM used the [CLS] token, and that after observing a large performance gap on Clevr/Dist, the evaluation was 'standardized' to the [CLS] token. This is post-hoc selection of the evaluation token, and the paper does not report the discarded numbers. Because Clevr/Dist is one of only two low-level tasks supporting the claim of richer low-level representations, please report both token choices for all methods, or otherwise demonstrate that the conclusion is insensitive to token choice.","section":"Section 3, Table 1(b)"},{"comment":"All results are single runs without variance estimates. Several differences are small (e.g., Clevr/Dist 67.8 for PiLaMIM vs 68.7 for BootMAE; Places365 47.0 vs 47.6 between PiLaMIM 600 and 800 epochs) and may be within run-to-run noise. To support the claim that PiLaMIM 'outperforms key baselines in most cases', please report means and standard deviations over multiple pretraining or linear-probing seeds, or at least provide a statistical analysis of the differences.","section":"Section 3, Table 1"},{"comment":"The BootMAE baseline was not retrained in the authors' environment but was evaluated using a publicly available checkpoint. Since pretraining details (data processing, augmentation, optimizer settings, and training length) may differ from those used for the other models, the comparison against BootMAE may reflect training-setup differences rather than the method itself. Please retrain BootMAE under identical conditions, or temper the claim of outperforming BootMAE and explicitly state the limitation in the main text.","section":"Appendix A.1"}],"minor_comments":[{"comment":"The notation D_t (or D_T) is used without definition; please define it consistently with D_x, and ensure the subscripts in the normalization factors match the text.","section":"Equation (4)"},{"comment":"The phrase 'proving its effectiveness' is stronger than what a finite set of empirical comparisons can establish; consider using 'indicating' or 'suggesting' to match the evidence level.","section":"Abstract"},{"comment":"The sentence 'For MAE and PiLaMIM, we used the [CLS] token from the same layer' is ambiguous because MAE in its original formulation does not use a [CLS] token for linear probing; please clarify whether a [CLS] token was added to MAE and, if so, how it was trained.","section":"Section 3"},{"comment":"The description of the learnable mask tokens M_M and N_M is easy to confuse with the index sets M; please explicitly state that these are distinct learnable vectors for the two decoders.","section":"Section 2"},{"comment":"The ablation of the [CLS] token is reported on only two datasets (CIFAR100 and Clevr/Count). If the claim is that [CLS] benefits both high-level and low-level tasks, consider adding at least one more dataset from each category.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"I agree with the reader's assessment that the Clevr/Dist token choice is the main load-bearing weakness. The authors should be asked to disclose the performance of every method under both token types, not just the one selected after seeing results. If the alternative numbers reverse the conclusion, the low-level claim would need to be substantially revised. The missing variance and the external BootMAE checkpoint are also important, but they are fixable within a revision. The paper is not fatally flawed, but it is not yet at the standard required to support the strong claims made in the abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a reasonably clean empirical paper with a modest but real contribution. The biggest weakness is a disclosed post-hoc evaluation choice, not the method itself.\n\nWhat's new: the specific architecture combining one shared encoder, two decoders, and CLS reconstruction with an EMA target. BootMAE already combined pixel and latent targets, and iBOT already used CLS, so the novelty is in the integration, not in any new principle. The losses are correctly specified, the code is linked, hyperparameters are fully listed, and the ablation shows CLS adding a consistent 1-2 points. The high-level classification results are solid: PiLaMIM beats MAE and I-JEPA across the board and beats BootMAE on most classification sets.\n\nSoft spots: the Clevr/Dist evaluation. The paper admits that after seeing a large gap between CLS and mean-pooled tokens, it standardized to CLS for that dataset. That means I-JEPA and BootMAE, which were evaluated with mean-pooled tokens following their original protocols, may have been penalized—or helped—by the switch. Without the alternative token numbers for those baselines, the low-level comparison is not controlled. This is selection after the fact, even if disclosed. Also, BootMAE was not retrained in the authors' environment; the official checkpoint may have different training details. Single runs make small differences like the 0.9 gap on Clevr/Dist uninterpretable.\n\nFor readers working on MIM, this is a useful data point. It doesn't change the landscape, but it shows a simple integration works and ships a reproduction. It deserves a serious referee because the method is sound and the flaw is fixable. In review, I'd ask for both token choices across all models, retrained BootMAE or a matched checkpoint, and error bars.\n\nRecommendation: send it to peer review, but with the expectation of heavy revision on the evaluation section.","headline":"Simple and mostly reproducible, but the low-level claim rests on a post-hoc token choice that should be controlled before the paper is trusted.","tokens_in":9169,"tokens_out":3169,"would_cite":false,"duration_ms":28318,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Jointly predicting pixels and latent features gives richer vision representations.","keywords":["masked image modeling","self-supervised learning","pixel reconstruction","latent representation","vision transformer","linear probing","CLS token","representation learning"],"falsifier":"Re-run the Clevr/Dist linear probing using each baseline's originally intended token—average pooled patch tokens for I-JEPA and BootMAE, [CLS] for MAE and PiLaMIM—and check whether PiLaMIM still beats them; if the gap narrows or reverses, the claimed low-level advantage is an artifact of token choice.","tokens_in":8239,"feed_emoji":"🖼️","tokens_out":4104,"duration_ms":110031,"temperature":0.7,"pith_summary":"This paper tries to show that masked image modeling gets richer visual representations when a single shared encoder is trained to reconstruct both raw pixels and latent features at once, rather than choosing one target. The authors build on MAE with two decoders, one for pixel values and one for latent representations produced by an EMA-updated target encoder, and add a loss on the [CLS] token to pull in global context. They report that the combined objective beats the single-target baselines MAE and I-JEPA, and also BootMAE, on most of the linear-probing tasks they try, including image classification on five datasets and object counting and distance prediction on CLEVR. If correct, the claim is that the two levels of visual detail are complementary and can be captured simultaneously without sacrificing either.","feed_headline":"One encoder, two reconstruction targets, richer vision features","feed_subtitle":"Combined pixel and latent masking beats MAE, I-JEPA, and BootMAE on classification and CLEVR counts.","key_machinery":"The load-bearing mechanism is the shared context encoder with two lightweight decoders operating on the same encoded visible patches. A pixel decoder reconstructs raw RGB values of masked patches; a latent decoder predicts the corresponding token representations produced by an exponentially moving-averaged target encoder. A third term, $L_{cls}$, predicts the target encoder's [CLS] token, which aggregates global image context, and the whole objective is simply $L = L_{pixel} + L_{latent} + L_{cls}$ with a masking ratio of 0.75 and no normalization of targets. The two decoders let one set of features be pulled toward low-level detail and high-level semantics simultaneously, while the [CLS] term discourages the encoder from collapsing to patch-local statistics.","core_discovery":"PiLaMIM's central claim is that a single context encoder trained with the sum of three mean-squared-error losses—pixel reconstruction on masked patches, latent reconstruction on masked patches, and latent reconstruction on the [CLS] token—produces representations that are simultaneously strong on semantic tasks and on geometric/low-level tasks. The latent targets come from a target encoder whose weights are an exponential moving average of the context encoder, so the model predicts its own evolving features rather than a fixed offline representation. In linear probing, PiLaMIM at 800 epochs reaches 69.2% on ImageNet-1K, 92.4% on CIFAR10, 74.2% on CIFAR100, 31.0% on iNaturalist2021, and 47.6% on Places365, while on CLEVR it reaches 83.8% for counting and 67.8% for distance, all above the reported baselines. The paper reads these numbers as evidence that neither pixel MIM nor latent MIM is sufficient alone and that their integration, plus the global [CLS] signal, yields a better balance.","pith_inferences":["A natural testable extension is to vary the weighting of the three losses; the paper fixes all weights at 1, and a trade-off curve would show whether the two targets interfere at other training budgets.","The RankMe scores in the appendix suggest the joint model's representation has higher effective rank; one could check whether that rank advantage predicts performance on dense prediction tasks like semantic segmentation, which the paper did not evaluate.","Because the target encoder is an EMA, the method inherits the same collapse-avoidance behavior as I-JEPA; it remains to be seen whether the pixel decoder's gradient through the shared encoder changes the EMA dynamics in a way that harms very long training runs."],"forward_implications":["Linear probing accuracy on out-of-domain classification improves by several points over I-JEPA and MAE, notably +6.1 on CIFAR10 and +8.1 on CIFAR100 compared with I-JEPA.","On low-level tasks, PiLaMIM beats MAE by 7.2 points on Clevr/Count and 2.8 points on Clevr/Dist, showing that the pixel decoder keeps fine detail that latent-only training discards.","The [CLS] token contributes 1.9 points on CIFAR100 and 1.2 points on Clevr/Count in the ablation, so global-context reconstruction helps both semantic and geometric probing.","PiLaMIM also outperforms BootMAE on most datasets, implying that the specific design of separate decoders plus the [CLS] term extracts more than simply summing the two objectives."],"supporting_citations":[{"why":"MAE supplies the pixel-reconstruction baseline and the overall masked-autoencoder structure that PiLaMIM extends.","marker":"[10]"},{"why":"I-JEPA defines latent MIM with an EMA target encoder and supplies the high-level/low-level task split used in evaluation.","marker":"[1]"},{"why":"BootMAE is the prior attempt to combine pixel and latent objectives and is the direct comparison for the proposed architecture.","marker":"[6]"},{"why":"iBOT contributes the idea of reconstructing the [CLS] token as a way to inject global context into the representation.","marker":"[25]"}],"fun_headline_variants":["Pixel plus latent MIM: one encoder, richer vision features","Combining pixel and latent reconstruction yields richer visual features","PiLaMIM: integrating pixel and latent targets for better representations","One encoder, two decoders: pixel and latent MIM combined","Self-supervised vision: fusing pixel and latent MIM outperforms baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison on Clevr/Dist assumes it is fair to judge all models with the [CLS] token, even though the token-type choice was made after seeing a performance gap and may not match the token type each baseline was designed to use.","fun_headline_variants_meta":{"raw":{"variants":["Pixel plus latent MIM: one encoder, richer vision features","Combining pixel and latent reconstruction yields richer visual features","PiLaMIM: integrating pixel and latent targets for better representations","One encoder, two decoders: pixel and latent MIM combined","Self-supervised vision: fusing pixel and latent MIM outperforms baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1517,"prompt_tokens":979,"completion_tokens":538,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":448}},"tokens_in":595,"tokens_out":538,"duration_ms":5472,"temperature":1.0,"reasoning_tokens":448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:58:18.099612+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Clevr/Dist linear probing using each baseline's originally intended token—average pooled patch tokens for I-JEPA and BootMAE, [CLS] for MAE and PiLaMIM—and check whether PiLaMIM still beats them; if the gap narrows or reverses, the claimed low-level advantage is an artifact of token choice.","supporting_citations":[{"cited_title":"Assran, Q","cited_arxiv_id":null,"evidence_quote":"I-JEPA defines latent MIM with an EMA target encoder and supplies the high-level/low-level task split used in evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BootMAE is the prior attempt to combine pixel and latent objectives and is the direct comparison for the proposed architecture."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"iBOT contributes the idea of reconstructing the [CLS] token as a way to inject global context into the representation."}],"review_version":1}