{"id":"f8335eed-0ad5-4395-a661-cda835d26640","arxiv_id":"2508.04181","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Small-scale ViT-22B models outperform standard ViT under matched parameter counts on CIFAR, and a proposed ViTUnet runs image-to-image translation, though without strong quantitative validation.","lead":"The paper trains small versions of the ViT-22B architecture and reports that they match or beat standard ViT at equal parameter counts, plus a new ViT-based Unet for image-to-image translation. It matters because it tests whether ViT-22B's gains come from its structure rather than just its scale, and offers a stabilization recipe for local training.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'ViT-22B outperforms ViT' claim is unestablished: Section 3.2 evaluates the LayerNorm-modified ViT-22B at ≤307M parameters, not the original 22B architecture.","rationale":"The reader's verdict of CONDITIONAL is reasonable, but the weakest assumption is more specific than 'scaled-down models transfer to 22B.' The experiments never evaluate the original ViT-22B architecture; they evaluate a modified version with an added LayerNorm on the parallel MLP output (and an unspecified gradient penalty). This makes the causal attribution in the headline claim unsupported. A clean ablation of the stabilization modification would settle whether the advantage over ViT is due to the ViT-22B parallel-block design or to the paper's modification. The released code is a plus, and the exploratory image-generation study is honest about limitations, but the classification claim is load-bearing and currently under-specified. The paper can be conditionally accepted if the authors add the ablation, numeric accuracies with variance, seed counts, and clearly distinguish the modified architecture from the original ViT-22B.","tokens_in":6890,"tokens_out":14597,"duration_ms":156871,"concrete_test":"Run a three-way classification ablation on CIFAR-100 at the Base size with at least 3 seeds, using identical data, optimizer, LR schedule, and augmentation: (a) standard ViT, (b) original unmodified ViT-22B architecture (parallel MLP+attention, QK-norm, no LayerNorm on MLP output) trained with best-effort standard stabilization (e.g., gradient clipping, small LR), and (c) the paper's modified ViT-22B (Equation 2). Report mean +/- std accuracy. If (b) does not beat (a) while (c) does, the headline claim must be attributed to the stabilization modification, not to ViT-22B. If (b) cannot be trained stably, the paper should rephrase all conclusions as applying only to the modified architecture.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline claim (abstract; Section 3.2) is that 'The ViT-22B model, trained from scratch, overall outperformed ViT under the same parameter size.' But the architecture compared is not the ViT-22B architecture from [6]: Section 3.1 modifies ViT-22B by adding LayerNorm to the output of the parallel linear networks (Equation 2), and Figure 1b indicates a gradient penalty that is never specified in the text. Section 3.1 shows the unmodified ViT-22B collapses during training, so the classification results in Section 3.2 necessarily use the modified variant. Thus any accuracy difference could be caused by the added LayerNorm (or the undisclosed penalty), not by the ViT-22B parallel-block design. No ablation on the original ViT-22B architecture is provided. Additionally, all 'ViT-22B' models in Table 1 have 5.6M–307M parameters; the actual 21.7B model is never trained, so statements about 'the ViT-22B model' require a scaling-transfer assumption for which no evidence is offered. Figure 2 contains no numeric accuracies, error bars, or seed counts, so the claimed superiority cannot be quantitatively verified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies local (small-scale) training of the ViT-22B vision transformer architecture. It observes gradient explosion in the original parallel-block design, proposes a modification that adds LayerNorm to the output of the parallel MLP, and compares classification accuracy of ViT-22B versus ViT on CIFAR-10/100. The paper also introduces 'ViTUnet', a ViT-based image-to-image translation architecture, and reports FID scores for two backbones. The central claims are that the ViT-22B structure outperforms ViT under matched parameter counts, and that the proposed modifications stabilize training.","tokens_in":7233,"tokens_out":5195,"duration_ms":53177,"significance":"If the classification and training-stability claims were rigorously established, the paper would provide practically useful guidance for training large vision transformers in resource-constrained settings and for choosing between ViT and ViT-22B-like blocks. The release of source code and the attempt to control for parameter count and pretraining data are positive aspects. However, the evidence as presented is not sufficient to support the headline claims: no numerical accuracy or seed statistics are given, the evaluated architecture is modified relative to ViT-22B, and the image-generation comparison rests on single-run FID values with acknowledged contradictions to qualitative results.","major_comments":[{"comment":"The headline claim that 'The ViT-22B model, trained from scratch, overall outperformed ViT' is not established by the reported experiments. The architecture evaluated in §3.2 is not the ViT-22B architecture from [6]: §3.1 adds LayerNorm to the output of the parallel MLP in Eq. (2), and Figure 1b also indicates an unspecified gradient penalty. No ablation separates the effect of these modifications from the parallel-block design. Moreover, Figure 2 shows no numerical accuracies, error bars, or number of seeds, so the claimed superiority cannot be quantitatively verified. Please provide numeric tables with seeds and an ablation comparing original ViT-22B, modified ViT-22B, and ViT.","section":"Abstract; §3.2, Eq. (2)"},{"comment":"All 'ViT-22B' models trained in this paper have 5.6M–307M parameters; the actual 21.7B model is never trained. Statements about 'the ViT-22B model' therefore require an unstated scaling-transfer assumption. This is especially problematic because §3.1 shows that the original scaled-down ViT-22B collapses, so the relative behavior at small scale may not reflect the 22B regime. Please either train at the intended scale (if feasible) or clearly reframe the conclusions as applying to a scaled-down ViT-22B-like block, and provide scaling evidence if transfer is claimed.","section":"Table 1; §3.1"},{"comment":"The training-stabilization claim rests on a single representative trajectory. Figure 1b is labeled '+LN +Penalty', but the gradient penalty is never defined in the text; the text says three methods (small LR, weight decay, LayerNorm) and omits the penalty. No seed count, quantitative gradient norm statistics, or success rate over runs is reported. To support the claim that the modification stabilizes training, please report multiple seeds, the penalty formulation/coefficient, and clipping threshold.","section":"Figure 1; §3.1"},{"comment":"FID is reported for a single run per configuration with no variance or statistical comparison. The relationship between FID and qualitative results is inconsistent: for Apple2Orange, ViT-22B-S has the worst FID (10.494 average) yet the text says it 'recorded the highest FID score', while the qualitative images are described as poor. This ambiguity, combined with the acknowledged mismatch, makes the generation conclusion difficult to assess. Please provide mean±std over seeds and clarify which FID direction is considered better in each sentence.","section":"Table 2; §4.2"},{"comment":"The ViTUnet architecture is described too vaguely to be reproducible: patch size p, number of transformer blocks per stage, downsampling/upsampling method, residual concatenation details, CNN residual structure, optimizer, loss weights, and training iterations are not specified. Since the image-generation experiment is one of the paper's main contributions, these details are load-bearing.","section":"§4.1"}],"minor_comments":[{"comment":"Typo in the title of Section 1: 'Introdcution' should be 'Introduction'.","section":"Throughout"},{"comment":"The sentence 'The results obtained from training from scratch show numerical values similar to the findings34' references two GitHub repositories but does not report the actual numbers or compare quantitatively. Please add the values or remove the vague reference.","section":"§3.2"},{"comment":"The axis labels in Figure 2 are garbled/unreadable in the PDF; please ensure all text is rendered properly.","section":"Figure 2"},{"comment":"The text says 'the largest model achieved the best FID score' but Table 2 shows only ViT-B/32 (44M) as the largest evaluated backbone; clarify this wording.","section":"§4.2"},{"comment":"The conclusion correctly hedges that ViT-22B does not consistently outperform in generation, but the abstract's unconditional 'overall outperformed' claim should be matched to the actual scope of the experiments.","section":"Abstract/Conclusion"}],"recommendation":"major_revision","confidential_remarks":"This is an early-stage empirical study with an accessible writing style and code release, but the main claims are not supported by the evidence as written. The authors would need substantial additional experiments—numerical accuracy tables with seeds, ablations separating the LayerNorm modification from the parallel-block design, a clear scaling-transfer argument or a reframed scope, and reproducible generation details—before the paper meets the standard for publication. The FID reporting and the discrepancy with qualitative results also need careful correction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the central claim is overreach, but there's a useful empirical kernel here.\n\nWhat's new: a matched-parameter comparison of ViT vs ViT-22B-style blocks at 5–300M parameters, and ViTUnet, a U-Net-like encoder-decoder built from ViT blocks for unpaired image translation. I haven't seen that exact comparison in the cited literature, and the code is out. That's real, though modest, value.\n\nThe paper also does something right: it shows the original ViT-22B block (with LayerNorm only on Q/K) blows up at 86M parameters, and that adding LayerNorm to the parallel MLP output plus a gradient penalty stabilizes training. That's a reproducible practical observation, even if the recipe is incomplete. The honesty about FID disagreeing with qualitative results is also a point in its favor.\n\nNow the soft spots, and they are load-bearing. The stress-test is right: Section 3.2 compares ViT to the modified 'ViT-22B + LN + Penalty', not to the architecture from [6]. Figure 1a shows the unmodified model collapses. So the abstract's 'ViT-22B model trained from scratch overall outperformed ViT' is unsupported. What is supported is that the modified block does better. That's a real finding, but it needs to be stated that way, with an ablation of the LayerNorm alone.\n\nBeyond that: no numeric accuracies, no error bars, no seed counts in Figure 2; the stability claim rests on a single trajectory; the gradient penalty in Figure 1b is never specified; Table 2's FID scores contradict the visuals and the paper doesn't reconcile them. The scaling from 5.6M–307M to 21.7B is assumed, not argued. ViTUnet is a new combination, but ViT-Unet hybrids exist in adjacent work, so the related-work claim should be tightened.\n\nWho is this for? People who want to train small ViT-22B-style blocks and want a quick warning about instability. That audience gets something. But as a claim about ViT-22B versus ViT, it doesn't hold.\n\nMy recommendation: don't desk-reject; send to a referee who can ask for the reframing and the missing statistics. If the authors reposition the paper as a stabilization recipe and a preliminary architecture study, it could be a minor but useful contribution. As is, the headline needs to change.","headline":"The paper's own experiments undercut its headline: the ViT-22B that 'outperformed' ViT is actually a modified ViT-22B with an added LayerNorm and an unspecified gradient penalty, so the central comparison does not test the architecture as published.","tokens_in":7638,"tokens_out":2658,"would_cite":false,"duration_ms":31143,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Local from-scratch training of the ViT-22B architecture shows its parallel linear branch explodes gradients unless normalized, and once normalized the architecture outperforms vanilla ViT at equal parameter counts on CIFAR-10 and CIFAR-100.","keywords":["Vision Transformer","ViT-22B","training stability","LayerNorm","image-to-image translation","ViTUnet","CIFAR-10/100","FID"],"falsifier":"Reproduce the paper's from-scratch CIFAR-10 training of a ViT-22B-style Small/Base model without LayerNorm on the parallel MLP output; if logits and gradients stay below 4000 past 3000 iterations instead of exploding near 1500, the attribution of the instability to the unnormalized parallel branch is falsified. Alternatively, a matched-size ViT-versus-ViT-22B comparison across many seeds on a larger dataset would test whether the 'overall outperformed' ranking holds.","tokens_in":6825,"feed_emoji":"🖼️","tokens_out":10453,"duration_ms":109148,"temperature":0.7,"pith_summary":"This paper asks whether the ViT-22B architecture can be understood and used outside the massive private-data regime where it was introduced. Training scaled-down versions from scratch locally, the paper finds that the parallel linear branch in the ViT-22B block drives logits and gradients to explode, and that appending LayerNorm to that branch's output—together with small learning rates and weight decay—stabilizes training for hundreds of epochs. With matched parameter counts and identical data, the ViT-22B block generally outperforms vanilla ViT on CIFAR-10 and CIFAR-100, though on small datasets the smallest models win. The paper also introduces ViTUnet, a ViT-based encoder-decoder for image-to-image translation, and reports that the ViT-22B backbone is not consistently better there: FID and visual quality diverge, with vanilla ViT backbones producing more intelligible Apple-to-Orange images while the ViT-22B variant scores better on Monet-to-Photo.","feed_headline":"One LayerNorm fix lets ViT-22B beat ViT at equal size from scratch","feed_subtitle":"Local CIFAR tests show the 22B-style block trains stably and usually beats vanilla ViT at matched size","key_machinery":"The load-bearing object is the modified ViT-22B residual block $y = x + \\mathrm{LayerNorm}(\\mathrm{MLP}(\\mathrm{LayerNorm}(x))) + \\mathrm{Attention}(\\mathrm{LayerNorm}(x))$. Inserting LayerNorm after the parallel MLP output is the paper's key intervention: it bounds logit and gradient magnitudes that otherwise explode. The second mechanism is ViTUnet, a patch-based encoder-decoder that downsamples through stacked transformer blocks, upsamples during decoding, and concatenates residuals across scales before CNN-based reconstruction, which lets a ViT backbone perform image-to-image translation.","core_discovery":"On its own terms, the paper's central claim is that the ViT-22B block is locally trainable once its parallel MLP branch is normalized. In the original block, a linear network runs in parallel with self-attention, and LayerNorm is applied only to queries and keys; the paper attributes the observed gradient explosion to the unnormalized parallel branch. Its modified update is $y = x + \\mathrm{LayerNorm}(\\mathrm{MLP}(y')) + \\mathrm{Attention}(y')$, with $y' = \\mathrm{LayerNorm}(x)$, and this stabilizes from-scratch training at scales from 5.6M to 307M parameters. The paper then claims that, trained from scratch on CIFAR-10 and CIFAR-100 with equal parameters and data, the ViT-22B structure 'ove","pith_inferences":["If the stabilization cause is general, any transformer variant with an unnormalized parallel branch should normalize that branch before summation; this would extend beyond ViT-22B to other parallel-branch designs.","ViTUnet's residual encoder-decoder could transfer to other dense prediction tasks such as segmentation or depth estimation, which currently bolt task-specific heads onto ViT backbones.","The FID-versus-quality inversion in Monet-to-Photo suggests FID may reward global style statistics over object structure; human ratings or object-detector checks could test this directly.","Since experiments are limited to CIFAR-scale data, whether the ViT-22B advantage persists at ImageNet scale remains open; the paper's matched-size result is a lower-bound test, not a scaling law."],"forward_implications":["ViT-22B-style architectures can be trained from scratch at modest scale without the original 4B-image pretraining, once the parallel branch is normalized.","Because ViT-22B wins at matched parameter count and data, part of its reported advantage is structural rather than merely a matter of scale or pretraining corpus.","On small datasets, smaller ViT-22B recipes outperform larger ones, implying model size should be chosen with dataset size in mind.","ViTUnet provides a ViT-native image-to-image translation path, and its FID-versus-quality divergence means FID alone can mislead when comparing generative backbones.","The ViT-22B backbone is not uniformly better for generation; Apple-to-Orange and Monet-to-Photo favor different backbones."],"supporting_citations":[{"why":"Introduces the ViT-22B architecture and original recipe whose local behavior the paper tests.","marker":"[6]"},{"why":"Defines the vanilla ViT baseline and patch-embedding formulation used in all comparisons.","marker":"[7]"},{"why":"Supplies the LayerNorm operation the paper inserts to stabilize the parallel MLP branch.","marker":"[1]"},{"why":"Documents transformer training instabilities and the LayerNorm-on-key/query rationale the paper extends.","marker":"[11]"},{"why":"Provides the U-Net encoder-decoder and residual-concatenation idea behind ViTUnet.","marker":"[25]"},{"why":"Supplies the CycleGAN unpaired translation training setup and the Apple2Orange/Monet2Photo tasks.","marker":"[32]"},{"why":"Supplies the PatchGAN discriminator with 70x70 receptive field used in generation experiments.","marker":"[12]"},{"why":"AdamW optimizer used for all from-scratch training runs.","marker":"[18]"},{"why":"Mixed precision training, one of the standard tricks that only delayed gradient explosion.","marker":"[21]"}],"fun_headline_variants":["LayerNorm on parallel branch stabilizes ViT-22B from scratch","ViT-22B beats ViT at equal size after LayerNorm fix","One LayerNorm tweak lets ViT-22B beat Vanilla ViT","Adding LayerNorm to MLP branch makes ViT-22B train stably","ViT-22B's parallel MLP needs LayerNorm to train from scratch"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The scaled-down ViT-22B models (5.6M to 307M parameters) faithfully represent the 21.7B-parameter architecture, so the observed gradient instability and the LayerNorm fix transfer to full-scale ViT-22B.","fun_headline_variants_meta":{"raw":{"variants":["LayerNorm on parallel branch stabilizes ViT-22B from scratch","ViT-22B beats ViT at equal size after LayerNorm fix","One LayerNorm tweak lets ViT-22B beat Vanilla ViT","Adding LayerNorm to MLP branch makes ViT-22B train stably","ViT-22B's parallel MLP needs LayerNorm to train from scratch"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1181,"prompt_tokens":693,"completion_tokens":488,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":437,"completion_tokens_details":{"reasoning_tokens":385}},"tokens_in":437,"tokens_out":488,"duration_ms":5019,"temperature":1.0,"reasoning_tokens":385,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:48:45.360619+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the paper's from-scratch CIFAR-10 training of a ViT-22B-style Small/Base model without LayerNorm on the parallel MLP output; if logits and gradients stay below 4000 past 3000 iterations instead of exploding near 1500, the attribution of the instability to the unnormalized parallel branch is falsified. Alternatively, a matched-size ViT-versus-ViT-22B comparison across many seeds on a larger dataset would test whether the 'overall outperformed' ranking holds.","supporting_citations":[{"cited_title":"Intriguing properties of transformer training instabilities, 2023","cited_arxiv_id":null,"evidence_quote":"Documents transformer training instabilities and the LayerNorm-on-key/query rationale the paper extends."},{"cited_title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","cited_arxiv_id":null,"evidence_quote":"Supplies the CycleGAN unpaired translation training setup and the Apple2Orange/Monet2Photo tasks."}],"review_version":1}