{"id":"714bf314-6392-4807-8294-ac0115a044b2","arxiv_id":"2504.14992","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PHD-Transformer repeats input tokens K times during pretraining with a custom attention mask, then drops the repeated tokens' KV cache, yielding average accuracy gains over a vanilla baseline at 1.2B scale.","lead":"This paper presents a transformer variant that repeats input tokens during pretraining and then discards the repeated tokens' attention caches, claiming better benchmark scores with the same inference memory. A generalist reader might care because it tests whether letting a model 'think longer' during training can be made nearly free at inference time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed pretraining length scaling is confounded with test-time compute: PHD models are trained and evaluated with K repeated tokens, while the vanilla baseline uses K=1; no cross-condition evaluation is reported.","rationale":"The paper offers a plausible engineering contribution: a KV-cache management scheme that lets a transformer reuse input tokens as parallel hidden decoding tokens, with latency and prefill measurements (Figure 9) that support the efficiency claims. Those measurements are independent evidence for the systems part of the paper, and I do not dispute them. The weakness is in the scientific attribution of the accuracy gains. The reader flagged that PHD models use K times more FLOPs and 1/K unique data during pretraining, so a compute- or epoch-matched vanilla baseline is missing; that is a real concern. My stress-test goes further: the accuracy evaluation itself appears to use K repetitions for the PHD models and K=1 for vanilla. The paper's own Section 3.4 and Table 2 describe the performance as scaling with decoding computation, and the prediction rule from Section 2.2 naturally carries over to evaluation. If evaluation uses K repetitions, then even a perfectly matched pretraining-compute baseline would not isolate the pretraining mechanism, because the test-time compute differs. The decisive experiment is cross-condition evaluation: vanilla evaluated with K>1 under the PHD forward pass, and PHD-trained models evaluated with K=1. Without these, the data are consistent with a test-time compute effect and do not establish 'efficient pretraining length scaling' as a distinct phenomenon. Since this is a missing control rather than an internal inconsistency, and since the engineering claims are credible, CONDITIONAL remains the right verdict; the reader's CONDITIONAL is unchanged by my analysis, but my reasoning emphasizes the evaluation confound rather than the training confound.","tokens_in":14041,"tokens_out":8969,"duration_ms":81208,"concrete_test":"Run two additional evaluation conditions on the 1.2B setting. (a) Take the vanilla 1.2B checkpoint and evaluate it with the PHD forward pass for K=2 and K=3: repeat the input tokens K times, apply the attention mask of Eq. 2, use the final copy's hidden state for scoring, and discard hidden-token KV cache. Compare HellaSwag, ARC-Challenge, and CommonsenseQA against Table 1. (b) Evaluate the PHD-CSWA-3-16-32 trained checkpoint with K=1 (standard vanilla forward pass) on the same benchmarks. If condition (a) recovers most of the Table 1 gains, the improvements are due to test-time compute, not pretraining length scaling. If condition (b) retains the gains, the pretraining objective itself has a residual effect. The combination of (a) and (b) cleanly separates the two factors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that repeating input tokens K times during pretraining improves downstream accuracy while keeping inference cost near vanilla. The main accuracy tables (Table 1, Table 2, Figures 5 and 8) compare PHD-CSWA-K / PHD-SWA-K models against a vanilla baseline. In every such comparison, the PHD model is both trained and evaluated with K repetitions: Section 3.4 is titled 'Decoding Token Scaling', Table 2's caption says performance scales with the 'increase of decoding computation', and the prediction rule in Section 2.2 uses the final copy of the token for next-token prediction. The vanilla baseline is trained and evaluated with K=1. Thus the reported gains bundle two effects: (i) the pretraining objective that uses K copies, and (ii) a K-fold increase in test-time forward computation per token. Because a vanilla transformer's weights can be run through the same PHD attention pattern (Eq. 2) at test time, the K>1 advantage may be available to the vanilla baseline without any PHD pretraining. The paper does not report a vanilla model evaluated with K>1, nor a PHD-trained model evaluated with K=1. Without these controls, the improvement is equally consistent with test-time length scaling, and the central 'pretraining length scaling' attribution is unsupported. The reader's concern about training FLOPs and unique-token counts is valid, but the eval-side confound is more direct: even if pretraining compute were matched, the reported accuracy comparison would still conflate training recipe with decode-time compute.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes the Parallel Hidden Decoding Transformer (PHD) family for what it calls pre-training length scaling: input sequences are repeated K times, the first ('original') copies retain global KV caches while the repeated ('hidden decoding') copies are discarded after use, and the next-token prediction loss is taken from the final copy. PHD-SWA adds a sliding window over hidden decoding tokens and PHD-CSWA chunks that window to avoid pre-fill growth. Experiments on 1.2B and 550M models trained on OLMo2-style data report lower training loss, roughly 1.5-2.0% average downstream accuracy gains over vanilla baselines (Tables 1-2), and near-vanilla pre-fill and decoding latency (Figure 9).","tokens_in":14319,"tokens_out":5013,"duration_ms":47129,"significance":"The architecture idea is clear and the efficiency measurements are genuinely useful: keeping the KV cache at the vanilla size while adding K parallel decoding tokens addresses a real inference bottleneck, and the chunked sliding-window variant is a reasonable engineering solution. The paper also includes ablations (Figures 6-7) that support the chosen window and chunk sizes. However, the empirical evidence for the central 'pre-training length scaling' claim is currently confounded in two ways: the PHD runs are not matched to the vanilla baseline in compute or unique data, and the evaluation protocol appears to increase K at test time for PHD models only. Until controlled comparisons isolate the pretraining effect, the observed gains could be explained by extra training FLOPs, repeated-data epochs, or test-time repetition. If the authors supply the missing controls and clarify the evaluation protocol, this could become a solid contribution to efficient pretraining and inference.","major_comments":[{"comment":"The main accuracy comparison is not matched in compute or data. If the shared 500B value on the Figure 5 x-axis is consumed tokens, then PHD-CSWA-2 and PHD-CSWA-3 process 500B tokens with K=2 and K=3 repetitions, i.e., only 250B and 167B unique tokens, while the vanilla baseline sees 500B unique tokens; the PHD runs also incur roughly K times more FLOPs. The reported loss drop and average accuracy gain are therefore compatible with a compute-scaling or repeated-epochs explanation rather than 'length scaling' as a distinct pretraining phenomenon. The paper should add a vanilla baseline with matched FLOPs (e.g., 1500B consumed tokens for the K=3 condition, either from additional data or from repeating the same corpus) and state the unique-token count and FLOP count for every run.","section":"Section 3.2, Table 1, Figure 5"},{"comment":"The evaluation protocol confounds the effect with test-time computation. The accuracy tables and Figure 8 compare PHD-K models, which appear to be evaluated with K repeated tokens, against a vanilla baseline evaluated with K=1. Since Equation (2) defines a valid attention mask for any transformer weights, a vanilla model can be run in the same repeated-token mode without any PHD-style pretraining; this is precisely what Section 3.4 calls 'decoding computation scaling.' Without cross-evaluations - vanilla evaluated at K>1 and, where possible, PHD-trained models evaluated at K=1 - the reported gains cannot be attributed to the pretraining objective. This is a load-bearing control and should be reported at least for the 550M/Table 2 setting.","section":"Section 3.2 and Section 3.4, Tables 1-2, Equation (2)"},{"comment":"The paper's own framing is ambiguous about what is being scaled. The abstract claims pre-training length scaling, while Section 3.4 is titled 'Decoding Token Scaling' and Table 2's caption states that 'downstream performance scales w.r.t. the increase of decoding computation.' The manuscript should state explicitly what changes during evaluation when K is increased, and then design the experiments so that the pretraining contribution is separately identifiable from the decoding-time contribution.","section":"Section 3.4 and Abstract"}],"minor_comments":[{"comment":"The x-axis label 'Cosumed T okens' contains typos and should read 'Consumed Tokens'.","section":"Figures 1, 5, 6, 7, 8"},{"comment":"The phrase 'origin tokens' is used inconsistently; the rest of the paper uses 'original tokens.'","section":"Section 2.2"},{"comment":"The claim that rearranging token positions 'reducing the complexity of attention computation to O(K)' is underspecified, since the original sequence length t also enters the complexity; please state the complexity in terms of both t and K.","section":"Section 2.3"},{"comment":"The window-size and chunk-size ablations report only training and validation loss; adding a downstream benchmark to these ablations would make the chosen hyperparameters more convincing.","section":"Section 3.3.1"},{"comment":"Figure 9 would benefit from a statement of the exact measurement protocol, including sequence length, batch size, and number of repeated runs, since the reported 20% decoding-latency increase is central to the efficiency claim.","section":"Section 3.5"}],"recommendation":"major_revision","confidential_remarks":"The novelty of the architecture is not the main concern; the gap is experimental isolation. The eval-side confound described in the stress-test note is real and should be addressed directly rather than through additional scaling-law arguments. Given the cost of 1.2B runs, requiring matched-compute baselines at that scale may be heavy, but the 550M setting is feasible and should be sufficient to validate the attribution with cross-evaluations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The PHD attention pattern is a real, useful engineering idea, and the paper's central claim about pretraining length scaling is not yet supported by the experiments. The good part: distinguishing original tokens from repeated hidden decoding tokens, keeping only the former in the global KV cache, and adding a chunk-wise sliding window for the latter is a clean way to get extra compute without linear KV growth. The kernel rearrangement and the latency measurements in Figure 9 give the efficiency story substance. The gains are consistent across benchmarks, and the ablations on window/chunk size are the right kind of tuning.\n\nThe soft spot is load-bearing. Every accuracy comparison pits PHD-SWA/CSWA with K repetitions against a vanilla baseline with K=1. That bundles three effects: K times more training FLOPs, 1/K as many unique tokens per update (at fixed consumed-token budget), and K times more forward passes at evaluation. The paper's own Section 3.4 is titled 'Decoding Token Scaling' and Table 2 says performance scales with increase of decoding computation—which is a test-time effect, not a pretraining effect. The missing control is straightforward: run the vanilla baseline with the same repeated-token attention pattern at test time (Eq. 2 applies to any model), and evaluate PHD-trained models with K=1. Without those cross-condition numbers, the improvement is equally explained by test-time repetition, and the 'pretraining length scaling' attribution collapses. The reader's concern about matched FLOPs/unique tokens points to the same missing baseline.\n\nMinor issues: some typos in the text, and the 1.5-2% average gain is modest even if the attribution holds. The ablation-based hyperparameter choice is standard and not circular.\n\nIf I were the editor, I would send this to review rather than desk-reject. The architecture is novel and the latency story is credible; the confound is fixable with a few controlled runs. A serious referee should ask for the vanilla-K>1 and PHD-K=1 evaluations, plus a compute-matched training comparison, before the 'length scaling' language is accepted.","headline":"Useful KV-cache-friendly token-repetition architecture, but the central 'pretraining length scaling' claim is unproven because the comparisons bundle repetition with more FLOPs, fewer unique tokens, and more test-time compute.","tokens_in":14902,"tokens_out":2095,"would_cite":false,"duration_ms":19720,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Repeating input tokens during pretraining is a viable length-scaling axis when the repeated tokens' KV cache is discarded after decoding, yielding accuracy gains without extra inference memory.","keywords":["pretraining length scaling","KV cache management","token repetition","hidden decoding tokens","parallel hidden decoding","sliding window attention","chunk-wise attention","large language models"],"falsifier":"Train a vanilla transformer with FLOPs matched to PHD-CSWA-2 and PHD-CSWA-3 on the same unique 500B tokens, for example by running two or three epochs over the same data or by increasing depth to match compute, and compare average accuracy on the six benchmarks. If the compute-matched vanilla model reaches or exceeds the 1.5 and 2.0 point gains, the central claim that the KV-cache design enables length scaling rather than extra compute would collapse.","tokens_in":13824,"feed_emoji":"🔁","tokens_out":9079,"duration_ms":76551,"temperature":0.7,"pith_summary":"The paper argues that length scaling, already useful in post-training, can also be harnessed during pretraining by simply repeating each input token K times. It introduces PHD-Transformer, a training scheme that keeps the KV cache of only the first copies of tokens and discards the cache of repeated copies after use, so inference memory and latency stay close to a vanilla transformer. Two variants, PHD-SWA and PHD-CSWA, add a small local window over the repeated tokens, with chunking to prevent prefill time from growing linearly. On 1.2B and 550M models, the scheme lowers training loss and raises average downstream accuracy by 1.5 to 2.0 percentage points over a vanilla baseline. A sympathetic reader would take the paper's contribution to be an inference-friendly way to buy pretraining quality by spending more compute on repeated tokens.","feed_headline":"Token repetition during pretraining boosts accuracy at no KV cost","feed_subtitle":"A cache trick keeps inference memory flat while 2–3x token repetition lifts six benchmarks by up to 2 points.","key_machinery":"The central mechanism is an attention-mask and KV-cache split between original tokens and hidden decoding tokens. Original tokens, the first copies, are globally attended to and their key/value vectors are retained, while hidden decoding tokens, the repeated copies, are decoded and then their key/value vectors are dropped, so only the original sequence length is cached and the final copy of each position carries the next-token loss. PHD-SWA adds a sliding window over recent hidden decoding tokens to keep some local cache with O(1) extra memory, and PHD-CSWA restricts the window to fixed chunks so that prefill time does not grow linearly with K.","core_discovery":"The paper claims that a transformer can be pretrained with its input sequence repeated K times, using the first copy of each token to generate the persistent KV cache while immediately discarding the KV cache of the hidden decoding tokens after they contribute to next-token prediction. This keeps the KV cache size identical to a vanilla transformer while giving the model extra computational depth per token. Empirically, PHD-CSWA-2-16-32 improves average accuracy on six benchmarks by 1.5 points over a 1.2B vanilla baseline, PHD-CSWA-3-16-32 improves it by 2.0 points, and PHD-SWA with K=5 improves a 550M baseline by 1.8 points; decoding latency stays within about 20 percent of vanilla even at K=256.","pith_inferences":["The decisive untested comparison is a compute-matched vanilla baseline: if a vanilla model trained with the same FLOPs on the same unique tokens matches the gains, then the contribution is extra compute rather than length scaling specifically.","The same discard-after-use KV trick could be applied to inference-time chain-of-thought or thinking tokens, giving models a bounded-memory way to spend more forward passes before answering.","Because only the final copy receives the next-token loss, the repeated copies act as an implicit depth increase; a natural extension is to vary K adaptively by token or training stage rather than fixing it for the whole corpus.","The reported gains are on commonsense and language-understanding benchmarks at 1.2B and below, so whether the pattern transfers to frontier-scale models or to math and reasoning tasks is still open."],"forward_implications":["Pretraining length scaling via token repetition is a workable axis: larger K gives lower loss and higher downstream accuracy, at least up to K=5 in the tested settings.","Inference cost stays near vanilla: KV cache size is unchanged, decoding latency grows by at most about 20 percent even at K=256, and CSWA keeps prefill time close to vanilla.","A small sliding window over hidden decoding tokens captures most of the benefit, so the extra KV memory is only O(1).","Chunk-wise sliding window attention removes the linear prefill growth of the sliding-window variant with negligible performance loss.","The attention-rearrangement and KV-sharing design is compatible with other kernel-level and cache-management optimizations, so it can be layered onto standard efficient transformers."],"supporting_citations":[{"why":"Supplies the pretraining codebase, model configuration, and training pipeline used for both the vanilla baseline and the PHD variants.","marker":"[37]"},{"why":"Earlier work inserting pause tokens into pretraining sequences; the token-repetition formulation is positioned against this line.","marker":"[17]"},{"why":"Latent-space reasoning embeddings are a prior length-scaling approach that PHD's hidden decoding tokens extend.","marker":"[20]"},{"why":"Implicit 2x length scaling by reusing hidden embeddings is a direct predecessor of repeating tokens.","marker":"[36]"},{"why":"Continuous-concept pretraining is contrasted with PHD's hidden decoding tokens.","marker":"[48]"},{"why":"Train-native sparse attention is the related method against which PHD's train-time attention patterns are compared.","marker":"[57]"},{"why":"Supplies the sliding-window attention pattern that PHD-SWA adapts for hidden decoding tokens.","marker":"[24]"}],"fun_headline_variants":["Repeat tokens in pretraining: same KV cache, better accuracy","Token repetition gives transformer depth without KV bloat","PHD-Transformer: more depth per token, flat inference memory","Pretrain with repeated tokens, keep KV size vanilla, gain points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's main experiments compare PHD models against a single vanilla baseline trained on the same 500B consumed tokens but with fewer unique tokens and much less compute, so the reported gains are only attributable to length scaling if extra compute and repeated exposure to the same data are not responsible.","fun_headline_variants_meta":{"raw":{"variants":["Repeat tokens in pretraining: same KV cache, better accuracy","Token repetition gives transformer depth without KV bloat","PHD-Transformer: more depth per token, flat inference memory","Pretrain with repeated tokens, keep KV size vanilla, gain points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1376,"prompt_tokens":862,"completion_tokens":514,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":453}},"tokens_in":478,"tokens_out":514,"duration_ms":5184,"temperature":1.0,"reasoning_tokens":453,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:36:13.026918+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a vanilla transformer with FLOPs matched to PHD-CSWA-2 and PHD-CSWA-3 on the same unique 500B tokens, for example by running two or three epochs over the same data or by increasing depth to match compute, and compare average accuracy on the six benchmarks. If the compute-matched vanilla model reaches or exceeds the 1.5 and 2.0 point gains, the central claim that the KV-cache design enables length scaling rather than extra compute would collapse.","supporting_citations":[{"cited_title":"Think before you speak: Training language models with pause tokens","cited_arxiv_id":null,"evidence_quote":"Earlier work inserting pause tokens into pretraining sequences; the token-repetition formulation is positioned against this line."},{"cited_title":"Cotformer: More tokens with attention make up for less depth","cited_arxiv_id":null,"evidence_quote":"Implicit 2x length scaling by reusing hidden embeddings is a direct predecessor of repeating tokens."}],"review_version":1}