{"id":"c660e5d7-85f0-4873-b413-bc97d40626bb","arxiv_id":"2604.06070","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Long-context retrieval transfers to student models through logit-based distillation on packed short sequences, aided by phase-wise RoPE scaling and observable positional propagation to output logits.","lead":"This paper shows that long-context retrieval skills can be transferred to a student transformer via logit distillation even when the student sees only packed short sequences inside a long window. A generalist might care because it points to a lower-cost path for building models that handle long documents without collecting rare long training data.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Distillation on packed repeated-token sequences may transfer superficial positional patterns rather than genuine long-context retrieval","rationale":"The reader's weakest assumption directly identifies the risk of superficial pattern matching; the repeated-token tracing experiment, while technically interesting for RoPE analysis, does not close that gap. With full text the experimental details are now visible, but the missing control still leaves the transfer claim conditional on further verification.","tokens_in":1677,"tokens_out":390,"duration_ms":31080,"concrete_test":"Re-run the distillation experiment of §4.2 but replace the repeated-token packs with random contiguous excerpts from a diverse corpus (same total tokens and packing density); evaluate both the original and new student on a needle-in-haystack retrieval task at lengths 2–4× the short sample size. If the performance gap between distilled and non-distilled students shrinks by >30% relative on the diverse data, the original claim rests on the repeated-token artifact.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that logit distillation from a teacher, when fed only short packed samples inside a long window, supplies a faithful signal for the student to acquire position-dependent retrieval at arbitrary distances. The paper traces how RoPE perturbations in query/key states propagate to output logits on repeated-token packs, showing systematic influence. However, repeated tokens create highly structured, low-entropy sequences where positional cues are unusually salient and content-based retrieval is trivial; this setup does not rule out the student learning to exploit the same artificial regularities (e.g., token repetition periodicity) rather than acquiring general long-range positional retrieval. No ablation is described that replaces repeated tokens with diverse natural text while keeping the same packing and distillation protocol, nor is there a control that severs the positional channel (e.g., position-agnostic teacher) to measure how much of the transfer depends on RoPE specifically.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that long-context retrieval capabilities can be transferred to student models through logit-based knowledge distillation even when training exclusively on packed short-context samples within a long-context window. It reports three empirical findings: phase-wise RoPE scaling maximizes long-context performance in distillation; logit distillation directly enables positional information transfer, demonstrated by tracing RoPE perturbations in query/key states through layers to output logits on packed repeated-token sequences; and long-context extension produces structured update patterns in the query state with distinct parameter spans showing differential sensitivity.","tokens_in":1862,"tokens_out":592,"duration_ms":28556,"significance":"If the central claim holds, the result would offer a practical route to long-context extension that avoids the full cost of long pre-training and long data collection, which is a meaningful efficiency advance for transformer scaling. The mechanistic tracing of how RoPE-induced perturbations propagate to logits and the identification of structured query-state updates constitute concrete strengths that could inform future distillation and position-encoding work. These elements provide falsifiable, layer-wise observations rather than purely black-box performance claims.","major_comments":[{"comment":"The experimental setup with packed repeated token sequences (described in the second finding and associated analysis) is load-bearing for the claim that distillation supplies a faithful signal for genuine long-context positional retrieval. Repeated tokens create low-entropy sequences in which positional cues are unusually salient and content-based retrieval is trivial; without an ablation that replaces repeated tokens with diverse natural text while preserving the same packing and distillation protocol, or a control that severs the positional channel (e.g., position-agnostic teacher), it remains possible that the student acquires superficial periodicity patterns rather than general long-range retrieval. This directly affects whether the reported transfer generalizes beyond the artificial setup.","section":"Second finding / experimental setup with packed repeated token sequences"},{"comment":"The abstract and results sections state three empirical findings yet supply no quantitative metrics, error bars, baseline comparisons, or ablation tables for the distillation experiments. Without these, the magnitude of the positional transfer, its statistical reliability, and its improvement over non-distillation long-context baselines cannot be assessed, weakening support for the central efficiency claim.","section":"Abstract and results on the three findings"}],"minor_comments":[{"comment":"Notation for RoPE phase-wise scaling and the precise definition of 'packed short-context samples within a long-context window' should be formalized with an equation or diagram in the methods section for reproducibility.","section":"Methods / RoPE scaling description"},{"comment":"The third finding on 'structured update patterns' and 'distinct parameter spans' would benefit from explicit identification of the affected parameter indices or layers and a quantitative sensitivity metric.","section":"Third finding / query state analysis"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the thoughtful and detailed comments, which highlight important aspects of our experimental design and presentation. We address each major comment below and commit to revisions that strengthen the evidence for positional transfer via distillation.","responses":[{"response":"The repeated-token packing was deliberately chosen to create a low-entropy regime that isolates positional signals, enabling direct tracing of RoPE perturbations from query/key states through layers to output logits without content-based confounds. This mechanistic probe supports the second finding on how distillation propagates positional information. We agree, however, that the setup alone does not fully demonstrate generalization to natural text. In the revised manuscript we will add an ablation using packed sequences of diverse natural language while keeping the same distillation protocol, along with a position-agnostic teacher control, to quantify whether the observed transfer relies on genuine long-range positional retrieval.","revision_made":"yes","referee_comment":"[Second finding / experimental setup with packed repeated token sequences] The experimental setup with packed repeated token sequences (described in the second finding and associated analysis) is load-bearing for the claim that distillation supplies a faithful signal for genuine long-context positional retrieval. Repeated tokens create low-entropy sequences in which positional cues are unusually salient and content-based retrieval is trivial; without an ablation that replaces repeated tokens with diverse natural text while preserving the same packing and distillation protocol, or a control that severs the positional channel (e.g., position-agnostic teacher), it remains possible that the student acquires superficial periodicity patterns rather than general long-range retrieval. This directly affects whether the reported transfer generalizes beyond the artificial setup."},{"response":"We acknowledge that the current presentation of the three findings would benefit from more explicit quantitative support. The manuscript already reports retrieval accuracies on standard long-context benchmarks, but we will expand the abstract and results section to include (i) numerical deltas with standard-error bars from multiple random seeds, (ii) direct comparisons against non-distillation long-context baselines, and (iii) ablation tables isolating the contribution of logit distillation, phase-wise RoPE scaling, and packing. These additions will allow readers to evaluate the magnitude and reliability of the reported transfer.","revision_made":"yes","referee_comment":"[Abstract and results on the three findings] The abstract and results sections state three empirical findings yet supply no quantitative metrics, error bars, baseline comparisons, or ablation tables for the distillation experiments. Without these, the magnitude of the positional transfer, its statistical reliability, and its improvement over non-distillation long-context baselines cannot be assessed, weakening support for the central efficiency claim."}],"tokens_in":1444,"tokens_out":551,"duration_ms":40504,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main claim is that you can transfer long-context retrieval to a student model through logit distillation even when the student trains only on short sequences packed inside a long window. That would be useful if it works, since it sidesteps collecting and training on full long documents.","headline":"Distillation on packed short repeated-token sequences can pass some positional signal from a long-context teacher, but the artificial data leaves open whether students learn real long-range retrieval or just exploit repetition patterns.","tokens_in":2343,"tokens_out":139,"would_cite":false,"duration_ms":43041,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"RoPE frequency superposition and logit propagation in packed-sequence distillation","alignment":"orthogonal","rationale":"The paper's core machinery (RoPE angle accumulation, non-monotonic cosine-similarity patterns from multi-frequency superposition, propagation of query/key perturbations through layers to position-modulated output logits, and phase-wise θ scaling) operates entirely within standard transformer training and distillation. It neither invokes nor parallels the RS forcing chain: no J-cost functional, no ratio-symmetric reciprocal cost, no golden-ratio fixed points or φ-ladder spacings, no 8-tick periodicity, and no parameter-free derivation of constants. The repeated-token isolation experiment is a diagnostic tool for positional sensitivity, not a recognition-cost or distinction-forcing construction. RS therefore has no opinion on the empirical claims or methodology.","tokens_in":49009,"confidence":"moderate","tokens_out":179,"duration_ms":25226,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Long-context retrieval transfers to students via logit distillation on packed short sequences.","keywords":["knowledge distillation","long context","rotary position embeddings","transformers","positional information","retrieval","model extension"],"falsifier":"Measure whether a distilled student can correctly retrieve a fact whose position lies beyond the length of any single packed short sample; if retrieval accuracy drops to chance levels while the teacher succeeds, the claim fails.","tokens_in":2604,"feed_emoji":"📏","tokens_out":648,"duration_ms":44159,"temperature":0.7,"pith_summary":"The paper shows that a teacher model's ability to retrieve information over long distances can be passed to a student model without ever showing the student real long sequences. Short samples are packed together to fill a long training window, and the student is trained to match the teacher's output probabilities at each step. This logit matching carries positional signals forward through the layers because the teacher's logits already reflect where tokens sit in the full window. Analysis with Rotary Position Embeddings confirms that phase-wise scaling of the embeddings gives the strongest transfer and that query-state updates during training follow repeatable patterns tied to distance. The result suggests long-context extension can be done with far less data and compute than direct pre-training on long text.","feed_headline":"Distillation gives long context from short packed data","feed_subtitle":"Student models acquire positional retrieval by matching teacher logits on short sequences packed inside long windows.","key_machinery":"Logit-based knowledge distillation applied to short sequences packed into a long window, with positional effects traced through Rotary Position Embeddings and layer-wise propagation to output logits.","core_discovery":"The authors establish that logit-based knowledge distillation on packed short-context samples inside a long-context window is sufficient to transfer long-context retrieval capabilities. They trace how positional perturbations in query and key vectors propagate through successive layers to shape the teacher's output distribution, thereby supplying a usable training signal to the student. Phase-wise RoPE scaling maximizes performance at each stage, and the query states exhibit structured sensitivity patterns across parameter spans when the context length is extended.","pith_inferences":["The same packing-plus-distillation recipe may work for other positional encodings such as ALiBi or learned absolute embeddings.","Downstream tasks that require cross-document reasoning could be used to test whether the transferred retrieval is functionally useful rather than only measured on synthetic probes.","If the method generalizes, training pipelines could generate synthetic short packs on the fly instead of storing large long-context corpora."],"forward_implications":["Long-context model development no longer requires collecting or training on native long documents.","Phase-wise RoPE scaling becomes the default schedule for distillation-based length extension.","Positional information flows measurably from teacher logits into the student even without explicit position labels.","Query-state parameter updates during extension follow repeatable distance-sensitive patterns that can be monitored."],"fun_headline_variants":["Logit distillation transfers long context from packed short data","Positional retrieval flows from teacher logits to student models","RoPE scaling optimizes long-context distillation performance","Query state patterns reveal sensitivity in context extension"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"That matching teacher logits on packed short samples inside a long window gives the student genuine long-range positional retrieval rather than just local pattern matching.","fun_headline_variants_meta":{"raw":{"variants":["Logit distillation transfers long context from packed short data","Positional retrieval flows from teacher logits to student models","RoPE scaling optimizes long-context distillation performance","Query state patterns reveal sensitivity in context extension"]},"model":"grok-4.3","cost_usd":0.004431,"raw_usage":{"total_tokens":2206,"prompt_tokens":652,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":44312000,"prompt_tokens_details":{"text_tokens":652,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1497,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":652,"tokens_out":57,"duration_ms":29880,"temperature":1.0,"reasoning_tokens":1497,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-10T18:54:52.076459+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Measure whether a distilled student can correctly retrieve a fact whose position lies beyond the length of any single packed short sample; if retrieval accuracy drops to chance levels while the teacher succeeds, the claim fails.","supporting_citations":[],"review_version":1}