{"id":"f556d753-7b36-46c2-9bec-d761fbb2a09a","arxiv_id":"2412.07171","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Giving each attention head its own RoPE base frequency lets a single continual-pretraining stage at 128k match or beat a three-stage schedule, per the paper's NiaH, PPL and RULER results.","lead":"A team from Kuaishou, Tsinghua and Beihang proposes HARPE, which trains a large language model to handle 128k-token texts in a single stage instead of the usual several. Each attention head gets its own position-encoding frequency, so one training run can imitate the gradual multi-stage schedules that are currently standard.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HARPE's headline 5.46% NiaH margin over multi-stage ABF (Tab 4) rests on a single hand-chosen three-stage schedule; Tab 1 shows the same stage recipe varies 67.83–81.36 with data scheduling alone, so the margin may not survive a tuned multi-stage baseline.","rationale":"The reader's CONDITIONAL verdict identifies the representativeness of the three-stage ABF baseline as the primary load-bearing assumption, and I agree this is the right focal point. The central claim is comparative: HARPE beats multi-stage. That comparison is only meaningful if the multi-stage baseline is strong or at least average for its budget. Table 1 demonstrates that the authors' own multi-stage recipe is extremely sensitive to data scheduling (67.83 vs 81.36), and the 5.46-point HARPE advantage is smaller than this internal swing. Without a schedule/data search or uncertainty quantification, the 'stage barrier' conclusion is underdetermined. The undefined fp/fv in Algorithm 1 is a genuine reproducibility flaw, but it is less load-bearing because Table 6 shows even uniform base distributions (84.30–85.30) outperform the 81.36 multi-stage baseline, so the head-adaptive idea does not depend on the opaque peak-valley search. I give credit for the budget-matched controlled setup, the use of the recent RULER benchmark, and the reasonable short-context results. The recommendation remains CONDITIONAL: the finding is plausible but the core empirical comparison needs a stronger, better-swept multi-stage baseline and variance reporting before it can be accepted.","tokens_in":16033,"tokens_out":5434,"duration_ms":49682,"concrete_test":"Reproduce the controlled comparison with a small sweep over multi-stage ABF schedules using the same 6B-token budget from Sec 4.4: vary stage lengths (e.g., 4k→16k→128k, 32k→64k→128k, 8k→32k→128k), per-stage token allocations (e.g., 1B/2B/3B), and base sequences (1m/2m/5m or NTK-style schedules), plus the two data-scheduling regimes from Tab 1. If any schedule reaches or exceeds HARPE's 86.82 average on the upgraded NiaH test, the claim that HARPE matches or surpasses multi-stage training is falsified. Run each configuration with at least 3 random seeds and report means and standard deviations to assess whether the remaining differences are noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that HARPE 'consistently matches or surpasses' multi-stage long-context training—is supported by Table 4, where HARPE averages 86.82 on the upgraded NiaH test versus 81.36 for 'ABF Multi-Stage' (Sec 4.1: bases 1m/2m/5m, lengths 32k/64k/128k, 6B total tokens). The load-bearing assumption is that this single three-stage schedule is a representative or strong multi-stage baseline. The paper's own Table 1 contradicts that: the same three-stage ABF recipe scores 67.83 with 'Uniform 2B Tokens' and 81.36 with 'Carefully Selected' data—a 13.5-point swing caused entirely by data scheduling. Since the authors do not search over multi-stage schedules, data allocations, or stage lengths, and report no seeds or error bars for either method, there is no evidence that a tuned multi-stage pipeline (or the production systems cited in Sec 1) would not erase or exceed the 5.46% margin. Additionally, HARPE's hyperparameters—specifically the stride=30k base set in Tab 2—were selected on this same upgraded NiaH benchmark (Tab 6), so part of the margin may reflect selection bias rather than a fundamental single-stage advantage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Head-Adaptive Rotary Position Encoding (HARPE), a single-stage continual pretraining method that assigns different RoPE base frequencies to different attention heads and directly trains Llama2-7B at the target 128k context length with a fixed 6B-token budget. The base set is chosen either by uniform distribution or by a peak-valley complementarity search (Algorithm 1), with the stride-30k configuration selected after ablations. The authors evaluate on sliding-window perplexity (Proof-pile, GovReport), an upgraded eight-task Needle-in-a-Haystack test, five short-context benchmarks, and the RULER benchmark, reporting that HARPE matches or outperforms the tested multi-stage ABF baseline and other long-context methods, including a 5.46% NiaH margin over multi-stage ABF. Section 7 notes that the method has only been studied for continual pretraining, not for supervised fine-tuning.","tokens_in":16321,"tokens_out":8746,"duration_ms":84944,"significance":"If substantiated, HARPE would provide a meaningful simplification of long-context adaptation: it removes the multi-stage scheduling and manual tuning that current pipelines require, while preserving short-context performance. The paper has clear strengths: the main comparisons are budget-matched at 6B tokens per method; the upgraded NiaH evaluation covers eight tasks and six lengths; the NiaH advantage is consistent across all seven base-configuration variants in Table 6; and Table 7 shows a competitive RULER score. These strengths make the empirical result worth taking seriously. However, the central claim that HARPE 'consistently matches or surpasses' multi-stage methods is not yet fully established, because the multi-stage baseline is a single hand-picked schedule, the peak-valley search is not reproducible as written, and the headline configuration was selected on the same benchmark used to report the main result.","major_comments":[{"comment":"Algorithm 1 is not self-contained: the functions fp(b) and fv(b), which map a base to 'peak positions' and 'valley positions in attention waveforms', are defined only by name in lines 1-2. The paper gives no formula, no specification of which attention scores are used (initial model, theoretical RoPE decay, or trained attention weights), and no exact pointer to the definition in Chen et al. (2024b). Since Table 2, the stride-30k base set, and therefore every HARPE result depend on this search, a reader cannot reproduce the central configuration. Please provide explicit definitions of fp and fv, or quote the exact equations from the cited paper and state any adaptations.","section":"Sec. 3.2, Algorithm 1"},{"comment":"The claim that HARPE 'consistently matches or surpasses' multi-stage methods rests on a single ABF Multi-Stage schedule (1m/32k, 2m/64k, 5m/128k, 6B tokens). Table 1 shows that the same three-stage recipe yields 67.83 or 81.36 on the upgraded NiaH test depending only on how tokens are scheduled, so one schedule does not establish the upper envelope of multi-stage training. A tuned multi-stage pipeline with different stage lengths, token allocations, or stage counts could shrink or erase the reported 5.46% margin, and the production models cited in Section 1 use exactly such tuned schedules. Please compare HARPE against additional multi-stage schedules or restrict the claim to the tested baseline.","section":"Sec. 4.1, Table 1 and Table 4"},{"comment":"The stride-30k peak-valley configuration was selected because it achieved the highest average score on the upgraded NiaH test (86.82 in Table 6), and the same benchmark and same configuration then produce the headline result in Table 4 (86.82 vs. 81.36). This selection-on-evaluation makes the reported margin optimistically biased; adjacent configurations in Table 6 differ by up to 3.14 points. Please validate the chosen configuration on a benchmark that was not used for hyperparameter selection, or explicitly report the selection procedure and quantify its bias, for example with a validation split or nested evaluation.","section":"Sec. 5.2, Table 6; Sec. 5.1, Table 4"},{"comment":"No run-to-run variance is reported for any experiment. For the main 'matches or surpasses' claim, the PPL differences are tiny (Table 3: 3.02 vs. 3.03 on Proof-pile; 3.54 vs. 3.57 on GovReport) and the short-context average differences are also small (Table 5: 62.75 vs. 62.69). With a single run, these differences may be within normal noise for 7B continual pretraining. Please provide at least three seeds, or error bars, or a statistical analysis for the central comparisons.","section":"Sec. 4.4 and Tables 3-5"}],"minor_comments":[{"comment":"There are typos in the abstract and body: 'pertaining' should be 'pretraining' in the abstract, 'revist' appears in Section 3.1, and 'Adjused' appears in Section 1; please copyedit throughout.","section":"Abstract and Section 1"},{"comment":"Section 2 contains an unresolved citation '?' after (Cai et al., 2024); please replace it with a proper reference or delete it.","section":"Section 2"},{"comment":"Since Llama2-7B has 32 attention heads per layer and many layers, state explicitly whether the 32 bases in Table 2 are reused in every layer or assigned per head across all layers.","section":"Section 3.2 and Table 2"},{"comment":"Define what 'Uniform 2B Tokens' and 'Carefully Selected' mean (per-stage token allocation, data mixture, or both) and state whether the 'Carefully Selected' pipeline is the configuration used for the ABF Multi-Stage results in Tables 3-5.","section":"Table 1"},{"comment":"The rows labeled 'same stride ascending order' and 'descending order' are described in the text as uniform-distribution experiments, but the table label is unclear; specify which stride (30k?) and which ordering refers to.","section":"Section 5.2 and Table 6"},{"comment":"Specify how the reported PPL values are aggregated over the 2k-128k lengths and document the sliding-window settings (window size, stride, and whether overlapping windows are used) precisely.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper would be materially strengthened by releasing code for Algorithm 1 with the definitions of fp and fv, and by showing that the 5.46% margin survives comparison with stronger or differently scheduled multi-stage baselines. The relationship to Chen et al. (2024b) should be clarified in the final version: if the peak-valley search is adopted verbatim, the novelty rests on single-stage per-head training rather than on the base selection method, and the text should make that explicit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper's core claim is that per-head RoPE base values let you skip multi-stage continual pretraining: train Llama2-7B for 6B tokens directly at 128k and match or beat a three-stage ABF schedule. That is a practically useful result if it holds, and the paper does the right thing by keeping compute fixed and comparing on PPL, NiaH, short-context, and RULER. The NiaH advantage (86.82 vs 81.36) is not tiny, and the short-context numbers don't collapse. The two ingredients—per-head bases and peak-valley base search—are inherited from cited papers, but the training-time single-stage application is new as far as I can tell. Give credit for that.\n\nNow the soft spots. The stress-test is on target. The multi-stage ABF baseline is one hand-picked schedule (1m/32k, 2m/64k, 5m/128k), and the paper's own Tab 1 shows that exact recipe swings from 67.83 to 81.36 just by changing data scheduling. So the 5.46% margin over that baseline is not yet evidence against multi-stage in general. A tuned multi-stage baseline, or a search over stage lengths and token allocations, could shrink it. Also, stride=30k was chosen because it scored best on the upgraded NiaH test (Tab 6), and that same test produces the headline (Tab 4). That's a selection-on-results concern, not a fatal one, but it should be disclosed and ideally corrected with a held-out benchmark.\n\nReproducibility is the bigger issue. Algorithm 1's fp and fv are never defined, so no one can regenerate the base set. There are no seeds or error bars, and some of the differences (Tab 3 PPL, Tab 5 short-context) are at noise level. No code or models are released. These are addressable, and the central finding—HARPE is competitive with a reasonable three-stage ABF recipe—looks defensible. The limitations section only mentions SFT unexplored; it should also mention the baseline and selection issues.\n\nWho is this for? Practitioners doing long-context continual pretraining, and researchers working on RoPE-based extension. It deserves a serious referee: the question is important, the experiment is budget-matched, and the result is plausible. I would ask for a baseline search, error bars, and a reproducible base-search algorithm before accepting. Not a desk reject.","headline":"A plausible and budget-matched single-stage long-context recipe whose headline margin is partly fitted to the test benchmark; worth a serious referee but needs a stronger baseline, error bars, and a reproducible base search.","tokens_in":16943,"tokens_out":2421,"would_cite":true,"duration_ms":21621,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single training stage reaches 128k context by giving each attention head its own RoPE base.","keywords":["long-context extension","RoPE base frequency","head-adaptive positional encoding","single-stage continual pretraining","multi-stage continual pretraining","needle-in-a-haystack","RULER benchmark","LLM context length"],"falsifier":"Reproduce Algorithm 1 with explicit peak and valley functions to see whether it yields the exact Table 2 base set; then, with the same 6B-token budget and data, compare HARPE against a multi-stage adjusted-base-frequency schedule that uses the paper's own carefully selected data ordering from Table 1. If that baseline reaches or exceeds the HARPE average of 86.82 on the upgraded needle-in-a-haystack test, the claim that a single stage breaks the stage barrier would fail.","tokens_in":15743,"feed_emoji":"🧠","tokens_out":9083,"duration_ms":79273,"temperature":0.7,"pith_summary":"LLMs trained on short contexts lose accuracy on long inputs, and the standard fix is to grow the context window through several expensive, hand-tuned training stages. This paper claims that the staged schedule can be collapsed into one stage: assign each attention head a different RoPE base frequency, run one continual-pretraining pass at the target 128k context, and the model learns long-context behavior that matches or beats staged training. If true, the result matters because it removes the need to design and tune intermediate context lengths, data mixes, and checkpoints. The reported evidence covers four benchmarks, with the largest gain on the upgraded needle-in-a-haystack test (86.82 average for HARPE versus 81.36 for the staged adjusted-base-frequency baseline).","feed_headline":"One training stage reaches 128k context with per-head RoPE bases","feed_subtitle":"A single continual-pretraining pass on Llama2-7B matches or beats staged schedules on long-context tests.","key_machinery":"The central mechanism is head-adaptive rotary position encoding (HARPE). In standard RoPE one base frequency controls the rotation angles for all attention heads; HARPE instead assigns each head its own base, effectively letting different heads specialize to different effective context lengths during a single training run. The main experiments use a peak–valley search (Algorithm 1) that selects bases whose attention-waveform valleys overlap other bases' peaks, producing the per-head bases in Table 2 (roughly 1M to 4.54M with a 30k search stride).","core_discovery":"The paper's central claim is that long-context capability does not require gradual context growth during continual pretraining. By giving each attention head its own rotary position encoding base value, chosen so that the peaks of one head's attention waveform fill the valleys of another's, a single 6B-token training run at 128k context equips Llama2-7B with long-context modeling that the authors report as outperforming the multi-stage adjusted-base-frequency baseline by 5.46% on the upgraded needle-in-a-haystack test, while preserving short-context performance on MMLU, HellaSwag, ARC-C, PIQA, and TriviaQA.","pith_inferences":["Editorial inference: if the result holds at larger scales, per-head RoPE bases could turn long-context extension from a scheduling problem into a base-set search problem, which is easier to automate.","Editorial inference: the peak–valley complementarity idea predicts that small-base heads handle short-range dependencies and large-base heads handle long-range ones; ablating either end of the base range should degrade the corresponding length regime.","Editorial inference: the 5.46% margin over the staged baseline may partly reflect the baseline's data schedule, so a fair cross-check would compare HARPE against production-style multi-stage schedules with matched token budgets.","Editorial inference: the paper leaves supervised fine-tuning and other later training stages unexplored; whether per-head bases survive those stages or need to be reset to a single base is an open question."],"forward_implications":["The training pipeline reduces to one stage: choose a per-head base set, run 6B tokens at 128k context, and evaluate.","Long-context gains do not have to come at the cost of short-context accuracy; the reported averages on five short-context tasks are comparable to or slightly better than the baselines.","The approach changes only the RoPE base per head during training, adding no new modules and no inference-time machinery."],"supporting_citations":[{"why":"Defines RoPE and the base-frequency parameter that HARPE varies per head.","marker":"(Su et al., 2024a)"},{"why":"Supplies the adjusted-base-frequency method and the multi-stage baseline that HARPE is compared against.","marker":"(Xiong et al., 2023)"},{"why":"Contributes the peak–valley search idea that HARPE adapts for base selection.","marker":"(Chen et al., 2024b)"},{"why":"Provides YaRN, a single-stage method used as a baseline and whose sliding-window perplexity setup the paper follows.","marker":"(Peng et al., 2023)"},{"why":"Defines the RULER benchmark and the upgraded multi-key, multi-value, and multi-query needle-in-a-haystack tasks used for the headline comparison.","marker":"(Hsieh et al., 2024)"},{"why":"Supplies the upsampled training dataset and training configuration used for all continual pretraining runs.","marker":"(Fu et al., 2024b)"},{"why":"Releases Llama2-7B, the base model used for all experiments.","marker":"(Touvron et al., 2023b)"},{"why":"Defines positional interpolation, the PI baseline that HARPE outperforms.","marker":"(Chen et al., 2023)"}],"fun_headline_variants":["Single-stage training matches multi-stage long context","Per-head RoPE bases enable one-pass 128k context","HARPE: break the stage barrier in long context","One training stage for long context, no gradual growth"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed advantage over multi-stage training rests on the paper's three-stage adjusted-base-frequency baseline being a representative multi-stage schedule, and on the reported per-head base values being reproducible from the search algorithm.","fun_headline_variants_meta":{"raw":{"variants":["Single-stage training matches multi-stage long context","Per-head RoPE bases enable one-pass 128k context","HARPE: break the stage barrier in long context","One training stage for long context, no gradual growth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1313,"prompt_tokens":882,"completion_tokens":431,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":367}},"tokens_in":498,"tokens_out":431,"duration_ms":4888,"temperature":1.0,"reasoning_tokens":367,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:06:21.807075+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce Algorithm 1 with explicit peak and valley functions to see whether it yields the exact Table 2 base set; then, with the same 6B-token budget and data, compare HARPE against a multi-stage adjusted-base-frequency schedule that uses the paper's own carefully selected data ordering from Table 1. If that baseline reaches or exceeds the HARPE average of 86.82 on the upgraded needle-in-a-haystack test, the claim that a single stage breaks the stage barrier would fail.","supporting_citations":[],"review_version":1}