{"id":"9d1908c6-f9f6-4a48-9208-82950f97e3c9","arxiv_id":"2504.12637","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A hierarchical synthetic question-answer pipeline extended a Llama-3.1-8B model to a one-million-token context with strong RULER and InfiniteBench scores and small general-task regression.","lead":"This paper shows how to create long-context training data for language models by having a short-context model summarize chunks of books and write questions about them, then stitching many documents together. The authors fine-tuned an 8-billion-parameter model to accept up to a million tokens and report strong long-context benchmark scores with only modest drops on general tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training data contains no QA requiring synthesis across spans beyond ~16K tokens, so 1M-token RULER/InfiniteBench gains may reflect local retrieval and RoPE scaling rather than true long-range reasoning.","rationale":"The reader's weakest assumption identifies essentially the same load-bearing risk: QA pairs generated from chunk summaries and local contexts may teach the model to exploit local cues instead of performing genuine long-range reasoning. I agree with that diagnosis and sharpen it with a concrete, checkable consequence. Section 3.1 states chunks are typically 4K tokens and multi-hop questions span 2-4 chunks, so no training example demands synthesis across more than ~16K tokens. The 1M model is therefore never trained on the behavior the headline claim asserts. In addition, the paper's own footnote in Section 4.2 says the 650K and 1M models are undertrained and that the observed drop is likely due to multi-node communication overhead; this is an explicit limitation that should be weighed against the 1M claim. The paper does have real independent support: systematic ablations at 100K/180K, robustness across three generator models, and reproducible data-generation code. These support the data-generation strategy at moderate lengths, but they do not close the gap between 'good retrieval over long inputs' and '1M-token reasoning.' A targeted cross-100K synthesis evaluation would settle whether the concern lands, and the verdict should remain conditional until that test is run.","tokens_in":20432,"tokens_out":3653,"duration_ms":41323,"concrete_test":"Construct a synthetic evaluation set in the style of RULER where each question requires combining two facts whose source sentences are separated by at least 100K tokens, with the question unanswerable from either local neighborhood or any summary. Run it on the 1M model, the 350K model, and the zero-shot RoPE baseline. If accuracy on these cross-100K synthesis questions is near baseline while standard RULER scores remain high, the headline claim loses support; if the model solves them, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the hierarchical synthetic data pipeline plus stepwise RoPE scaling yields a model that 'performs well' at 1M-token contexts. Section 3.1 and Algorithm 1 generate every QA from a single 4K chunk, a local summary, or 2-4 chunks within a 12K medium section; multi-hop questions therefore span at most roughly 16K tokens. The 1M training context is assembled by concatenating documents, but each generated question's answer is recoverable from one local segment or summary. Nothing in the training data requires the model to bind evidence separated by 100K+ tokens. The RULER and InfiniteBench gains could accordingly reflect improved local search and attention over long inputs, not integration of distant context. The authors' own footnote in Section 4.2 that the 650K and 1M models are undertrained, with results dropping due to multi-node training overhead, compounds this: the flagship 1M numbers come from an incompletely trained model, and no evaluation isolates cross-100K synthesis. This is not an internal inconsistency, but it is a gap between the evidence and the load-bearing 1M-understanding claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a post-training pipeline that uses short-context LLMs to generate long-context instruction-tuning data hierarchically. Documents are split into small and medium chunks, summarized, and used to produce hierarchical, multi-hop, and diverse QA pairs; multiple documents are then concatenated to reach 180K, 350K, 650K, and 1M token training contexts. The authors fine-tune LLaMA-3.1-8B-Instruct with stepwise RoPE scaling on this data and report results on RULER, InfiniteBench, LongBench, and MMLU, claiming strong performance at up to 1M tokens while preserving general abilities. They also include ablations at 100K and 180K context lengths and comparisons against a Gradient AI 1M-context model.","tokens_in":20643,"tokens_out":8255,"duration_ms":77826,"significance":"If the empirical claims hold, the work would be a useful contribution to long-context instruction tuning: it provides a scalable synthetic-data recipe, systematic ablations of data composition choices, and evidence that data quality matters beyond positional-embedding scaling alone. The paper's strengths include explicit ablations at 100K and 180K, validation with multiple generator models, and an attempt to preserve short-context performance. However, the headline 1M-token claim is not yet established: the training data never requires cross-evidence synthesis beyond roughly 16K tokens, the 1M model is acknowledged to be undertrained, RULER comparisons lack error bars, and the main baseline comparison is confounded by different base models. These issues are addressable and do not invalidate the core data-generation idea, but they currently prevent a strong endorsement of the central claim.","major_comments":[{"comment":"The QA supervision generated by the pipeline never requires the model to integrate evidence separated by more than roughly 16K tokens: every question is produced from a 4K chunk, a summary, or 2-4 chunks within a 12K medium section (Algorithm 1; Section 3.1), and the multi-document examples concatenate documents without creating cross-document reasoning targets beyond revisiting prior documents (Section 3.2). As a result, the RULER and InfiniteBench improvements at 1M could reflect improved local retrieval and attention over long inputs rather than genuine synthesis across distant evidence. Please add an evaluation that explicitly manipulates the distance between evidence spans (e.g., multi-hop RULER tasks with needles 100K+ tokens apart) or otherwise demonstrate that the model can bind distant evidence.","section":"Section 3.1, Algorithm 1, Appendix C"},{"comment":"The footnote in Section 4.2 states that the 650K and 1M models are undertrained and that results dropped due to multi-node training overhead. The flagship 1M RULER numbers in Table 8 and Figure 4 are therefore produced by a model that did not complete training. This is load-bearing: the main contribution is the 1M-context result, and the paper does not show what a fully trained model would achieve. Please retrain to convergence (or address the training instability) and report results, or explicitly reframe the 1M results as preliminary and support them with an analysis of training dynamics.","section":"Section 4.2, footnote"},{"comment":"The paper reports a single point estimate for each model/task, and the RULER numbers in Table 8 have no error bars or repeated-generation variance, although the tasks are sampled from a generative benchmark. Many of the central comparisons are small differences (e.g., Table 1: 180K 59.45 vs 350K 59.26; Table 7: h-h-s-fixed 58.58 vs h-h-h-randomized 58.14), and 'significantly exceeding' in Section 4.2 is not backed by any statistical test. Please provide variance estimates, seeds, and significance tests for the reported differences.","section":"Tables 1-8"},{"comment":"The comparison against gradientai/Llama-3-8B-Instruct-Gradient-1048k in Tables 1-6 is confounded with base model: the Gradient model starts from Llama-3-8B, whereas all proposed models start from LLaMA-3.1-8B-Instruct, which is substantially stronger on MMLU (68.21 vs 60.48 in Table 3). The claim that the method 'outperforms' the Gradient model is therefore not an apples-to-apples comparison. A matched evaluation using the same base model (or a Llama-3.1-based 1M baseline) is needed.","section":"Tables 1, 2, 3; Section 4.2"},{"comment":"Section 4.2 describes 'stepwise rope scaling' but does not specify the scaling method, factors, schedule, or the exact training recipe (e.g., whether NTK-aware or YaRN interpolation was used, how the RoPE base frequency was changed at each stage). Since the result depends on this recipe as much as on the synthetic data, the missing details prevent reproduction and make it impossible to attribute the gains to the data pipeline. Please provide the full configuration (or point to released code) in the main text or appendix.","section":"Section 4.2"}],"minor_comments":[{"comment":"The text 'hs-hs-fixed-no-sum scored 58.03, slightly below hs-hs-hs-fixed (58.58)' does not match Table 7, which shows h-h-s-fixed-no-sum at 58.03 and h-h-s-fixed at 58.58, while hs-hs-hs-fixed is 59.45. Please correct the configuration labels and numbers.","section":"Section 4.4, Table 7"},{"comment":"The RULER evaluation description says 130 tasks per context length were sampled, but no seed or version of RULER is given; please report the exact configuration for reproducibility.","section":"Section 4.1"},{"comment":"The claim that no open-source instruction-tuning dataset exceeds 100K tokens is not accompanied by a statement of whether the generated 180K/350K/650K/1M dataset will be released; the reproducibility section only provides code. Please clarify the release status.","section":"Abstract and Introduction"},{"comment":"'Significantly exceeding' is used for differences without statistical tests; please either add tests or use descriptive language.","section":"Section 4.2"},{"comment":"The RULER curves for Llama-3.1-8B and Qwen-2.5-7B generators are not accompanied by the numerical table that exists for the Qwen-2-72B generator (Table 8); please add the analogous numbers for reproducibility.","section":"Section 4.3, Figure 5"},{"comment":"There are minor typographical errors such as 'Table 2, , our models' in Section 4.2 and '4 hierachies' in Appendix E; please proofread the manuscript.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"For the editor: The paper's main empirical claim is plausible but currently overstated; the RULER and Gradient comparisons need strengthening, and the 1M result is weakened by the authors' own undertraining statement. The paper is nevertheless suited to the long-context community if the revision addresses the load-bearing issues. I would not recommend rejection because the core data-generation pipeline and ablation studies are useful and the issues are addressable in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this paper has a genuinely useful, well-ablated recipe for turning short-context LLMs into generators of long-context instruction data. Second, the headline claim that this yields a model that \"performs well\" at 1M tokens is undercut by both the data construction and the authors' own training note.\n\nThe real contribution is the hierarchical ordering plus diverse question generation, validated by systematic ablations. The 180K two-document ablation (Table 7) is convincing: fixed hierarchical + diverse questions score 59.45 on InfiniteBench versus 51.31 for the base LLaMA-3.1-8B-Instruct. The 100K ablations (Table 11) also show multi-hop and diverse questions each help. Not many open-source efforts do this level of ablation, and the robustness check with smaller generators (7B, 8B) is a plus.\n\nNow the soft spots, in proportion. The stress-test concern is correct: the generated QA pairs never require synthesis across spans beyond about 16K tokens. Every question is answered from a 4K chunk, a local summary, or 2–4 chunks within a 12K medium section. The 1M training context is a concatenation of documents, but each individual question can be answered by finding one local segment. So the measured RULER/InfiniteBench gains at 1M could reflect improved local search plus RoPE scaling, not genuine cross-100K reasoning. The paper does not isolate long-range synthesis, and the abstract's \"performs well\" overstates what the evidence shows.\n\nSecond, the authors' own footnote in Section 4.2 states that the 650K and 1M models are undertrained due to multi-node training overhead. That is the model used for the flagship 1M numbers. This is not a minor caveat; it means the headline result comes from an incomplete training run.\n\nThird, the comparison to the Gradient AI model is confounded: that model is based on Llama-3-8B, a weaker base than Llama-3.1-8B, and was trained on 1.4B tokens with a different recipe. The comparison tells you little about the method's relative merit.\n\nFourth, RULER results have no error bars, and the claimed open dataset is not released — only code to generate it. That matters because the abstract explicitly mentions the lack of openly available >100K instruction data.\n\nWho is this for? People working on context extension or synthetic instruction data will find the ablations and the multi-document concatenation recipe valuable. The method is likely to be useful at 180K–350K, where the evidence is decent. The 1M claim needs targeted evaluation — for example, generating questions that require binding evidence from distant parts of the context, and training the model to completion.\n\nMy recommendation: this deserves serious peer review, but with major revision. The reviewers should push for the 1M claim to be either softened or backed by evals that force long-range synthesis, and for the Gradient comparison to be redone with a matched base model.","headline":"Solid, well-ablated synthetic-data recipe for long-context instruction tuning, but the 1M-token understanding claim is not supported by the training data or the model's own undertrained state.","tokens_in":21190,"tokens_out":2117,"would_cite":true,"duration_ms":23043,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hierarchical synthetic-data pipeline plus stepwise RoPE scaling extends LLaMA-3.1-8B-Instruct to a 1M-token context window, beating prior baselines on long-context benchmarks with minimal loss on general tasks.","keywords":["long-context language models","synthetic data generation","instruction tuning","hierarchical question-answer generation","RoPE scaling","million-token context","RULER benchmark","multi-document concatenation"],"falsifier":"Build a probe set of 1M-token examples where the answer depends on facts in two chunks separated by more than half the context, with no intermediate summary containing either fact. If the trained model's accuracy on such bridge-gap questions collapses at 1M while its accuracy on questions answerable from a single chunk stays high, the RULER-style gains reflect local or summary reliance rather than true long-range integration. The same probe run on the 350K model would show whether the degradation appears as context grows.","tokens_in":20224,"feed_emoji":"📚","tokens_out":7502,"duration_ms":63595,"temperature":0.7,"pith_summary":"The paper argues that the bottleneck for long-context language models is not architecture but training data: high-quality instruction pairs with truly long contexts are scarce and expensive to annotate. It proposes generating such data synthetically by splitting documents into chunks, summarizing each chunk, and composing question-answer pairs that range from single-chunk details to multi-hop and cross-document synthesis. Combined with stepwise rotation-scaling of position embeddings (180K to 350K to 650K to 1M tokens), the pipeline extends LLaMA-3.1-8B-Instruct to a 1M-token window. On RULER the trained model keeps meaningful accuracy out to 1M tokens, it beats both the untuned base model and a 1M-capable baseline on InfiniteBench, and its LongBench and MMLU scores stay close to the original model's. If correct, the result means million-token instruction tuning no longer requires million-token human annotations.","feed_headline":"Synthetic data pushes an 8B model to a 1M-token context","feed_subtitle":"Chunk-level summaries plus stepwise position scaling keep long-context gains without wrecking general-task scores.","key_machinery":"The load-bearing mechanism is the two-level hierarchical data generator: a document is split into small (4K-token) and medium (12K-token) chunks, each is summarized by an off-the-shelf short-context LLM, and Algorithm 1 walks question generation through the resulting hierarchy, starting from the whole-document summary, then a random medium chunk, then a small chunk inside it, then alternating between deeper dives into the current chunk and moves to new chunks, so the generated conversation mirrors the document's global-to-local structure. Running alongside this are hierarchical-aware, multi-hop, and specific-detail question prompts, and when several books are concatenated, revisits to earlier documents with a 60 percent probability create cross-document questions. The second mechanism is stepwise RoPE scaling: the model is fine-tuned successively at 180K, 350K, 650K, and 1M tokens, combining the hierarchical data at each length with rotary position embedding rescaling in the spirit of positional interpolation and NTK-aware scaling.","core_discovery":"The central claim is that a locality-guided, hierarchy-aware synthetic data strategy, rather than raw long text, is what teaches an instruction-tuned model to reason across very long contexts. The authors split each document into 4K-token chunks nested inside 12K-token medium sections, summarize every level, and walk a question generator from the global summary down to specific chunks and back, alternating exploitation of the current chunk with exploration of new ones; multi-hop questions stitch 2-4 chunks together, and multiple documents are concatenated, with probabilistic revisits to earlier documents, to reach arbitrarily long contexts. Training proceeds in stages of increasing context length with scaled rotary position embeddings. The 1M-token model scores 62.95 percent on RULER at the full 1M length, against 48.81 percent for zero-shot RoPE scaling of the base model, averages 54.80 on InfiniteBench versus 51.31 for the base model and 41.04 for a 1M-capable comparative model, and lands within about three MMLU points of the base model. The authors themselves note that the 650K and 1M models appear undertrained because of multi-node communication overhead, which they believe depressed those results.","pith_inferences":["The ablations leave open a sharper test the paper does not run: whether the gains persist when questions require integrating facts from widely separated chunks with no summarizing bridge. A natural experiment is to adversarially construct 1M-token questions whose answers are spread across non-adjacent documents and compare accuracy against same-length questions answerable from a single chunk.","The hierarchical summaries act as a learned compression bottleneck; one could test the same data pipeline on models with different position encodings, such as ALiBi, to see whether the data itself, rather than the RoPE rescaling, is doing the work.","The paper's own note that the 650K and 1M models are undertrained suggests the reported numbers may be a floor, and retraining with better parallelism or more epochs could raise the headline RULER and InfiniteBench scores.","Because the pipeline is data-side, it composes with architectural advances; feeding the same hierarchical data into linear-attention or state-space models would test whether data scarcity or attention cost dominates at very long context."],"forward_implications":["Million-token instruction datasets can be built by concatenating ordinary books with LLM-generated hierarchical QA pairs, removing the need for expensive long-context human annotations.","The stepwise RoPE schedule suffices to push an off-the-shelf 8B instruct model to a 1M window using only a few hundred 1M-token samples at the final stage (200 in this paper).","Models trained this way keep most of their short-context ability, with LongBench staying within roughly 1.7 points and MMLU within roughly 3 points of the base 8B model at 1M context.","Smaller or same-size generator models (7B and 8B) produce data that transfers nearly as well as data from a 72B generator, suggesting the pipeline does not depend on a top-tier generator."],"supporting_citations":[{"why":"Supplies RULER, the synthetic long-context benchmark whose effective-context-length curves are the paper's primary evidence of 1M-token capability.","marker":"(Hsieh et al., 2024)"},{"why":"Supplies InfiniteBench, the 100K-plus benchmark where the trained models beat both the base model and the comparative 1M model.","marker":"(Zhang et al., 2024)"},{"why":"Supplies LongBench, the short-context benchmark used to show general task performance is preserved after long-context training.","marker":"(Bai et al., 2024)"},{"why":"Supplies MMLU, the general-knowledge benchmark showing only modest regression from the base model.","marker":"(Hendrycks et al., 2021)"},{"why":"Defines RoPE, the rotary position embedding whose stepwise scaling is the training backbone of the method.","marker":"(Su et al., 2023)"},{"why":"Positional interpolation is the base rescaling technique that the stepwise RoPE training schedule extends.","marker":"(Chen et al., 2023)"},{"why":"NTK-aware RoPE scaling is the other extrapolation method underlying the stepwise context-extension schedule.","marker":"(Peng et al., 2023)"},{"why":"Self-Instruct is the synthetic instruction-generation lineage that the paper's QA-generation prompts extend from short to long context.","marker":"(Wang et al., 2023)"}],"fun_headline_variants":["1M-token context via hierarchical synthetic data generation","Chunk-summary training pushes LLM to million-token context","Stepwise position scaling enables 1M-token LLM context","Synthetic data hierarchy scales instruction-tuned LLM to 1M"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole 1M-token claim rests on the assumption that question-answer pairs built from chunk summaries and local excerpts teach the model to genuinely integrate distant context, rather than to answer from whichever nearby summary or local passage holds the answer.","fun_headline_variants_meta":{"raw":{"variants":["1M-token context via hierarchical synthetic data generation","Chunk-summary training pushes LLM to million-token context","Stepwise position scaling enables 1M-token LLM context","Synthetic data hierarchy scales instruction-tuned LLM to 1M"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1411,"prompt_tokens":973,"completion_tokens":438,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":365}},"tokens_in":589,"tokens_out":438,"duration_ms":4880,"temperature":1.0,"reasoning_tokens":365,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:26:22.474302+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a probe set of 1M-token examples where the answer depends on facts in two chunks separated by more than half the context, with no intermediate summary containing either fact. If the trained model's accuracy on such bridge-gap questions collapses at 1M while its accuracy on questions answerable from a single chunk stays high, the RULER-style gains reflect local or summary reliance rather than true long-range integration. The same probe run on the 350K model would show whether the degradation appears as context grows.","supporting_citations":[],"review_version":1}