Pith. sign in

REVIEW 2 major objections 7 minor 1 cited by

LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions

T0 review · 2 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Aligned LLMs can write their own long-context questions, and training on those self-written questions beats the existing long-context instruction datasets.

desk verdict Solid Magpie extension for long-context SFT data, with a fair head-to-head setup; the missing decontamination check and single-run results are the real caveats, not the method. read the letter →

arxiv 2505.17134 v2 pith:JFU4VDSQ submitted 2025-05-22 cs.CL cs.AI

Chaochen Gao , Xing Wu , Zijia Lin , Debing Zhang , Songlin Hu This is my paper
classification cs.CLcs.AI
keywords long-contextinstructiondataself-synthesistuningdocument-querygenerationp-Mixmulti-documentreasoningalignmentsynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LongMagpie aims to show that high-quality long-context instruction data can be synthesized without human annotators, predefined templates, or seed questions. The method feeds an aligned long-context LLM a document followed by the special tokens that normally precede a user turn, and the model auto-regressively continues with a contextually relevant question. The authors then generate an answer, filter the queries, and optionally mix in extra documents to create harder multi-document tasks. Training Llama-3-8B on this self-generated data tops HELMET, RULER, and LongBench-v2 within the long-instruction-data comparison, and the proposed p-Mix schedule keeps short-context performance competitive. If the result holds, long-context alignment data, currently proprietary, can be produced openly at scale and at much lower cost.

What carries the argument

The load-bearing mechanism is auto-regressive document-query generation: for a document $D$ and a pre-query template $T_{pre}$ containing the user-turn tokens (such as `\<|im_start|>user`), an aligned LLM $M$ generates a query $Q$ by sampling from $p_M(Q \mid D, T_{pre}) = \prod_{i=1}^{k} p_M(q_i \mid D, T_{pre}, q_{<i})$. This lets LongMagpie harvest document-query pairs without prompting or annotation. The second mechanism is p-Mix, which begins each training sequence with a short instruction and then probabilistically appends long or short segments up to length $L_{max}$, balancing long- and short-context capabilities.

What would settle it

Generate queries with an untuned base model (no instruction tuning) using the same documents and user-turn tokens; after the question-mark and length filters, count how many usable queries survive. If the yield is near zero, the method depends on instruction-tuning's internalized document-query patterns as claimed; if the yield is high and the resulting fine-tuned model still reaches the reported benchmarks, then those patterns are not the operative mechanism.

Watch

Extended reading notes

Core claim

The central discovery is that instruction-tuned long-context LLMs internalize document-query relationship patterns, so when they are given a document and the start of a user turn, they will generate queries about that document as their natural continuation. LongMagpie harvests these document-query pairs, adds model-generated responses, filters out non-interrogative continuations, and then optionally concatenates random extra documents to create multi-document reasoning tasks. Models trained on the resulting data achieve the best long-context scores among the compared instruction datasets, and the p-Mix strategy maintains competitive short-context accuracy by starting each training sequence with a short instruction and then probabilistically mixing long and short segments.

Load-bearing premise

The load-bearing premise is that an aligned long-context LLM, given a document plus the tokens that open a user turn, will reliably produce a relevant question rather than continue the document; the paper's own filtering step shows this is not automatic, especially for smaller generators.

Editorial extensions

If this is right

  • Long-context instruction data can be generated at scale from any sufficiently aligned long-context LLM, removing the need for human reading of thousands of tokens to write questions.
  • The p-Mix schedule lets a model trained mostly on long-context data retain competitive accuracy on standard short-instruction benchmarks, so long-context alignment need not trade away general instruction following.
  • Mixing extra random documents into the prompt (up to about ten) makes the synthesized tasks harder and improves downstream long-context scores, while too many documents hurts performance.
  • Larger generator models and larger synthesized datasets both improve downstream long-context performance, so the method's value scales with the generator and the compute available for synthesis.
  • The query-style, reward-score, and embedding analyses indicate the generated queries are more diverse and higher-scored than those from template-based pipelines, supporting the claim that self-synthesis is not just cheaper but also higher quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the generation trigger is just the model's own chat template, LongMagpie should transfer to any newly released instruction-tuned model without retraining the pipeline; testing on a multilingual or code-specialized model would show whether the document-query pattern generalizes across domains.
  • Beyond the paper: the question-mark and length filters likely bias the dataset toward short, factoid questions; the authors' own limitation section notes long-form outputs are undercovered, so a natural extension is to add continuation-style tasks that require extended reasoning.
  • Beyond the paper: the multi-document setting with randomly sampled negative documents implicitly creates retrieval-style training examples; this suggests LongMagpie data could double as a resource for RAG fine-tuning, a use the paper does not evaluate.
  • Beyond the paper: the reward-model and embedding analyses compare LongMagpie to other synthetic datasets, not to human-authored long-context data; a direct human-vs-synthetic benchmark would sharpen the claim that the method is a true substitute for annotation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper introduces LongMagpie, a self-synthesis method for creating long-context instruction data. The key idea is to feed a document to an aligned long-context LLM followed by the special tokens that normally precede a user turn, and then harvest the auto-regressively generated query and the model's answer. The method is extended to multi-document settings by randomly adding distractor documents, and a p-Mix strategy is proposed to interleave short and long instruction segments during fine-tuning. Experiments use Llama-3-8B-NExtLong-512K-Base with a fixed 1B-token training budget, and report that LongMagpie-trained models achieve the best scores in the compared set on HELMET (62.10), RULER (91.17), and LongBench-v2 (34.4), with ablations over the number of documents, mixing strategy, data volume, source model size, and data quality/diversity.

Significance. If the empirical claims hold, LongMagpie is a genuinely useful contribution: it offers a simple, scalable, and low-cost way to synthesize long-context instruction data without human annotation or seed questions, and the p-Mix strategy addresses a real problem of short-context degradation. The paper has concrete strengths: same base model, same token budget, and same training configuration are used across all compared datasets; evaluation is on external benchmarks (HELMET, RULER, LongBench-v2); and the code is released. The reproduction of the pipeline is straightforward. However, the headline 'leading performance' claim is currently not fully established because the paper does not check for overlap between its FineWeb-Edu source corpus and the real-document evaluation benchmarks, and because all results come from single training runs without variance estimates.

major comments (2)
  1. [Section 3.1 / 3.1.2] The paper reports no decontamination analysis between the FineWeb-Edu documents used for LongMagpie data generation and the HELMET and LongBench-v2 evaluation documents. Since the method trains on queries and answers derived from the same source documents that may also appear in these real-document benchmarks, exact or near-duplicate overlap could inflate the reported gains over ChatQA and LongAlign (e.g., +1.87 on HELMET, +3.6 on LongBench-v2). This is load-bearing for the central claim stated in Section 3.2. The authors should add an overlap analysis (exact-match, substring, or embedding-similarity based), rerun the main comparison after removing overlapping documents, or otherwise demonstrate that the observed gains are not an artifact of train/test contamination. RULER is synthetic and largely immune, but the LongA VG average includes HELMET and LongBench-v2, so contamination there is sufficient to affect the headline.
  2. [Table 1 / Section 3.2] All experimental results are single runs with no error bars, multiple seeds, or significance tests. Several of the differences that support the 'leading performance' and p-Mix claims are small: LongMagpie exceeds Tulu on HELMET by 0.17 (62.10 vs 61.93), and p-Mix exceeds Simple Mix on ShortA VG by only 0.06 (64.10 vs 64.04). Without an estimate of run-to-run variance, these differences are not statistically distinguishable. The authors should either report results across at least a few seeds for the main comparisons or provide a clear justification for treating the reported single-run numbers as stable.
minor comments (7)
  1. [Section 2.2.2] The multi-document construction is ambiguous: the text says x documents {D1,...,Dx} are sampled as negative documents and Dmulti is their concatenation, but then states that n=0 reduces to the single-document QA setting. If x=0, the concatenation would be empty, not a single document. Please clarify whether the query-bearing document is always included in addition to x distractor documents, and how the special separator is placed.
  2. [Section 2.2.1] The query filtering step (question-mark ending and <1.5k characters) is described but the paper does not report the fraction of generated queries that are discarded, nor whether the filter changes the task distribution in a way that affects the diversity or difficulty of the final data. A short statistics table or a sentence quantifying the filter rates would strengthen Section 4.5.
  3. [Section 4.6 / Figure 3c] The claim that LongMagpie processes 'averaging 1.6K tokens' per generated instruction is inconsistent with the multi-document extension, which adds distractor documents and can make arbitrarily long contexts. Please specify whether the 1.6K average is computed for the single-document setting only, and report the average token consumption for the actual multi-document data used in the main experiments.
  4. [Section 4] There is a typo in the introduction to Section 4: 'finally assesses the its resource efficiency' should be 'finally assesses its resource efficiency'.
  5. [Table 5] The header 'Data V olume' contains a typo; it should be 'Data Volume'.
  6. [References] Several entries in the reference list do not appear to be cited in the text (e.g., Azar et al. 2023, Chen et al. 2023a, Fu et al. 2024, Jin et al. 2024, Ouyang et al. 2022, Peng et al. 2023, Rafailov et al. 2023, Xiao et al. 2023, Zhu et al. 2023). Please either cite them where relevant or remove them.
  7. [Section 5.1] The related work mentions WildLong, LOGO, and GATEAU as representative long-context instruction synthesis methods, but Table 1 does not compare against any of them. Please clarify why these are excluded from the comparison, or temper the 'leading performance' language to reflect the set of baselines actually evaluated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the data-generation pipeline is externally evaluated, and self-citations are background rather than load-bearing.

full rationale

No circular step is present. The paper's core premise (Sec. 2.1) is that aligned long-context LLMs auto-regressively generate document-relevant queries when given a document followed by user-turn tokens; this is an empirical claim about model behavior, not a definition of the evaluation outcome. LongMagpie data is synthesized from FineWeb-Edu using Qwen2.5-70B-Instruct and used to fine-tune a fixed base model (Llama-3-8B-NExtLong-512K-Base), and the headline results are measured on external benchmarks (HELMET, RULER, LongBench-v2). No parameter of the synthesis pipeline is fitted to those benchmark test sets, and no evaluated quantity is defined in terms of the generated data itself. The n=10 multi-document setting in Sec. 4.1 is selected by a sweep on the same benchmarks, which is a benchmark-selection/overfitting concern rather than a circular derivation; it does not make the reported comparison equal to its input by construction. Self-citations to Quest, NExtLong, and Magpie appear as prior work or as the source of the base model, but the base model is held fixed across all compared datasets, so the dataset-level comparison is not forced by those citations. The central claim is therefore an externally falsifiable empirical result, not a renaming or re-derivation of its inputs.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a behavioral assumption about aligned models (document-query auto-regression), on hyperparameters tuned via benchmarks (PL, NS, n), and on the validity of three long-context benchmarks. No new physical entities are postulated.

free parameters (4)
  • PL (long-context selection probability in p-Mix) = 0.4
    Chosen from the ablation in Table 7 as best LongA VG with NS=1; tuned on the evaluation benchmarks.
  • NS (number of pre-pended short-context samples) = 1
    Selected from Table 7; the main result uses NS=1, which gives the best long-short balance.
  • n (max number of documents in multi-document setting) = 10
    Selected from Table 2 as the best LongA VG; a model-capacity-dependent choice.
  • Query filter thresholds (question-mark ending, length under 1.5k chars) = Ends with '?'; under 1500 characters
    Heuristic thresholds introduced in Section 2.2.1 to remove continuations and non-questions; they shape the generated task distribution.
assumptions (3)
  • domain assumption Aligned long-context LLMs auto-generate contextually relevant queries when given a document plus user-turn special tokens.
    Stated as the key insight in Section 2.1; all generated data depends on this behavior.
  • domain assumption The evaluation benchmarks (HELMET, RULER, LongBench-v2) are valid measurements of long-context capabilities.
    Used without questioning their validity; the central claim is defined by these benchmarks.
  • domain assumption Llama-3-8B-NExtLong-512K-Base is an appropriate testbed for comparing instruction data quality.
    All comparisons are made on this single base model, so results may not transfer to other base models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions." pith.science (2026). https://pith.science/paper/JFU4VDSQ

@misc{pith2026250517134,
  author       = {Pith},
  title        = {Pith review of: LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JFU4VDSQ}},
  note         = {Machine review of arXiv:2505.17134}
}
read the original abstract

High-quality long-context instruction data is essential for aligning long-context large language models (LLMs). Despite the public release of models like Qwen and Llama, their long-context instruction data remains proprietary. Human annotation is costly and challenging, while template-based synthesis methods limit scale, diversity, and quality. We introduce LongMagpie, a self-synthesis framework that automatically generates large-scale long-context instruction data. Our key insight is that aligned long-context LLMs, when presented with a document followed by special tokens preceding a user turn, auto-regressively generate contextually relevant queries. By harvesting these document-query pairs and the model's responses, LongMagpie produces high-quality instructions without human effort. Experiments on HELMET, RULER, and Longbench v2 demonstrate that LongMagpie achieves leading performance on long-context tasks while maintaining competitive performance on short-context tasks, establishing it as a simple and effective approach for open, diverse, and scalable long-context instruction data synthesis.

Figures

Figures reproduced from arXiv: 2505.17134 by the authors.

Figure 1
Figure 1. LongMagpie pipeline overview. Stage one: a document serves as a system prompt, a special [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Analysis of LongMagpie-generated data quality and diversity. (a) higher reward model [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Visualizations of LongMagpie characteristics: (a,b) t-SNE visualizations of query embed [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Impact of the multi-document setting on model performance. As the number of documents [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Libra: Large Chinese-based Safeguard for AI Content

    cs.AI 2025-07 conditional novelty 6.0 of 10

    Libra-Guard, a Chinese-language AI content safety model, reaches 86.8% accuracy on the new 5,720-sample Libra-Test benchmark, beating open-source baselines and approaching GPT-4o and Claude-3.5-Sonnet.

Reference graph

Works this paper leans on

44 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [3]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508,

    arXiv

  3. [4]

    Longalign: A recipe for long context alignment of large language models

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. Longalign: A recipe for long context alignment of large language models. arXiv preprint arXiv:2401.18058, 2024a. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longb...

    arXiv

  4. [5]

    Extending context window of large language models via positional interpolation

    Chi Han Chen, Sang Michael Liu, Tianyi Xiao, Zhilin Xu, Rameswar Panda, Xin Eric Xu, Yikang Ma, Kevin Xie Lin, Soheil Feizi, and Pradip Mittal. Extending context window of large language models via positional interpolation. In Proceedings of the 40th International Conference on Machine Learning, 2023a. 10 Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, ...

    arXiv

  5. [7]

    Ultrachat: A large-scale auto-generated data for diverse conversations with large language models

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Sheng Si, Yun Liu, Zhiyuan Zhang, Yu Wu, Chao Li, et al. Ultrachat: A large-scale auto-generated data for diverse conversations with large language models. arXiv preprint arXiv:2305.14233,

    arXiv

  6. [8]

    Rlhf workflow: From reward modeling to online rlhf

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863,

    arXiv

  7. [10]

    Quest: Query-centric data synthesis approach for long-context scaling of large language model

    Chaochen Gao, Xing Wu, Qi Fu, and Songlin Hu. Quest: Query-centric data synthesis approach for long-context scaling of large language model. arXiv preprint arXiv:2405.19846, 2024a. Chaochen Gao, Xing Wu, Zijia Lin, Debing Zhang, and Songlin Hu. Nextlong: Toward effective long-context training without long documents. arXiv preprint arXiv:2501.12766,

    arXiv

  8. [12]

    Gemini model updates: March 2025, March

    Google DeepMind. Gemini model updates: March 2025, March

    work page 2025

Show all 44 references
  1. [13]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  2. [14]

    Retrieval augmented language model pre-training

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Retrieval augmented language model pre-training. arXiv preprint arXiv:2002.08909,

    2002 arXiv

  3. [16]

    Unnatural instructions: Tuning language models with (almost) no human labor

    Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689,

    arXiv

  4. [18]

    Llm self-extension: Teaching large language models to use tools by generating canonical api calls

    Cheng Jin, Wenli Zhang, Aniruddha Li, Yang Zhang, Xiang Yuan, Wenjie Liu, Bin Zhao, Shiyu Chang, Yongfeng Zhang, and Yueting Zhuang. Llm self-extension: Teaching large language models to use tools by generating canonical api calls. arXiv preprint arXiv:2403.08233,

    arXiv

  5. [19]

    Longform: Optimizing instruction tuning for long text generation with corpus extraction

    Abdullatif Köksal, Timo Schick, Anna Korhonen, and Hinrich Schütze. Longform: Optimizing instruction tuning for long text generation with corpus extraction. In Findings of EMNLP 2024,

    2024

  6. [20]

    T \" ulu 3: Pushing frontiers in open language model post-training

    11 Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T \" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124,

    arXiv

  7. [21]

    Synthetic data (almost) from scratch: Generalized instruction tuning for language models

    Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si- Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. Synthetic data (almost) from scr...

    arXiv

  8. [23]

    Self-alignment with instruction backtranslation

    Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.06259, 2024b. Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet V ong, and "Tekn...

    arXiv

  9. [24]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. ...

    2007 arXiv

  10. [25]

    Chatqa: Building gpt-4 level conversational qa models

    Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa: Building gpt-4 level conversational qa models. CoRR, 2024b. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,

    arXiv

  11. [26]

    The lambada dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031,

    arXiv

  12. [28]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290,

    arXiv

  13. [29]

    Code llama: Open foundation models for code

    12 Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Liu, Denis Lebowitz, Piero Molino Ferrer, Tom Cochrane, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950,

    arXiv

  14. [31]

    In-context pretraining: Language modeling beyond document boundaries

    Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Gergely Szilvasy, Rich James, Xi Victoria Lin, Noah A Smith, Luke Zettlemoyer, et al. In-context pretraining: Language modeling beyond document boundaries. arXiv preprint arXiv:2310.10638,

    arXiv

  15. [32]

    Gateau: Selecting influential sample for long context alignment

    Shuzheng Si, Haozhe Zhao, Gang Chen, Yunshui Li, Kangyang Luo, Chuancheng Lv, Kaikai An, Fanchao Qi, Baobao Chang, and Maosong Sun. Gateau: Selecting influential sample for long context alignment. arXiv preprint arXiv:2410.15633,

  16. [33]

    Rui Sun, Zhiwei Sun, Yang Li, Yi Ren, and Wei Bi

    URL https: //arxiv.org/abs/2409.10173. Rui Sun, Zhiwei Sun, Yang Li, Yi Ren, and Wei Bi. Efficient training of ultra-long context large language models. arXiv preprint arXiv:2504.06214,

    arXiv

  17. [34]

    Logo–long context alignment via efficient preference optimization

    Zecheng Tang, Zechen Sun, Juntao Li, Qiaoming Zhu, and Min Zhang. Logo–long context alignment via efficient preference optimization. arXiv preprint arXiv:2410.18533,

    arXiv

  18. [35]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpaca: A strong, replicable instruction-following model. arXiv preprint arXiv:2303.16199,

    arXiv

  19. [36]

    Llama 3: Third-generation open foundation language models

    Llama team at Meta. Llama 3: Third-generation open foundation language models. arXiv preprint arXiv:2405.08456,

    arXiv

  20. [38]

    Self-instruct: Aligning language models with self-generated instructions

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022a. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa ...

    arXiv

  21. [39]

    Effective long-context scaling of foundation models

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039,

    arXiv

  22. [40]

    Wizardlm: Empowering large language models to follow complex instructions

    13 Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023a. Peng Xu, Wei Ping, Xianchao Wu, Chejian Xu, Zihan Liu, Moha...

    arXiv

  23. [41]

    Helmet: A hierarchical efficient benchmark for long context evaluation with modular units and comprehensive taxonomy

    Tse-Yu Yen, Tsu-Jui Cheng, Paul Pu Liang, Xiang Dong, Tianyu Zhao, Wenhan Liu, Wenhao Wang, Min Peng, Oleksiy Shliazhko, Li Zhang, et al. Helmet: A hierarchical efficient benchmark for long context evaluation with modular units and comprehensive taxonomy. arXiv preprint arXiv:...

    arXiv

  24. [42]

    Metamath: Bootstrap your own mathematical questions for large language models

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In ICLR 2024,

    2024

  25. [43]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    1905 arXiv

  26. [45]

    Effective data synthesis for long-context instruction tuning

    Wenhao Zhu, Pinzhen Chen, Hanxu Hu, Shujian Huang, Fei Yuan, Jiajun Chen, and Alexandra Birch. Effective data synthesis for long-context instruction tuning. arXiv preprint arXiv:2502.15592, 2025b. 14 A Detailed Experimental Results A.1 Training Config We employ the AdamW (Losh...

    arXiv

  27. [2008]

    Bootstrap your own context length

    Liang Wang, Nan Yang, Xingxing Zhang, Xiaolong Huang, and Furu Wei. Bootstrap your own context length. arXiv preprint arXiv:2412.18860,

    arXiv

  28. [2016]

    Yarn: Efficient context window extension of large language models

    Bowen Peng, Ye Zheng, Tie Xiao, Guangtao Liu, Siyuan Wang, Yutao Yang, and Aston Zhang. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071,

    arXiv

  29. [2018]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691,

    arXiv

  30. [2019]

    Davis, Lingjiao Chen, Boris Hanin, Ion Stoica, Joseph E

    Alan Zhu, Parth Asawa, Jared Q. Davis, Lingjiao Chen, Boris Hanin, Ion Stoica, Joseph E. Gonzalez, and Matei Zaharia. Bare: Combining base and instruction-tuned language models for better synthetic data generation. arXiv preprint arXiv:2502.01697, 2025a. Dawei Zhu, Zhihong Xia...

    arXiv

  31. [2020]

    Lm- infinite: Simple on-the-fly length generalization for large language models

    Chi Han, Qifan Wang, Wenhan Wang, Yun Bai, Eric P Xing, Xiang Ren, and Fan Yang. Lm- infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137,

    arXiv

  32. [2021]

    Multitask prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207,

    arXiv

  33. [2022]

    Ruler: Discrimination-aware long-context benchmarking

    Tianyu Hsieh, Gupta Otkrist, Jeff Wu, Devamanyu Lin, Yuntian Li, Yue Tian, Yann LeCun, and Wenhan Xiong. Ruler: Discrimination-aware long-context benchmarking. arXiv preprint arXiv:2405.17781,

    arXiv

  34. [2023]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Kevin Swersky, Daniel Tarlow, Satinder Sundararajan, Mohammad Hashemi, Danilo Jimenez Rezende, Stephan Gouws, Wenjie Zhao, Yunhao Li, Edward Gibson, et al. A general theoretical paradigm to understand learning from human preferences. arXiv preprint ar...

    arXiv

  35. [2024]

    Data engineering for scaling language models to 128k context

    Yury Fu, Peter Levin, Nikos Casas, Orhan Firat, and Rohan Anil. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402.10171,

    arXiv

  36. [2025]

    How to train long-context lan- guage models (effectively)

    Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context lan- guage models (effectively). 2024b. URL https://api.semanticscholar.org/CorpusID: 273098476. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Hao...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.