{"id":"897f0f82-c8ee-467c-9e24-fca2ad9a2335","arxiv_id":"2607.06160","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LongCrafter generates long-context SFT data via a 32-task taxonomy and evidence-graph-guided instruction synthesis; trained 7B/8B models outperform prior synthetic SFT baselines on LongBench, LongBench v2, and LooGLE.","lead":"LongCrafter is a method for automatically building long-context training questions by first turning documents into evidence graphs, then asking an LLM to write questions and cited answers from those graphs. Models fine-tuned on the resulting 2,000 examples beat earlier synthetic-data baselines on three long-document benchmarks and come close to or exceed official instruction-tuned models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unmeasured correctness of the evidence graph is load-bearing: the main ablation attributes ~12 points to EGC, yet no node/edge accuracy against source text is reported.","rationale":"I read the paper as a data-construction method paper: its claim is that the LongCrafter pipeline (taxonomy + evidence graph + citation-based responses) produces training data that yields better long-context models. The evidence in Table 2 is multi-faceted and the ablations are the right shape. The weakest link is not the benchmark evaluation, which is shared across baselines, nor the lack of release, which is a reproducibility concern rather than a correctness one. It is the unmeasured correctness of the intermediate evidence graph. The paper's own Stage 2 says a valid graph must have nodes and edges that are indispensable and 'uniquely support the correct answer.' That is a factual property of the source documents; it is asserted, not verified. The reader identified this same assumption. I agree, and I would add that the ablation in Table 3 makes it load-bearing: removing EGC is the single largest performance change (roughly 12 points), so the mechanism is not decorative. The response-quality evaluation in Figure 5 and the human evaluation in Section 3.4 measure outputs, not graph correctness; a confident, well-cited wrong graph can still score high on faithfulness because the LLM judge checks consistency with the prompt, and the prompt is the same synthesized context that may encode the graph's errors. A graph-accuracy audit against source text is therefore the check that would settle whether the evidence grounding is real or is an artifact of a competent generator producing self-consistent but ungrounded graphs. I do not think this warrants rejection: the method could still be a useful recipe even if graphs have moderate error, and the downstream gains are consistent across two backbones. But it does keep the verdict conditional, matching the reader's CONDITIONAL verdict; no adjustment is needed.","tokens_in":17736,"tokens_out":7349,"duration_ms":71664,"concrete_test":"Stratified audit of 200 LongCrafter instances across the 32 task types. For each instance, two independent annotators—or a second LLM given access only to the context, not to the generated response—verify: (a) every node span occurs verbatim in the indicated chunk; (b) deleting any node changes the answer, i.e., the graph is necessary; (c) each directed edge's relation type is entailed by the two cited spans and not by parametric knowledge; and (d) the response's cited evidence chain is exactly the graph path. Report node precision/recall and edge-relation accuracy. If accuracy is below, say, 90% on (a)–(d), retrain LongCrafter with manually corrected graphs and compare; if scores do not change, graph correctness is not load-bearing, while if they improve, unmeasured graph quality is the binding constraint. A cheaper companion check is to re-run the w/o-EGC ablation while matching instru","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the Stage-2 evidence graph (V,E) is a correct, minimal, and uniquely-supporting decomposition of the context. Everything downstream—instruction generation, response citations, and the claimed 'faithful, traceable reasoning'—inherits this graph. Yet the paper reports no accuracy for graph construction against gold annotations: no precision/recall for node spans, no verification that edge labels (temporal/causal/coreference) are entailed by the cited spans, and no count of how often the 'uniquely supports the correct answer' condition actually holds. The validation LLM checks only that an instruction is answerable and has a unique answer; it does not independently verify the graph against the source text. Because the same generator (GLM-5) both builds the graph and writes the response, a graph can be self-consistent but wrong: it may quote verbatim spans while connecting them with hallucinated or non-entailed relations, or may omit a necessary span while still producing a plausible answer from parametric knowledge. This is not only a data-quality footnote: Table 3 shows that removing EGC drops average scores by 12.64 points on Qwen2.5-7B and 11.89 on LLaMA-3.1-8B, so the framework's main claimed driver is precisely the component whose correctness is unmeasured. If graph errors are frequent, the downstream gains could be an artifact of difficulty/diversity control (harder questions) rather than evidence grounding, and the faithfulness advantage in Figure 5 would be illusory. This does not require assuming bad faith; it is a standard unmeasured intermediate variable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"LongCrafter proposes a structured pipeline for synthesizing long-context SFT data: a hierarchical taxonomy of 32 fine-grained task types, a long-context construction stage, an evidence-constraint graph that decomposes task-relevant spans and cross-paragraph dependencies, and evidence-grounded instruction/response generation with verbatim citations. The paper fine-tunes Qwen2.5-7B and LLaMA-3.1-8B on 2,000 LongCrafter samples and reports the highest average scores on LongBench, LongBench v2, and LooGLE (45.15% and 45.71%), surpassing four SFT baselines and the official post-trained models. Additional analyses claim greater data diversity, controllable difficulty, higher response faithfulness, and improved evidence-position robustness. The central claim is that evidence-graph-guided synthesis is a more effective and more faithful source of long-context SFT data than existing automatic methods.","tokens_in":18112,"tokens_out":7305,"duration_ms":65490,"significance":"If the empirical claims hold, the contribution is substantial: a scalable, annotation-free synthesis recipe that outperforms existing automatic data-construction methods on three public long-context benchmarks across two model families. The paper has genuine strengths: evaluation on external public benchmarks reduces circularity; results reproduce across two backbones; component and data-distribution ablations isolate parts of the pipeline; human–LLM agreement (ICC 0.82/0.87, Spearman 0.63/0.74) supports the LLM-judge protocol; and the position-robustness and attention-mass analyses directly address the 'lost in the middle' problem. The evidence-graph idea is well motivated, and the citation-format responses provide a practical, testable form of faithfulness supervision. However, the load-bearing correctness of the evidence graph is unmeasured, and several experimental-reporting gaps prevent the claims from being fully supported as written.","major_comments":[{"comment":"Graph correctness is unmeasured. The graph is asserted to be minimal and its node set must 'uniquely support the correct answer,' yet the paper reports no span precision/recall, no verification that edge labels (temporal/causal/coreference) are entailed by the cited spans, and no uniqueness rate. The Stage-3 validation LLM checks only answerability and unique answer of the instruction, not graph validity; since GLM-5 constructs the graph and writes the response, self-consistent but wrong graphs can pass. This is load-bearing: removing EGC costs 12.64/11.89 average points (Table 3). Add a human-annotated graph-accuracy study (span F1, edge-label accuracy, uniqueness rate) and report downstream performance conditioned on graph quality.","section":"§2, Stage 2; Table 3"},{"comment":"The EGC ablation is confounded with instruction difficulty and generation quality. The paper states that removing EGC lowers Global/Deep difficulty from 2.59 to 1.74 and that 30% of 100 inspected w/o-EGC samples are low-quality. Thus the large downstream drop may reflect easier or weaker instructions rather than the absence of evidence constraints. Please add matched controls: generate w/o-EGC instructions with a matched difficulty distribution, or corrupt the graph's edges/nodes (e.g., shuffled edges, random labels, redundant nodes) while keeping the vertex set fixed, and compare benchmark scores. This would isolate the contribution of V and E rather than overall instruction difficulty.","section":"§4.1, Table 3"},{"comment":"Almost no error bars or significance tests are reported. Only LongBench v2 is averaged over three runs; all other benchmark scores and all ablations are single-run at temperature 0. The headline claims ('highest average', gains of 2.41/5.25 points) rest on point estimates. Please run at least three seeds for main results and ablations, report mean±std, and state whether observed differences are significant. This is especially important for subtask-level conclusions and for the position-robustness and EAM analyses, which are also single-run.","section":"§4.1, Tables 2–3"},{"comment":"The method cannot be reproduced from the manuscript. Only evaluation prompts are supplied; the Stage-2/3 synthesis prompts—the core of LongCrafter—are absent, and data/code are not released. Since the paper's contribution is an empirical recipe, please release the generated data (or a substantial sample), the full synthesis prompts, and training code, or provide them as supplementary material. Without these artifacts, the central empirical claims are not independently verifiable.","section":"Appendix 'Prompts'; §6"}],"minor_comments":[{"comment":"The abstract lists 'three limitations' (narrow task coverage, insufficient difficulty, lack of faithfulness supervision), while the introduction states 'two major limitations.' Please align the wording.","section":"Abstract vs. §1"},{"comment":"LongCrafter's average input length is 10,269 tokens, shorter than LongAlign (16,465) and LongReward (22,697). Report the length distribution and maximum length to support the 'long-context' characterization and to clarify the comparison with baselines.","section":"Appendix Table A.1"},{"comment":"The multi-panel captions do not clearly identify every panel and axis. In particular, the right panel of Figure 4 lacks a clear y-axis label and the reader must infer the grouping of 'single-info' vs. 'multi-info.'","section":"Figures 4 and 5"},{"comment":"Training hyperparameters are incomplete: no batch size, optimizer, gradient accumulation, sequence packing, or maximum training length is reported. These details matter for reproducing the LoRA fine-tuning recipe.","section":"§4.1"},{"comment":"The Evidence Attention Mass formula refers to 'visible gold evidence tokens' E_t, but visibility is not defined. State which tokens are included and how many layers, heads, and samples are averaged in Figure 7.","section":"§4.3 and Eq. (1)"},{"comment":"'Exhaustive candidate span extraction' overclaims: the extraction is performed by an LLM and is likely recall-limited. Consider using 'candidate span extraction' or report extraction recall against gold spans.","section":"§2, Stage 2"},{"comment":"The manual spot-check claiming 99.5% agreement between GPT-5 judgments and human verification gives no sample size or selection procedure. Please specify these details.","section":"Appendix"}],"recommendation":"major_revision","confidential_remarks":"This is a promising and well-executed empirical paper, but the missing measurement of evidence-graph correctness is the main risk to the central claim. I would condition acceptance on (a) a graph-accuracy evaluation, (b) controlled ablations that isolate graph structure from difficulty, (c) error bars for the main results, and (d) release of data and synthesis prompts. The paper's framing would also benefit from toning down 'exhaustive' and 'uniquely supports' until verified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing to know up front: the main result is credible on external benchmarks. Models trained on LongCrafter data beat other SFT recipes and official post-trained models on LongBench, LongBench v2, and LooGLE on both Qwen2.5-7B and LLaMA-3.1-8B. The gains are not circular—those are public benchmarks, not metrics defined by the pipeline. The new thing here is the ordering: a 32-type task taxonomy and an evidence-constraint graph built before instruction generation, with responses that cite spans verbatim. That distinguishes it from CGMIS (graph after the fact, multi-hop only) and LongFaith (attribution added post hoc).\n\nWhat the paper does well: the pipeline is described clearly; the ablations separate the evidence graph (EGC), the citations (EBC), and the data distribution, and each removal hurts. The human–LLM agreement study on 500 instances gives some external anchor to the LLM-judged quality numbers. The position-robustness and attention-mass analyses are a useful addition.\n\nThe soft spot is exactly what the stress-test note says: the evidence graph itself is never scored. No precision/recall on node spans, no check that edge labels (temporal, causal, coreference) are entailed by the cited text. The same model builds the graph and writes the response, so a graph can be self-consistent and wrong. The ablation attributes ~12 points to EGC, so this is load-bearing for the internal story. But I would not call it fatal: the external benchmark gains stand regardless, and the EBC ablation plus the faithfulness scores from the human-agreed judge suggest the evidence grounding does real work. The alternative explanation—that EGC mostly buys difficulty and diversity—is not fully ruled out, though. The paper would be much stronger with a direct graph-accuracy measurement against a human-annotated sample.\n\nOther issues are more minor: most numbers are single-run with no error bars; LongBench v2 is averaged over three runs but the rest is not. Data, prompts, and code are not released, which is a real reproducibility problem. The 99.5% GPT-5–human agreement spot-check is a single appendix line, and the human–LLM agreement study covers difficulty and quality, not benchmark correctness.\n\nWho this is for: anyone building long-context SFT data. The recipe is clearly specified and the external results are worth testing. It deserves a serious referee. I would send it to review and ask for artifacts, variance, and a graph-accuracy evaluation. If those come back, this is a solid accept.","headline":"A credible synthetic-data recipe with external benchmark gains, but the unmeasured evidence-graph accuracy is the gap that needs closing.","tokens_in":18588,"tokens_out":3512,"would_cite":true,"duration_ms":32721,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LongCrafter claims that building an explicit evidence graph before generating instruction–response pairs yields long-context SFT data that outperforms prior synthesis pipelines and official post-trained models.","keywords":["long-context understanding","instruction synthesis","evidence graph","faithfulness supervision","supervised fine-tuning","task taxonomy","lost in the middle","data diversity"],"falsifier":"Audit 100 LongCrafter samples across task types: have human judges mark the minimal set of evidence spans needed to answer each question, and compare with the evidence graph's node set and edge labels. If a large fraction (say >20%) of graphs contain extraneous, missing, or wrongly typed evidence, yet those samples still produce correct, faithful answers, then the graph is not the true carrier of the signal — the generator model's own ability is. A complementary check is to rebuild the same pipeline with a different graph-building model and re-run the three benchmarks; if the gains persist, th","tokens_in":17663,"feed_emoji":"🧩","tokens_out":7939,"duration_ms":61775,"temperature":0.7,"pith_summary":"LongCrafter is a framework for synthesizing long-context supervised fine-tuning data. Its central claim is that structure, not volume, drives long-context ability: instead of generating questions directly from documents, the pipeline first extracts verbatim evidence spans and arranges them into a directed graph of cross-paragraph dependencies (temporal, causal, coreference), then writes instructions and step-by-step cited answers from that graph. Models fine-tuned on just 2,000 such samples outperform all SFT baselines and the official post-trained Qwen and LLaMA models on LongBench, LongBench v2, and LooGLE, with the largest margins on high-difficulty tasks and on evidence located at any position. The paper argues that the evidence graph provides two things prior methods lack: controllable difficulty (because the graph fixes what evidence is needed and how it connects) and explicit faithfulness supervision (because every reasoning step must quote its supporting span).","feed_headline":"Evidence-graph training data beats official long-context models","feed_subtitle":"LongCrafter's 2,000 synthesized samples lift Qwen and LLaMA scores by 2.41 and 5.25 points over post-trained models.","key_machinery":"The evidence-constraint graph (evidence graph, for short) is the load-bearing object. Its nodes are verbatim evidence spans, each tied to a paragraph location; its directed edges are typed cross-paragraph dependencies (temporal, causal, coreference). It carries the argument by defining, before any question exists, which evidence is necessary and sufficient to answer a target task type, thereby making instruction difficulty a controllable design parameter and enabling response generation that is grounded span-by-span in verified citations.","core_discovery":"The paper's discovery is that an evidence-constraint graph — a minimal set of verbatim spans with typed directed edges capturing cross-paragraph relations — can serve as the generative skeleton for long-context instruction data. By making the graph before the question, the synthesis process can target a specific difficulty level (the graph dictates how many spans and hops are required) and can produce responses that cite each reasoning step to a located span, yielding faithfulness supervision that is absent from direct-generation or concept-graph methods. The empirical payoff: on three long-context benchmarks and two 7B/8B backbones, models trained on 2,000 LongCrafter samples reach average","pith_inferences":["A direct test of the paper's mechanism would be to measure evidence-graph quality (span precision/recall, edge correctness) against human annotations; the paper does not report such a measurement, and since the same generator model builds the graphs and the responses, a portion of the faithfulness gain might reflect generator habits rather than the graph structure itself.","The 32-task taxonomy is hand-built; it would be useful to know whether a taxonomy derived from existing benchmark categories yields the same gains, which would suggest the pipeline's effect comes from the graph construction, not the specific task list.","The paper evaluates evidence position at the document level (placing the gold document at positions 1, 5, 10, 15, 20 among 20 documents); a stricter test of 'lost in the middle' would place the gold evidence sentence at varying depths inside a single document, where within-document attention is more fragile.","Given the small data scale, an untested extension is whether scaling LongCrafter to 10,000–20,000 samples (with harder graphs, more task types, or multi-turn tasks) continues to improve long-context ability or saturates; the paper's results imply the current 2,000-sample curve is still improving."],"forward_implications":["If the claim holds, high-quality long-context SFT data can be produced at 2,000 samples with no human annotation, matching or beating official post-trained models that are presumably trained on far more data.","The evidence-graph-first recipe generalizes beyond multi-hop QA: because the graph encodes dependency structure, it can pace difficulty for summarization, tracking, ordering, and calculation tasks that prior synthesis methods could not control.","Explicit citation supervision transfers to held-out tasks, giving position-robust evidence localization and mitigating the 'lost in the middle' effect, so models rely on content rather than position.","The ablation pattern — graph removal costs ~12 points, citation removal ~5 points — indicates that structural constraint and traceability, not raw diversity alone, drive the gains."],"fun_headline_variants":["Evidence graphs lift 8B models past official long-context SFT","LongCrafter: 2k graph-grounded samples top post-trained models","Graph-built SFT data cracks long-context benchmarks","Traceable reasoning data beats official long-context tuning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The pipeline assumes that the generator LLM builds a correct evidence graph — that the selected spans are verbatim, jointly necessary and sufficient, and uniquely support the correct answer — because any wrong node or edge propagates into the instruction and the cited response, and the paper does not measure graph accuracy against gold annotations.","fun_headline_variants_meta":{"raw":{"variants":["Evidence graphs lift 8B models past official long-context SFT","LongCrafter: 2k graph-grounded samples top post-trained models","Graph-built SFT data cracks long-context benchmarks","Traceable reasoning data beats official long-context tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000601,"raw_usage":{"total_tokens":2667,"prompt_tokens":788,"completion_tokens":1879,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":1816}},"tokens_in":532,"tokens_out":1879,"duration_ms":60906,"temperature":1.0,"reasoning_tokens":1816,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T04:25:14.918137+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit 100 LongCrafter samples across task types: have human judges mark the minimal set of evidence spans needed to answer each question, and compare with the evidence graph's node set and edge labels. If a large fraction (say >20%) of graphs contain extraneous, missing, or wrongly typed evidence, yet those samples still produce correct, faithful answers, then the graph is not the true carrier of the signal — the generator model's own ability is. A complementary check is to rebuild the same pipeline with a different graph-building model and re-run the three benchmarks; if the gains persist, th","supporting_citations":[],"review_version":2}