{"id":"d200c3b2-9524-4190-8077-e53d16fe9ddf","arxiv_id":"2411.08147","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLMs can improve their own long-context reasoning by fine-tuning on outputs selected by Minimum Bayes Risk consensus, without external annotations.","lead":"The paper introduces SEALONG, a training method in which a large language model samples many answers to long-context questions, scores them by how similar they are to each other, and then fine-tunes itself on the most consistent answers. Across five long-document QA benchmarks, the method lifts Llama-3.1-8B-Instruct by 4.2 points on average, with no human-written or GPT-4-generated training labels.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control: fine-tuning on randomly selected self-generated outputs would test whether MBR selection, not self-training alone, drives the reported gains.","rationale":"We read the paper as claiming that self-supervision via MBR consistency—not merely fine-tuning on self-generated outputs—improves long-context reasoning. The reader's weakest_assumption (that MBR score tracks correctness) is real, but the paper provides partial evidence in its favor: MBR decoding outperforms greedy and random selection at inference (Table 7, Figure 1). The more direct vulnerability is the lack of a fine-tuning control that randomizes the selection. The design compares against external datasets and an expert-model dataset, but never against the same self-generated data with a non-MBR selection rule. Such a control is cheap and would decisively test whether the reported gains come from the consistency filter or from exposure to any self-generated in-domain data. The paper's small evaluation scale (200 questions per task, single runs) adds urgency: the headline 4.2-point average could plausibly shift under a different seed, and a paired random-selection control would provide a within-run comparison. We do not see evidence of internal inconsistency; the method is clearly described and the code is promised. We therefore set the verdict to UNCHANGED (CONDITIONAL remains appropriate), with the concrete random-selection control as the condition.","tokens_in":18554,"tokens_out":4494,"duration_ms":43769,"concrete_test":"Fine-tune Llama-3.1-8B-Instruct with ORPO on 2K examples from MuSiQue, using the same 32 sampled outputs per question, but selecting the 'chosen' output uniformly at random instead of by MBR score (keeping the rejected output randomly selected among low-scoring ones, as in the paper). Evaluate on the same LongBench tasks. If the random-selection model matches SEALONG's 55.0 average (within noise), the improvement is not attributable to MBR selection; if it falls significantly short, the MBR filter is necessary.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MBR-consistency scoring of self-sampled outputs provides useful supervision for long-context reasoning (§3.1, §3.2). The load-bearing premise is that the selected output is a better training target than a random sample. The paper's evidence for this is indirect: Table 5 compares SEALONG against other datasets and GPT-4o-MuSiQue, and Section 4.4 compares scoring methods only at decoding time, not after fine-tuning. There is no control where the 'chosen' output is selected randomly (or the lowest-scoring output) from the same 32 samples. Without this control, the reported 4.2-point gain could stem from fine-tuning on self-generated in-domain data, rather than from the MBR filter. The paper's own Figure 1 shows MBR decoding still lags far behind the oracle sample, so the selected outputs are only partially correct; a random baseline could plausibly yield similar improvements. The absence of error bars and the small task sizes (200 questions per task) make this concern acute, since a 4.2-point average difference may be within run-to-run noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SEALONG, a self-improvement method for long-context reasoning. For each training question, the method samples N outputs from the base model, scores them with Minimum Bayes Risk (MBR) using sentence-embedding similarity, and then fine-tunes the model with ORPO on the highest-scoring output as the chosen response and a low-scoring output as the rejected response. Experiments on Llama-3.1 and Qwen-2.5 models across five LongBench tasks report an average SubEM improvement of 4.2 points for Llama-3.1-8B-Instruct (50.8 to 55.0), with Qwen-2.5-14B-Instruct exceeding its 32B variant. The authors also compare SEALONG with prior human- or expert-annotated datasets and find that it outperforms GPT-4o-MuSiQue (55.0 vs. 50.9). Additional analyses examine scoring methods, the number of synthetic examples, the number of samples per question, and short-context performance stability.","tokens_in":18756,"tokens_out":6033,"duration_ms":56392,"significance":"If the reported results are robust, the contribution is significant: SEALONG would demonstrate that consensus-based MBR selection over a model's own sampled outputs can provide effective training supervision for long-context reasoning without human or expert-model annotations. This would lower the cost of data synthesis and offer a practical route to self-improvement. The paper also includes useful control analyses: a token-count check showing that gains are not explained by longer outputs (Table 4), a comparison against several prior datasets with a fair 2K-example budget (Table 5), and an evaluation showing minimal short-context degradation (Table 8). However, the central claim is currently only partially supported because of three load-bearing gaps: MuSiQue is used both as the training-data source and as an evaluation task, no control exists for random selection of self-generated training targets, and no error bars or significance tests are reported. These issues are addressable within the manuscript's scope.","major_comments":[{"comment":"MuSiQue serves both as the source of training questions (§4.1: 'we leverage the training dataset of MuSiQue') and as one of the five evaluation tasks in Table 2. For Llama-3.1-8B-Instruct, the largest gain is on MuSiQue (49.5 to 58.5, +9.0 points), and the headline 50.8-to-55.0 average includes this in-domain result. The caption of Table 2 correctly frames the other tasks as demonstrating generalization, but the reported average does not. To support the claim of self-improvement in long-context reasoning, report the average over the four out-of-domain tasks separately, and ideally add a held-out multi-hop QA task that does not overlap with MuSiQue in question style or distribution.","section":"§4.1, Table 2"},{"comment":"There is no control where the training target is selected randomly (or by lowest MBR score) from the same N sampled outputs. Table 7 shows that MBR decoding selects better outputs than random at inference time, but that does not establish that MBR-selected training targets are superior to randomly selected ones after fine-tuning. The 4.2-point improvement could therefore arise from fine-tuning on self-generated in-domain outputs alone rather than from the MBR filter. Add an ablation that fine-tunes on randomly sampled outputs (and, ideally, on lowest-scoring outputs) from the same pool, with identical ORPO hyperparameters; if the random control achieves a similar gain, the central claim about MBR-based self-supervision is not supported.","section":"§3.2, §4.4, Table 7"},{"comment":"All results are reported as point estimates without error bars, multiple seeds, or significance tests. The evaluation sets contain only 150–200 questions per task (Table 3), so a 4.2-point average difference may be within run-to-run or sampling noise; this is especially relevant for the comparison of Qwen-2.5-14B+SEALONG (54.7) versus Qwen-2.5-32B (53.1). Report bootstrap confidence intervals for the main averages, or per-task significance tests (e.g., McNemar's test), and run at least two fine-tuning seeds for the headline results.","section":"Tables 2, 5, 7, 8; Figs. 3–4"},{"comment":"The central premise that 'correct reasoning trajectories typically exhibit higher semantic consistency' is only partially supported. Figure 1 shows that MBR decoding remains far below the oracle sample even at N=128, and the Limitations section acknowledges this gap. This weakness is not fatal by itself, but it makes the missing random-selection control (second major comment) more acute. A quantitative analysis on a small labeled subset of the training questions, reporting how often the MBR-selected output is correct, would strengthen the premise and help interpret the fine-tuning gains.","section":"§3.1, Fig. 1, Limitations"}],"minor_comments":[{"comment":"The description of the embedding model is inconsistent: §3.1 says 'a lightweight RoBERTa-based model' while §4.1 says 'jina-embeddings-v3 serving as the sentence embedding model.' Clarify which model is actually used and whether it is part of the self-supervision pipeline.","section":"§3.1, §4.1"},{"comment":"There are several typographical errors: 'Minimum Bayesian Risk' in the Limitations section should read 'Minimum Bayes Risk'; 'dose not' should read 'does not'; 'particularlly' should read 'particularly'; and 'see also in Tab. 1' in §4.4 likely refers to Fig. 1.","section":"Limitations, §4.3, §4.4"},{"comment":"The caption of Table 7 reports the performance of the highest-scoring output for each method but does not include the oracle upper bound. Adding the oracle score would help readers gauge the remaining headroom for the scoring methods.","section":"Table 7"},{"comment":"The construction of synthetic contexts is underspecified: 'we randomly sample some unrelated documents' — please state how unrelated documents are selected, whether they are drawn from the same MuSiQue corpus, and whether answer leakage across questions is possible.","section":"§4.1"},{"comment":"The captions for Figures 3 and 4 do not specify the exact metric plotted. State that the vertical axis is the average SubEM over the five LongBench tasks (or otherwise specify the metric).","section":"Figs. 3 and 4"},{"comment":"The sentence 'SEALONG first reveals the underestimated potential of LLMs in long-context reasoning' overstates the contribution; consider softening to 'provides evidence for the underestimated potential.'","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practical question, and the core idea is plausible. The main issue is that the experimental validation does not yet rule out simpler explanations: fine-tuning on self-generated in-domain data without MBR selection, and evaluation leakage of MuSiQue into the benchmark. The missing random-selection control and error bars are essential before the claims can be trusted. I recommend major revision rather than rejection because the missing experiments are well-defined and within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the quick take: the paper applies a known recipe—sample multiple outputs, pick the one most consistent with the rest under sentence-embedding similarity, fine-tune on it with ORPO—to long-context multi-hop QA. That is a legitimate extension, and the empirical result is new relative to the cited prior work: consistent gains on five LongBench tasks across Llama-3.1 and Qwen-2.5 models, with a 4.2-point average gain on Llama-3.1-8B and no drop in short-context scores. The authors also check the obvious token-count shortcut and compare against several existing synthetic datasets, where their self-generated data beats GPT-4o-generated data. They are transparent about the remaining gap between MBR-selected and oracle outputs.\n\nThe soft spots are real but not disqualifying. The most important is the missing control the stress-test flags: fine-tuning on randomly selected self-generated outputs. Table 7 shows MBR beats random at selection time, so the premise has some support, but that does not tell us whether the post-fine-tuning gain comes from the filter or just from training on self-generated in-domain trajectories. Without that control, the attribution to MBR is under-supported. Second, there are no error bars or multiple seeds; with 200 questions per task, a 4.2-point average could be within run-to-run noise. Third, the training data is drawn from MuSiQue's training set while MuSiQue is one of the evaluation tasks; even though the test instances are distinct, the distributional overlap means part of the gain may be domain matching rather than general long-context ability. The generalization to other tasks helps, but does not fully remove that concern.\n\nNone of this breaks the central claim that self-improvement via consensus filtering is worth exploring in long-context reasoning. The paper is a solid, clearly-written empirical study. It deserves a serious referee. I would send it to review, and ask for the random-output fine-tuning control, repeated seeds, and ideally a held-out task from the same training family. For the reading group, it is a good paper to discuss precisely because it surfaces how hard it is to isolate the effect of a selection signal in self-training.\n\nRecommendation: send to peer review with those requests. The result as it stands is conditional, not refuted.","headline":"A legitimate first application of MBR-style consensus self-training to long-context multi-hop QA, with consistent gains but a missing control that leaves the headline attribution under-supported.","tokens_in":19284,"tokens_out":2663,"would_cite":true,"duration_ms":27445,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Language models can improve their own long-context reasoning by learning from the most self-consistent of their sampled answers, with no human or expert-model labels.","keywords":["self-improvement","long-context reasoning","Minimum Bayes Risk","preference optimization","ORPO","multi-hop question answering","consensus-based supervision"],"falsifier":"Construct a long-context multi-hop question where a majority of sampled outputs contain the same wrong answer and only a minority contain the correct one; if MBR decoding selects the wrong majority output more often than the oracle and fine-tuning on those selections lowers held-out accuracy compared with random-sample fine-tuning, the consensus-as-correctness premise is falsified.","tokens_in":18371,"feed_emoji":"🧠","tokens_out":9986,"duration_ms":80579,"temperature":0.7,"pith_summary":"SEALONG claims that a large language model can improve its own long-context reasoning without any human or stronger-model annotations. The recipe is to sample many reasoning trajectories for each question, score each output by how semantically similar it is to the other sampled outputs (Minimum Bayes Risk), and then fine-tune the model on the highest-scoring output, optionally using a low-scoring output as a rejected example for preference optimization. On five long-context QA tasks the method lifts Llama-3.1-8B-Instruct from 50.8 to 55.0 average substring exact match, lets Qwen-2.5-14B-Instruct beat its 32B sibling (54.7 vs. 53.1), and leaves short-context performance unchanged. If this holds, it matters because it removes the reliance on human experts or advanced teacher models for long-context training data, turning the model's own sampled outputs into a scalable supervision signal.","feed_headline":"No GPT-4 needed: LLMs self-improve on long-context reasoning","feed_subtitle":"Sampling many answers, scoring by consensus, then fine-tuning on the winner lifts Llama-3.1-8B from 50.8 to 55.0.","key_machinery":"The load-bearing object is the MBR score $s(y) \\approx (1/N) \\sum_{i=1}^{N} u(y, y_i)$, where $u(y,y')$ is the inner-product similarity between sentence embeddings of two sampled outputs, computed with jina-embeddings-v3. It turns the intuition that correct reasoning is semantically consistent into a label-free ranking of sampled trajectories; the highest-scoring output becomes the SFT target or the chosen response in ORPO, and a randomly selected low-scoring output becomes the rejected response. The other supporting piece is plan-and-solve prompting, which elicits structured step-by-step trajectories from the model before scoring.","core_discovery":"The paper's central claim is that consensus among a model's own sampled outputs is a usable correctness signal for long-context reasoning, and that fine-tuning on the consensus-selected output converts that signal into durable capability. Concretely, SEALONG samples $N=32$ plan-and-solve reasoning trajectories per question, scores each with Minimum Bayes Risk using sentence-embedding similarity as the utility, and either supervises fine-tuning with the top-scoring output or applies ORPO with a high-scoring output preferred over a randomly chosen low-scoring one. The method raises the average SubEM of Llama-3.1-8B-Instruct from 50.8 to 55.0 across Qasper, MultiFieldQA-En, HotpotQA, MuSiQue, and 2WikiMQA, and enables Qwen-2.5-14B-Instruct to reach 54.7, exceeding the 53.1 of Qwen-2.5-32B-Instruct. It also outperforms fine-tuning on existing human- or expert-model-annotated long-context datasets at matched 2K-example budgets, indicating the self-supervision itself, not a larger or higher-quality external dataset, drives the gain.","pith_inferences":["If consensus-tracks-correctness holds at larger scales, self-improvement could compound: each generation of a model family could bootstrap its own long-context training data, removing the need to wait for a stronger teacher, which would change how long-context capability is scaled.","The paper's own oracle-versus-MBR gap suggests the scoring function, not the sampling budget, is the current bottleneck; a correctness-aware utility that verifies factual claims against the context rather than only semantic overlap might close that gap.","Because all synthetic training data comes from MuSiQue-style multi-hop questions, the method's scope is currently limited to that question type; applying the same loop to full-context reasoning, code, or math prompts would test whether the mechanism generalizes beyond multi-hop QA.","The comparison against expert-annotated datasets is at matched 2K examples; at larger data budgets the relative advantage could shift, so the claim of superiority over human/expert data should be read as specific to this scale and setup."],"forward_implications":["A model trained with SEALONG improves on long-context multi-hop QA across held-out tasks it never saw during data synthesis, so the benefit is not task-specific memorization.","SEALONG improves Qwen-2.5-7B-Instruct enough to close the gap with its 14B version, and makes 14B surpass 32B, showing self-improvement can substitute for some parameter scale.","The gain is not an artifact of longer outputs: average output token counts stay nearly unchanged, and short-context performance on six Open LLM Leaderboard tasks is flat.","At equal 2K-example budgets, SEALONG beats training on existing long-context datasets such as LongAlign, LongReward, and GPT-4o-MuSiQue, so the self-generated data is at least as useful as current human- or expert-annotated data.","Performance saturates around 1K synthetic examples and 32 samples per question, suggesting the method unlocks latent capability rather than teaching a new skill that needs more data."],"supporting_citations":[{"why":"Supplies the Minimum Bayes Risk definition that SEALONG uses to score outputs by expected utility under the model distribution.","marker":"Bickel and Doksum, 1977"},{"why":"Supports the premise that less consistent outputs are more likely to be hallucinations, motivating consensus-based scoring.","marker":"Manakul et al., 2023"},{"why":"Provides the semantic-consistency view of hallucination detection that SEALONG extends to long-context reasoning trajectories.","marker":"Farquhar et al., 2024"},{"why":"Provides the plan-and-solve prompting strategy used to sample reasoning trajectories for self-supervision.","marker":"Wang et al., 2023a"},{"why":"Supplies the MuSiQue questions and documents used to synthesize the self-supervised training examples.","marker":"Trivedi et al., 2022"},{"why":"Supplies the ORPO preference-optimization objective used to fine-tune on MBR-scored chosen and rejected outputs.","marker":"Hong et al., 2024"},{"why":"Supplies jina-embeddings-v3, the sentence embedding model used as the MBR utility metric.","marker":"Sturua et al., 2024"},{"why":"Defines the Llama-3.1-8B-Instruct model whose 4.2-point gain is the paper's main result and the comparison baselines.","marker":"Dubey et al., 2024"},{"why":"Defines the Qwen-2.5 models used to show that 14B plus SEALONG beats the 32B variant.","marker":"Yang et al., 2024a"}],"fun_headline_variants":["LLMs self-improve on long-context reasoning, no GPT-4 needed","Consensus scoring turns LLM outputs into training data","Self-generated answers fine-tune LLMs for long-context gain","Sampling, MBR, and fine-tuning: self-improvement for long-context","Model's own consensus lifts long-context reasoning by 4.2 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method works only if the most semantically consistent output among a model's sampled answers is usually the correct one; if the model confidently repeats a wrong answer across many samples, consensus scoring will pick that wrong answer and fine-tuning will reinforce it.","fun_headline_variants_meta":{"raw":{"variants":["LLMs self-improve on long-context reasoning, no GPT-4 needed","Consensus scoring turns LLM outputs into training data","Self-generated answers fine-tune LLMs for long-context gain","Sampling, MBR, and fine-tuning: self-improvement for long-context","Model's own consensus lifts long-context reasoning by 4.2 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000258,"raw_usage":{"total_tokens":1610,"prompt_tokens":1001,"completion_tokens":609,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":512}},"tokens_in":617,"tokens_out":609,"duration_ms":5121,"temperature":1.0,"reasoning_tokens":512,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:55:06.220723+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a long-context multi-hop question where a majority of sampled outputs contain the same wrong answer and only a minority contain the correct one; if MBR decoding selects the wrong majority output more often than the oracle and fine-tuning on those selections lowers held-out accuracy compared with random-sample fine-tuning, the consensus-as-correctness premise is falsified.","supporting_citations":[],"review_version":1}