{"id":"f7d59d3e-b29e-4d59-bde9-729173f410dd","arxiv_id":"2412.14780","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A shuffle-based token classifier plus group-level loss reweighting improves supervised fine-tuning of LLM agents on tool-use benchmarks.","lead":"The paper proposes SHAD, a method that sorts tokens in agent-training data into 'reasoning' and 'boilerplate' labels by shuffling inputs with outputs and watching which tokens become harder to predict, then fine-tunes models while upweighting reasoning tokens. This gives a cheap recipe to improve LLM agent skills on tool-use benchmarks without changing the model architecture.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SHAD's zero-threshold loss-change rule is the load-bearing assumption; an inverted/random-label ablation would test whether RFT gains actually stem from identifying reasoning tokens.","rationale":"I considered whether the lack of variance estimates or test-set selection of tau is the more load-bearing issue, and those are real threats to the empirical claim. However, the SHAD classification rule is the paper's stated novelty and the linchpin of its explanation: RFT is only 'reasoning-highlighted' if SHAD actually separates reasoning from boilerplate. The Limitations section explicitly admits a failure mode of this assumption, and Appendix D validates the classifier only on the easiest subset of tokens. An inverted-label or random-label ablation is a direct, low-cost experiment that would settle whether the reported gains require the specific SHAD labels. Until that control is reported, a conditional verdict is appropriate; the reader's CONDITIONAL verdict already captures this, so I do not change it.","tokens_in":16222,"tokens_out":7089,"duration_ms":53829,"concrete_test":"Run SHAD+RFT on ToolBench+APIGen with three label conditions: (i) SHAD labels as proposed; (ii) inverted labels (boilerplate treated as reasoning and vice versa); (iii) random labels with the same group proportions. Evaluate on the four benchmarks from Table 1 (StableToolBench, BFCL, T-eval, Nexus) with at least 3 seeds. If condition (ii) or (iii) matches or exceeds (i), the performance gains are not attributable to identifying reasoning tokens; if (i) clearly wins on average, the classification rule is load-bearing as claimed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is the SHAD classifier in Section 3.1: a token is boilerplate iff LD(yk) = ls(yk) - lo(yk) <= 0 after tuning on 1% shuffled data. The paper's own Limitations (Section 7) concede that high boilerplate diversity can break the assumption that boilerplate losses decrease while reasoning losses increase. The only direct validation (Appendix D.2) is on regex-identifiable format tokens (misclassification 0.82%/2.62%), not on template-connecting tokens or on reasoning tokens. If the zero threshold mislabels a substantial fraction of reasoning tokens, RFT's softmax reweighting (Section 3.2, Equations 4-5) will upweight boilerplate instead, and any measured gains could come from a generic high-loss emphasis rather than from 'reasoning highlighting.' Because the paper does not report an inverted-label or random-label control, the empirical gains in Table 1 do not establish that SHAD's specific labels are responsible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that when fine-tuning LLMs on agent-task datasets, tokens play different roles: reasoning tokens carry sample-specific problem-solving content, while boilerplate tokens (format markers and template-connecting phrases) are repetitive and easier to learn. The authors propose SHAD, which labels tokens by fine-tuning a small reference model on 1% of the data with input–output pairs shuffled, then classifying a token as boilerplate iff its loss decreases relative to the original model and as reasoning otherwise. They then introduce RFT, a fine-tuning loss that applies softmax weights to the group totals of reasoning and boilerplate token losses, emphasizing the higher-loss group. Experiments on LLaMA3-8B and LLaMA3.1-8B compare SHAD+RFT with SFT, Regex, Rho-1, RewardFT, and two variants (SHAD+α-FT, Regex+RFT) on StableToolBench, BFCL, T-eval, and Nexus, reporting that SHAD+RFT generally outperforms baselines. The paper also includes case studies, a temperature sensitivity plot, and a short limitations section.","tokens_in":16414,"tokens_out":4184,"duration_ms":29199,"significance":"If the central claim is validated, the paper offers a simple, low-cost heuristic for token-level differentiation during agent fine-tuning, with a plausible mechanistic story (shuffling disrupts reasoning tokens but not boilerplate tokens). The writing is clear, the method is easy to reproduce from the description, and the authors are candid about limitations. The paper does not currently ship code or checkable artifacts, but the experimental design is within the scope of the claims. The main issue is that the empirical evidence does not yet establish that the SHAD labels themselves are responsible for the gains, because the threshold rule is un-sensitivity-tested, no variance or significance information is provided, the temperature is selected on the evaluation benchmarks, and the direct classification validation covers only regex-identifiable format tokens.","major_comments":[{"comment":"The classification rule 'boilerplate if LD(yk) ≤ 0; reasoning otherwise' is the load-bearing component of the method, but neither the zero threshold nor the 1% shuffle ratio is sensitivity-tested. The authors' own Limitations section (Section 7) states that the method may fail when boilerplate diversity is high, which is exactly the regime where the loss-change sign assumption can break. To support the claim that SHAD's labels drive the Table 1 gains, the paper needs ablations that vary the LD threshold around 0 and the shuffle ratio, plus controls with inverted labels and random labels. Without such controls, the observed improvements could come from a generic high-loss emphasis rather than from identifying reasoning tokens.","section":"Section 3.1, Eq. (2), Table 1"},{"comment":"Table 1 reports a single number per method and benchmark, with no standard deviation, number of seeds, or significance tests. Some differences are small (e.g., 89.4 vs 89.3 on BFCL with LLaMA3.1-8B), and even the larger gaps (e.g., Nexus 27.8 vs 14.0 for LLaMA3-8B) cannot be assessed for stability from one run. The authors should report mean±std over at least three seeds and, where appropriate, paired significance tests or confidence intervals. This is necessary to support the claim that SHAD+RFT 'outperforms all baselines' on the evaluated benchmarks.","section":"Table 1, Section 4.2"},{"comment":"The temperature τ is selected by varying 1/τ and observing the averaged performance on the same evaluation benchmarks that appear in Table 1, with no mention of a separate validation split. If τ is tuned on the test benchmarks, the comparison in Table 1 is not a clean evaluation of the final method. Please specify a held-out validation protocol for hyperparameter selection and report the chosen τ (and the α used for SHAD+α-FT, which is not stated anywhere).","section":"Section 5, Figure 7"},{"comment":"The direct validation of SHAD's classification quality is limited to formatting tokens that can be identified by regular expressions (misclassification 0.82% on ToolBench and 2.62% on APIGen). This subset is the easiest part of boilerplate, and the paper's own examples show that template-connecting tokens and reasoning tokens are more ambiguous. The absence of any quantitative validation on these harder categories—for example, human annotations on a sample of template-connecting and reasoning tokens—means the central mechanism remains unverified. The downstream improvements in Table 1 are suggestive but cannot substitute for evidence that SHAD labels are accurate on the categories it is designed to distinguish.","section":"Appendix D.2, Section 5"}],"minor_comments":[{"comment":"In the paragraph before the three steps, 'tree steps' should be 'three steps'.","section":"Section 3.1"},{"comment":"The table header 'warmup radio' should be 'warmup ratio'.","section":"Appendix B, Table 3"},{"comment":"The color coding for token types is inconsistent across figures: Figure 1 uses green for reasoning and yellow/blue for boilerplate, while Figures 5 and 8 use blue for reasoning and brown for boilerplate. Please harmonize the colors and ensure captions describe the current figure.","section":"Figures 1, 5, and 8"},{"comment":"For Example 1, the 'Response (SFT):' block appears empty in the provided text, making the qualitative comparison incomplete. Please fill in the missing response.","section":"Appendix F, Figure 9"},{"comment":"The entry for 'Yang et al.' appears without a year in the reference list, and the same work appears twice ('Yang et al., 2023' in two forms). Please unify and complete the citation information.","section":"References"},{"comment":"The numbers '83.81' and '80.31' for Regex+RFT appear to carry a footnote marker '1' with no corresponding footnote in the text; please clarify or remove the marker.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a simple and timely idea, and the topic fits well with the journal's scope. My main concern is the gap between the central claim and the evidence: the SHAD labeling rule is the entire mechanism, but it is validated only on the easiest subset of tokens and never tested against an inverted-label control. The missing variance and the test-set-based hyperparameter selection strengthen the concern. These issues are fixable within the paper's scope by adding sensitivity analyses, controls, and a stricter evaluation protocol, so I am not recommending rejection. I would also encourage the authors to release code to aid reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is not a generic token-reweighting paper. The SHAD idea—shuffle input-output pairs across samples, fine-tune a small reference model, then classify tokens by whether the loss dropped—is genuinely new to me and not in the cited baselines. Empirically, the gains are consistent: SHAD+RFT beats SFT by roughly 4-7 average points across StableToolBench, BFCL, T-eval, and Nexus on both LLaMA3-8B and LLaMA3.1-8B. That is a useful result even if part of the effect is just high-loss emphasis.\n\nThe paper does several things right. It reports component ablations (SHAD vs regex, RFT vs fixed alpha), training-loss curves that show reasoning-token loss drops more under RFT, and an honest limitations section. The method is cheap: 1% shuffled data, one extra fine-tune, offline classification. Those details matter.\n\nThe soft spots are about attribution, not about the headline numbers. Table 1 reports single runs with no variance, seeds, or significance tests; that is common here, but it means the 4-7 point spread is credible rather than established. Second, tau is selected using the evaluation sets themselves in Figure 7, with no described held-out split. Third, the stress-test concern is right: the zero-threshold loss-change rule is the load-bearing assumption, and the only quantitative validation is on regex-identifiable format tokens (0.82% and 2.62% misclassification). There is no inverted-label or random-label ablation, so we can't confidently attribute the gains to SHAD's specific labels. The Limitations section even admits the consistency assumption can fail with high boilerplate diversity. A random-label control would resolve this, and its absence is the biggest gap.\n\nNone of this is fatal. As a training recipe, the empirical claim holds up: the method beats SFT and the baselines tried. As a mechanism claim—that it specifically highlights reasoning tokens—it is plausible but under-supported. The authors are clear about the limitations.\n\nThis paper is for anyone working on agent fine-tuning or instruction tuning who wants a practical token-reweighting recipe. It deserves a serious referee: the idea is new, the results are consistent, and the gaps are addressable with modest additional experiments. I'd engage with it.","headline":"It is a genuinely new shuffle-based token classifier with consistent gains across benchmarks; the mechanism is plausible but not fully isolated, and it deserves a serious referee.","tokens_in":16920,"tokens_out":3707,"would_cite":true,"duration_ms":24027,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Shuffle-aware fine-tuning beats SFT by reweighting reasoning tokens","keywords":["reasoning tokens","boilerplate tokens","shuffle-aware discriminator","token reweighting","fine-tuning","LLM agents","tool use","supervised fine-tuning"],"falsifier":"Construct a synthetic agent dataset with known ground-truth token roles, for example by randomizing the output format while copying the reasoning text verbatim across samples. Run the SHAD shuffle-tune-and-compare procedure and count how many tokens with decreased loss are actually content-bearing reasoning tokens; a substantial misclassification rate would falsify the claim that shuffling cleanly separates the two groups.","tokens_in":16035,"feed_emoji":"🧠","tokens_out":6432,"duration_ms":52045,"temperature":0.7,"pith_summary":"The paper argues that not all tokens in agent-training data deserve equal weight during supervised fine-tuning. It proposes a cheap shuffle-based discriminator that automatically separates sample-specific reasoning tokens from repetitive boilerplate tokens such as output-format and template-connecting phrases, then adaptively upweights the reasoning tokens during training. If the method works, LLM agent capabilities such as tool use and multi-step planning can be improved without manual annotation, regular expressions, or extra reward models. The paper reports consistent gains over standard SFT and several token-differentiation baselines across held-in and held-out agent benchmarks.","feed_headline":"Shuffle trick tells LLMs which tokens to learn hardest","feed_subtitle":"A 1% shuffle of agent training data reveals which tokens carry reasoning, and weighting them up lifts tool-use and planning scores.","key_machinery":"The load-bearing mechanism is the loss-difference classifier $LD(y_k)=l_s(y_k)-l_o(y_k)$, where $l_s$ is the token loss of a model fine-tuned on shuffled input-output pairs and $l_o$ is the loss of the original model. The rule is: boilerplate if $LD(y_k) \\leq 0$, otherwise reasoning. Shuffling destroys the input-output correspondence for reasoning tokens while leaving repetitive boilerplate tokens predictable, which is what lets the loss shift separate the two groups. RFT then uses softmax group weights $\\omega_b$ and $\\omega_r$ over the total losses of each token group to emphasize the reasoning part.","core_discovery":"The paper claims that a Shuffle-Aware Discriminator (SHAD) can label each token as reasoning or boilerplate by fine-tuning a model on a small shuffled subset of the data and comparing token-level losses before and after. A token whose loss decreases after shuffle tuning is classified as boilerplate; a token whose loss increases is classified as reasoning. Building on these labels, Reasoning-highlighted Fine-Tuning (RFT) weights the total loss of the reasoning group and the boilerplate group with a softmax, so the harder, more informative reasoning part receives more gradient pressure. In their experiments with LLaMA-3-8B and LLaMA-3.1-8B trained on ToolBench and APIGen data, the paper reports that SHAD+RFT outperforms all baselines on all held-in and held-out evaluation datasets, with the single exception of BFCL on LLaMA-3-8B.","pith_inferences":["The shuffle-loss signal could be reused as a general measure of how template-like any token is, not only in agent outputs, making it a diagnostic for format overfitting before training even begins.","The zero threshold in the classifier is likely fragile; a margin-based or percentile-based threshold would be a natural robustness improvement, especially under the high boilerplate diversity the paper itself lists as a limitation.","Because the classification step is cheap, it could be combined with token-level reward or loss weighting methods to separate hard reasoning substeps within a single sample, an extension the paper mentions but does not pursue."],"forward_implications":["Agent fine-tuning pipelines can label every token automatically with no manual regex, using only a small shuffled subset of the training data and one forward pass.","Models trained this way should overfit less to output formats and spend more of their learning budget on sample-specific tool-use and planning content.","The same shuffle-loss diagnostic should transfer to other structured-output SFT tasks, such as JSON generation or code generation, where boilerplate tokens are shared across samples.","On the benchmarks studied, held-out tool-use and planning scores rise while held-in scores stay at least comparable, suggesting the gains reflect better generalization rather than memorization of training formats."],"supporting_citations":[{"why":"Supplies the ToolBench training data and the StableToolBench evaluation setup used in the experiments.","marker":"Qin et al., 2024"},{"why":"Supplies the APIGen function-calling training dataset used alongside ToolBench.","marker":"Liu et al., 2024"},{"why":"Provides the AgentTuning practice of mixing general data with agent data to preserve general capabilities.","marker":"Zeng et al., 2023"},{"why":"Defines the Rho-1 baseline that identifies and masks noisy tokens during fine-tuning, which SHAD+RFT is compared against.","marker":"Lin et al., 2024"},{"why":"Provides the token-level reward fine-tuning baselines that SHAD+RFT must beat.","marker":"Yang et al.; Rafailov et al., 2024"},{"why":"Agent-Flan is the closest prior work on token differentiation for agent tuning and motivates the focus on boilerplate tokens.","marker":"Chen et al., 2024b"},{"why":"Supplies the T-eval held-out benchmark for step-level tool-use evaluation.","marker":"Chen et al., 2024a"},{"why":"Supplies the Nexus held-out benchmark for complex single-step nested tool use.","marker":"team, 2023"}],"fun_headline_variants":["Shuffle identifies reasoning tokens for smarter fine-tuning","Focus fine-tuning on reasoning tokens via shuffle discrimination","Weight reasoning tokens, not boilerplate, to boost LLM tuning","Shuffle trick separates reasoning from boilerplate in LLM data","How shuffling data reveals which tokens LLMs should learn"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method depends on boilerplate tokens being nearly identical across samples so shuffling leaves them predictable; if boilerplate tokens vary a lot, or reasoning tokens repeat fixed phrases, the loss-difference labels flip and the reweighting can backfire.","fun_headline_variants_meta":{"raw":{"variants":["Shuffle identifies reasoning tokens for smarter fine-tuning","Focus fine-tuning on reasoning tokens via shuffle discrimination","Weight reasoning tokens, not boilerplate, to boost LLM tuning","Shuffle trick separates reasoning from boilerplate in LLM data","How shuffling data reveals which tokens LLMs should learn"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1573,"prompt_tokens":891,"completion_tokens":682,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":602}},"tokens_in":507,"tokens_out":682,"duration_ms":5169,"temperature":1.0,"reasoning_tokens":602,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:54:46.237347+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a synthetic agent dataset with known ground-truth token roles, for example by randomizing the output format while copying the reasoning text verbatim across samples. Run the SHAD shuffle-tune-and-compare procedure and count how many tokens with decreased loss are actually content-bearing reasoning tokens; a substantial misclassification rate would falsify the claim that shuffling cleanly separates the two groups.","supporting_citations":[],"review_version":1}