{"id":"85a69b64-bec0-4a9b-b36d-ed63601b16d5","arxiv_id":"2507.20252","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A training framework that uses the sequence space after the answer token for self-evaluation and reward prediction, with inference stopped at a new marker, improves reasoning accuracy over standard SFT and RL baselines.","lead":"This paper trains language models to keep generating self-evaluations and reward scores after the answer, then stops them at a new marker during inference so there is no extra cost. It reports accuracy gains on math and reasoning benchmarks, pointing to a training-time reinforcement signal that does not slow deployed models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Exact-match evaluation may inflate PCL's gains: the paper admits its criteria are 'more stringent' and may lower baseline scores, so a format-agnostic rescoring is needed before the central claim is accepted.","rationale":"The reader's weakest assumption is exactly the concern I find most load-bearing: exact-match scoring on the region, combined with PCL's dedicated format training, can inflate the reported gains over baselines that were not trained to follow the same format. The paper's own admission that its criteria are 'relatively more stringent' and 'may result in lower baseline performance metrics' makes this a concrete, testable threat to the central claim. The presence of the 'w/ eval output' baseline, which includes format rewards and still trails PCL by 2 points, suggests the effect may not be entirely an artifact, but the primary comparisons against SFT and SFT+RL are not clean. I agree with the reader that a conditional verdict is appropriate: the method is plausible, the ablations are directionally consistent, and the 'w/ eval output' comparison offers some internal control, but the evaluation protocol and the lack of released code/data leave the magnitude of the claimed improvement unverified. The theoretical sections (Theorems 1-3) are largely asserted rather than derived, but they are secondary to the empirical claim. No ad hominem is intended; the issue is with the evaluation design, not the authors' integrity.","tokens_in":11962,"tokens_out":4029,"duration_ms":44938,"concrete_test":"Ask the authors to release generation logs for Tables 2-4 and re-score every method with a standard math-verify-style answer extractor over the full response (regex-based, normalizing whitespace and units) instead of exact matching on the region. Report both exact-match and extracted-answer accuracy. If PCL's average improvement over SFT+RL falls below roughly 1 point on GSM8K, the headline result is largely an evaluation-protocol artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the evaluation protocol. The paper explicitly states: 'our evaluation criteria are relatively more stringent, which may result in lower baseline performance metrics compared to other studies' (Experiments, Main Results). Accuracy is measured by exact matching on the region for every method, while PCL is the only method whose training includes a dedicated format reward and SFT on the PCL format. Baselines SFT and SFT+RL (Table 1) are not trained to emit / tags reliably, so their answers can be discarded by the parser even when mathematically correct. Consequently, the reported +3.45-point average improvement over SFT+RL on GSM8K and +4.56 on StrategyQA may partly measure format compliance rather than reasoning gains. The 'w/ eval output' baseline, which does include format rewards, shows a smaller +2.00-point gap, suggesting that some benefit may remain; but without scoring all methods by a format-agnostic answer extractor, the magnitude of the claimed 'enhanced output quality' is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Post-Completion Learning (PCL), a training framework in which the model continues generating a self-evaluation and reward-prediction segment after the answer, separated by a dedicated token. Training combines dual-track SFT (one track for reasoning, one for evaluation) with GRPO-style RL using accuracy, format, and consistency rewards. At inference the model stops at the token, so no extra tokens are generated. Experiments on GSM8K, StrategyQA, and MathQA across several small models report consistent gains over SFT and SFT+RL baselines, with ablations on GSM8K attributing parts of the gain to the evaluation SFT and the consistency reward. The paper also presents an information-theoretic and convergence analysis intended to justify the method.","tokens_in":12288,"tokens_out":7563,"duration_ms":78119,"significance":"If the empirical claims hold, PCL is a practically appealing idea: it internalizes self-evaluation during training while adding zero inference-time tokens, and the GSM8K ablations give a credible decomposition of the gains into reasoning SFT, evaluation SFT, format rewards, and consistency rewards. The teacher-data filtering step (retaining only samples where the teacher's self-evaluation matches the recalculated reward) is a sensible anti-circularity measure, and the consistency reward is computed against fixed ground-truth-based reward functions rather than against the model's own outputs. The main value is the training recipe and the empirical comparison, not the theoretical section as written. The central weakness is that all methods are scored by exact matching on the region while only PCL and one control are trained to emit the required tags, so the reported margins may partly measure format compliance rather than reasoning quality. The paper would be strengthened by a format-agnostic rescoring, by significance information, and by either proving or explicitly reframing the theoretical claims.","major_comments":[{"comment":"The evaluation protocol is not symmetric across methods. Accuracy is computed by exact matching on the region for every method, while PCL is trained with format rewards and evaluation SFT to emit the required tags; SFT and SFT+RL are not. The paper itself states that the criteria are \"relatively more stringent, which may result in lower baseline performance metrics compared to other studies.\" A baseline that produces a correct answer outside the tags is scored as wrong, whereas PCL is rewarded for tag compliance. The reported gaps (e.g., +3.45 on GSM8K, +4.56 on StrategyQA) therefore conflate format compliance with reasoning gains. Please re-score all methods with a format-agnostic answer extractor (e.g., regular-expression extraction from the full generation) and report both exact-match and format-agnostic accuracies. The Table 2 \"w/ eval output\" row, which receives format rewards and post-completion outputs, narrows the GSM8K gap to about +2.00, suggesting a residual effect, but that control is absent from Tables 3 and 4, so the magnitude of the claimed \"enhanced output quality\" is not yet established.","section":"Experiments (Main Results), Tables 2–4"},{"comment":"The main theoretical claims are asserted rather than proved. Theorem 1 needs the strict inequality I(θ;e,r|x,y)>0 in Eq. (8), but the chain rule alone gives only non-negativity; the \"not redundant\" assumption is exactly what must be shown, especially because in the data-generation process e and r are constructed from y and fixed reward rules. Theorem 2's Eq. (11) states O(log n/n) versus O(sqrt(log n)/sqrt n) sample complexity without defining the error classes, the learning algorithm, or the \"parameter separation property,\" and Eqs. (9)–(10) are not derivations. Theorem 3 likewise relies on an undefined claim that |Θ_R ∩ Θ_E| ≪ |Θ_R ∪ Θ_E| and on unspecified \"standard regularity conditions.\" If these results are intended as a theoretical contribution, they need real assumptions and proofs; otherwise the section should be reframed as intuition, because the current \"Proof\" labels overclaim.","section":"Theoretical Analysis, Theorems 1–3, Eqs. (7)–(13)"},{"comment":"The cross-dataset evidence is weaker than the text implies. StrategyQA and MathQA report only three settings (SFT, SFT+RL, PCL), so neither the ablations nor the format-trained \"w/ eval output\" control is available on those datasets. MathQA results for the two smaller models are -0.04 and +0.24 points, which the paper itself calls \"mixed results\" in the Limitations section, yet the conclusion claims \"consistent improvements\" across datasets and scales. With no repeated runs, confidence intervals, or significance tests, these near-zero differences cannot support the cross-dataset claim. Please add variance information, include ablation or control rows for at least StrategyQA, and analyze the MathQA non-improvement quantitatively instead of attributing it to an unplotted reward oscillation.","section":"Tables 3–4 and Limitations"},{"comment":"Equation (5) presents the RL loss as REINFORCE with the raw sum of rewards as the weighting term, but the text says the method uses GRPO with 8 sampled responses, group-relative contrast, and a KL constraint of β=0.04. None of these ingredients appears in Eq. (5). The exact objective matters for reproducibility and for the claimed connection to GRPO. Please write the actual loss, including the advantage normalization and KL penalty, or explicitly state that Eq. (5) is schematic and give the precise training objective in an appendix.","section":"Method, Unified Training Framework, Eq. (5)"}],"minor_comments":[{"comment":"The sentence \"PCL demonstrates consistent and improvements over all baseline methods\" is missing a noun; it should read \"consistent and significant improvements\" or similar.","section":"Experiments, Main Results"},{"comment":"Model names are inconsistent: Table 2 uses \"LlaMA3.2\" while Table 3 uses \"LlaMA3.1\"; the standard spelling is \"LLaMA.\"","section":"Table 2 / Table 3"},{"comment":"The text refers to \"RFT (Joint SFT+RL) w/ eval output\" but the table row is labeled \"w/ eval output + post-completion output\"; please align the nomenclature.","section":"Table 2 / Main Results"},{"comment":"The description of the teacher-distillation baseline is confusing: the table says \"Teacher distillation only\" is SFT on reasoning+evaluation data, while the text calls it \"corrected responses from teacher models\"; clarify what is distilled and what is filtered.","section":"Method, Evaluation Data Preparation"},{"comment":"The example text \"Then, 7 + 5 = 14 is wrong\" is confusing because 7+5=12; the intended point is that the earlier step was wrong. Rephrase the self-evaluation to say explicitly that 7+5=12, so the stated final answer of 14 is incorrect.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The empirical idea is interesting and potentially publishable if the evaluation asymmetry is resolved. I would be willing to review a revision that rescors all methods format-agnostically, reports uncertainty, and either proves or substantially softens the theoretical claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nHere's my read on arXiv:2507.20252. The core idea is genuinely new: train the model to continue generating evaluation and reward tokens after the answer marker, then stop at a new token at inference. That gives you a training-time self-evaluation signal with zero inference overhead. The consistency reward—comparing the model's verbalized scores to scores computed from ground truth and format rules—is a concrete supervision target that I haven't seen in the training literature. The experiments cover three datasets and multiple model sizes, and the ablations attribute parts of the gain to the evaluation SFT and the consistency reward. That is a real contribution.\n\nThe soft spots are real too. The evaluation protocol is the biggest one. Accuracy is measured by exact matching on the field for every method, while PCL is the only method trained with a dedicated format reward. The paper admits this: the criteria are 'more stringent' and may lower baseline scores. The 'w/ eval output' baseline, which does include the format rewards, only narrows the gap to about +2 points on GSM8K rather than the +3.45 claimed over SFT+RL. So a meaningful part of the headline improvement looks like format compliance, not reasoning. This needs to be addressed with a format-agnostic re-scoring before the central claim is accepted.\n\nThe theory section is largely hand-waving. The mutual information expansion assumes the term I(θ;e,r|x,y) > 0 without proof; the sample complexity comparison is asserted, not derived; and the 'parameter separation' argument is vague. I'd either make it rigorous or cut it. Also, no code or data is released, which makes it hard to verify the results. MathQA for smaller models shows mixed results, and the paper acknowledges that.\n\nOverall, the central idea holds up as a plausible training-time method, and the empirical work is suggestive, but the magnitude of the benefit is uncertain because of the evaluation bias. I'd send it to review rather than desk-reject—the novelty is clear and a good referee could push the authors to fix the evaluation. I'd cite it as a related method, but not as strong evidence.\n\nBest,","headline":"A genuinely new training-time idea with plausible gains, but the exact-match evaluation protocol and hand-wavy theory keep the headline numbers from being fully convincing.","tokens_in":12698,"tokens_out":4454,"would_cite":true,"duration_ms":42389,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a language model to keep writing after its final answer—generating a self-evaluation and predicted reward scores in a region—improves reasoning accuracy on math benchmarks while adding zero inference tokens…","keywords":["post-completion learning","self-evaluation","white-box reinforcement learning","consistency reward","GRPO","chain-of-thought reasoning","language model training","inference efficiency"],"falsifier":"Re-run the comparison with a lenient answer extractor that searches the full response for the final number, as traditional math-verify scoring does, and see whether PCL's accuracy advantage over SFT and SFT+RL survives; if the gap collapses, the gains are format compliance rather than reasoning.","tokens_in":11814,"feed_emoji":"🧠","tokens_out":10567,"duration_ms":85537,"temperature":0.7,"pith_summary":"Post-Completion Learning (PCL) is a training scheme that lets a language model keep generating after it has produced its final answer, writing a self-assessment and predicted reward values in a new region. At deployment the model stops at that marker, so the reflection costs nothing at inference time. The paper reports that models trained this way score higher on GSM8K, StrategyQA, and MathQA than models trained with supervised fine-tuning, sequential SFT-then-RL, or joint SFT+RL, with average gains over SFT of 7.81, 6.11, and 6.27 percentage points respectively. The central bet is that forcing a model to compute and verbalize reward-function scores during training makes it internalize self-evaluation, and that this internalized evaluation improves the answers that are actually served.","feed_headline":"Self-evaluation after answers improves reasoning, no extra tokens","feed_subtitle":"Models that grade their own output after answering beat SFT and RL baselines on math and reasoning benchmarks.","key_machinery":"The load-bearing object is the token, a separator that splits the sequence into a reasoning region ( + ) and a reflection region ( + ). Around it the paper builds a white-box reinforcement learning loop: the model is given the reward rules in its prompt, works through accuracy and format scoring inside the evaluation passage, outputs predicted values in the reward block, and is supervised by a consistency reward that compares those predictions with the true reward-function outputs. Dual-track SFT trains the two regions separately, and GRPO samples eight responses per group and optimizes the sum of accuracy, format, and consistency rewards. The token is what makes the scheme zero-cost at inference: because it is a stop word for generation, the reflection region is trained but never decoded.","core_discovery":"PCL's central claim is that the sequence space after the end-of-sequence token is a usable training resource. Instead of ending learning at , the model is trained to emit , then an passage that recomputes the accuracy and format rewards, then a block containing predicted scores. A consistency reward $R_c = 1 - |R_{pred} - R_{true}|_1$ aligns those predicted scores with true reward-function outputs, GRPO optimizes the full sequence, and dual-track SFT keeps the reasoning loss (think + answer) and evaluation loss (evaluation + reward) separate. At inference is registered as a stop word, so the reflection content is never generated. On GSM8K, StrategyQA, and MathQA, PCL outperforms SFT, SFT+RL, and joint SFT+RL baselines on exact-match answer accuracy, with average improvements over SFT of +7.81, +6.11, and +6.27 percentage points and over SFT+RL of +3.45, +4.56, and +1.14 percentage points respectively.","pith_inferences":["If PCL's benefit comes from internalized self-evaluation rather than from the extra output tokens, the recipe should transfer to code generation and multi-turn dialogue, where the model could reflect on compiler errors or conversation goals during training without paying for that reflection at serving time.","Because the consistency reward penalizes mismatches between predicted and true reward scores, PCL is implicitly a calibration training method; a testable prediction is that PCL models would show better confidence calibration on out-of-domain questions, not merely higher accuracy.","The white-box format also creates an audit trail: since the model writes out its reward calculations, a deployed system could read the evaluation text to detect fabricated self-assessments, turning reward hacking into an inspectable failure mode instead of a hidden one."],"forward_implications":["PCL models serve answers at the same latency as baselines: the token stops generation before any self-evaluation text is produced, so the training-time reflection is free at inference.","Smaller models get the largest bumps on structured reasoning, with LLaMA3.2-3B up 10.47 points and Qwen2.5-1.5B up 7.20 points over SFT on GSM8K, suggesting the self-evaluation signal helps most when raw capacity is limited.","Both evaluation-track SFT and the consistency reward carry weight: dropping the evaluation SFT lowers GSM8K accuracy to 67.02 and dropping the consistency reward lowers it to 66.08, compared with 67.93 for the full method.","Training reasoning and evaluation objectives jointly in one stage beats sequential SFT-then-RL in the reported experiments, matching the paper's claim that avoiding the forgetting penalty improves sample efficiency.","The gains appear across three model families (Qwen2.5 and LLaMA3.x) and three reasoning benchmarks, which supports treating PCL as a general training recipe rather than a dataset-specific adjustment."],"supporting_citations":[{"why":"Supplies the GSM8K benchmark used as the primary arithmetic-reasoning evaluation.","marker":"(Cobbe et al. 2021)"},{"why":"Supplies StrategyQA, the multi-hop reasoning benchmark evaluated in the main results.","marker":"(Geva et al. 2021)"},{"why":"Supplies MathQA, the large-scale math benchmark where PCL's gains are smallest and scale-dependent.","marker":"(Amini et al. 2019)"},{"why":"Provides the GRPO algorithm used for the reinforcement-learning stage of PCL.","marker":"(Shao et al. 2024)"},{"why":"Cited as prior proof that mixed SFT and RL training converges, supporting Theorem 3 of the paper.","marker":"(Fu et al. 2025)"}],"fun_headline_variants":["Train past to boost reasoning and self-grading","Self-evaluate after answer: better reasoning, no extra tokens","Post-completion training improves reasoning without inference cost","Use the space after for training, not just stopping"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume exact matching on the region is equally fair to PCL and to baseline models, even though PCL is the only method explicitly trained with a format reward to produce those tags.","fun_headline_variants_meta":{"raw":{"variants":["Train past to boost reasoning and self-grading","Self-evaluate after answer: better reasoning, no extra tokens","Post-completion training improves reasoning without inference cost","Use the space after for training, not just stopping"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1323,"prompt_tokens":956,"completion_tokens":367,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":296}},"tokens_in":572,"tokens_out":367,"duration_ms":4277,"temperature":1.0,"reasoning_tokens":296,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:46:47.554425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison with a lenient answer extractor that searches the full response for the final number, as traditional math-verify scoring does, and see whether PCL's accuracy advantage over SFT and SFT+RL survives; if the gap collapses, the gains are format compliance rather than reasoning.","supporting_citations":[],"review_version":1}