{"id":"cffc63f8-46d7-4e04-9d2c-1d2afebb5caa","arxiv_id":"2504.18376","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"GRPO reinforcement learning with chain-of-thought improves NLI accuracy on adversarial benchmarks for 7B-32B Qwen models, and most gains survive 4-bit AWQ quantization.","lead":"This paper trains large language models for natural language inference using reinforcement learning with chain-of-thought prompting, removing the need for human-written explanations. The authors report accuracy gains on adversarial benchmarks that persist under 4-bit quantization, with code and adapters made public.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward scores only the final label, so the CoT need not be causally responsible; Appendix E invalid rationales undermine the robust-reasoning claim.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the reward function in Section 3.2 only scores the final label, so the CoT is assumed to be causal. Appendix E confirms the risk with concrete invalid rationales that still receive full reward. I agree this is the most load-bearing concern because it attacks the paper's stated contribution (CoT learning, robust reasoning) rather than secondary issues like baseline fairness. The within-base-model accuracy gains in Table 3 are real evidence that GRPO improves accuracy, and the public release of adapters, training data, and code is a genuine strength. However, the interpretation of those gains as evidence of robust reasoning is not supported without a CoT-fidelity check. Since the reader already assigned CONDITIONAL based on this concern, my stress-test does not move the verdict; it sharpens the required condition: either demonstrate that the CoT is logically faithful or reframe the claims as accuracy improvements without the reasoning interpretation. The proposed audit is feasible and would settle whether the Appendix E examples are isolated or systemic.","tokens_in":24776,"tokens_out":8359,"duration_ms":86193,"concrete_test":"Using the released pablomiralles22/Qwen2.5-32B-AWQ-GRPO_r64-NLI adapter, sample greedy-decoding outputs on 200 randomly selected ANLI R3 test examples. Have three annotators independently judge whether the <think> rationale is logically sufficient for the predicted <answer>, using a rubric calibrated on the Appendix E examples. If the proportion of correct-label examples rated invalid by at least two annotators exceeds 25%, or if annotators cannot agree (Fleiss' kappa < 0.6), the CoT is not demonstrably the cause of correct predictions and the robust-reasoning claim should be withdrawn or substantially weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GRPO yields chain-of-thought NLI with robust reasoning (Abstract) rests on the assumption that the sampled <think> rationale is the cause of the correct <answer>. But Section 3.2's reward function grades only the final label; nothing in the optimization verifies the CoT. Appendix E provides direct counterexamples from the 32B AWQ model: manslaughter is treated as entailing murder, and 'optional' is treated as 'not needed,' yet the final labels match gold. This shows the policy can increase reward while producing logically invalid reasoning, i.e., it may be fitting ANLI-specific labeling conventions rather than learning general inference. Because the paper's novelty and motivation are explicitly about CoT learning and robust reasoning (Abstract, Section 1), the reported accuracy gains on static benchmarks do not establish the central claim unless CoT fidelity is demonstrated. The public adapters make this testable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes applying Group Relative Policy Optimization (GRPO), a reinforcement learning algorithm, to fine-tune large language models (Qwen2.5 7B, 14B, 32B) with LoRA/QLoRA for natural language inference, using the ANLI R3 training set filtered to examples missed by a 7B model. The model is prompted to emit a chain-of-thought before a final entailment/neutral/contradiction answer, and only the final label is rewarded. The authors report consistent accuracy gains over the same-base models across standard and adversarial NLI benchmarks, with the largest gains for the 32B AWQ-quantized model, which they claim surpasses state-of-the-art results on 7 of 11 adversarial sets (or all of them using their own replication of a competing method) within a 22GB memory footprint. The paper also analyzes training dynamics and includes an appendix with seven examples where the GRPO model's rationale is logically invalid despite matching the gold label.","tokens_in":24862,"tokens_out":5261,"duration_ms":51033,"significance":"If the central claims hold, the work is a useful empirical demonstration that GRPO-style RL with parameter-efficient fine-tuning can improve NLI accuracy on adversarial benchmarks, and the release of code, adapters, and the filtered training set is a strength that enables reproduction and extension. The same-base comparisons in Section 5.3 are internally consistent and show that GRPO improves average accuracy over the base Qwen models in most configurations. However, the paper's headline claims about 'robust reasoning' and 'state-of-the-art' performance are not yet supported: the reward does not verify the chain-of-thought, the paper's own Appendix E contains counterexamples to robust reasoning, and the SOTA comparison relies on outdated baselines and the authors' own unreplicated implementation of a competing method. The core empirical contribution is therefore sound but the interpretation overreaches.","major_comments":[{"comment":"The claim that the 32B AWQ model 'surpasses state-of-the-art results on 7 out of 11 adversarial sets—or on all of them considering our replication' is not adequately supported. Table 2 compares against RoBERTa-large baselines from 2020-2022 (e.g., ANLI r3 SOTA of 47.70%), which are not current state-of-the-art for these benchmarks. The 'all of them' variant rests on the authors' own replication of Kavumba et al. [1], reported in Appendix B, where the replication scores are far below the published numbers (e.g., 52.10% vs 81.80% on ANLI r1). Because the replication may itself be faulty, the authors are effectively both contestant and referee for this baseline. I recommend either using verified current leaderboard numbers, or clearly limiting the claim to 'exceeds the listed baselines' and removing the 'all of them' phrasing.","section":"Abstract, Section 5.1, Table 2, Appendix B"},{"comment":"The paper's central motivation is that GRPO enables chain-of-thought learning and 'robust reasoning' (Abstract, Section 1), but the reward function in Section 3.2 scores only the final label and does not verify the sampled rationale. The paper's own Appendix E shows multiple cases where the GRPO-trained 32B model produces logically invalid rationales (e.g., 'optional' treated as 'not needed' in Example 1; manslaughter treated as entailing murder in Example 6; misattribution treated as contradiction in Example 7) yet the final label matches the gold label. This indicates the policy can increase reward while learning dataset-specific labeling conventions rather than general inference, which directly undermines the 'robust reasoning' claim. Since the adapters are public, the authors should either demonstrate CoT fidelity (e.g., a consistency or human-evaluation study) or substantially soften the robustness claim in the abstract and conclusions.","section":"Section 3.2, Section 5.7, Appendix E"},{"comment":"All training runs use the ANLI R3 subset filtered to examples missed by a Qwen2.5-7B-Instruct model ('one-missed'). This filter is a design choice that affects which examples produce nonzero advantages, and no sensitivity analysis is provided (e.g., unfiltered R3, a different filter model, or a filter based on the same model being adapted). It is possible that the reported gains are partially an artifact of this particular selection. I recommend ablating the filter or at least reporting the size of the filtered set and training on unfiltered R3 as a control.","section":"Section 3.3"}],"minor_comments":[{"comment":"The sentence 'predictions that contradict the explanation are rarely sampled' is asserted without supporting statistics; if the authors have such data, it should be reported, otherwise the claim should be removed.","section":"Section 3.2"},{"comment":"The prompt template contains obvious typos such as 'f oll ow ing' and 'de te rm ine'; these should be corrected in the final version.","section":"Listing 1, Appendix D"},{"comment":"The column layout in Tables 4 and 5 is dense and hard to read; restructuring them (e.g., separate columns for No-AWQ and AWQ with a clear Δ) would improve clarity.","section":"Tables 4 and 5"},{"comment":"The phrase '7 out of 11 adversarial sets' is ambiguous because Table 2 shows the 32B model beating all listed baselines on all 11 adversarial rows; the abstract should specify that the '7 out of 11' refers to comparison with the reported numbers of Kavumba et al. [1].","section":"Section 5.1"},{"comment":"The 7B (AWQ) results in Table 3 use batch size 8, while Table 2 uses batch size 32 (as noted in the caption of Table 2); this difference should be mentioned in Section 5.3 to avoid confusion.","section":"Section 5.3"},{"comment":"The replication section would be stronger if it reported the training setup for the replication (e.g., learning rate, number of steps, checkpoint selection) and not only the final scores, so readers can judge the replication's fidelity.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper's SOTA claim is entangled with the authors' own failed replication of Kavumba et al. [1], which is a conflict-of-interest situation in evaluation. The editor may wish to ask the authors to compare against published leaderboard numbers or clearly separate 'our evaluation of prior work' from 'reported prior work'. The core empirical study (same-base GRPO gains, public artifacts) is valuable and likely within scope for a CL journal if the claims are calibrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the real contribution is credible. GRPO training with LoRA moves the needle on NLI for Qwen2.5 models, and the authors ship adapters, data, and code. The packaging overstates it: the \"state of the art\" claim leans on unreplicated and size-unfair baselines, and the \"robust reasoning\" framing is not supported by the evidence they provide.\n\nWhat is actually new: this is the first application of GRPO/CoT reinforcement learning to NLI without labeled rationales, letting them train on ANLI R3. They test 7B, 14B, and 32B models, quantized and not, across LoRA ranks. The same-base comparisons are the strong part: GRPO improves average accuracy over base Qwen at every size, and the largest model gains on nearly every adversarial set. The quantization story is also useful: LoRA recovers most of the AWQ loss, and the 32B model runs in 22GB. Public artifacts make this genuinely reproducible.\n\nSoft spots, in proportion:\n\n- The \"robust reasoning\" claim is the main one. The reward scores only the final label, so nothing forces the CoT to be the cause of the answer. Appendix E shows concrete cases where the GRPO model emits logically invalid rationales (manslaughter entailing murder; optional entailing not needed) yet hits the gold label. That means accuracy gains on static benchmarks do not establish robust reasoning. The paper should claim improved label accuracy, not sound CoT.\n- The SOTA claim is weaker than it looks. Most Table 2 baselines are much smaller RoBERTa models, which the authors themselves call unfair. The prior SOTA [1] could not be replicated, so the 7-of-11 vs all-11 line depends on a comparison the authors ran themselves. I would reframe as competitive within same-base comparisons or add comparable-scale baselines.\n- The hard-example filter (one-missed on ANLI R3) has no ablation. It is plausible, but there is no evidence it helped.\n- No variance is reported in the main tables, though the text says three seeds were averaged. Report the seed spread, at least for Table 2.\n\nNone of this kills the paper. The core same-base result is solid, the citation pattern is fine, and the attempted replication of [1] with released code is good practice. The paper is for people working on RL for NLI, efficient fine-tuning, and fact-checking. It deserves a serious referee and likely acceptance after a revision that reframes the robustness claim and adds the missing ablations and variance.","headline":"A solid, artifact-rich first look at GRPO-trained CoT for NLI; same-base gains are real, but the SOTA and robust-reasoning claims outrun the evidence.","tokens_in":25493,"tokens_out":2796,"would_cite":true,"duration_ms":29703,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper tries to establish that reinforcement learning can replace labeled rationales in natural-language-inference training, using Group Relative Policy Optimization over a model's own chain-of-thought completions, and that this works…","keywords":["natural language inference","GRPO","chain-of-thought reasoning","LoRA","AWQ quantization","adversarial benchmarks","reinforcement learning","ANLI"],"falsifier":"If, on a newly constructed set of adversarial premise-hypothesis pairs that avoid annotation conventions, a GRPO-trained model shows no accuracy advantage over its untrained base, or if a version trained to output only the label without the required explanation matches its accuracy, then the claim that chain-of-thought reasoning is the source of the gains is refuted.","tokens_in":1539,"feed_emoji":"🧠","tokens_out":1708,"duration_ms":55769,"temperature":0.7,"pith_summary":"The paper argues that Group Relative Policy Optimization (GRPO), a memory-efficient reinforcement learning algorithm, can replace supervised fine-tuning for natural language inference (NLI). Instead of requiring human-written rationales, GRPO samples chain-of-thought explanations from the model itself and rewards only the final label. The authors fine-tune 7B, 14B, and 32B quantized LLMs with LoRA and report that the 32B model surpasses prior state-of-the-art on 7 out of 11 adversarial NLI sets (or all of them under their own replication of the competing method) while running in about 22GB of CUDA memory. If correct, this makes robust NLI practical for memory-constrained hardware and avoids the need for costly explanation annotations.","feed_headline":"RL-trained 32B model tops 7 of 11 adversarial NLI sets","feed_subtitle":"The quantized model runs in 22GB and reasons through NLI with no labeled rationales.","key_machinery":"The load-bearing mechanism is GRPO, a policy-gradient reinforcement-learning algorithm that samples a group of completions per prompt and computes a normalized advantage without requiring a critic model, paired with a KL-divergence penalty that keeps the policy close to the base model. LoRA and QLoRA make the fine-tuning parameter-efficient, while the prompt forces the model to emit a <think>...</think> explanation before an <answer>...</answer> label, and the reward is 1 for a correct label and 0 otherwise. Because the explanation itself is never supervised, the model explores its own rationales during training. The training data is ANLI Round 3 filtered to examples that a 7B model initially misses, which gives the group-advantage signal a nonzero spread and avoids wasted updates on already-solved examples.","core_discovery":"The paper's central claim is that GRPO training with LoRA is an effective, memory-lean way to learn chain-of-thought reasoning for NLI without human-written rationales. Applying the GRPO objective to 7B, 14B, and 32B instruct models, while sampling online explanations and scoring only the final label, yields consistent gains over the untrained base on ANLI, Counter-NLI, HANS, NLI Diagnostic, SNLI, and MultiNLI. The 32B quantized model reaches 82.46% average accuracy across the selected sets and surpasses prior state-of-the-art results on 7 of 11 adversarial sets, or all of them under the authors' own replication of the competing method. The paper also claims that AWQ quantization costs only about a point of average accuracy after GRPO training, so aggressive compression does not erase the robustness gains.","pith_inferences":["Beyond the paper: because the reward never inspects the rationale, the method may be learning label conventions that hold on ANLI but not on genuinely novel adversarial data; the appendix's examples of logically invalid rationales that still match the gold label make this a live risk.","Beyond the paper: one can test whether the chain of thought is causal by training a control that outputs only the label with the same reward; if the control matches GRPO accuracy, the explanation format is not the source of the gain.","Beyond the paper: the reported flat completion lengths suggest that, unlike mathematics reinforcement learning, NLI gains come from better label selection rather than longer deliberation, so combining GRPO with a small supervised rationale signal could push robustness further."],"forward_implications":["NLI training no longer depends on datasets with human explanation annotations, so harder adversarial corpora such as ANLI can be used directly as training data.","A 32B quantized model fitting in 22GB of CUDA memory can deliver state-of-the-art adversarial NLI accuracy on a single 24GB or 48GB GPU.","Quantization-induced accuracy drops shrink after GRPO training with higher LoRA ranks, so memory savings and reasoning quality are not necessarily in conflict.","Larger base models generalize to adversarial benchmarks even before fine-tuning, suggesting that pretraining capacity sets the ceiling for what GRPO can unlock.","If the claim holds, the accuracy-versus-model-size frontier for NLI shifts, making robust inference feasible at substantially lower compute budgets."],"supporting_citations":[{"why":"Supplies the GRPO objective and training procedure used throughout the paper.","marker":"[32]"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method for the non-quantized models.","marker":"[34]"},{"why":"Provides QLoRA, enabling fine-tuning of AWQ-quantized bases.","marker":"[35]"},{"why":"Defines AWQ quantization, the compression method whose memory savings and accuracy effects are central to the results.","marker":"[36]"},{"why":"Introduces ANLI, the adversarial benchmark used both for GRPO training data and for evaluation.","marker":"[11]"},{"why":"Introduces HANS, the syntactic-heuristic adversarial test set the models are evaluated on.","marker":"[9]"},{"why":"Introduces Counter-NLI, the counterfactual benchmark used to test resilience to hypothesis-only bias.","marker":"[10]"},{"why":"Introduces NLI Diagnostic, the fine-grained linguistic-phenomena evaluation set used in the paper.","marker":"[8]"},{"why":"Provides the prior state-of-the-art explanation-fine-tuning approach that the paper compares against and attempts to replicate.","marker":"[1]"},{"why":"Informs the decision to remove the standard-deviation division from the advantage normalization in the GRPO objective.","marker":"[33]"}],"fun_headline_variants":["GRPO teaches 32B model NLI reasoning without rationales","32B quantized NLI model beats SOTA on 7 adversarial sets","RL with GRPO yields robust NLI at 22GB, no labeled rationales","Without rationales, GRPO-trained 32B tops 7 of 11 adversarial NLI","22GB quantized model uses RL to reason through NLI without rationales"],"cache_read_input_tokens":27648,"weakest_assumption_plain":"The reward scores only the final answer, so the sampled chain of thought is assumed to be genuine reasoning that causes the correct label; if the model is instead matching dataset-specific labeling conventions, the robustness gains will not transfer.","fun_headline_variants_meta":{"raw":{"variants":["GRPO teaches 32B model NLI reasoning without rationales","32B quantized NLI model beats SOTA on 7 adversarial sets","RL with GRPO yields robust NLI at 22GB, no labeled rationales","Without rationales, GRPO-trained 32B tops 7 of 11 adversarial NLI","22GB quantized model uses RL to reason through NLI without rationales"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000524,"raw_usage":{"total_tokens":2525,"prompt_tokens":931,"completion_tokens":1594,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":1489}},"tokens_in":547,"tokens_out":1594,"duration_ms":12279,"temperature":1.0,"reasoning_tokens":1489,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:17:55.886527+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If, on a newly constructed set of adversarial premise-hypothesis pairs that avoid annotation conventions, a GRPO-trained model shows no accuracy advantage over its untrained base, or if a version trained to output only the label without the required explanation matches its accuracy, then the claim that chain-of-thought reasoning is the source of the gains is refuted.","supporting_citations":[{"cited_title":"Shao et al","cited_arxiv_id":null,"evidence_quote":"Supplies the GRPO objective and training procedure used throughout the paper."},{"cited_title":"LoRA: Low-Rank Adaptation of Large Language Models","cited_arxiv_id":null,"evidence_quote":"Provides LoRA, the parameter-efficient fine-tuning method for the non-quantized models."},{"cited_title":"QLoRA: Efficient Finetuning of Quantized LLMs","cited_arxiv_id":null,"evidence_quote":"Provides QLoRA, enabling fine-tuning of AWQ-quantized bases."},{"cited_title":"AWQ: Activation-Aware Weight Quantization for on-Device LLM Compression and Acceleration","cited_arxiv_id":null,"evidence_quote":"Defines AWQ quantization, the compression method whose memory savings and accuracy effects are central to the results."},{"cited_title":"Learning the Difference That Makes a Difference with Counterfactually- Augmented Data","cited_arxiv_id":null,"evidence_quote":"Introduces Counter-NLI, the counterfactual benchmark used to test resilience to hypothesis-only bias."}],"review_version":1}