Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Pushing the boundary on Natural Language Inference

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper tries to establish that reinforcement learning can replace labeled rationales in natural-language-inference training, using Group Relative Policy Optimization over a model's own chain-of-thought completions, and that this works…

desk verdict A solid, artifact-rich first look at GRPO-trained CoT for NLI; same-base gains are real, but the SOTA and robust-reasoning claims outrun the evidence. read the letter →

arxiv 2504.18376 v2 pith:JQH43LAQ submitted 2025-04-25 cs.CL cs.AI

Pablo Miralles-González , Javier Huertas-Tato , Alejandro Martín , David Camacho This is my paper · ORCID
classification cs.CLcs.AI
keywords naturallanguageinferenceGRPOchain-of-thoughtreasoningLoRAAWQquantizationadversarialbenchmarksreinforcementlearningANLI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that Group Relative Policy Optimization (GRPO), a memory-efficient reinforcement learning algorithm, can replace supervised fine-tuning for natural language inference (NLI). Instead of requiring human-written rationales, GRPO samples chain-of-thought explanations from the model itself and rewards only the final label. The authors fine-tune 7B, 14B, and 32B quantized LLMs with LoRA and report that the 32B model surpasses prior state-of-the-art on 7 out of 11 adversarial NLI sets (or all of them under their own replication of the competing method) while running in about 22GB of CUDA memory. If correct, this makes robust NLI practical for memory-constrained hardware and avoids the need for costly explanation annotations.

What carries the argument

The load-bearing mechanism is GRPO, a policy-gradient reinforcement-learning algorithm that samples a group of completions per prompt and computes a normalized advantage without requiring a critic model, paired with a KL-divergence penalty that keeps the policy close to the base model. LoRA and QLoRA make the fine-tuning parameter-efficient, while the prompt forces the model to emit a <think>...</think> explanation before an <answer>...</answer> label, and the reward is 1 for a correct label and 0 otherwise. Because the explanation itself is never supervised, the model explores its own rationales during training. The training data is ANLI Round 3 filtered to examples that a 7B model initially misses, which gives the group-advantage signal a nonzero spread and avoids wasted updates on already-solved examples.

What would settle it

If, on a newly constructed set of adversarial premise-hypothesis pairs that avoid annotation conventions, a GRPO-trained model shows no accuracy advantage over its untrained base, or if a version trained to output only the label without the required explanation matches its accuracy, then the claim that chain-of-thought reasoning is the source of the gains is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that GRPO training with LoRA is an effective, memory-lean way to learn chain-of-thought reasoning for NLI without human-written rationales. Applying the GRPO objective to 7B, 14B, and 32B instruct models, while sampling online explanations and scoring only the final label, yields consistent gains over the untrained base on ANLI, Counter-NLI, HANS, NLI Diagnostic, SNLI, and MultiNLI. The 32B quantized model reaches 82.46% average accuracy across the selected sets and surpasses prior state-of-the-art results on 7 of 11 adversarial sets, or all of them under the authors' own replication of the competing method. The paper also claims that AWQ quantization costs only about a point of average accuracy after GRPO training, so aggressive compression does not erase the robustness gains.

Load-bearing premise

The reward scores only the final answer, so the sampled chain of thought is assumed to be genuine reasoning that causes the correct label; if the model is instead matching dataset-specific labeling conventions, the robustness gains will not transfer.

Editorial extensions

If this is right

  • NLI training no longer depends on datasets with human explanation annotations, so harder adversarial corpora such as ANLI can be used directly as training data.
  • A 32B quantized model fitting in 22GB of CUDA memory can deliver state-of-the-art adversarial NLI accuracy on a single 24GB or 48GB GPU.
  • Quantization-induced accuracy drops shrink after GRPO training with higher LoRA ranks, so memory savings and reasoning quality are not necessarily in conflict.
  • Larger base models generalize to adversarial benchmarks even before fine-tuning, suggesting that pretraining capacity sets the ceiling for what GRPO can unlock.
  • If the claim holds, the accuracy-versus-model-size frontier for NLI shifts, making robust inference feasible at substantially lower compute budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the reward never inspects the rationale, the method may be learning label conventions that hold on ANLI but not on genuinely novel adversarial data; the appendix's examples of logically invalid rationales that still match the gold label make this a live risk.
  • Beyond the paper: one can test whether the chain of thought is causal by training a control that outputs only the label with the same reward; if the control matches GRPO accuracy, the explanation format is not the source of the gain.
  • Beyond the paper: the reported flat completion lengths suggest that, unlike mathematics reinforcement learning, NLI gains come from better label selection rather than longer deliberation, so combining GRPO with a small supervised rationale signal could push robustness further.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes applying Group Relative Policy Optimization (GRPO), a reinforcement learning algorithm, to fine-tune large language models (Qwen2.5 7B, 14B, 32B) with LoRA/QLoRA for natural language inference, using the ANLI R3 training set filtered to examples missed by a 7B model. The model is prompted to emit a chain-of-thought before a final entailment/neutral/contradiction answer, and only the final label is rewarded. The authors report consistent accuracy gains over the same-base models across standard and adversarial NLI benchmarks, with the largest gains for the 32B AWQ-quantized model, which they claim surpasses state-of-the-art results on 7 of 11 adversarial sets (or all of them using their own replication of a competing method) within a 22GB memory footprint. The paper also analyzes training dynamics and includes an appendix with seven examples where the GRPO model's rationale is logically invalid despite matching the gold label.

Significance. If the central claims hold, the work is a useful empirical demonstration that GRPO-style RL with parameter-efficient fine-tuning can improve NLI accuracy on adversarial benchmarks, and the release of code, adapters, and the filtered training set is a strength that enables reproduction and extension. The same-base comparisons in Section 5.3 are internally consistent and show that GRPO improves average accuracy over the base Qwen models in most configurations. However, the paper's headline claims about 'robust reasoning' and 'state-of-the-art' performance are not yet supported: the reward does not verify the chain-of-thought, the paper's own Appendix E contains counterexamples to robust reasoning, and the SOTA comparison relies on outdated baselines and the authors' own unreplicated implementation of a competing method. The core empirical contribution is therefore sound but the interpretation overreaches.

major comments (3)
  1. [Abstract, Section 5.1, Table 2, Appendix B] The claim that the 32B AWQ model 'surpasses state-of-the-art results on 7 out of 11 adversarial sets—or on all of them considering our replication' is not adequately supported. Table 2 compares against RoBERTa-large baselines from 2020-2022 (e.g., ANLI r3 SOTA of 47.70%), which are not current state-of-the-art for these benchmarks. The 'all of them' variant rests on the authors' own replication of Kavumba et al. [1], reported in Appendix B, where the replication scores are far below the published numbers (e.g., 52.10% vs 81.80% on ANLI r1). Because the replication may itself be faulty, the authors are effectively both contestant and referee for this baseline. I recommend either using verified current leaderboard numbers, or clearly limiting the claim to 'exceeds the listed baselines' and removing the 'all of them' phrasing.
  2. [Section 3.2, Section 5.7, Appendix E] The paper's central motivation is that GRPO enables chain-of-thought learning and 'robust reasoning' (Abstract, Section 1), but the reward function in Section 3.2 scores only the final label and does not verify the sampled rationale. The paper's own Appendix E shows multiple cases where the GRPO-trained 32B model produces logically invalid rationales (e.g., 'optional' treated as 'not needed' in Example 1; manslaughter treated as entailing murder in Example 6; misattribution treated as contradiction in Example 7) yet the final label matches the gold label. This indicates the policy can increase reward while learning dataset-specific labeling conventions rather than general inference, which directly undermines the 'robust reasoning' claim. Since the adapters are public, the authors should either demonstrate CoT fidelity (e.g., a consistency or human-evaluation study) or substantially soften the robustness claim in the abstract and conclusions.
  3. [Section 3.3] All training runs use the ANLI R3 subset filtered to examples missed by a Qwen2.5-7B-Instruct model ('one-missed'). This filter is a design choice that affects which examples produce nonzero advantages, and no sensitivity analysis is provided (e.g., unfiltered R3, a different filter model, or a filter based on the same model being adapted). It is possible that the reported gains are partially an artifact of this particular selection. I recommend ablating the filter or at least reporting the size of the filtered set and training on unfiltered R3 as a control.
minor comments (6)
  1. [Section 3.2] The sentence 'predictions that contradict the explanation are rarely sampled' is asserted without supporting statistics; if the authors have such data, it should be reported, otherwise the claim should be removed.
  2. [Listing 1, Appendix D] The prompt template contains obvious typos such as 'f oll ow ing' and 'de te rm ine'; these should be corrected in the final version.
  3. [Tables 4 and 5] The column layout in Tables 4 and 5 is dense and hard to read; restructuring them (e.g., separate columns for No-AWQ and AWQ with a clear Δ) would improve clarity.
  4. [Section 5.1] The phrase '7 out of 11 adversarial sets' is ambiguous because Table 2 shows the 32B model beating all listed baselines on all 11 adversarial rows; the abstract should specify that the '7 out of 11' refers to comparison with the reported numbers of Kavumba et al. [1].
  5. [Section 5.3] The 7B (AWQ) results in Table 3 use batch size 8, while Table 2 uses batch size 32 (as noted in the caption of Table 2); this difference should be mentioned in Section 5.3 to avoid confusion.
  6. [Appendix B] The replication section would be stronger if it reported the training setup for the replication (e.g., learning rate, number of steps, checkpoint selection) and not only the final scores, so readers can judge the replication's fidelity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core results are external-benchmark measurements, with only a disclosed replication caveat that is a comparison-protocol concern rather than a circular derivation.

full rationale

The paper does not derive its central results from fitted parameters or from self-citations. The GRPO reward in Section 3.2 scores only the final label, and the training signal is the gold label on a filtered ANLI R3 training set, while evaluation is performed on held-out public benchmarks (SNLI, MultiNLI, ANLI R1-R3, Counter-NLI, HANS, NLI-Diagnostic). No predicted quantity is equivalent by construction to a fitted input. The Appendix E examples show that the trained model can produce logically invalid rationales while matching gold labels; this is a validity threat to the robust-reasoning claim, not a circularity, because the reported accuracies are still measured against external labels. The 'all 11 adversarial sets' claim depends on the authors' own replication of prior work [1], disclosed in Appendix B; this is a fairness-of-comparison issue rather than a derivation-from-inputs issue, and the paper explicitly separates reported results from replication results in Table 7. There are no load-bearing self-citations or imported uniqueness theorems. The paper is self-contained against external benchmarks, so the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central empirical claim rests on the standard RL training setup and on the base model's pretrained competence. The main choices that are not independently justified are the one-missed training filter, the reward design that ignores chain-of-thought fidelity, and the decision to compare against small RoBERTa baselines. No new entities are introduced.

free parameters (5)
  • LoRA rank for reported models = 64
    Selected based on the ablation in Section 5.5; rank 128 gives slightly higher averages in some setups but rank 64 is used for the main comparison table.
  • GRPO KL regularization coefficient beta = 0.01
    Fixed hyperparameter in Table 6; controls deviation from the base model.
  • ANLI R3 one-missed filter = examples where Qwen2.5-7B-Instruct predicts incorrectly
    Defines the GRPO training set in Section 3.3; the filter is not ablated, so its contribution to the reported gains is unknown.
  • Batch size = 32 for 7B, 8 for 14B and 32B
    Set by memory constraints; affects the group-relative advantage statistics in GRPO.
  • Learning rate = 5e-6
    Fixed hyperparameter in Table 6.
assumptions (5)
  • domain assumption Qwen2.5 base models have sufficient semantic and logical knowledge from pretraining to benefit from GRPO on NLI.
    Section 1: if base transformer models do not acquire sufficient semantic understanding from pretraining, they will not be able to learn the task through fine-tuning on small, biased datasets.
  • domain assumption A reward based only on the final label is an adequate training signal for NLI.
    Section 3.2: the reward enforces the output format and correct label, but does not check whether the chain-of-thought is faithful; the paper assumes this still yields robust inference.
  • domain assumption The group-relative advantage without division by standard deviation is stable for this setting.
    Section 3.1 follows the recommendation in Dr. GRPO [33].
  • domain assumption Gold labels in the evaluation sets are reliable.
    Section 5.7 and Appendix E note that some gold labels appear ambiguous or incorrect, which contradicts the assumption.
  • domain assumption The evaluation benchmarks have not leaked into the pretraining corpora.
    Sections 5.1 and 6 acknowledge this concern and restrict reliable comparisons to same-base models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pushing the boundary on Natural Language Inference." pith.science (2026). https://pith.science/paper/JQH43LAQ

@misc{pith2026250418376,
  author       = {Pith},
  title        = {Pith review of: Pushing the boundary on Natural Language Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JQH43LAQ}},
  note         = {Machine review of arXiv:2504.18376}
}
abstract

Natural Language Inference (NLI) is a central task in natural language understanding with applications in fact-checking, question answering, and information retrieval. Despite its importance, current NLI systems heavily rely on supervised learning with datasets that often contain annotation artifacts and biases, limiting generalization and real-world applicability. In this work, we apply a reinforcement learning-based approach using Group Relative Policy Optimization (GRPO) for Chain-of-Thought (CoT) learning in NLI, eliminating the need for labeled rationales and enabling this type of training on more challenging datasets such as ANLI. We fine-tune 7B, 14B, and 32B language models using parameter-efficient techniques (LoRA and QLoRA), demonstrating strong performance across standard and adversarial NLI benchmarks. Our 32B AWQ-quantized model surpasses state-of-the-art results on 7 out of 11 adversarial sets$\unicode{x2013}$or on all of them considering our replication$\unicode{x2013}$within a 22GB memory footprint, showing that robust reasoning can be retained under aggressive quantization. This work provides a scalable and practical framework for building robust NLI systems without sacrificing inference quality.

Figures

Figures reproduced from arXiv: 2504.18376 by the authors.

Figure 1
Figure 1. Average accuracy on our dataset selection against model size on GPU. Models fine-tuned by GRPO have a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Effect of GRPO training on ANLI accuracy by model size, test set and decoding temperature. For non-greedy [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. presents the training dynamics—specifically, the reward, completion length, and KL divergence—of a selection of Qwen2.5-7B-Instruct checkpoints trained with LoRA ranks of 8, 16, 32, 64, and 128, using the GRPO algorithm. All curves are smoothed using an exponential moving average. The final reward exhibits a slight upward trend as the LoRA rank increases, suggesting that further gains in reward may be achievable by … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Training metrics by number of parameters for [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Confusion matrices between the base and GRPO models’ predictions, and between predictions and gold [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MERGE: Minimal Expression-Replacement GEneralization Test for Natural Language Inference

    cs.CL 2025-10 conditional novelty 6.0 of 10

    A new automatic method for creating minimal reasoning-preserving variants of NLI problems shows that 14 models drop 4 to 20 percent in accuracy on those variants.

Reference graph

Works this paper leans on

41 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [2]

    A Large Annotated Corpus for Learning Natural Language Inference

    S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning. “A Large Annotated Corpus for Learning Natural Language Inference”. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Ed. by L. Màrquez, C. Callison-Burch, and J. Su. Lisbon, Portugal: Association for Computational Linguistics, Sept. 2015, pp. 632–642. DOI: 10....

    doi:10.18653/v1/d15-1075 2015

  2. [3]

    E-SNLI: Natural Language Inference with Natural Language Explanations

    O.-M. Camburu, T. Rocktäschel, T. Lukasiewicz, and P. Blunsom. “E-SNLI: Natural Language Inference with Natural Language Explanations”. In: Advances in Neural Information Processing Systems . Ed. by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett. V ol. 31. Curran Asso- ciates, Inc., 2018. URL: https : / / proceedings . n...

    work page 2018

  3. [4]

    A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

    A. Williams, N. Nangia, and S. Bowman. “A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference”. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Proceedings of the 2018 Conference of the North American Chapter...

    2018

  4. [5]

    Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment

    M. Tsuchiya. “Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment”. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Ed. by N. Calzolari et al. Miyazaki, Japan: European Language Resources Association (ELRA), May 2018. URL: https://aclanthology.org/L18-1239/

    work page 2018

  5. [6]

    Annotation Artifacts in Natural Language Inference Data

    S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. Bowman, and N. A. Smith. “Annotation Artifacts in Natural Language Inference Data”. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Proceedings of the 2018 Conference of the No...

    2018

  6. [7]

    Hypothesis Only Baselines in Natural Language Inference

    A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, and B. Van Durme. “Hypothesis Only Baselines in Natural Language Inference”. In: Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. New Orleans, Louisiana: Association for Computational Ling...

    doi:10.18653/v1/s18-2023 2018

  7. [8]

    GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman. “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding”. In: Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks f...

    doi:10.18653/v1/w18-5446 2018

  8. [9]

    Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

    T. McCoy, E. Pavlick, and T. Linzen. “Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference”. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computation...

    doi:10.18653/v1/p19-1334 2019

Show all 41 references
  1. [10]

    Learning the Difference That Makes a Difference with Counterfactually- Augmented Data

    D. Kaushik, E. Hovy, and Z. Lipton. “Learning the Difference That Makes a Difference with Counterfactually- Augmented Data”. In: International Conference on Learning Representations . 2020. URL: https : / / openreview.net/forum?id=Sklgs0NFvr

    2020

  2. [11]

    Adversarial NLI: A New Benchmark for Natural Language Understanding

    Y . Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela. “Adversarial NLI: A New Benchmark for Natural Language Understanding”. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Proceedings of the 58th Annual Meeting of the...

    2020 doi

  3. [12]

    W ANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation

    A. Liu, S. Swayamdipta, N. A. Smith, and Y . Choi. “W ANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation”. In: Findings of the Association for Computational Linguistics: EMNLP 2022. Abu Dhabi, United Arab Emirates: Association for Computational L...

    2022

  4. [13]

    Attention Is All You Need

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. “Attention Is All You Need”. In: Advances in Neural Information Processing Systems. V ol. 30. Curran Associates, Inc., 2017

    2017

  5. [14]

    BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding”. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog...

    2019 doi

  6. [15]

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. July 26, 2019. arXiv: 1907.11692 [cs]. URL: http://arxiv.org/abs/1907.11692 (visited on 09/01/2023). Pre-published

    2019 arXiv

  7. [16]

    DeBERTa: Decoding-Enhanced BERT with Disentangled Attention

    P. He, X. Liu, J. Gao, and W. Chen. “DeBERTa: Decoding-Enhanced BERT with Disentangled Attention”. In: International Conference on Learning Representations. 2021. URL: https://openreview.net/forum?id= XPZIaotutsD

    2021

  8. [17]

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu. “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”. In: Journal of Machine Learning Research 21.1 (Jan. 2020). ISSN : 1532-4435

    2020

  9. [18]

    Multitask Prompted Training Enables Zero-Shot Task Generalization

    V . Sanh et al. “Multitask Prompted Training Enables Zero-Shot Task Generalization”. In:International Confer- ence on Learning Representations. 2022. URL: https://openreview.net/forum?id=9Vrb9D0WI4

    2022

  10. [19]

    Using Natural Language Explanations to Improve Robustness of In-Context Learning

    X. He, Y . Wu, O.-M. Camburu, P. Minervini, and P. Stenetorp. “Using Natural Language Explanations to Improve Robustness of In-Context Learning”. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Proceedings o...

    2024

  11. [20]

    TRUE: Re-Evaluating Factual Consistency Evaluation

    O. Honovich, R. Aharoni, J. Herzig, H. Taitelbaum, D. Kukliansy, V . Cohen, T. Scialom, I. Szpektor, A. Hassidim, and Y . Matias. “TRUE: Re-Evaluating Factual Consistency Evaluation”. In:Proceedings of the Second DialDoc Workshop on Document-Grounded Dialogue and Conversationa...

    2022

  12. [21]

    HypoNLI: Exploring the Artificial Patterns of Hypothesis-Only Bias in Natural Language Inference

    T. Liu, Z. Xin, B. Chang, and Z. Sui. “HypoNLI: Exploring the Artificial Patterns of Hypothesis-Only Bias in Natural Language Inference”. In: Proceedings of the Twelfth Language Resources and Evaluation Conference. Ed. by N. Calzolari et al. Marseille, France: European Languag...

    2020

  13. [22]

    Adversarial Filters of Dataset Biases

    R. L. Bras, S. Swayamdipta, C. Bhagavatula, R. Zellers, M. Peters, A. Sabharwal, and Y . Choi. “Adversarial Filters of Dataset Biases”. In: Proceedings of the 37th International Conference on Machine Learning . Ed. by H. D. III and A. Singh. V ol. 119. Proceedings of Machine L...

    2020

  14. [23]

    Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets

    Y . Wu, M. Gardner, P. Stenetorp, and P. Dasigi. “Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets”. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Proceedings of the ...

    2022

  15. [24]

    An Empirical Study on Model-Agnostic Debiasing Strategies for Robust Natural Language Inference

    T. Liu, Z. Xin, X. Ding, B. Chang, and Z. Sui. “An Empirical Study on Model-Agnostic Debiasing Strategies for Robust Natural Language Inference”. In: Proceedings of the 24th Conference on Computational Natural Language Learning. Proceedings of the 24th Conference on Computatio...

    2020 doi

  16. [25]

    Info{BERT}: Improving Robustness of Language Models from an Information Theoretic Perspective

    B. Wang, S. Wang, Y . Cheng, Z. Gan, R. Jia, B. Li, and J. Liu. “Info{BERT}: Improving Robustness of Language Models from an Information Theoretic Perspective”. In: International Conference on Learning Representations

  17. [26]

    Supervising Model Attention with Human Explanations for Robust Natural Language Inference

    J. Stacey, Y . Belinkov, and M. Rei. “Supervising Model Attention with Human Explanations for Robust Natural Language Inference”. In: Proceedings of the AAAI Conference on Artificial Intelligence . V ol. 36. 10. 2022, pp. 11349–11357

    2022

  18. [27]

    Narang, C

    S. Narang, C. Raffel, K. Lee, A. Roberts, N. Fiedel, and K. Malkan. WT5?! Training Text-to-Text Models to Explain Their Predictions. Apr. 30, 2020. DOI: 10.48550/arXiv.2004.14546 . arXiv: 2004.14546 [cs] . URL: http://arxiv.org/abs/2004.14546 (visited on 04/09/2025). Pre-published

    2020 doi arXiv

  19. [28]

    Chain of Thought Prompting Elicits Reasoning in Large Language Models

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou. “Chain of Thought Prompting Elicits Reasoning in Large Language Models”. In: Advances in Neural Information Processing Systems. Ed. by A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho....

    2022

  20. [29]

    MathPile: A Billion-Token-Scale Pretraining Corpus for Math

    Z. Wang, X. Li, R. Xia, and P. Liu. “MathPile: A Billion-Token-Scale Pretraining Corpus for Math”. In: Advances in Neural Information Processing Systems. Ed. by A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang. V ol. 37. Curran Associates, Inc....

    2024

  21. [30]

    MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning

    X. Yue, X. Qu, G. Zhang, Y . Fu, W. Huang, H. Sun, Y . Su, and W. Chen. “MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning”. In: The Twelfth International Conference on Learning Representations. 2024. URL: https://openreview.net/forum?id=yLClGs770I

    2024

  22. [31]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    DeepSeek-AI et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning . Jan. 22, 2025. DOI: 10.48550/arXiv.2501.12948 . arXiv: 2501.12948 [cs] . URL: http://arxiv.org/ abs/2501.12948 (visited on 01/28/2025). Pre-published

    2025 doi arXiv

  23. [32]

    Shao et al

    Z. Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Apr. 27,

  24. [33]

    Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin.Understanding R1-Zero-Like Training: A Critical Perspective. Mar. 26, 2025. DOI: 10.48550/arXiv.2503.20783. arXiv: 2503.20783 [cs]. URL: http://arxiv.org/abs/2503.20783 (visited on 04/08/2025). Pre-published

    2025 doi arXiv

  25. [34]

    LoRA: Low-Rank Adaptation of Large Language Models

    E. J. Hu, y. shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. “LoRA: Low-Rank Adaptation of Large Language Models”. In: International Conference on Learning Representations. 2022. URL: https: //openreview.net/forum?id=nZeVKeeFYf9

    2022

  26. [35]

    QLoRA: Efficient Finetuning of Quantized LLMs

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. “QLoRA: Efficient Finetuning of Quantized LLMs”. In: Thirty-Seventh Conference on Neural Information Processing Systems. 2023. URL: https://openreview. net/forum?id=OUIFPHEgJU

    2023

  27. [36]

    AWQ: Activation-Aware Weight Quantization for on-Device LLM Compression and Acceleration

    J. Lin, J. Tang, H. Tang, S. Yang, W. -M. Chen, W. -C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han. “AWQ: Activation-Aware Weight Quantization for on-Device LLM Compression and Acceleration”. In:Pro- ceedings of Machine Learning and Systems . Ed. by P. Gibbons, G. Pekhimenko, a...

    2024

  28. [37]

    S. Wang, H. Fang, M. Khabsa, H. Mao, and H. Ma. Entailment as Few-Shot Learner . Apr. 29, 2021. DOI: 10.48550/arXiv.2104.14690 . arXiv: 2104.14690 [cs] . URL: http://arxiv.org/abs/2104.14690 (visited on 04/09/2025). Pre-published

    2021 doi arXiv

  29. [38]

    Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation

    Q. Zhu, Q. Cheng, R. Peng, X. Li, R. Peng, T. Liu, X. Qiu, and X. Huang. “Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation”. In: Findings of the Association for Computational Linguistics: EMNLP 2024. Findings of the Association for ...

    2024

  30. [42]

    optional

    Miami, Florida, USA: Association for Computational Linguistics, 2024, pp. 9113–9129. DOI: 10.18653/ v1 / 2024 . findings - emnlp . 532. URL: https : / / aclanthology . org / 2024 . findings - emnlp . 532 (visited on 04/25/2025). 16 Pushing the boundary on NLI A Experiment deta...

    2024

  31. [2021]

    URL: https://openreview.net/forum?id=hpH98mK5Puk

  32. [2024]

    arXiv: 2402.03300 [cs]

    DOI: 10.48550/arXiv.2402.03300. arXiv: 2402.03300 [cs]. URL: http://arxiv.org/abs/2402. 03300 (visited on 04/08/2025). Pre-published

    2025 doi arXiv

  33. [6860]

    URL: https://aclanthology.org/2020.lrec-1.846/

    ISBN : 979-10-95546-34-4. URL: https://aclanthology.org/2020.lrec-1.846/

    2020

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.