Pith. sign in

REVIEW 4 major objections 5 minor 37 references

Solver-verified counterfactual normative reasoning shows that models can reach perfect verdict accuracy while learning no falsification competence, and that a class-conditioned reward teaches structural change fields that final-answer rewar

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:43 UTC pith:YM4G6ZSQ

load-bearing objection Solver-verified counterfactual benchmark with a genuinely useful reward-geometry result, but ground truth rests on an unverified and possibly order-sensitive solver, so treat as a strong conditional. the 4 major comments →

arxiv 2607.03957 v2 pith:YM4G6ZSQ submitted 2026-07-04 cs.CL cs.AI

NormWorlds-CF: Solver-Verified Counterfactual Normative Reasoning with Metamorphic-Relation GRPO

Xinqi Zhang This is my paper
classification cs.CL cs.AI
keywords counterfactual normative reasoningsolver-verified benchmarkdefeasible logicmetamorphic testingGRPOreward designfalsification certificatesstructural change records
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

NormWorlds-CF argues that in normative reasoning, where verdicts come from a tiny label set, a language model can reach perfect answer accuracy while learning none of the structural competence—proof, falsification, and counterfactual change—that makes a verdict trustworthy. To test this, the paper builds a synthetic environment of executable rule worlds in which a deterministic solver emits answers, certificates, and paired-world change labels, so supervision and evaluation never rely on an LLM judge. The staged SFT diagnostics show that answer-only supervision saturates verdict accuracy but scores zero on joint falsification, while full-mix training with targeted replay reaches strong all-task accuracy. On the paired-world change task, a class-conditioned reward, MR-GRPO, measurably improves structural change fields compared with sparse or answer-only rewards, leading the paper to conclude that verified counterfactual structure can shape post-training beyond final answers and that reward geometry, not final-answer accuracy, is what determines whether structure is learned.

Core claim

The paper's central discovery is a reward geometry: different supervision targets and rewards learn different slices of the same normative structure, and final-answer supervision is not a proxy for dialectical competence. Answer-only SFT reaches perfect answer accuracy but scores zero on joint falsification certificates; adding proof targets saturates proofs yet still leaves falsification at zero, and only full-mix training with targeted replay reaches 0.99 all-task accuracy. In the compact paired-world change task, answer-only GRPO shifts the relation prior toward change labels and improves answer-local fields at the expense of relation-family structure, while sparse exact/schema GRPO prese

What carries the argument

The load-bearing object is the deterministic solver S(w) = (a, C), a closed symbolic procedure that maps an executable rule world to a final answer and a certificate (decisive rules, defeated alternatives, argument statuses, minimal support), together with its change-differencing operator M(w, τ) = (r_τ, D[S(w), S(τ(w))]) that derives paired-world metamorphic relation labels and fine-grained change fields by comparing two solver traces. This machinery carries the argument because every supervision target, GRPO reward, and reported metric is a downstream function of these solver outputs, never an LLM judge. The second key object is the class-conditioned metamorphic reward A_class, which group

Load-bearing premise

The whole benchmark rests on the unreleased deterministic solver being a correct and complete implementation of the intended defeasible normative semantics; if the solver's design choices misrepresent dialectical reasoning, every ground-truth label, reward, and result inherits the error.

What would settle it

Take the prose description of the solver, reimplement the forward-chaining, attack-only-on-incompatible-labels, priority-by-directed-reachability, and minimal-support algorithm, and compare its outputs against the paper's reported ground truth on all 270 root families and 1080 canonical-to-variant pairs; a single mismatch in any relation family label would falsify the benchmark's ground truth and, with it, the reward-geometry conclusions.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the paper is right, post-training for judgment-like tasks should not use answer-only rewards as a competence proxy: perfect verdicts can coexist with zero falsification ability.
  • Answer-only reward can actively distort structure in a low-entropy label setting: it biases the relation prior toward change labels, converting invariant examples into wrong-family errors.
  • Sparse exact rewards can starve GRPO when the target structure is nested rather than a rare token: zero within-group reward variance removes the ranking signal.
  • Class-conditioned partial credit along a diagnostic hierarchy (invariant vs change families) densifies structural reward without collapsing to the most visible answer field.
  • Solver-verified environments make failure attribution typed (answer vs proof vs falsification vs counterfactual errors), turning 'the model is wrong' into a researchable diagnosis.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The scarcity-inversion framing generalizes: any domain with a small verdict label set (medical triage, compliance checks, policy classification) should place reward on the structure that distinguishes a right answer from a right reason, not on the answer itself.
  • Because the solver is unreleased, the benchmark is not yet independently auditable; opening the solver or providing a formal specification would let other labs regenerate worlds and test reward designs against a checkable ground truth.
  • The proof-to-falsification non-transfer suggests critique is a separate capability from construction; a testable extension is whether training on falsification first transfers to proof, or whether the two require separate curricula.
  • MR-GRPO's class-conditioning could be extended from the invariant/change boundary to finer subtype distinctions (e.g., support_delta vs priority_delta) or adapted for OOD transfer by conditioning on structural invariants rather than family names.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces NormWorlds-CF, a solver-verified environment for counterfactual normative reasoning in executable rule worlds. A deterministic solver S (Eq. 1) produces final answers, proof/falsification certificates, argument statuses, minimal supports, and paired-world change labels via a metamorphic relation M (Eq. 2), so supervision, reward, and evaluation avoid LLM judges. Staged SFT diagnostics show that answer-only supervision saturates verdict accuracy but leaves joint falsification at zero, while full-mix training with targeted replay reaches 0.99 overall. On the compact paired-world change-record task, the paper compares answer-only, sparse, and class-conditioned MR-GRPO rewards under a matched Qwen3-1.7B continuation and a Qwen3-4B three-seed from-SFT validation. The central claim is a reward geometry: sparse reward preserves coarse relation labels, answer-only reward trades family structure for answer-local gains, and MR-GRPO improves answer/support/status change fields while keeping family structure, with exact full-record generation, invariant subtype recognition, and OOD transfer remaining open.

Significance. If the solver is correct and released, this is a valuable contribution to post-training evaluation and reward design for structured judgment tasks. The deterministic scoring, typed failure taxonomy, compact change-record interface, and the explicit separation of final-answer supervision from structural competence are constructive; the paper also states its limitations clearly and uses McNemar tests for the 1.7B held-out items. However, the significance is conditional on the correctness and order-invariance of the unreleased solver, on the statistical robustness of the three-seed 4B means, and on the availability of code/configurations for independent verification.

major comments (4)
  1. [§2.2, Eq. (2)] The description of minimal-support search is not order-invariant. The sentence 'Minimal support searches subsets of decisive rules sufficient to sustain the answer while defeating conflicting alternatives' specifies no tie-breaking rule. If two minimal supports exist (e.g., two independent rules both permit the same action), a surface_invariant edit that reorders facts/rules/priorities can change which support is selected, so D(S(w),S(τ(w))) can report spurious support or attack changes on an invariant edit; conversely, a support_delta edit can switch to an alternative support and report an empty support-change field. Since every SFT target and every reward term in Eqs. (4)–(7) is a function of these labels, the central MR-GRPO comparison inherits any such artifact. Please specify deterministic tie-breaking (e.g., lexicographic over rule IDs), provide an order-ablation experiment (permut
  2. [§4.3, Table 7] The three-seed means are reported without variance, confidence intervals, or per-seed significance tests. The headline 4B claims ('MR-GRPO leads on answer-, support-, and status-change fields') rest on differences of 0.020–0.043 (e.g., support change 0.621 vs 0.597; status change 0.600 vs 0.571; all-changes exact 0.443 vs 0.435). With three seeds these differences are consistent with noise. Provide per-seed tables with CIs or paired tests, or explicitly hedge these as directional. Also report whether the sparse-vs-MR relation-family difference (0.790 vs 0.781) is within noise.
  3. [§4.2, Table 6] The 1.7B continuation is a single run per reward. The abstract and §4.2 say MR-GRPO 'improves' relation accuracy and 'reduces' wrong-family error, with McNemar p-values computed on held-out items. Those p-values quantify item-level agreement within one run, not run-to-run variance; the gains (relation +0.096, wrong-family −0.142) are plausible but need at least one additional seed or a bootstrap over runs to support the reward-geometry conclusion. The paper does acknowledge single runs, but the conclusion 'verified counterfactual structure can shape post-training' is carried largely by this 1.7B result.
  4. [Reproducibility (abstract, §2.2, §4.3)] No code, solver implementation, or supplementary material is provided, despite repeated references to 'supplementary material' for full weights (Eqs. 4–7), seeds/hardware, per-seed tables, and field breakdowns. Since the entire benchmark is generated by the unreleased solver S and all training uses reward coefficients that are not listed, none of the headline numbers can be independently reproduced or checked. Please make the solver, data-generation code, training configs, and coefficient values available, or include them in an appendix.
minor comments (5)
  1. [§2.3, Table 5] The caption '896 / 280 from base' is unexplained; state what the two numbers denote (records and steps) and what 'from base' means.
  2. [References] The reference entry beginning 'sch 2021' is incomplete; provide full author, title, and venue.
  3. [Eq. (7)] Some symbols used in A_class (η_same, A_same, η_chg, A_chg, η_prs, A_prs, η_E) are not explicitly defined in the text. Please define all coefficients and score fields in one place or a table.
  4. [§4.2] The term 'matched' is used loosely; Table 4 shows only initializer and steps. Specify which data, optimizer, group size, and hyperparameters are identical across the three 1.7B branches.
  5. [§2.3] The sentence 'OOD uses held-out structural family types (e.g., priority chains and distractor-conflict variants)' should clarify whether these are subtypes of the same four root-level families or new family types, since this affects interpretation of the OOD results.

Circularity Check

0 steps flagged

No significant circularity: central claims rest on held-out exact metrics; reward-aligned soft scores are disclosed as secondary.

full rationale

NormWorlds-CF is an oracle-based benchmark construction: the deterministic solver S (Eq. 1) defines ground truth, and SFT targets, GRPO rewards, and evaluation metrics are all functions of the same solver labels. This self-containedness is not circularity, because the paper's empirical claims concern whether models can learn and generalize those labels under different supervision/reward geometries, using held-out root families ('Splits are by root family, so variants never leak across train/dev/ID/OOD'). The one by-construction alignment is that the class-conditioned MR score Aclass (Eq. 7) and change-presence are components of the MR-GRPO reward, and the paper explicitly labels them 'secondary diagnostics, not substitutes for exact family checks'; they are not used to support the central geometry claim. Primary metrics (relation, relation-family, wrong-family, exact answer/support/status change fields) are scored by deterministic parsers on held-out pairs, and although some are also rewarded, the matched-protocol comparison isolates reward geometry rather than renaming a fitted quantity as a prediction. No load-bearing self-citation, imported uniqueness theorem, or smuggled ansatz is present; the solver determinism and minimal-support tie-breaking concern is a correctness/verification risk, not a circularity. Therefore no significant circularity is found.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 2 invented entities

The paper's headline empirical contributions presuppose: (i) the correctness of the unreleased solver whose outputs are used as supervision, reward, and evaluation; (ii) the validity of the four-family metamorphic taxonomy; (iii) standard RL assumptions from GRPO. The central set of free parameters is the reward coefficient vector, which is not visible in the main text.

free parameters (1)
  • MR-GRPO and baseline reward coefficients (α_r, α_fam, α_opt, α_wrong, γ_0, γ_Δ, η_I, η_C, η_same, η_chg, η_prs, ω_*, λ_* = Not reported in main text; 'full weights in the supplementary material'
    The reward geometry that MR-GRPO is compared on is set by these hand-chosen weights (Eqs 4-7). If tuned on dev, the comparison is not purely about reward form; the paper only says coefficients are fixed before held-out evaluation.
axioms (4)
  • domain assumption The defeasible reasoning semantics implemented by the solver (forward-chaining to fixed point; attacks only on incompatible labels; priority by directed graph reachability; serialization order obligated>forbidden>permitted) is the correct ground truth for normative reasoning.
    §2.2 'Label construction is a closed symbolic procedure...'. If these semantics are wrong or buggy, every certificate, reward, and reported metric is wrong while still deterministic.
  • domain assumption The four root-level metamorphic family labels (surface_invariant, support_delta, priority_delta, distractor_invariant) are valid and meaningful ground-truth categories.
    §2.2 and Table 3: family labels are 'determined by its edit operator and validated against solver-computed pre- and postconditions'. The taxonomy is an author design choice; no external benchmark confirms these are the right normative invariants.
  • standard math Standard GRPO group-relative advantage normalization (Eq 3) and LoRA training are valid and correctly implemented.
    Borrowed from Shao et al. 2024 and Hu et al. 2022; standard assumptions for this regime.
  • domain assumption Splitting by root family prevents leakage between train/dev/ID/OOD.
    Table 3 splits by root family; the paper assumes variants of a root do not let the model memorize templates across splits. The paper itself reports OOD transfer remains open, which is consistent with this assumption being load-bearing.
invented entities (2)
  • Paired-world change-record schema (relation + answer/support/status/attack fields) independent evidence
    purpose: A compact model-output target that is scoreable repeatedly during GRPO and distinguishes coarse relation recognition from fine-grained change description (§2.3, Table 2).
    Defined operationally by solver trace differencing (Eq 2); a re-implementation can check the fields, but no released code exists, so the handle is grounded only in the paper's solver.
  • Root-level metamorphic families (surface_invariant, support_delta, priority_delta, distractor_invariant) no independent evidence
    purpose: Counterfactual substrate pairing each canonical world with four diagnostic variants; family labels are the main classification target and the basis of partial credit in MR-GRPO.
    Family taxonomy is a design choice assembled from residual failure types; no external benchmark or independent oracle confirms these are the right normative invariants, and labels are computed by the paper's own solver/operator definitions.

pith-pipeline@v1.3.0-alltime-deepseek · 11608 in / 20082 out tokens · 209923 ms · 2026-08-02T08:43:45.538693+00:00 · methodology

0 comments
read the original abstract

Language models can reach the right normative verdict for the wrong reason. We introduce NormWorlds-CF, a solver-verified environment for counterfactual normative reasoning in executable rule worlds. Its deterministic solver produces final answers, proof and falsification certificates, argument statuses, support sets, and paired-world change labels, enabling supervision and evaluation without LLM judges. The benchmark contains staged SFT diagnostics and a compact paired-world task with 270 root families and 1080 canonical-to-variant pairs. The SFT diagnostics show that final-answer supervision can saturate verdict accuracy without inducing falsification competence: answer-only SFT reaches perfect answer accuracy but scores zero on joint falsification certificates, while full-mix training with targeted replay reaches strong all-task accuracy (0.99). For the structured-change task, we introduce metamorphic-relation GRPO (MR-GRPO), a class-conditioned reward for GRPO that gives partial credit for relation families and solver-visible change fields. In matched Qwen3-1.7B continuation experiments, MR-GRPO improves held-out relation accuracy and relation-family correctness, and reduces wrong-family error, compared to sparse and answer-only GRPO. In Qwen3-4B three-seed validation, sparse reward preserves coarse relation labels best, answer-only reward improves answer-change but weakens relation-family structure, and MR-GRPO leads on answer-, support-, and status-change fields as well as class-conditioned MR and change-presence. These results show that verified counterfactual structure can shape post-training beyond final answers, while exact full change-record generation, invariant subtype recognition, and out-of-distribution (OOD) transfer remain open problems.

Figures

Figures reproduced from arXiv: 2607.03957 by Xinqi Zhang.

Figure 1
Figure 1. Figure 1: Overview of NormWorlds-CF. Executable rule worlds are solved by deterministic procedures that produce verified answers, certificates, and paired-world change records. These solver-generated structures support staged SFT diagnostics and a matched GRPO comparison among answer-only, sparse exact/schema, and MR-aware change-field rewards, all evaluated by deterministic scorers. 2 NormWorlds-CF NormWorlds-CF se… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 11 linked inside Pith

  1. [1]

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, Y. K. and Wu, Y. and Guo, Daya , title =. arXiv preprint arXiv:2402.03300 , year =

    Pith/arXiv arXiv

  2. [2]

    Nature , volume =

    Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Zhang, Ruoyu and Ma, Shirong and Bi, Xiao and others , title =. Nature , volume =

  3. [3]

    and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , title =

    Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , title =. International Conference on Learning Representations , year =

  4. [4]

    arXiv preprint arXiv:2505.09388 , year =

    Yang, An and Li, Anfeng and Yang, Baosong and Zhang, Beichen and Hui, Binyuan and Zheng, Bo and Yu, Bowen and Gao, Chang and Huang, Chengen and Lv, Chenxu and others , title =. arXiv preprint arXiv:2505.09388 , year =

    Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:2406.05397 , year =

    Li, Rui and Liu, Huai and Poon, Pak-Lok and Towey, Dave and Sun, Chang-Ai and Zheng, Zheng and Zhou, Zhi Quan and Chen, Tsong Yueh , title =. arXiv preprint arXiv:2406.05397 , year =

    Pith/arXiv arXiv

  6. [6]

    and van Merri

    Weston, Jason and Bordes, Antoine and Chopra, Sumit and Rush, Alexander M. and van Merri. Towards. arXiv preprint arXiv:1502.05698 , year =

    Pith/arXiv arXiv

  7. [7]

    Lawrence and Girshick, Ross , title =

    Johnson, Justin and Hariharan, Bharath and van der Maaten, Laurens and Li, Fei-Fei and Zitnick, C. Lawrence and Girshick, Ross , title =. 2017 IEEE Conference on Computer Vision and Pattern Recognition (

    2017

  8. [8]

    Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence,

    Clark, Peter and Tafjord, Oyvind and Richardson, Kyle , title =. Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence,

  9. [9]

    Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages =

    Tafjord, Oyvind and Dalvi, Bhavana and Clark, Peter , title =. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages =. 2021 , publisher =

    2021

  10. [10]

    Mind , volume =

    von Wright, Georg Henrik , title =. Mind , volume =

  11. [11]

    Handbook of Logic in Artificial Intelligence and Logic Programming, Volume 3: Nonmonotonic Reasoning and Uncertain Reasoning , editor =

    Nute, Donald , title =. Handbook of Logic in Artificial Intelligence and Logic Programming, Volume 3: Nonmonotonic Reasoning and Uncertain Reasoning , editor =

  12. [12]

    , title =

    Antoniou, Grigoris and Billington, David and Governatori, Guido and Maher, Michael J. , title =. ACM Transactions on Computational Logic , volume =

  13. [13]

    Artificial Intelligence , volume =

    Prakken, Henry and Sartor, Giovanni , title =. Artificial Intelligence , volume =

  14. [14]

    Guha, Neel and Nyarko, Julian and Ho, Daniel E. and R. Advances in Neural Information Processing Systems , volume =

  15. [15]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Chalkidis, Ilias and Jana, Abhik and Hartung, Dirk and Bommarito, Michael and Androutsopoulos, Ion and Katz, Daniel and Aletras, Nikolaos , title =. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2022 , publisher =

    2022

  16. [16]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages =

    Fei, Zhiwei and Shen, Xiaoyu and Zhu, Dawei and Zhou, Fengzhe and Han, Zhuo and Huang, Alan and Zhang, Songyang and Chen, Kai and Yin, Zhixin and Shen, Zongwen and Ge, Jidong and Ng, Vincent , title =. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages =. 2024 , publisher =

    2024

  17. [17]

    arXiv preprint arXiv:2408.10343 , year =

    Pipitone, Nicholas and Alami, Ghita Houir , title =. arXiv preprint arXiv:2408.10343 , year =

    Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2403.03883 , year =

    Colombo, Pierre and Pires, Telmo Pessoa and Boudiaf, Malik and Culver, Dominic and Melo, Rui and Corro, Caio and Martins, Andr. arXiv preprint arXiv:2403.03883 , year =

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2309.11325 , year =

    Yue, Shengbin and Chen, Wei and Wang, Siyuan and Li, Bingxuan and Shen, Chenchen and Liu, Shujun and Zhou, Yuxuan and Xiao, Yao and Yun, Song and Huang, Xuanjing and Wei, Zhongyu , title =. arXiv preprint arXiv:2309.11325 , year =

    Pith/arXiv arXiv

  20. [20]

    Fundamental Research , year =

    Cui, Jiaxi and Ning, Munan and Li, Zongjian and Li, Hao and Ya, Yang and Chen, Bohua and Ling, Bin and Tian, Yonghong and Yuan, Li , title =. Fundamental Research , year =

  21. [21]

    Proceedings of the 31st International Conference on Computational Linguistics , pages =

    Fei, Zhiwei and Zhang, Songyang and Shen, Xiaoyu and Zhu, Dawei and Wang, Xiao and Ge, Jidong and Ng, Vincent , title =. Proceedings of the 31st International Conference on Computational Linguistics , pages =. 2025 , publisher =

    2025

  22. [22]

    arXiv preprint arXiv:2110.14168 , year =

    Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and Hesse, Christopher and Schulman, John , title =. arXiv preprint arXiv:2110.14168 , year =

    Pith/arXiv arXiv

  23. [23]

    International Conference on Learning Representations , pages =

    Lightman, Hunter and Kosaraju, Vineet and Burda, Yuri and Edwards, Harrison and Baker, Bowen and Lee, Teddy and Leike, Jan and Schulman, John and Sutskever, Ilya and Cobbe, Karl , title =. International Conference on Learning Representations , pages =

  24. [24]

    and Leike, Jan and Brown, Tom and Martic, Miljan and Legg, Shane and Amodei, Dario , title =

    Christiano, Paul F. and Leike, Jan and Brown, Tom and Martic, Miljan and Legg, Shane and Amodei, Dario , title =. Advances in Neural Information Processing Systems , volume =

  25. [25]

    arXiv preprint arXiv:1707.06347 , year =

    Schulman, John and Wolski, Filip and Dhariwal, Prafulla and Radford, Alec and Klimov, Oleg , title =. arXiv preprint arXiv:1707.06347 , year =

    Pith/arXiv arXiv

  26. [26]

    and Stiennon, Nisan and Wu, Jeffrey and Brown, Tom B

    Ziegler, Daniel M. and Stiennon, Nisan and Wu, Jeffrey and Brown, Tom B. and Radford, Alec and Amodei, Dario and Christiano, Paul and Irving, Geoffrey , title =. arXiv preprint arXiv:1909.08593 , year =

    Pith/arXiv arXiv 1909

  27. [27]

    and Leike, Jan and Lowe, Ryan , title =

    Ouyang, Long and Wu, Jeffrey and Jiang, Xu and Almeida, Diogo and Wainwright, Carroll and Mishkin, Pamela and Zhang, Chong and Agarwal, Sandhini and Slama, Katarina and Ray, Alex and Schulman, John and Hilton, Jacob and Kelton, Fraser and Miller, Luke and Simens, Maddie and Askell, Amanda and Welinder, Peter and Christiano, Paul F. and Leike, Jan and Lowe...

  28. [28]

    Chen, Tsong Yueh and Cheung, Shing-Chi and Yiu, Siu-Ming , title =

  29. [29]

    Chen, Tsong Yueh and Kuo, Fei-Ching and Liu, Huai and Poon, Pak-Lok and Towey, Dave and Tse, T. H. and Zhou, Zhi Quan , title =. ACM Computing Surveys , volume =

  30. [30]

    International Conference on Learning Representations , year =

    Kaushik, Divyansh and Hovy, Eduard and Lipton, Zachary , title =. International Conference on Learning Representations , year =

  31. [31]

    Findings of the Association for Computational Linguistics: EMNLP 2020 , pages =

    Gardner, Matt and Artzi, Yoav and Basmov, Victoria and Berant, Jonathan and Bogin, Ben and Chen, Sihao and Dasigi, Pradeep and Dua, Dheeru and Elazar, Yanai and Gottumukkala, Ananth and others , title =. Findings of the Association for Computational Linguistics: EMNLP 2020 , pages =. 2020 , publisher =

    2020

  32. [32]

    2021 , publisher =

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages =. 2021 , publisher =

    2021

  33. [33]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages =

    Sanyal, Soumya and Liao, Zeyi and Ren, Xiang , title =. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages =. 2022 , publisher =

    2022

  34. [34]

    Allaway, Emily and McKeown, Kathleen , title =. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages =. 2025 , publisher =

    2025

  35. [35]

    Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP , pages =

    Ozeki, Kentaro and Ando, Risako and Morishita, Takanobu and Abe, Hirohiko and Mineshima, Koji and Okada, Mitsuhiro , title =. Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP , pages =. 2025 , publisher =

    2025

  36. [36]

    and Jain, Rajiv and Linna, Jr., Daniel W

    Servantez, Sergio and Lawsky, Sarah B. and Jain, Rajiv and Linna, Jr., Daniel W. and Hammond, Kristian J. , title =. Findings of the Association for Computational Linguistics: ACL 2026 , pages =. 2026 , publisher =

    2026

  37. [37]

    arXiv preprint arXiv:2507.17746 , year =

    Gunjal, Anisha and Wang, Anthony and Lau, Elaine and Nath, Vaskar and He, Yunzhong and Liu, Bing and Hendryx, Sean , title =. arXiv preprint arXiv:2507.17746 , year =

    Pith/arXiv arXiv