pith. sign in

arxiv: 2606.08932 · v1 · pith:D43IHX4Enew · submitted 2026-06-08 · 💻 cs.CL · cs.AI· cs.CE

From Statute to Control Flow: Span-Grounded Deontic Trees for Defeasible Scope Parsing

Jian Chen , Siyuan Li , Chucheng Wan , Zixuan Yuan This is my paper

Pith reviewed 2026-06-27 17:14 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.CE
keywords defeasible scope parsingsilent scope omissiondeontic treeslegal NLPnorm understandingcontrol flowexception handlingstatutory parsing
0
0 comments X

The pith

Span-Grounded Deontic Trees raise whole-tree fidelity and defeater recovery when LLMs parse nested exceptions in statutes.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces NormBench, a dataset of 2,290 provisions across languages and domains, to measure how models identify which clauses override others in policies. It diagnoses two LLM failure modes: recursion decay on deep defeaters and an auditability trap where spans are retrieved but control flow is assembled incorrectly. Span-Grounded Deontic Trees are presented as a compiler-style intermediate representation that forces every branch to be anchored to source text spans and equipped with explicit exclusion guards. Requiring models to output this structure improves fidelity on exception-active cases while leaving aggregate scores mixed when the added structure is unnecessary.

Core claim

Span-Grounded Deontic Trees (SG-DT) serve as a constrained intermediate output for defeasible scope parsing that anchors every logical branch to source spans and requires explicit exclusion guards, enabling deterministic compilation and audit. Using SG-DT improves whole-tree fidelity and defeater recovery, with utility concentrated on exception-active, SSO-prone cases.

What carries the argument

Span-Grounded Deontic Trees (SG-DT), an intermediate representation that grounds deontic logic branches to text spans with explicit exclusion guards.

If this is right

  • LLM performance on scope parsing drops sharply as the number of nested defeaters increases.
  • Models can retrieve relevant spans yet still assemble incorrect control flow between them.
  • Gains from SG-DT appear mainly on cases that contain active exceptions rather than uniform rules.
  • Aggregate accuracy can stay flat or decline when the added structure is unnecessary or parser fidelity is low.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Agent systems that adopt SG-DT parsing may reduce silent failures on real policies even if end-task accuracy looks similar.
  • The same intermediate format could be tested on non-legal rule sets that also contain nested exceptions.
  • Low-fidelity SG-DT parsers would need separate fixes before the fidelity gains become reliable.

Load-bearing premise

The benchmark provisions and their manually constructed SG-DT annotations accurately reflect the defeasible logical structure that real statutes and policies require agents to respect.

What would settle it

A new collection of statutes with deep nested exceptions on which models forced to emit SG-DT still produce incorrect override relations or drop counter-exceptions.

Figures

Figures reproduced from arXiv: 2606.08932 by Chucheng Wan, Jian Chen, Siyuan Li, Zixuan Yuan.

Figure 1
Figure 1. Figure 1: Non-monotonic rules and silent omissions. In statutes and policies, later clauses can narrow or override earlier ones [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
read the original abstract

Rule-following agents tasked with executing policies and regulations often fail via Silent Scope Omission (SSO): a model applies a general rule but silently drops nested exceptions or counter-exceptions, producing outputs that appear compliant yet break on important edge cases. Although such failures are often framed as an agentic-systems problem, the underlying bottleneck is statutory and policy understanding, a capability typically studied in legal NLP. However, most existing legal NLP benchmarks emphasize end-task outcomes, which can overlook the structural omissions that cause SSO. To diagnose and mitigate SSO, we introduce NormBench, a benchmark of 2,290 provisions spanning Chinese (laws and local policies), English (U.S. tax law, GDPR, and corporate policies), and cross-lingual settings, designed for defeasible scope parsing: identifying precisely which clause overrides which. NormBench uses Span-Grounded Deontic Trees (SG-DT), a compiler-style intermediate representation that anchors every logical branch to source spans and requires explicit exclusion guards, enabling deterministic compilation and audit. Evaluations of frontier LLMs reveal two recurring pathologies: (1) Recursion Decay, where performance drops sharply as defeater depth increases, and (2) an Auditability Trap, where models retrieve relevant spans but fail to assemble correct control flow. Using SG-DT as a constrained intermediate output improves whole-tree fidelity and defeater recovery, and downstream experiments show that its utility is mechanism-specific: gains concentrate on exception-active, SSO-prone cases, while aggregate accuracy can be mixed when the added structure is unnecessary or parser fidelity is low.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces NormBench, a benchmark of 2,290 provisions drawn from Chinese laws/policies, U.S. tax law, GDPR, and corporate policies, annotated with Span-Grounded Deontic Trees (SG-DT) as a compiler-style intermediate representation that anchors logical branches to source spans and requires explicit exclusion guards. It identifies two LLM pathologies—Recursion Decay (performance drop with defeater depth) and Auditability Trap (failure to assemble correct control flow despite span retrieval)—and claims that constraining outputs to SG-DT improves whole-tree fidelity and defeater recovery, with utility concentrated on exception-active, SSO-prone cases.

Significance. If the SG-DT annotations are shown to faithfully capture defeasible statutory structure, the work could advance legal NLP by shifting focus from end-task metrics to auditable control-flow representations, offering a practical mechanism for mitigating Silent Scope Omission in rule-following agents. The cross-lingual scope and emphasis on deterministic compilation are constructive contributions, though significance is limited by the absence of reported quantitative evaluation details.

major comments (2)
  1. [NormBench construction paragraph] NormBench construction paragraph: The description states that SG-DT trees are 'manually constructed' and 'require explicit exclusion guards' but provides no inter-annotator agreement metrics, expert legal review, or cross-check against statutory commentary. This is load-bearing for the central empirical claim, because reported gains in defeater recovery on 'exception-active, SSO-prone cases' could be artifacts of annotator alignment rather than genuine improvements if the gold trees systematically mis-specify overriding relations.
  2. [Abstract and evaluation description] Abstract and evaluation description: The text reports two pathologies and claims improvement from SG-DT but supplies no quantitative results, error analysis, or details on how the 2,290 provisions were selected and annotated. Without these, the magnitude and reliability of the fidelity gains cannot be assessed.
minor comments (1)
  1. The acronym 'SSO' for Silent Scope Omission is introduced without reference to prior work on scope omission or defeasible reasoning in legal NLP; adding such citations would clarify novelty.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the thoughtful comments, which highlight important aspects of benchmark validation and result presentation. We address each major comment below with specific plans for revision where warranted.

read point-by-point responses

  1. Referee: NormBench construction paragraph: The description states that SG-DT trees are 'manually constructed' and 'require explicit exclusion guards' but provides no inter-annotator agreement metrics, expert legal review, or cross-check against statutory commentary. This is load-bearing for the central empirical claim, because reported gains in defeater recovery on 'exception-active, SSO-prone cases' could be artifacts of annotator alignment rather than genuine improvements if the gold trees systematically mis-specify overriding relations.

    Authors: We agree that explicit reporting of annotation reliability is necessary. The SG-DT trees were produced by annotators with relevant legal domain expertise (native speakers for Chinese provisions, tax/GDPR specialists for English sources), following detailed guidelines that require span grounding and explicit exclusion guards. We will add a new subsection to Section 3 detailing the annotation protocol, inter-annotator agreement statistics (Cohen's kappa on span identification and tree structure), and any cross-checks performed against official commentaries or secondary sources. This addition will allow readers to assess the robustness of the gold trees independently of the model results. revision: yes

  2. Referee: Abstract and evaluation description: The text reports two pathologies and claims improvement from SG-DT but supplies no quantitative results, error analysis, or details on how the 2,290 provisions were selected and annotated. Without these, the magnitude and reliability of the fidelity gains cannot be assessed.

    Authors: The full manuscript contains quantitative tables (Section 4) reporting whole-tree fidelity, defeater recovery rates, and breakdowns by recursion depth, plus error analysis in Section 5.2 that quantifies Recursion Decay and Auditability Trap. Provision selection is described in Section 3.1 (stratified sampling across jurisdictions and exception density). We acknowledge that the abstract remains high-level and does not preview these numbers. We will revise the abstract to include one or two key quantitative findings and ensure the evaluation section explicitly cross-references the selection and annotation details. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical benchmark evaluation with no derivations or self-referential reductions

full rationale

The paper introduces NormBench and SG-DT as a new representation for defeasible scope parsing, then reports empirical LLM evaluations showing improvements in fidelity when constraining outputs to SG-DT. No equations, fitted parameters, or derivation chains appear in the abstract or described structure. Claims rest on benchmark performance rather than any step that reduces by construction to its own inputs or prior self-citations. The manual construction of annotations is a standard benchmark-creation step and does not create the enumerated circularity patterns (self-definitional, fitted-input prediction, etc.). This is a self-contained empirical study against an external benchmark.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no free parameters, axioms, or invented entities are described or can be inferred.

pith-pipeline@v0.9.1-grok · 5826 in / 1128 out tokens · 13431 ms · 2026-06-27T17:14:43.703795+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 9 canonical work pages · 2 internal anchors

  1. [1]

    2025.https://www.anthropic.com/news/claude-opus-4-5 KDD 2026, August 9–13, 2026, Jeju Island, Republic of Korea

    Anthropic. 2025.https://www.anthropic.com/news/claude-opus-4-5 KDD 2026, August 9–13, 2026, Jeju Island, Republic of Korea. Jian Chen, Siyuan Li, Chucheng Wan, and Zixuan Yuan

    2025

  2. [2]

    Farid Ariai, Joel Mackenzie, and Gianluca Demartini. 2025. Natural language pro- cessing for the legal domain: A survey of tasks, datasets, models, and challenges. Comput. Surveys58, 6 (2025), 1–37

    2025

  3. [3]

    Tara Athan, Guido Governatori, Monica Palmirani, Adrian Paschke, and Adam Wyner. 2015. LegalRuleML: Design principles and foundations. InReasoning Web International Summer School. Springer, 151–188

    2015

  4. [4]

    Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider

  5. [5]

    InProceedings of the 7th linguistic annotation workshop and interoperability with discourse

    Abstract meaning representation for sembanking. InProceedings of the 7th linguistic annotation workshop and interoperability with discourse. 178–186

  6. [6]

    Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androut- sopoulos, Daniel Katz, and Nikolaos Aletras. 2022. LexGLUE: A benchmark dataset for legal language understanding in English. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 4310–4330

    2022

  7. [7]

    Jian Chen, Zhenyan Chen, Xuming Hu, Peilin Zhou, Yining Hua, Han Fang, Cissy Hing Yee Choy, Xinmei Ke, Jingfeng Luo, and Zixuan Yuan. 2025. DeKeyNLU: Enhancing Natural Language to SQL Generation through Task Decomposition and Keyword Extraction.arXiv preprint arXiv:2509.14507(2025)

    work page arXiv 2025

  8. [8]

    Jian Chen, Peilin Zhou, Yining Hua, Loh Xin, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. 2024. Fintextqa: A dataset for long-form financial question answering. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 6025–6047

    2024

  9. [9]

    Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E Ho. 2024. Large legal fictions: Profiling legal hallucinations in large language models.Journal of Legal Analysis16, 1 (2024), 64–93

    2024

  10. [10]

    Yu Fan, Jingwei Ni, Jakob Merane, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Etienne Salimbeni, Florian Geering, Oliver Dreyer, et al

  11. [11]

    Lexam: Benchmarking legal reasoning on 340 law exams.arXiv preprint arXiv:2505.12864(2025)

    work page arXiv 2025

  12. [12]

    You don’t need a university degree to comprehend data protection this way

    Vincent Freiberger, Arthur Fleig, and Erik Buchmann. 2025. " You don’t need a university degree to comprehend data protection this way": LLM-Powered Interactive Privacy Policy Assessment. InProceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. 1–12

    2025

  13. [13]

    Thomas F Gordon. 1995. The pleadings game. InThe Pleadings Game: An Artificial Intelligence Model of Procedural Justice. Springer, 109–169

    1995

  14. [14]

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zam- brano, et al. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models.Advances in neural information pro- cessing systems36 (2023), 44123–44279

    2023

  15. [15]

    Eilif Hjelseth and Nick Nisbet. 2011. Capturing normative constraints by use of the semantic mark-up RASE methodology. InProceedings of CIB W78-W102 Conference. 1–10

    2011

  16. [16]

    Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2020. A dataset for statutory reasoning in tax law entailment and question answering. arXiv preprint arXiv:2005.05257(2020)

    work page arXiv 2020

  17. [17]

    Nils Holzenberger and Benjamin Van Durme. 2021. Factoring Statutory Reasoning as Language Understanding Challenges. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2742–2758

    2021

  18. [18]

    Liane Huttner and Denis Merigoux. 2022. Catala: moving towards the future of legal expert systems.Artificial intelligence and law(2022), 1–24

    2022

  19. [19]

    Yongjae Kim and Wonjae Lee. 2025. Where Does Legal AI Fail? Evaluating RAG Pipelines. InProceedings of the 34th ACM International Conference on Information and Knowledge Management. 1396–1405

    2025

  20. [20]

    Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang, and Yiqun Liu

  21. [21]

    Lexeval: A comprehensive chinese legal benchmark for evaluating large language models.Advances in Neural Information Processing Systems37 (2024), 25061–25094

    2024

  22. [22]

    Siyuan Li, Jian Chen, Rui Yao, Xuming Hu, Peilin Zhou, Weihua Qiu, Simin Zhang, Chucheng Dong, Zhiyao Li, Qipeng Xie, et al. 2025. Compliance-to-Code: Enhancing Financial Compliance Checking via Code Generation.arXiv preprint arXiv:2505.19804(2025)

    work page arXiv 2025

  23. [23]

    Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al . 2025. Deepseek- v3. 2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  24. [24]

    James Mohun and Alex Roberts. 2020. Cracking the code: Rulemaking for humans and machines.OECD Working Papers on Public Governance42 (2020), 0_1–109

    2020

  25. [25]

    OpenAI. 2025. Introducing GPT-5.2 | OpenAI. https://openai.com/index/ introducing-gpt-5-2/

    2025

  26. [26]

    Eunkyu Park, Wesley Hanwen Deng, Vasudha Varadarajan, Mingxi Yan, Gunhee Kim, Maarten Sap, and Motahhare Eslami. 2025. Critical or Compliant? The Double-Edged Sword of Reasoning in Chain-of-Thought Explanations.arXiv preprint arXiv:2511.12001(2025)

    work page arXiv 2025

  27. [27]

    Henry Prakken and Giovanni Sartor. 1996. A dialectical model of assessing conflicting arguments in legal reasoning.Artificial Intelligence and Law4, 3 (1996), 331–368

    1996

  28. [28]

    Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading.arXiv preprint arXiv:1809.01494(2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  29. [29]

    Fabio Vitali and Flavio Zeni. 2007. Towards a country-independent data format: the Akoma Ntoso experience. InProceedings of the V legislative XML workshop, Vol. 6786. European Press Academic Publishing Firenze

    2007

  30. [30]

    Wenhan Yu, Xinbo Lin, Lanxin Ni, Jinhua Cheng, and Lei Sha. 2025. Bench- marking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Models.arXiv preprint arXiv:2511.07979(2025)

    work page arXiv 2025

  31. [31]

    unless”, “provided that

    Xuanming Zhang, Yuxuan Chen, Yuan Yuan, and Minlie Huang. 2025. Towards Exception Safety Code Generation with Intermediate Representation Agents Framework.arXiv preprint arXiv:2410.06949(2025). A Diagnostic Analysis: Defeasibility and Silent Scope Omission Takeaway.We move beyond surface-level keywords to a mechanism-based evaluation using aStrong Judgeto...

    work page arXiv 2025

  32. [32]

    Annotation reliability Post-adjudication convergence reaches 0.91 for deon- tic label, 92.4% for exact span match, 86.8% for skeleton attachment, and 88.6% for tree-structure match

    changes routing by only Δroute ≈ 0.03; recon- ciled gold remains stable with Tree-ES≥0.98. Annotation reliability Post-adjudication convergence reaches 0.91 for deon- tic label, 92.4% for exact span match, 86.8% for skeleton attachment, and 88.6% for tree-structure match. Ambiguity handling 198 units (8.6%) are marked ambiguous, dominated by overlap/prece...