pith. sign in

arxiv: 2605.01486 · v1 · submitted 2026-05-02 · 💻 cs.AI

MAP-Law: Coverage-Driven Retrieval Control for Multi-Turn Legal Consultation

Qinchuan Cheng , Ruixuan Xie , Jiaqi Liu , Xiaoya Yuan , Yuxin Liu This is my paper

Pith reviewed 2026-05-09 14:19 UTC · model grok-4.3

classification 💻 cs.AI
keywords legal consultationretrieval controlmulti-turn agentscoverage metricsretrieval-augmented generationlabor lawgraph state representationaction selection
0
0 comments X

The pith

MAP-Law controls multi-turn legal retrieval by tracking coverage of required legal elements in a joint graph state rather than using fixed search depth.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces MAP-Law to manage retrieval depth during extended legal consultations where agents must gather authoritative support without missing key issues or overloading context. It represents the process as a structured graph linking legal issues to required elements and evidence pieces. After each retrieval round the system scores Element Coverage, Evidence Coverage, and Marginal Gain to decide whether to retrieve more, redirect, or stop and answer. This replaces arbitrary fixed rounds with decisions grounded in legal argumentative structure. Tests on fifty labor-law cases show the method reaches 0.860 element coverage using roughly three rounds and six evidence items on average.

Core claim

MAP-Law models consultation as a controlled retrieval process over a joint structured state of issue nodes, legal element nodes, and evidence nodes. After each round the agent computes Element Coverage, Evidence Coverage, and Marginal Gain to choose continuation, redirection, or final response generation. This converts stopping into an interpretable decision aligned with legal structure. On a self-constructed set of fifty cases spanning eight labor-law scenarios, MAP-Law with DeepSeek as selector reaches 0.860 Element Coverage using 2.9 retrieval rounds and 5.8 evidence pieces on average, cutting evidence volume by over 80 percent and rounds by 58 percent versus a fixed seven-round baseline.

What carries the argument

A joint graph state of issue nodes, legal element nodes, and evidence nodes, together with the Element Coverage, Evidence Coverage, and Marginal Gain metrics that drive LLM-based action selection for retrieval control.

If this is right

  • Achieves 0.860 Element Coverage with only 2.9 retrieval rounds and 5.8 evidence pieces on average across the tested labor-law cases.
  • Reduces evidence volume by more than 80 percent and retrieval rounds by 58 percent compared with fixed seven-round retrieval.
  • Makes stopping decisions interpretable by tying them directly to coverage of legal elements in the graph.
  • Ablation results show separate contributions from coverage-driven stopping, the joint graph representation, and LLM action selection.
  • Demonstrates consistent performance across eight distinct labor-law scenarios in the evaluation set.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The graph-based state representation could support audit trails that trace which legal elements were covered before a recommendation is issued.
  • Similar coverage-driven control might apply to other multi-turn domains that require structured evidence gathering before conclusion, such as medical or financial advising.
  • Testing the correlation between the paper's coverage metrics and human expert ratings of legal sufficiency on larger or more diverse case sets would strengthen claims of practical adequacy.
  • Integration with broader legal knowledge bases could further reduce average retrieval needs while preserving the same element coverage targets.

Load-bearing premise

The self-constructed fifty-case dataset and the newly defined Element Coverage and Evidence Coverage metrics accurately reflect when evidence is legally sufficient for a recommendation.

What would settle it

An independent evaluation by legal experts rating the sufficiency and accuracy of MAP-Law responses versus fixed-round baselines on a fresh set of cases, checking whether high Element Coverage scores reliably match expert judgments of recommendation readiness.

Figures

Figures reproduced from arXiv: 2605.01486 by Jiaqi Liu, Qinchuan Cheng, Ruixuan Xie, Xiaoya Yuan, Yuxin Liu.

Figure 1
Figure 1. Figure 1: Positioning of MAP-Law in prior research. Existing studies on legal RAG and grounded consultation improve retrieval quality and workflow organization, while planning-oriented language-agent methods provide reasoning-action loops, structured memory, and tool-use mechanisms. However, these lines of work rarely model when retrieval should stop in multi-turn legal consultation and whether the stopping decision… view at source ↗
Figure 2
Figure 2. Figure 2: Overall framework of MAP-Law. Starting from the user’s legal query and conversational context, the system performs issue parsing, element extraction, plan-graph initialization, joint graph memory maintenance, coverage evaluation, LLM-based action planning, targeted retrieval, evidence linking, and final answer generation. Plan graph. The plan graph is a directed acyclic graph containing issue nodes and ele… view at source ↗
Figure 3
Figure 3. Figure 3: Flowchart view of the MAP-Law control loop. The loop alternates between state initialization, coverage computation, stopping evaluation, LLM action selection, retrieval execution, evidence linking, graph update, and marginal-gain tracking. RQ2: What are the respective contributions of structured state representation and LLM-based action selection? This question is addressed through graph-removal and rule-b… view at source ↗
Figure 4
Figure 4. Figure 4: Primary experimental results: (a) Element Coverage by system; (b) average retrieval rounds by system. The dashed line marks the stopping threshold θE = 0.60. Evaluation metrics. We report four metrics: • Element Coverage (EC): proportion of fully supported legal elements; • Evidence Coverage (EVC): proportion of retrieved evidence ultimately cited; • Rounds: average number of retrieval rounds executed; • E… view at source ↗
Figure 5
Figure 5. Figure 5: Extended experimental results: (c) evidence count; (d) efficiency trade-off; (e) ablation study; (f) radar summary; (g) EC progression across rounds; (h) token efficiency. increase of approximately 65%, while keeping rounds and evidence nearly unchanged. This answers part of RQ2: the LLM contributes not merely by generating more retrieval activity, but by prioritizing more useful retrieval directions under… view at source ↗
read the original abstract

Legal consultation is a high-stakes, knowledge-intensive task that requires agents to identify relevant legal issues, retrieve authoritative support, and determine when evidence is sufficient for a recommendation. Although retrieval-augmented generation has improved grounding in legal question answering, many multi-turn legal agents still rely on fixed retrieval depth or coarse heuristic control. This often leads to either insufficient support for key legal elements or excessive retrieval that increases context burden and weakens answer focus. We propose MAP-Law, a coverage-driven framework for retrieval control in multi-turn legal consultation. MAP-Law models consultation as a controlled retrieval process over a joint structured state consisting of issue nodes, legal element nodes, and evidence nodes. After each retrieval round, the agent computes Element Coverage, Evidence Coverage, and Marginal Gain, and uses these signals to decide whether to continue retrieval, redirect the search, or generate the final response. In this way, MAP-Law turns stopping from a fixed hyperparameter into an interpretable and auditable decision aligned with legal argumentative structure. Experiments on a self-constructed dataset of 50 cases across eight labor-law scenarios show that MAP-Law with DeepSeek as the action selector achieves an Element Coverage of 0.860 using only 2.9 retrieval rounds and 5.8 evidence pieces on average. Compared with a fixed seven-round baseline, it reduces evidence volume by over 80% and retrieval rounds by 58%. Ablation results further confirm the independent contributions of coverage-driven stopping, joint graph representation, and LLM-based action selection.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript introduces MAP-Law, a coverage-driven framework for retrieval control in multi-turn legal consultation. It models the process as a joint graph over issue nodes, legal element nodes, and evidence nodes; after each round the agent computes Element Coverage, Evidence Coverage, and Marginal Gain to decide whether to continue retrieval, redirect, or terminate. Experiments on a self-constructed 50-case dataset spanning eight labor-law scenarios report that MAP-Law (with DeepSeek action selection) reaches 0.860 Element Coverage using 2.9 rounds and 5.8 evidence pieces on average, reducing evidence volume by >80% and rounds by 58% relative to a fixed seven-round baseline; ablations are said to confirm the contributions of coverage stopping, the joint graph, and LLM action selection.

Significance. If the newly defined coverage metrics prove to be reliable proxies for legal sufficiency, the work would offer a concrete, interpretable mechanism for dynamic retrieval control in high-stakes RAG agents, addressing the common problems of under- or over-retrieval. The structured graph representation and explicit marginal-gain signals are technically appealing and could generalize beyond labor law. The reported efficiency gains are large enough to be practically interesting, but the absence of external validation against expert judgments or downstream correctness measures limits the strength of the significance claim at present.

major comments (3)
  1. [Experiments] Experiments section: the reported Element Coverage of 0.860, 2.9 retrieval rounds, and 5.8 evidence pieces are presented without any definition or formula for how coverage is computed over the joint issue-element-evidence graph, without error bars, and without statistical tests comparing against the fixed-round baseline; these omissions make it impossible to assess whether the claimed 80% evidence reduction and 58% round reduction are robust.
  2. [Experiments] Dataset construction and metric validation: the 50-case dataset is described only as 'self-constructed' across eight labor-law scenarios with no details on case selection, annotation protocol, or inter-annotator agreement; moreover, no correlation is reported between the proposed Element/Evidence Coverage scores and human lawyer judgments of legal sufficiency or any downstream outcome (e.g., correctness of final advice).
  3. [Method] Metrics definition: the central thesis that coverage-driven stopping is 'aligned with legal argumentative structure' rests on the unvalidated assumption that the newly introduced Element Coverage, Evidence Coverage, and Marginal Gain signals accurately indicate when evidence is legally sufficient; without ground-truth sufficient-evidence sets or expert correlation, the efficiency numbers do not yet demonstrate safe control.
minor comments (2)
  1. [Experiments] The abstract and experiments mention 'DeepSeek as the action selector' but supply no prompt templates, temperature settings, or few-shot examples used for the LLM-based decision policy.
  2. Figure or table captions could more explicitly state the exact definitions and thresholds used for the coverage and marginal-gain signals.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback, which identifies key areas where the manuscript can be strengthened in terms of transparency, rigor, and validation. We address each major comment point by point below and will incorporate revisions to improve the paper.

read point-by-point responses

  1. Referee: [Experiments] Experiments section: the reported Element Coverage of 0.860, 2.9 retrieval rounds, and 5.8 evidence pieces are presented without any definition or formula for how coverage is computed over the joint issue-element-evidence graph, without error bars, and without statistical tests comparing against the fixed-round baseline; these omissions make it impossible to assess whether the claimed 80% evidence reduction and 58% round reduction are robust.

    Authors: We agree that the Experiments section should explicitly define the coverage metrics and provide supporting statistical analysis. In the revised manuscript, we will add the precise formulas for Element Coverage (fraction of required legal elements covered by retrieved evidence), Evidence Coverage (fraction of supporting evidence nodes populated), and Marginal Gain (incremental coverage improvement per round) as computed over the joint graph. We will also report standard deviations across the 50 cases and include statistical significance tests (e.g., paired t-tests or Wilcoxon signed-rank tests) comparing MAP-Law against the fixed-round baseline to substantiate the reported reductions in rounds and evidence volume. revision: yes

  2. Referee: [Experiments] Dataset construction and metric validation: the 50-case dataset is described only as 'self-constructed' across eight labor-law scenarios with no details on case selection, annotation protocol, or inter-annotator agreement; moreover, no correlation is reported between the proposed Element/Evidence Coverage scores and human lawyer judgments of legal sufficiency or any downstream outcome (e.g., correctness of final advice).

    Authors: We acknowledge the need for greater transparency on dataset construction. The revised manuscript will expand the Experiments section to detail case selection criteria (representative labor-law queries drawn from public sources across eight common scenarios), the annotation protocol for labeling issue, element, and evidence nodes, and any inter-annotator agreement measures used. Regarding correlation with human lawyer judgments or downstream correctness, our study did not collect such external validation data; we will explicitly note this as a limitation and discuss how the internal graph-based metrics and ablation results provide initial evidence of utility, while proposing expert correlation studies as future work. revision: partial

  3. Referee: [Method] Metrics definition: the central thesis that coverage-driven stopping is 'aligned with legal argumentative structure' rests on the unvalidated assumption that the newly introduced Element Coverage, Evidence Coverage, and Marginal Gain signals accurately indicate when evidence is legally sufficient; without ground-truth sufficient-evidence sets or expert correlation, the efficiency numbers do not yet demonstrate safe control.

    Authors: The referee is correct that the alignment claim rests on a modeling assumption. The joint graph is explicitly constructed to reflect standard legal argument structure (issues decomposed into elements supported by evidence), and the coverage signals are designed to operationalize sufficiency within that structure. However, we agree that without ground-truth sufficient-evidence sets or expert correlation, the safety of the stopping decisions cannot be fully demonstrated. In revision, we will clarify this assumption in the Method section, temper the language around 'safe control,' and add a dedicated limitations paragraph discussing the risks of unvalidated metrics while highlighting that the reported high coverage (0.860) with reduced retrieval provides empirical support for the approach. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation or experimental claims

full rationale

The paper defines a new coverage-driven retrieval framework with Element Coverage, Evidence Coverage, and Marginal Gain computed directly from its proposed joint issue-element-evidence graph state. These quantities are introduced as part of the method, used for stopping decisions, and then measured in experiments on a self-constructed dataset against a fixed-round baseline. No equations, fitted parameters, or self-citations are shown that would make the reported coverage numbers or efficiency gains equivalent to the inputs by construction. The evaluation remains an independent empirical comparison rather than a tautological renaming or self-referential fit.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 1 invented entities

The framework rests on the assumption that legal sufficiency can be captured by node coverage counts and that the self-constructed dataset reflects real consultation dynamics; no free parameters are explicitly fitted in the abstract, but implicit thresholds for the three signals are required.

free parameters (1)
  • coverage and marginal-gain thresholds
    Decision boundaries for continue/redirect/stop actions are not numerically specified and must be chosen to produce the reported behavior.
axioms (1)
  • domain assumption Legal consultations can be faithfully represented as a joint graph of issue nodes, legal-element nodes, and evidence nodes.
    This modeling choice is the foundation of the coverage calculations.
invented entities (1)
  • Element Coverage, Evidence Coverage, and Marginal Gain signals no independent evidence
    purpose: To provide interpretable stopping criteria aligned with legal structure
    These three quantities are introduced by the paper to replace fixed retrieval depth.

pith-pipeline@v0.9.0 · 5584 in / 1409 out tokens · 45704 ms · 2026-05-09T14:19:04.132971+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 6 canonical work pages · 1 internal anchor

  1. [1]

    Artificial intelligence and law: An overview.Georgia State University Law Review, 35, 2019

    Harry Surden. Artificial intelligence and law: An overview.Georgia State University Law Review, 35, 2019

    2019

  2. [2]

    How does NLP benefit legal system: A summary of legal artificial intelligence

    Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. How does NLP benefit legal system: A summary of legal artificial intelligence. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5218–5230, 2020

    2020

  3. [3]

    Karl Branting, Jack G

    Serena Villata, Michal Araszkiewicz, Kevin Ashley, Trevor Bench-Capon, L. Karl Branting, Jack G. Conrad, and Adam Wyner. Thirty years of artificial intelligence and law: the third decade.Artificial Intelligence and Law30:561–591, 2022

    2022

  4. [4]

    Jinqi Lai, Wensheng Gan, Jiayang Wu, Zhenlian Qi, and Philip S. Yu. Large language models in law: A survey.AI Open5:181–196, 2024

    2024

  5. [5]

    Natural language processing in the legal domain.arXiv preprint arXiv:2302.12039, 2023

    Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, and Nikolaos Aletras. Natural language processing in the legal domain.arXiv preprint arXiv:2302.12039, 2023

    work page internal anchor Pith review arXiv 2023

  6. [6]

    Muddamsetty, Thomas Gammeltoft-Hansen, Henrik Palmer Olsen, and Thomas B

    Karen McGregor Richmond, Satya M. Muddamsetty, Thomas Gammeltoft-Hansen, Henrik Palmer Olsen, and Thomas B. Moeslund. Explainable AI and law: An evidential survey. Digital Society3(1), 2024

    2024

  7. [7]

    Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E. Ho. Large legal fictions: Profiling legal hallucinations in large language models.arXiv preprint arXiv:2401.01301, 2024

    work page arXiv 2024

  8. [8]

    Chatlaw: Open-source legal large language model with integrated external knowledge bases,

    Jiaxi Cui, Munan Ning, Zongjian Li, Bohua Chen, Yang Yan, Hao Li, Bin Ling, Yonghong Tian, and Li Yuan. Chatlaw: A multi-agent collaborative legal assistant with knowledge graph enhanced mixture-of-experts large language model.arXiv preprint arXiv:2306.16092, 2023

    work page arXiv 2023

  9. [9]

    LexRAG: Benchmarking retrieval-augmented generation in multi-turn legal consultation conversation.arXiv preprint arXiv:2502.20640, 2025

    Haitao Li, Yifan Chen, Yiran Hu, Qingyao Ai, Junjie Chen, Xiaoyu Yang, Jianhui Yang, Yueyue Wu, Zeyang Liu, and Yiqun Liu. LexRAG: Benchmarking retrieval-augmented generation in multi-turn legal consultation conversation.arXiv preprint arXiv:2502.20640, 2025. 20

    work page arXiv 2025

  10. [10]

    Lawluo: A chinese law firm co-run by llm agents

    Jingyun Sun, Chengxiao Dai, Zhongze Luo, Yangbo Chang, and Yang Li. LawLuo: A multi- agent collaborative framework for multi-round chinese legal consultation.arXiv preprint arXiv:2407.16252, 2024

    work page arXiv 2024

  11. [11]

    Legalbench-rag: A benchmark for retrieval- augmented generation in the legal domain

    Nicholas Pipitone and Ghita Houir Alami. LegalBench-RAG: A benchmark for retrieval- augmented generation in the legal domain.arXiv preprint arXiv:2408.10343, 2024

    work page arXiv 2024

  12. [12]

    Finding the law: Enhancing statutory article retrieval via graph neural networks

    Antoine Louis, Gijs van Dijck, and Gerasimos Spanakis. Finding the law: Enhancing statutory article retrieval via graph neural networks. InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2761–2776, 2023

    2023

  13. [13]

    Beling, Michael A

    Faraz Dadgostari, Mauricio Guim, Peter A. Beling, Michael A. Livermore, and Daniel N. Rockmore. Modeling law search as prediction.Artificial Intelligence and Law29(1):3–34, 2021

    2021

  14. [14]

    LexGLUE: A benchmark dataset for legal language understanding in english

    Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. LexGLUE: A benchmark dataset for legal language understanding in english. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 4310–4330, 2022

    2022

  15. [15]

    LEXTREME: A multi-lingual and multi-task benchmark for the legal domain

    Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis. LEXTREME: A multi-lingual and multi-task benchmark for the legal domain. InFindings of the Association for Computational Linguistics: EMNLP 2023, pp. 3056–3080, 2023

    2023

  16. [16]

    Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, and Daniel E. Ho. MultiLe- galPile: A 689GB multilingual legal corpus. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp. 15077–15094, 2024

    2024

  17. [17]

    Predicting judicial decisions of the European Court of Human Rights: A natural language processing perspective.PeerJ Computer Science2:e93, 2016

    Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preoţiuc-Pietro, and Vasileios Lampos. Predicting judicial decisions of the European Court of Human Rights: A natural language processing perspective.PeerJ Computer Science2:e93, 2016

    2016

  18. [18]

    Paragraph-level rationale extraction through regularization: A case study on European Court of Human Rights cases

    Ilias Chalkidis, Manos Fergadiotis, Dimitrios Tsarapatsanis, Nikolaos Aletras, Ion An- droutsopoulos, and Prodromos Malakasiotis. Paragraph-level rationale extraction through regularization: A case study on European Court of Human Rights cases. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguist...

    2021

  19. [19]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InProceedings of the International Conference on Learning Representations, 2023

    2023

  20. [20]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. InAdvances in Neural Information Processing Systems, 2023

    2023

  21. [21]

    Towards end-to-end reinforcement learning of dialogue agents for information access

    Bhuwan Dhingra, Lihong Li, Xiujun Li, Jianfeng Gao, Yun-Nung Chen, Faisal Ahmed, and Li Deng. Towards end-to-end reinforcement learning of dialogue agents for information access. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp. 484–495, 2017. 21

    2017