Pith. sign in

REVIEW 4 major objections 6 minor 30 references

KoBLEX: Open Legal Question Answering with Multi-hop Reasoning

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that using LLM-generated statutory guesses as search queries is the best way to retrieve supporting provisions and answer open-ended multi-hop legal questions in Korean law.

desk verdict Solid bilingual legal QA benchmark and a usable retrieval pipeline; the new metric has a circularity problem and the retrieval trick's key assumption is untested. read the letter →

arxiv 2509.01324 v1 pith:LPAUEEL3 submitted 2025-09-01 cs.CL

Jihyung Lee , Daehui Kim , Seonjeong Hwang , Hyounghun Kim , Gary Lee This is my paper
classification cs.CL
keywords legalquestionansweringmulti-hopreasoningstatuteretrievalretrieval-augmentedgenerationKoreanlawbenchmarkLLM-as-a-judgefidelityevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

KoBLEX is a new bilingual Korean–English benchmark of 226 scenario-based legal questions, each requiring one to three statutory provisions to answer. The paper argues that existing legal benchmarks, mostly multiple-choice or classification tasks, do not test open-ended answers that cite the actual law, and that KoBLEX fills that gap. It then proposes ParSeR, a retrieval method that first asks an LLM to write the statutes it thinks apply, then uses each generated provision as a query in a retrieve–rerank–select pipeline, and finally answers the question with the selected real provisions. On KoBLEX, ParSeR is claimed to beat all baselines across all LLMs and metrics: with GPT-4o, provision retrieval F1 rises by +37.91 over one-time retrieval, and LF-Eval, a new legal-fidelity metric, rises by +30.81. LF-Eval is an LLM-as-a-judge metric that scores answers against the question, the expected answer, and the supporting provisions, and it correlates with human judgments at 84.90 Pearson.

What carries the argument

The machinery is ParSeR's parametric provision. Given a multi-hop legal question, the LLM first generates a list of statute-like clauses that would support an answer, explicitly instructed to produce the most plausible clause when no real statute comes to mind. Each generated clause becomes a query: BM25 retrieves the top-100 provisions from a 233,544-paragraph Korean statute corpus, a Korean-finetuned cross-encoder reranks them, and the LLM selects the single best provision per parametric clause. The collected provisions are then fed back to the LLM for provision-grounded answer generation. The paper also introduces LF-Eval, a G-Eval-style judge that scores answers on answer relevance, lega

What would settle it

Run ParSeR on a statute corpus in a language or legal domain where the generator has little parametric knowledge, or force the generator to produce paraphrases that share no content words with the real statutes, and compare provision-retrieval F1 against one-time retrieval; if the margin vanishes, the parametric-provision guidance is what does the work, not the pipeline itself.

Watch

Extended reading notes

Core claim

The central claim is that a pipeline which lets the LLM first produce plausible statute texts from its own knowledge, then treats each such text as a separate search query against a real statute corpus, recovers the gold provisions far better than one-shot retrieval, iterative retrieval (IRCoT, FLARE), or sub-question decomposition (ProbTree, BeamAggr), and that this better retrieval translates directly into better answers. The paper reports that with GPT-4o, ParSeR improves provision retrieval F1 by +37.91 and EM by +19.91 over one-time retrieval, and improves answer quality by +19.39 token-level F1 and +30.81 LF-Eval; against the strongest baseline (ProbTree), gains are +12.23 token F1 and

Load-bearing premise

The load-bearing premise is that the LLM's parametric knowledge of Korean law is strong enough that its fabricated provisions overlap lexically with the real statutes, so BM25 plus reranking can surface the gold provisions; if that overlap disappears, ParSeR collapses toward one-time retrieval.

Editorial extensions

If this is right

  • If ParSeR's gains are real, the best recipe for statute-grounded legal QA is to decompose a hard question into several LLM-written statute guesses and search for each one separately, rather than retrieving once with the original question.
  • Provision retrieval becomes a controllable bottleneck: the three-stage pipeline lifts provision F1 from about 21.5 with one-time retrieval to 59.4 with GPT-4o, and that retrieval improvement is what drives answer quality.
  • LF-Eval can replace token-overlap metrics for legal-fidelity evaluation; its 84.90 Pearson correlation with human raters is far above token F1 (61.25) and a prior faithfulness score (6.87) on this benchmark.
  • The method is claimed to stay best across 1-, 2-, and 3-hop questions and across 8B to 32B models, so it does not depend on a single large model.
  • ParSeR is also the most token-efficient: it reaches the highest LF-Eval with the fewest generated tokens, so better retrieval reduces rather than increases inference cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would expect the parametric-provision trick to transfer to other codified-law jurisdictions only where the LLM's parametric knowledge of that statute book is strong enough to produce lexical overlap; in lower-resource legal systems, the gains may shrink or vanish.
  • A natural extension, not tested here, is to use ParSeR's selected provisions as an attribution signal and verify whether the generated answer's citations match the retrieved set, turning retrieval accuracy into an answer-explainability check.
  • LF-Eval could plausibly serve as a reward model for reinforcement learning or as a filter in self-training loops, since it is cheap, interpretable, and aligns well with expert judgment.
  • The benchmark's construction pipeline, especially the check that rejects questions answerable by only a subset of the provided provisions, suggests a reusable recipe for building multi-hop QA datasets in other expert domains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces KoBLEX, a bilingual Korean-English benchmark of 226 provision-grounded, multi-hop legal QA instances, constructed through a hybrid LLM-and-human-expert pipeline and accompanied by gold supporting statutory provisions. It also proposes ParSeR, a retrieval pipeline that first prompts an LLM to generate 'parametric provisions' (LLM-written statute-like clauses), then uses each generated provision as a query in a retrieve–rerank–select loop over a 233,544-paragraph Korean statute corpus, and finally generates an answer from the selected provisions. A third contribution is LF-Eval, a G-Eval-style LLM-as-a-judge metric that scores answers against the expected output and the gold provisions. Experiments on five LLMs report that ParSeR consistently outperforms standard prompting, one-time retrieval, IRCoT, FLARE, ProbTree, and BeamAggr on retrieval F1/EM, token F1, and LF-Eval; ablations and hyperparameter analyses support the contribution of each pipeline component.

Significance. If the central claims hold, this is a useful package: a carefully validated legal QA benchmark in a non-English jurisdiction, a practical three-stage retrieval method for provision-grounded reasoning, and an interpretable automatic legal-fidelity metric. The paper has clear strengths: the benchmark is bilingual, the construction pipeline is documented with agreement statistics, the statute corpus and dataset are released, the method is evaluated across five models, and LF-Eval is validated against human judgments with a reported Pearson correlation of 84.90. The authors also candidly acknowledge the small scale and civil-law focus. However, two load-bearing aspects need strengthening before the claims can be accepted at face value: the unexamined lexical-overlap assumption that drives ParSeR's query-generation mechanism, and the confounding of retrieval quality with LF-Eval's scoring criteria. The lack of any significance testing or confidence intervals on a 226-instance benchmark is an additional concern for the 'consistently outperforms' claim.

major comments (4)
  1. [Section 4.1, Figure 19, Table 3] The core mechanism of ParSeR is that LLM-generated parametric provisions—including fabricated clauses produced when the model is told to 'generate the most plausible clause'—share enough lexical overlap with the true statutes for BM25 plus BGE reranking to surface gold provisions. The paper never measures this interface directly. It does not report recall@k of gold provisions using only the parametric provisions as queries, nor token/lexical overlap between generated and actual provisions. Table 3 shows retrieval F1 dropping from 48.74 to 21.41 when provision generation is replaced by original-question top-k retrieval, but that row also removes reranking and selection, so it does not isolate Step 1 under the full pipeline. Because all evaluated models have strong Korean legal knowledge, the gain may reflect memorized statute text rather than a general query-expansion mechanism. Please re
  2. [Section 5, Figure 5, Table 2] LF-Eval's evaluation prompt instructs the judge to 'heavily penalize when the legal conclusion differs in detail from the expected output' and to 'heavily penalize if the prediction contradicts or omits any specific elements from the context.' Since ParSeR explicitly feeds retrieved gold provisions into the generator, its outputs are structurally more likely to contain the exact context elements that LF-Eval rewards, independently of genuine legal reasoning quality. The +30.81 LF-Eval gap over one-time retrieval for GPT-4o is therefore not a clean measure of answer quality. Because LF-Eval is part of the paper's headline claims, add a sensitivity analysis—for example, scoring answers with the context omitted from the judge prompt, or controlling for retrieved provision quality—and, if possible, report human correlation separately for ParSeR-generated answers and baseline-generated answer
  3. [Section 7, Tables 2 and 10] All experimental results are single-run point estimates on a 226-instance benchmark, with no error bars, confidence intervals, or significance tests. The text uses 'significantly surpasses' (Section 7) and 'consistently outperforms' without statistical support. Some comparisons are large, but differences in retrieval EM in Table 10 are small, and Tables 5–6 show sensitivity to k and l. Please report at least bootstrap confidence intervals and paired significance tests for the main ParSeR-versus-baseline comparisons, and for the small-model appendix results.
  4. [Section 6.4, Table 2] The one-time retrieval baseline is described as having 'oracle access to the same number of gold reference provisions as the required number of reasoning hops,' but the results indicate it actually retrieves top-n provisions using the original question. ParSeR instead issues several LLM-generated parametric queries, each with a top-100 candidate pool, reranking to top-10, and LLM selection. ParSeR therefore benefits from a much larger retrieval budget, not only from better queries. To attribute the improvement to parametric provision generation, add an equal-budget control that sends the original question through the same retrieve–rerank–select pipeline with the same number of queries and the same k/l values.
minor comments (6)
  1. [Section 3.5] Typo: 'Koean data' should be 'Korean data.'
  2. [Section 8 heading] The heading 'Effect of Retreiver Type' contains a typo: 'Retriever.'
  3. [Section 4.2 vs Section 6.1] Section 4.2 describes the retriever as a 'Bi-encoder retriever based on cosine similarity,' but the experiments use BM25 as the retriever and BGE only as a reranker. Please align the method description with the actual experimental configuration.
  4. [Section 6.4] The phrase 'oracle access to the same number of gold reference provisions' is misleading. Clarify that the baseline uses top-n retrieval with n equal to the number of hops, not oracle access to gold provisions.
  5. [Appendix G.1, Table 8] The annotation study does not state the sample size, the number of responses scored, or how the two annotator groups were split. Also, 'ragree = 73.97' is not defined. Please provide these details, and fix the 'Faithfullness' typo.
  6. [Notation throughout] The method name is written inconsistently as 'PARSER' and 'ParSeR' (e.g., abstract vs. body). Use one spelling consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central claims rest on empirical comparisons and not on self-referential definitions.

full rationale

After walking the derivation chain, I find no circular step that reduces a prediction to an input by construction. ParSeR's parametric provisions are generated from the question via prompting (Figure 19) and used only as intermediate retrieval queries; they are not fitted to the gold provisions, and retrieval quality is measured independently against the gold set (Eq. 1). The ablation in Table 3 shows empirical contributions of each component; removing the provision step degrades F1, which is an empirical finding, not a tautology. LF-EVAL (Figure 5) is a reference-based LLM judge that compares predictions to the expected output and context; this makes it a strict fidelity metric, and ParSeR's higher LF-EVAL is an empirical consequence of retrieving and using the gold provisions, not a definitional equivalence. The metric would penalize any method equally for omitting context details; there is no fitted parameter or self-citation that forces the result. The only self-citations (e.g., Cho and Lee 2025) are related-work mentions and are not load-bearing. The paper's limitations (small 226-instance benchmark, civil-law focus, expert dependency) are honestly stated and do not hide a circular derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The paper's claims rest primarily on three empirical constructs: a benchmark whose questions are assumed derivable only from the gold provision set (verified by LLM checks plus Korean law graduates, not by an independent legal ground truth), a retrieval mechanism that assumes BM25 lexical overlap between fabricated parametric provisions and true statute text, and an evaluation metric anchored to human ratings via a study whose size and annotator qualifications are not fully reported. Hyperparameters k and l are selected on the same 226-instance benchmark used for the final headline results. No free parameters beyond these, and no invented entities beyond the internal query-scaffold construct.

free parameters (3)
  • k (retrieval scope) = 100
    Top-k provisions retrieved per parametric provision; chosen from the k in {50,100,200,300} ablation in Table 5, run on the same 226-instance benchmark used for final evaluation.
  • l (reranking scope) = 10
    Top-l provisions after reranking, from which the LLM selects one; chosen from l in {5,10,20,30} in Table 6, again tuned on the benchmark test set.
  • FLARE log-probability thresholds = -1.5 (Qwen/EXAONE), -0.6 (GPT-4o)
    Baseline-specific thresholds tuned to achieve a roughly 50% retrieval rate as reported in the original FLARE paper; this is baseline tuning rather than part of the central claim.
assumptions (4)
  • domain assumption The gold provision set C is both necessary and sufficient to answer Q; necessity is enforced by the Partial Check powerset test and sufficiency by the Full Check derivability criterion (Section 3.3).
    The entire benchmark and its retrieval evaluation assume answers are uniquely derivable from C alone with no external legal knowledge; verified by GPT-4o checks and Korean law graduates, not by an independent legal ground truth.
  • domain assumption BM25 lexical similarity between LLM-fabricated parametric provisions and true statute text is a usable retrieval signal (Sections 4.1-4.2).
    The mechanism assumes the fabricated provisions overlap lexically with the corpus; no analysis of when this fails is provided.
  • domain assumption The 608-statute, 233,544-paragraph corpus is the correct and complete retrieval pool for all gold provisions (Section 6.3).
    Gold provisions are drawn from this corpus by construction, so retrievability is guaranteed, but the claim that this pool mirrors realistic deployment is assumed.
  • domain assumption GPT-4o as judge (LF-Eval) tracks legal fidelity, validated by a Pearson correlation of 84.90 with human judgments (Appendix G.2).
    Sample size, annotator legal expertise, and the response set for the human study are not reported in the main text; the correlation is the sole external anchor for the metric.
invented entities (1)
  • Parametric provision (LLM-fabricated statute text)
    purpose: Generated query scaffold used to retrieve real provisions from the statute corpus in Step 1 of ParSeR (Section 4.1).
    An internal retrieval artifact with no falsifiable handle outside the pipeline; its only role is to bridge the query-document lexical gap.

how reviews work

0 comments
Cite this review

Pith. "Pith review of KoBLEX: Open Legal Question Answering with Multi-hop Reasoning." pith.science (2026). https://pith.science/paper/LPAUEEL3

@misc{pith2026250901324,
  author       = {Pith},
  title        = {Pith review of: KoBLEX: Open Legal Question Answering with Multi-hop Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LPAUEEL3}},
  note         = {Machine review of arXiv:2509.01324}
}
read the original abstract

Large Language Models (LLM) have achieved remarkable performances in general domains and are now extending into the expert domain of law. Several benchmarks have been proposed to evaluate LLMs' legal capabilities. However, these benchmarks fail to evaluate open-ended and provision-grounded Question Answering (QA). To address this, we introduce a Korean Benchmark for Legal EXplainable QA (KoBLEX), designed to evaluate provision-grounded, multi-hop legal reasoning. KoBLEX includes 226 scenario-based QA instances and their supporting provisions, created using a hybrid LLM-human expert pipeline. We also propose a method called Parametric provision-guided Selection Retrieval (ParSeR), which uses LLM-generated parametric provisions to guide legally grounded and reliable answers. ParSeR facilitates multi-hop reasoning on complex legal questions by generating parametric provisions and employing a three-stage sequential retrieval process. Furthermore, to better evaluate the legal fidelity of the generated answers, we propose Legal Fidelity Evaluation (LF-Eval). LF-Eval is an automatic metric that jointly considers the question, answer, and supporting provisions and shows a high correlation with human judgments. Experimental results show that ParSeR consistently outperforms strong baselines, achieving the best results across multiple LLMs. Notably, compared to standard retrieval with GPT-4o, ParSeR achieves +37.91 higher F1 and +30.81 higher LF-Eval. Further analyses reveal that ParSeR efficiently delivers consistent performance across reasoning depths, with ablations confirming the effectiveness of ParSeR.

Figures

Figures reproduced from arXiv: 2509.01324 by the authors.

Figure 1
Figure 1. Overview of KOBLEX structure and task design. Given a complex legal question, the system is required to reason over multiple statutory provisions. (Zhao et al., 2025), leading to the development of diverse benchmarks across general domains (Hendrycks et al., 2021; Cobbe et al., 2021; Zhou et al., 2023; Zheng et al., 2023; Rein et al., 2024). As LLMs increasingly demonstrate expert-level ca￾pabilities, interest in th… view at source ↗
Figure 2
Figure 2. Generation and validation pipeline for KOBLEX. The pipeline consists of three stages: (A) context construction from either randomly sampled segment of statute corpus or reference provisions in precedents, (B) question–answer generation using GPT-4o based on the selected legal context, and (C) multi-stage validation. An initial LLM-based evaluation filters out incomplete or unsupported pairs based on predefined crite… view at source ↗
Figure 3
Figure 3. Example QA instance from the KOBLEX (translated from Korean). This multi-hop question requires interpreting multiple statutes. Yellow texts highlight key legal information essential for deriving the correct answer. 3 KOBLEX In this section, we describe the construction process of KOBLEX. We first generate initial drafts using an LLM, then filter and revise them through a multi￾stage validation process involving both… view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: Illustration of PARSER. (Step 1) The LLM initially generates parametric provisions. (Step 2) Parametric provisions are then used as queries for Retrieve, Rerank, and Selection retrieval. (Step 3) Finally, collected supporting legal provisions are used to multi-hop reas…
Figure 5
Figure 5. Figure 5: Prompt of Legal Fidelity Evaluation (LF￾EVAL). {placeholder} indicates a slot to be filled with the corresponding value for evaluation. provision-grounded reasoning. The parametric pro￾visions generated in the initial stage facilitate multi￾hop reasoning by enabling PA…
Figure 6
Figure 6. Figure 6: Retrieval performance (F-1) and generation performance ( [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Average PARSER scores with respect to aver￾age number of generated tokens by GPT-4o. methods such as FLARE and ProbTree outperform other baselines. Specifically, FLARE ranks sec￾ond at the 1-hop level, while ProbTree excels in deeper reasoning at the 2-hop and 3-hop le…
Figure 8
Figure 8. Figure 8: Distribution of Statutes in KOBLEX. shows the most frequently appearing statutes in the bench￾mark—specifically, those cited in at least nine QA instances. Statutes referenced fewer than nine times are aggre￾gated into the Others category for clarity. The numbers outsi…
Figure 9
Figure 9. Figure 9: Inter-annotator agreement for each evaluation metric. We report the number and proportion of QA instances where at least two annotators agreed (Two or More Agree, blue) and where all three annotators provided identical labels (Three Agree, orange). Labels are categoriz…
Figure 10
Figure 10. Figure 10: Prompt templates used for generating legal Q&A pairs from statutory text. Part 1 describes instructions [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Prompt template for scenario-based reformulation of legal Q&A pairs. Part 3 rewrites a statutory Q&A [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Prompt templates for validating legal Q&A pairs. Part 4 checks answerability based on context, while [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Prompt for Standard Prompting (SP). We employ 5 different examples. {placeholder} indicates a slot to be filled with the corresponding value for evaluation. Chain-of-Thought prompt You are a helpful assistant. Whenever you are given a legal (or similar) question, stru…
Figure 14
Figure 14. Figure 14: Prompt for Chain-of-Thought (CoT). We employ 5 different examples. {placeholder} indicates a slot to be filled with the corresponding value for evaluation [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Prompt for retrieval augmented question answering. The system prompt and inclusion of reasoning [PITH_FULL_IMAGE:figures/full_fig_p028_15.png]
Figure 16
Figure 16. Figure 16: Prompt for Self-ask. We employ 5 different examples. {placeholder} indicates a slot to be filled with the corresponding value for evaluation [PITH_FULL_IMAGE:figures/full_fig_p028_16.png]
Figure 17
Figure 17. Figure 17: Prompt for IRCoT and FLARE. We employ 5 different examples. {placeholder} indicates a slot to be filled with the corresponding value for evaluation [PITH_FULL_IMAGE:figures/full_fig_p029_17.png]
Figure 18
Figure 18. Figure 18: Prompt for ProbTree and BeamAggr. We employ 5 different examples. {placeholder} indicates a slot to be filled with the corresponding value for evaluation [PITH_FULL_IMAGE:figures/full_fig_p030_18.png]
Figure 19
Figure 19. Figure 19: Prompt for PARSER. For parametric provision generation, we instruct LLM to generate list of parametric provisions. For selection, we instruct LLM to select most relevant provision among top-10 candidates. We employ 5 different examples. {placeholder} indicates a slot …
Figure 20
Figure 20. Figure 20: Case study of a single English instance from [PITH_FULL_IMAGE:figures/full_fig_p032_20.png]
Figure 21
Figure 21. Figure 21: Case study of a single English instance from [PITH_FULL_IMAGE:figures/full_fig_p033_21.png]
Figure 22
Figure 22. Figure 22: Case study of a single Korean instance from [PITH_FULL_IMAGE:figures/full_fig_p034_22.png]
Figure 23
Figure 23. Figure 23: Case study of a single Korean instance from [PITH_FULL_IMAGE:figures/full_fig_p035_23.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 25 canonical work pages

  1. [1]

    Select one provision from the given context that is legally meaningful and suitable for generating a question

    work page

  2. [2]

    Proceedings of the International Confer- ence on Learning Representations (ICLR)

    Measuring massive multitask language under- standing. Proceedings of the International Confer- ence on Learning Representations (ICLR). Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large lan- guage models: Principles, t...

    work page 2025

  3. [3]

    Provide an accurate legal answer that can be answered solely based on the selected provision

    work page

  4. [4]

    Not applicable

    If none of the given provisions are appropriate for generating a question, output only the line: "Not applicable" Output Format: question: [Generated legal question] answer: [Accurate answer based on the provision] selected_context: [Only include the provision used for the Q&A] Example: {Example} <Query> Context: {context} Part 2. Instruction for Q&A Gene...

    work page

  5. [5]

    Not applicable

    If none of the remaining provisions are appropriate for expansion, output only the line: "Not applicable" Constraints

    work page

  6. [7]

    The results show consistent patterns with robust LLMS as shown in the main body of the paper represented in Table 2

    and EXAONE-3.5-7.8B (Research, 2024). The results show consistent patterns with robust LLMS as shown in the main body of the paper represented in Table 2. PARSER reliably achieves the highest performance across metrics. These findings confirm that our approach is effective even with limited model capacity, highlighting its scalability and generalization. ...

    work page 2024

  7. [9]

    Create a clear and legally relevant question that reflects the core content of the selected provision

    work page

  8. [12]

    The current_context, which contains the provision(s) used for the existing Q&A

    work page

Show all 30 references
  1. [13]

    The remain_context, which contains additional legal provisions not yet used

  2. [14]

    Your task is to:

    Your task is to expand the original question into a (k+1)-hop question that logically incorporates one new provision from remain_context. Your task is to:

  3. [15]

    Select one provision from remain_context that logically connects to the existing question or answer

  4. [16]

    Generate an expanded (k+1)-hop legal question that requires both current_context and the newly selected provision to be answered

  5. [17]

    Provide a new answer that integrates both contexts

  6. [18]

    Output the newly selected provision as selected_context

  7. [20]

    The new question must logically build upon the existing question

  8. [21]

    The answer must not be answerable using only one of the contexts (neither current_context nor the selected_context alone)

  9. [22]

    The question must be in Korean, concise, and naturally phrased. Output Format: question: [Expanded k+1-hop legal question] answer: [New answer that depends on both current_context and selected_context] selected_context: [One newly selected sentence from remain_context] Example...

  10. [23]

    Create a short but concrete fact pattern (case scenario) that would require applying the same legal reasoning

  11. [24]

    Rewrite the original question to match the scenario

  12. [25]

    Keep the original legal answer, with minor edits if needed to match the scenario. Output Format: background_scenario: {BACKGROUND SCENARIO} question: {MULTI-HOP LEGAL QUESTION} answer: {ANSWER} <Query> Question: {question} Answer: {answer} Context: {context} Figure 11: Prompt ...

  13. [26]

    In other words, assess whether the question is logically and legally consistent with the context without requiring outside legal knowledge

    Scenario Consistent: Determine whether the background_scenario + question could have been composed using only the explicit legal content given in context. In other words, assess whether the question is logically and legally consistent with the context without requiring outside...

  14. [27]

    Correct: Evaluate whether the proposed answer is legally correct based on the background_scenario, question, and the context

  15. [28]

    Use Korean for the justifications

    Derivable: Assess whether the proposed answer can be logically and completely derived from the provided context alone, without requiring any unstated assumptions. Use Korean for the justifications. — Input: background_scenario: {background_scenario} question: {question} answer...

  16. [29]

    Reasoning:

    “Reasoning:” – List each logical step you took to arrive at the conclusion. – Use numbered or bulleted items, each beginning with a brief statement of fact or rule. – Make the chain of thought clear and concise

  17. [30]

    reasons for appeal

    “Answer:” – Provide a single, direct, definitive answer to the question in one or two sentences. – Do not include any additional reasoning here—just the conclusion. <Example> Question: Person A was sentenced to six months in prison in the first instance for assault. Both the d...

  18. [2021]

    arXiv preprint arXiv:2110.14168

    Training verifiers to solve math word prob- lems. arXiv preprint arXiv:2110.14168. Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E Ho. 2024. Large legal fictions: Profiling le- gal hallucinations in large language models. Journal of Legal Analysis, 16(1):64–93. David Fr...

    2024 arXiv

  19. [2023]

    NPJ digital medicine, 6(1):210

    A study of generative large language model for medical research and healthcare. NPJ digital medicine, 6(1):210. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language mod- els. In Findin...

    2023

  20. [2024]

    arXiv preprint arXiv:2402.11548

    Kmmlu: Measuring massive multitask lan- guage understanding in korean. arXiv preprint arXiv:2402.11548. Zhongxiang Sun. 2023. A short survey of viewing large language models in legal aspect. arXiv preprint arXiv:2303.09136. Qwen Team. 2025. Qwen3. Dietrich Trautmann, Natalia O...

    2023 arXiv

  21. [2025]

    What is the definition of a car?

    A survey of large language models. Preprint, arXiv:2303.18223. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-b...

    2023 arXiv

  22. [9474]

    Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, et al

    Curran Associates, Inc. Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, et al. 2024a. Legalagentbench: Evalu- ating llm agents in legal domain. arXiv preprint arXiv:2412.17259. Haitao Li, You Chen, Qingyao Ai, Y...

    2004 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.