{"id":"dca41d81-b780-4539-80bb-73d3601245da","arxiv_id":"2608.01292","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CrossLex evaluates LLMs on same-fact, cross-jurisdictional legal reasoning with citation grounding, showing that models often answer correctly but fail to bind conclusions and citations to the right legal system.","lead":"CrossLex is a new benchmark of 6,149 legal questions that reuse the same fact patterns across Chinese, Californian, and German law. It tests whether large language models can keep legal conclusions and citations tied to the correct jurisdiction, and finds that many models cannot.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's core empirical claim depends on gold labels being unique and correct for each jurisdiction, but the paper provides no inter-annotator agreement or released data to verify this; the reported 6,149-instance count is also not derivable from the stated 55×7×3 construction.","rationale":"I agree with the reader that the benchmark's validity hinges on the Stage I assumption of functional comparability and single defensible answers. I would make the same concern more pointed: the paper's own numbers do not add up under the stated construction, which weakens confidence that the 6,149 instances were produced by the described professional validation rather than by an undocumented generation or filtering process. The multi-model consensus check (Stage IV) and two-annotator professional review (Stage V) are real evidence of care, but consensus among LLMs and adjudicated review do not establish legal uniqueness; agreement statistics are needed. The strongest claim about T3 is load-bearing because the whole contribution is a diagnostic benchmark: if the gold bindings are noisy, the measured 'binding gap' is not a model property. This is not a disagreement with current consensus; it is an internal verification gap. The appropriate verdict remains conditional: accept only after the data, the instance inventory, and external legal validation are provided.","tokens_in":13471,"tokens_out":9501,"duration_ms":89860,"concrete_test":"Ask the authors to release, at minimum, a construction inventory mapping every one of the 6,149 instances to a fact-group ID, issue ID, task type, and jurisdiction, and to run an external double-annotation study on a stratified random sample of 100 fact groups: two licensed lawyers per jurisdiction independently derive the conclusion and citation for each fact group, blinded to CrossLex gold labels. Compute per-jurisdiction agreement (Cohen's kappa) and check whether any gold conclusion has a plausible legal alternative. If kappa is below about 0.8, or multiple defensible answers appear, then T1/T2/T3 accuracy and the T3 binding gap partly reflect label noise, not model capability. The construction inventory also settles whether 6,149 is derivable from 55×7 fact groups; if it is not, the reported scores need to be recomputed on a clearly defined sample.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that current LLMs 'fail to associate each conclusion with the correct governing jurisdiction' (T3) presupposes that CrossLex's gold jurisdiction-conclusion bindings are both correct and unique. Stage I asserts 'uniqueness guardrails to ensure that each multiple-choice item has exactly one legally defensible answer,' but the paper never reports the inter-annotator agreement from Stage V, never states the qualifications or count of the two legal reviewers per item, and does not release the data. If a nontrivial fraction of gold bindings is contested (e.g., because an issue is not functionally comparable across CN/CA/DE, or because a fact group admits a second defensible answer under one jurisdiction), then T3's exact-match binding accuracy will be depressed by label noise rather than by model confusion. This is not a hypothetical: Stage V lists 'hard-case adjudication' as a review dimension, which implies borderline cases exist; the paper does not say how many were excluded or adjudicated. The same under-specification appears in the arithmetic: 55 issues × 7 fact groups = 385 groups, and Stage III says each group is 'transformed into three task-specific QA formats,' which yields 1,155 instances, not the reported 6,149 (even counting T1 per-jurisdiction gives 1,925). Without a release or a construction table, the experimental numbers cannot be tied to the described pipeline. Since the data are withheld until acceptance, the central empirical claim is currently not independently checkable, and the headline result could be an artifact of gold-label ambiguity or of an undocumented instance pool.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CrossLex, a source-grounded benchmark for cross-jurisdictional legal reasoning across China, California, and Germany. The benchmark is built from 55 functionally comparable legal issues across five domains, expanded into 385 fact groups and 6,149 QA instances, and organized into three tasks: single-jurisdiction judgment (T1), multi-jurisdiction joint judgment (T2), and fine-grained jurisdiction–conclusion binding (T3). A new metric, Grounded Joint (GJ), jointly scores answer correctness and citation grounding. The paper also reports evaluation results for a range of closed- and open-weight LLMs, showing high T1 and T2 accuracy but much lower T3 binding accuracy and citation quality, and presents experiments with multilingual BM25-RAG and LoRA fine-tuning. The central claim is that current LLMs possess substantial single-jurisdiction legal knowledge yet fail to associate legal conclusions with the correct governing jurisdiction when facts are held constant.","tokens_in":13771,"tokens_out":4219,"duration_ms":42106,"significance":"If the benchmark is valid, it fills a real gap: existing legal benchmarks do not align identical fact patterns across jurisdictions, so they cannot directly test jurisdiction–conclusion binding. The three-task progression is a well-designed way to separate single-jurisdiction knowledge from cross-jurisdictional comparison and fine-grained binding. The empirical T2-versus-T3 gap is a falsifiable and potentially impactful finding: it suggests that answer-level accuracy can mask systematic jurisdiction confusion. The paper also ships practical infrastructure decisions (group-aware splits, leakage audits, normalized citation identifiers) that support reproducibility once the data are released. The main empirical results that motivate the benchmark are the low citation-F1 and GJ scores, and the high T2 scores that conceal low T3 binding accuracy; these are plausible but, as detailed below, they rest on unverified label quality and on an unexplained instance count.","major_comments":[{"comment":"There is an unexplained arithmetic mismatch between the stated construction pipeline and the reported instance count. Stage III says each validated fact group is transformed into three task-specific QA formats, and Stage II yields 385 = 55 × 7 fact groups; that gives 385 × 3 = 1,155 instances, not the 6,149 reported in the abstract, Figure 2, and Stage V. Even if T1 is generated separately per jurisdiction (which would give 385 × 5 = 1,925), the count still does not reach 6,149. The paper must provide a construction table that reconciles the per-task and per-jurisdiction instance counts with the reported total; otherwise the denominators for the per-task numbers in Table 2 cannot be verified.","section":"Stage III and Stage V (Figure 2)"},{"comment":"The paper does not report any inter-annotator agreement or adjudication statistics. It states that each item was independently reviewed by two legally trained annotators and that disagreements were resolved through adjudication, but it gives no Cohen's kappa or similar measure, no count of disagreements, no number of items excluded or revised under the 'hard-case adjudication' review dimension, and no release of the data. Because the central claim that T3 binding accuracy is low presupposes that the gold jurisdiction–conclusion bindings are both correct and unique, unquantified label noise could depress T3 scores just as easily as model confusion. The paper should report IAA, adjudication outcomes, exclusion counts, and ideally a sample or release of the data to make this assumption testable.","section":"Stage V: Professional Validation and Final Audit"},{"comment":"The definition of Citation-F1 and its use inside Grounded Joint is under-specified. Eq. (9) computes GJ as the mean over instances of 1[y_hat = y] times F1_cite_n, but the paper never states how a model's free-form citations are segmented, normalized, matched to the gold citation set, or assigned partial credit. It also does not specify whether the instance-level F1 is computed over sets of canonical citation identifiers, how duplicate or extra citations are counted, and what happens when a model cites a jurisdictionally correct rule but with an incomplete authority. All citation-related numbers in Table 2 and Figure 4 depend on this definition, so the paper must make the citation-matching procedure explicit and reproducible.","section":"Evaluation Metrics, Eq. (9)"}],"minor_comments":[{"comment":"The label 'Gold Catition' in the Stage II panel appears to be a typo for 'Gold Citation' and should be corrected.","section":"Figure 2"},{"comment":"The sentence ending '...associate each conclusion with its governing legal.' is grammatically incomplete; 'legal' lacks its noun (e.g., 'legal system' or 'governing jurisdiction').","section":"Introduction, last paragraph before 'To address this gap'"},{"comment":"The column groups for T3 are difficult to read: 'All CN CA DE Citation-F1 GJ' does not clearly separate the accuracy columns from the Citation-F1 and GJ columns, and 'GJ-Avg' is not defined in the table caption. Please clarify the subheadings and define GJ-Avg as the macro-average over T1, T2, and T3 GJ scores.","section":"Table 2"},{"comment":"The paper repeatedly refers to 'aligned, jurisdiction-neutral fact patterns' but does not show a concrete example of the exact same fact text across the three jurisdictions; the liquidated-damages example in Figure 2 is illustrative only. A sample aligned fact group in the main text or appendix would help readers verify the neutrality claim.","section":"Stage II and Figure 2"},{"comment":"The caption says 'unified exact-set scoring with multiple denominators' but the figure does not explain what the denominator is for each metric; please state the scoring rules per task in the caption or the main text.","section":"Figure 4"},{"comment":"The phrase 'authoritative legal sources' is vague in the abstract; the concrete sources (China National Laws Database, California Legislative Information, Bundesministerium der Justiz) appear only in the Figure 2 diagram and should be named in the text as well.","section":"Abstract and Stage I"}],"recommendation":"major_revision","confidential_remarks":"The benchmark addresses a timely and useful evaluation gap, and the three-task design is conceptually strong. However, the unexplained 6,149-instance count, the absence of inter-annotator agreement and adjudication statistics, and the under-specified citation-matching procedure are all load-bearing for the paper's central empirical claim. I would condition acceptance on a concrete construction table, IAA/adjudication details, and a precise citation-F1 definition; a data release with at least a sample would substantially increase confidence. The manuscript's claim that the data are 'released upon acceptance' means reviewers currently cannot independently verify the gold labels, which is a risk for a benchmark paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, well-scoped benchmark with a genuinely useful task design, but it currently hides the details that would let a reader verify its construction numbers. The core finding—models near ceiling on single-jurisdiction accuracy and joint comparisons, then fall apart on jurisdiction-conclusion binding—is believable and worth publishing. The paper just needs to open the hood.\n\nWhat's genuinely new: the aligned-fact setup across CN/CA/DE with three task levels. T1 to T3 gives a clean way to separate legal knowledge from jurisdiction-binding. The Grounded Joint metric is a small but sensible contribution. The authors also did the right things operationally: professional legal review, leakage audit, group-aware splits, deduplication.\n\nNow the soft spots, in order of importance.\n\nFirst, the instance math doesn't work on the face of it. 55 issues × 7 fact groups = 385 groups. If each group becomes exactly one T1, one T2, one T3, that's 1,155 instances, not 6,149. Even if T1 is per-jurisdiction (3 per group), you get 1,925. The paper never explains where 6,149 comes from. Maybe there are multiple questions per group per task, but then Stage III's description is misleading. A construction table mapping groups to counts would clear this up. As is, the experimental numbers float free from the pipeline.\n\nSecond, the gold-label verification is under-specified. Stage V says two legal professionals review each item and adjudicate disputes, but we get no inter-annotator agreement, no count of items that were revised or excluded, and no detail on reviewer qualifications beyond 'legal training.' Since T3's exact-match binding accuracy is the headline result, label noise in those bindings would depress scores directly. That need not be fatal—human review can be good—but you have to show it.\n\nThird, data is withheld until acceptance. That's normal but it does mean the central empirical claim isn't independently checkable right now. A sample or a detailed construction table would help.\n\nOne thing the reader flagged that I disagree with: the claim that Table 1 credits MultiLegalBench with aligned facts. It doesn't—the table marks aligned facts ✗ for MultiLegalBench, consistent with the text. That particular contradiction isn't in the paper.\n\nBottom line: this is a useful benchmark for anyone evaluating LLMs on legal reasoning. The design is thoughtful and the headline result is plausible. It deserves to go to peer review, but the authors should be asked to reconcile the instance count, report IAA or at least adjudication statistics, and release the data or a representative sample before acceptance.","headline":"A useful aligned-fact legal benchmark whose headline claim is plausible, but the construction math doesn't add up and the gold-label verification is under-reported; send to review with requests for transparency.","tokens_in":14299,"tokens_out":4171,"would_cite":false,"duration_ms":34353,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark shows that language models often pair a correct legal conclusion with the wrong jurisdiction.","keywords":["cross-jurisdictional legal reasoning","legal benchmark","citation grounding","jurisdiction binding","large language models","comparative law","Grounded Joint","legal question answering"],"falsifier":"Ask a panel of legal professionals from each of the three jurisdictions, blind to CrossLex's gold labels, to independently answer a sample of the T3 fact groups and cite the governing source; if the experts disagree with the gold binding on a substantial share of divergent items, then the measured binding gap is partly an artifact of contested gold labels rather than a pure model failure.","tokens_in":13294,"feed_emoji":"⚖️","tokens_out":7815,"duration_ms":63788,"temperature":0.7,"pith_summary":"This paper introduces CrossLex, a benchmark built to test whether large language models can keep legal systems apart when the facts stay the same. It aligns 55 legal issues across China, California, and Germany, turning them into 385 fact groups and 6,149 expert-validated question-answer instances with supporting citations. The authors design three tasks: single-jurisdiction judgment, joint cross-jurisdictional comparison, and fine-grained jurisdiction-conclusion binding, plus a Grounded Joint metric that credits an answer only when both the conclusion and its legal-source citations are correct. Across many current models, single-jurisdiction accuracy is high and joint comparison is near ceiling, but jurisdiction binding and citation grounding lag far behind. The paper concludes that evaluating legal AI on answer accuracy alone is misleading, and that jurisdiction-aware, source-grounded assessment should be the standard.","feed_headline":"Legal AI gives right answers under the wrong law","feed_subtitle":"A 6,149-question benchmark across China, California, and Germany shows citation grounding lags far behind accuracy.","key_machinery":"The load-bearing object is the CrossLex benchmark itself: same-fact, jurisdiction-aligned questions with gold answers and gold citations. Its construction chain runs from 55 functionally comparable legal issue schemas, through seven controlled fact groups per issue (385 total), to three task formats (T1 single-jurisdiction, T2 joint CN-CA-DE comparison, T3 jurisdiction-conclusion binding). The measuring instrument is the Grounded Joint metric, defined as $GJ = \\frac{1}{N}\\sum_n \\mathbb{1}[\\hat{y}_n = y_n]\\cdot F1^{\\mathrm{cite}}_n$, where answer correctness is multiplied by citation F1 over normalized legal-source identifiers. The mechanism this exposes is the binding gap: recognizing a plausible rule and knowing which legal system it belongs to are distinct capabilities, and current models possess the first without the second.","core_discovery":"The paper's central claim is that there is a measurable binding gap in current LLMs: given an identical neutral fact pattern, models can often identify plausible legal conclusions for China, California, and Germany, but they frequently attach a conclusion to the wrong governing legal system and cannot supply the correct supporting authority. This is established with three increasingly difficult tasks on the same fact groups, where the hardest task (T3) requires assigning each jurisdiction its correct legal conclusion from a shared candidate pool. Models reach near-ceiling scores on joint comparison yet drop sharply on exact jurisdiction-binding accuracy, and their Grounded Joint scores are consistently lower than answer accuracy. The paper argues this gap is not captured by existing single-jurisdiction or multilingual benchmarks, and that moving beyond answer accuracy toward jurisdiction-aware and source-grounded evaluation is necessary for legal AI.","pith_inferences":["If the binding gap holds beyond these three systems, the same aligned-fact design should expose similar jurisdiction confusion in other pairs of legal systems; the benchmark is a template that can be extended rather than a one-off evaluation.","The separation between T1 competence and T3 binding fault suggests a concrete architectural target: models may need an explicit jurisdiction-attribution component that ties each retrieved rule to its legal system before stating a conclusion.","A testable extension would take the divergent fact groups and ask lawyers from each jurisdiction to write the outcome and citation from memory; comparing those human protocols with model outputs would show whether the gap is a recall problem, a source-grounding problem, or a genuine confusion about legal-system boundaries."],"forward_implications":["Legal AI evaluation should report a source-grounded score like Grounded Joint alongside answer accuracy; an answer without the correct citation no longer counts as fully correct.","Near-ceiling performance on joint comparison can conceal a severe jurisdiction-binding failure, so aggregate accuracy measures are insufficient for legal benchmarks.","Retrieval augmentation alone will not fix cross-jurisdictional reasoning, because a model can retrieve the right statute and still bind the conclusion to the wrong legal system.","CrossLex is usable as supervision: lightweight models fine-tuned on its training set substantially improve citation quality and Grounded Joint scores while preserving answer accuracy.","Future legal benchmarks for multilingual or multi-system law should align identical fact patterns across jurisdictions, not just pool independent single-jurisdiction questions."],"supporting_citations":[{"why":"LexGLUE is the single-jurisdiction English legal benchmark against which CrossLex positions itself as measuring cross-jurisdictional reasoning.","marker":"Chalkidis et al. 2022"},{"why":"LegalBench is the collaborative legal reasoning benchmark that CrossLex contrasts as neither cross-jurisdictional nor citation-grounded.","marker":"Guha et al. 2023"},{"why":"LawBench is the Chinese-oriented legal knowledge benchmark used as an example of single-jurisdiction evaluation.","marker":"Fei et al. 2024"},{"why":"MultiEURLEX is the multilingual document classification dataset used to show that multilingual coverage does not imply aligned cross-jurisdiction facts.","marker":"Chalkidis, Fergadiotis, and Androutsopoulos 2021"},{"why":"LEXTREME is the multilingual and multi-task legal benchmark whose tasks are evaluated in isolation rather than under aligned facts.","marker":"Niklaus et al. 2023"},{"why":"MultiLegalBench is the closest prior multilingual cross-jurisdiction benchmark that CrossLex says lacks fully aligned facts and citation grounding.","marker":"Ovcharov 2026"}],"fun_headline_variants":["Legal AI knows the law but not which law rules","Same facts, three laws: AI picks the wrong one","AI legal answers are right but cite the wrong jurisdiction","New benchmark reveals AI's legal citation gap across borders"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark is valid only if the 55 issues are genuinely functionally comparable across China, California, and Germany, and only if each fact group has exactly one legally defensible gold conclusion and citation set per jurisdiction; the paper reports professional review but does not provide inter-annotator agreement data or a detailed legal-theory defense of comparability.","fun_headline_variants_meta":{"raw":{"variants":["Legal AI knows the law but not which law rules","Same facts, three laws: AI picks the wrong one","AI legal answers are right but cite the wrong jurisdiction","New benchmark reveals AI's legal citation gap across borders"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1571,"prompt_tokens":979,"completion_tokens":592,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":528}},"tokens_in":595,"tokens_out":592,"duration_ms":5730,"temperature":1.0,"reasoning_tokens":528,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:08:06.108637+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask a panel of legal professionals from each of the three jurisdictions, blind to CrossLex's gold labels, to independently answer a sample of the T3 fact groups and cite the governing source; if the experts disagree with the gold binding on a substantial share of divergent items, then the measured binding gap is partly an artifact of contested gold labels rather than a pure model failure.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LegalBench is the collaborative legal reasoning benchmark that CrossLex contrasts as neither cross-jurisdictional nor citation-grounded."},{"cited_title":"Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages =","cited_arxiv_id":null,"evidence_quote":"LawBench is the Chinese-oriented legal knowledge benchmark used as an example of single-jurisdiction evaluation."},{"cited_title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages =","cited_arxiv_id":null,"evidence_quote":"MultiEURLEX is the multilingual document classification dataset used to show that multilingual coverage does not imply aligned cross-jurisdiction facts."},{"cited_title":"Findings of the Association for Computational Linguistics: EMNLP 2023 , pages =","cited_arxiv_id":null,"evidence_quote":"LEXTREME is the multilingual and multi-task legal benchmark whose tasks are evaluated in isolation rather than under aligned facts."}],"review_version":2}