Pith. sign in

REVIEW 3 major objections 5 minor 68 references

LexKairos: Benchmarking Legal Temporal Capabilities in LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LexKairos, a new Chinese legal benchmark, shows that even the strongest LLMs struggle to recall exact statutory dates and to reason over procedural time limits.

desk verdict LexKairos fills a real gap in legal AI evaluation, but the measurement validity of Tasks 2 and 3 hinges on an under-documented gold-label pipeline and a prompt/label information mismatch that needs to be addressed before the scores can be trusted. read the letter →

arxiv 2608.09106 v1 pith:HWQHEF2X submitted 2026-08-10 cs.CL

Chenyang Li , Zejia Feng , Yuqin Huang , Yuxiao Ye , Huiyuan Xie This is my paper · ORCID
classification cs.CL
keywords legaltemporalreasoningbenchmarkChineselawLLMevaluationstatutoryknowledgecasechronologyproceduraltimelimits
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces LexKairos, a benchmark for measuring how well large language models handle time in Chinese legal settings. It organizes legal time into three dimensions—statutory temporal knowledge, case temporal modeling, and statute-case temporal reasoning—and builds nine subtasks from real judicial cases and statutes. The paper evaluates eight LLMs under vanilla, chain-of-thought, and thinking settings, and reports that even the strongest model, Gemini-3-Flash with thinking mode, reaches only 84.57 overall, with clear weaknesses on recalling exact effective dates and reasoning over procedural deadlines. The intended contribution is a reliable, multi-dimensional test for an underexplored capability, plus evidence that current models are not dependable for legal tasks where 'when' matters.

What carries the argument

The central object is the LexKairos benchmark itself, with its capability taxonomy: three primary dimensions (Statutory Temporal Knowledge, Case Temporal Modeling, Statute-Case Temporal Reasoning) mapped to nine subtasks. The conceptual machinery is the 'Legal Time' distinction between normative time (statute versions, effective periods, deadlines) and factual time (event anchoring, timeline construction, period calculation), together with Allen's interval relations used to define event-relation labels. The benchmark's curation pipeline—rule-based synthesis from statutory metadata, SQL-based QA generation over a manually audited event database, and expert-reviewed procedural time-limit cases—converts the taxonomy into measurable scores.

What would settle it

An independent re-annotation study: take a random sample of the 293-instance temporal database and the 289 procedural time-limit instances, have legal experts annotate them from scratch without seeing the original labels, and measure agreement. Low inter-annotator agreement, or systematic disagreement with the published labels, would show the benchmark does not measure what it claims.

Watch

Extended reading notes

Core claim

The central claim is that legal temporal capability is a distinct, measurable competence that existing legal AI benchmarks overlook, and that LexKairos operationalizes it. The paper defines 'Legal Time' at two levels—statutes (effective dates, versions, validity) and cases (event timestamps, timelines, durations)—and builds nine subtasks to test recall, modeling, and joint reasoning. Evaluating eight models, it finds that case chronology tasks are comparatively easier, statutory metadata recall and procedural time-limit reasoning remain hard, and thinking mode helps but does not fix a persistent precision deficit in version-identifier recall. The paper's discovery, in its own terms, is a capability taxonomy and a measurement showing current LLMs are not temporally reliable in law.

Load-bearing premise

The benchmark's measurements rest on the accuracy of the gold labels for case-chronology and procedural-time-limit tasks, which were produced by an automated extraction system and then reviewed by experts; if that review missed systematic temporal errors, every reported score on those tasks could be wrong.

Editorial extensions

If this is right

  • If LexKairos is a valid measure, legal temporal capability should be treated as a separate axis in legal LLM evaluation, distinct from semantic and logical reasoning.
  • The reported scores imply that legal assistants built on current LLMs should not be trusted for version identification or peremptory-period decisions without external verification.
  • Thinking mode consistently improves scores but does not remove the version-identifier precision deficit, so the bottleneck is knowledge recall, not reasoning effort.
  • Task-specific legal prompting achieves comparable performance on procedural time-limit reasoning with up to 3.4x shorter outputs, suggesting structured prompts are a token-efficient alternative to unconstrained thinking mode.
  • The benchmark's day-level granularity restriction means multi-scale temporal reasoning (mixed month/day, sub-day) remains untested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to test whether the reported hierarchy—case chronology easiest, statutory metadata hardest—holds in other legal systems, such as US or EU statutes with different revision conventions.
  • The 'tag only' error asymmetry between omission-prone and hallucination-prone models suggests calibration strategies tailored to each model family could be more effective than a single prompting recipe.
  • One could use LexKairos's structured task taxonomy as a training curriculum: models that first master event anchoring and timeline construction might then improve on statute-case reasoning.
  • If the gold labels inherit biases from the automated extraction model, the benchmark's scores could partly reflect that model's temporal blind spots; independent re-annotation would clarify.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript introduces LexKairos, a benchmark for evaluating the temporal capabilities of LLMs in the Chinese legal domain, organized into three dimensions (statutory temporal knowledge, case temporal modeling, and statute-case temporal reasoning) with nine sub-tasks. The data are constructed from statutes and real civil judgments, and the authors evaluate eight LLMs under vanilla, CoT, and thinking settings, reporting that Gemini-3-Flash (Thinking) achieves the highest overall score (84.57) while all models show marked weaknesses in statutory metadata recall and procedural time-limit reasoning. The paper also presents a fine-grained error analysis for Effective Version Identification and a token-efficiency study of task-specific legal prompting on the procedural tasks.

Significance. If the benchmark's measurements are valid, LexKairos addresses a genuine gap: existing legal AI benchmarks focus on semantic and logical reasoning while largely ignoring the temporal dimension, which is objectively verifiable and practically important. The task taxonomy is clear, the evaluation protocol is transparent (temperature 0, 8,000-token output limit, explicit prompts in Appendix B), Task 1 is generated by a rule-based pipeline with manual filtering, and the public release of data and code supports reproducibility. The fine-grained error decomposition for Task 1-2 is a useful contribution, and the central empirical finding—that strong general LLMs remain unreliable on precise statutory metadata and procedural deadlines—is plausible and interesting. However, the benchmark's validity hinges on the gold-label construction for Tasks 2 and 3; the paper currently does not supply enough evidence that those labels are correct, complete, and derivable from the prompts alone.

major comments (3)
  1. [Section 3.3 / Appendix A / Table 2] The gold labels for Task 2 (the 293-instance structured database) and Task 3 (180 and 109 instances) are produced by DeepSeek-V3.2 extraction followed by manual audit or expert review, but the paper reports no inter-annotator agreement, no number or qualifications of the auditors/experts, and no audit protocol. Because every Task 2 and Task 3 score in Table 2 is a measurement against these labels, this is load-bearing. The Limitations section asserts expert manual verification for every test instance, but without measurable reliability evidence this remains an assertion. The authors should report Cohen's kappa or equivalent on a re-annotated sample, describe the audit procedure, and address the risk that systematic extraction bias from DeepSeek-V3.2 could shift cross-model comparisons, since DeepSeek-R1 and DeepSeek-V4 are among the evaluated models.
  2. [Appendix B, Tables 11-14 / Section 3.3] The task prompts present abridged case facts (indicated by ellipses), while Appendix A states that Task 3 gold answers were audited and refined by legal experts using the full source judgments. If gold annotations used information absent from the prompt, low model scores reflect incomplete input rather than deficient temporal reasoning; conversely, if the gold is derivable from the prompt alone, the paper should demonstrate this. As written, I could not verify that every gold answer is derivable from the prompt alone. The authors should either include the full relevant fact text in the released benchmark or provide a derivability check, such as an annotation sample answered solely from the prompt, because this is necessary for the validity of Tasks 2 and 3.
  3. [Section 4.3 / Table 2] No confidence intervals or statistical tests are reported, and Task 3 has only 180 and 109 instances. Headline claims such as "Gemini-3-Flash achieves the strongest overall performance" and the cross-model comparisons for Task 3 rest on point estimates whose sampling variability is unknown; with n=109, a difference of a few F1 points can easily be within chance. The paper should add bootstrap confidence intervals or exact tests, at least for Task 3 and for the overall ranking, so that the reader can distinguish reliable differences from noise.
minor comments (5)
  1. [Table 2] The "Overall" column is not defined; the paper should state whether it is an unweighted mean of the nine sub-task scores and whether sub-tasks with different instance counts are weighted.
  2. [Section 4.4 / Figure 3 and Figure 4] The text says "As shown in Figure 4" when discussing the three-way error decomposition, but the figure captions indicate that Figure 3 is the error-decomposition figure and Figure 4 is the sub-category distribution; this cross-reference should be corrected.
  3. [Section 3.2.2 / Table 6] The text states that Task 2-1 uses three Allen interval relations (after, meet, during), but the option set in Table 6 includes a fourth relation (Event B strictly occurs before Event A); the description should be updated to match the actual option set.
  4. [Table 1 / Appendix A] The relation between the 293 verified instances in Appendix A and the much larger Task 2 volumes (450, 550, 500, 594) should be clarified, presumably multiple SQL-generated questions per case; this is currently left implicit.
  5. [Table 1] Task 2-4 is listed as SLC & MLC with a single data volume of 594; the paper should provide the split between the single-label and multi-label formats.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LexKairos is an external measurement benchmark with rule-based Task 1 labels, audited LLM-assisted Task 2/3 labels, and no fitted parameter or self-citation chain that forces the reported results.

full rationale

LexKairos is a benchmark-construction and evaluation paper, not a derivation. Its central claims are empirical measurements: task scores and error decompositions obtained by prompting fixed LLMs with fixed prompts and comparing against gold labels. Task 1 labels are synthesized by a rule-based pipeline from statutory metadata and manually filtered; Task 2 and Task 3 labels are LLM-assisted candidate extractions followed by manual audit or expert review, with the source material being public judicial decisions and statutes. Nothing in the paper fits a parameter to a subset of the data and then re-predicts the same quantities, and no result is defined in terms of itself. The use of DeepSeek-V3.2 during annotation and the later evaluation of DeepSeek-R1/V4 could in principle introduce curation bias, but the audit step and the external statute/case sources break any by-construction equivalence: the reported scores could have come out differently and are not algebraically forced by the curation pipeline. The paper's self-citations (e.g., LexChain, LegalOne) are contextual references and are not load-bearing for the benchmark's measurement claims; no uniqueness theorem or prior derivation is invoked to preclude alternative conclusions. Therefore, under the specified circularity criteria, the paper exhibits no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark introduces no fitted numerical parameters and no new theoretical entities. The reliability of the central empirical claim rests entirely on the accuracy of the gold labels and source metadata, which are domain assumptions listed above.

assumptions (4)
  • domain assumption Allen's interval relations (after, meet, during) are sufficient to classify all temporal relations in Task 2-1.
    Section 3.2.2 restricts Event Relation Discerning to three Allen relations; if a case contains relations outside this set, the options may be incomplete.
  • domain assumption Statutory metadata used to build Task 1 (effective dates, revision history, validity status) is accurate, complete, and legally correct.
    Appendix A states metadata was collected from public sources and filtered manually, but no primary source citations or cross-validation against an official legal database are provided.
  • domain assumption The event extraction and temporal anchoring in Task 2, performed by DeepSeek-V3.2 and then manually audited, are correct and deterministic.
    Appendix A reports a 'meticulous manual audit' of 293 instances but provides no inter-annotator agreement, no second reviewer, and no way to assess audit error.
  • domain assumption The legal expert reviews for Task 3 produce gold labels that are legally sound, mutually consistent, and reproducible.
    Section 3.3 says expert legal review was used for every test instance, but the paper does not state the number of experts, their qualifications, or any agreement measure, so label reliability is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LexKairos: Benchmarking Legal Temporal Capabilities in LLMs." pith.science (2026). https://pith.science/paper/HWQHEF2X

@misc{pith2026260809106,
  author       = {Pith},
  title        = {Pith review of: LexKairos: Benchmarking Legal Temporal Capabilities in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HWQHEF2X}},
  note         = {Machine review of arXiv:2608.09106}
}
read the original abstract

Large language models (LLMs) have demonstrated strong performance across a wide range of legal tasks. In legal practice, time is a critical concept that governs the validity of statutes, the progression of legal cases, and the enforcement of procedural deadlines. However, legal temporal capabilities remain underexplored in existing legal AI benchmarks. To address this gap, we propose LexKairos, a comprehensive benchmark for evaluating the temporal capabilities of LLMs in the Chinese legal context across three dimensions: statutory temporal knowledge, case temporal modeling, and statute-case temporal reasoning. LexKairos comprises nine sub-tasks drawn from real-world Chinese judicial cases and statutes. We conduct systematic evaluations of eight LLMs under multiple inference settings, including vanilla, Chain-of-Thought (CoT), and thinking modes. Our results show that Gemini-3-Flash achieves the strongest overall performance, yet even the best-performing model exhibits notable limitations on tasks demanding precise time-sensitive statutory metadata recall or complex reasoning in time limits, indicating that legal temporal knowledge and reasoning remain open challenges for current LLMs. Data and code are available at https://github.com/thunlp/LexKairos.

Figures

Figures reproduced from arXiv: 2608.09106 by the authors.

Figure 1
Figure 1. Task list and capability mapping in LEXKAIROS. The primary tasks are mapped into nine sub-tasks, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Results (vanilla setting) of LLMs evaluated [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Error decomposition for the Effective Version [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Sub-category distribution within ‘tag only’ [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 44 canonical work pages

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    1983

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  5. [5]

    Dan Gusfield , title =. 1997

    1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  7. [7]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  8. [9]

    The CLC - UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunal

    Xie, Huiyuan and Steffek, Felix and De Faria, Joana and Carter, Christine and Rutherford, Jonathan. The CLC - UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunal. Proceedings of the Natural Legal Language Processing Workshop 2024. 2024

    work page 2024

Show all 68 references
  1. [10]

    ACM Trans

    Su, Weihang and Ai, Qingyao and Wu, Yueyue and Xie, Anzhe and Wang, Changyue and Ma, Yixiao and Li, Haitao and Wu, Zhijing and Liu, Yiqun and Zhang, Min , title =. ACM Trans. Inf. Syst. , month = jul, articleno =. 2025 , issue_date =

    2025

  2. [11]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    LexChain: Modeling Legal Reasoning Chains for Chinese Tort Case Analysis , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  3. [12]

    Proceedings of the nineteenth international conference on artificial intelligence and law , pages=

    Legal syllogism prompting: Teaching large language models for legal judgment prediction , author=. Proceedings of the nineteenth international conference on artificial intelligence and law , pages=

  4. [13]

    Findings of the association for computational linguistics: ACL 2023 , pages=

    Exploring the effectiveness of prompt engineering for legal reasoning tasks , author=. Findings of the association for computational linguistics: ACL 2023 , pages=

    2023

  5. [14]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    Tram: Benchmarking temporal reasoning for large language models , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  6. [15]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Towards benchmarking and improving the temporal reasoning capability of large language models , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  7. [16]

    Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

    Lawbench: Benchmarking legal knowledge of large language models , author=. Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

    2024

  8. [17]

    Advances in Neural Information Processing Systems , volume=

    Lexeval: A comprehensive chinese legal benchmark for evaluating large language models , author=. Advances in Neural Information Processing Systems , volume=

  9. [19]

    L ex T ime: A Benchmark for Temporal Ordering of Legal Events

    Barale, Claire and Barrett, Leslie and Bajaj, Vikram Sunil and Rovatsos, Michael. L ex T ime: A Benchmark for Temporal Ordering of Legal Events. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025

    2025

  10. [20]

    Advances in neural information processing systems , volume=

    Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models , author=. Advances in neural information processing systems , volume=

  11. [21]

    Proceedings of the 31st International conference on computational linguistics , pages=

    LAiW: A Chinese legal large language models benchmark , author=. Proceedings of the 31st International conference on computational linguistics , pages=

  12. [23]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    LexTempus: Enhancing Temporal Generalizability of Legal Language Models Through Dynamic Mixture of Experts , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  13. [24]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Timebench: A comprehensive evaluation of temporal reasoning abilities in large language models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  14. [25]

    The Fourteenth International Conference on Learning Representations , year=

    Harnessing Temporal Databases for Systematic Evaluation of Factual Time-Sensitive Question-Answering in LLMs , author=. The Fourteenth International Conference on Learning Representations , year=

  15. [26]

    Advances in Neural Information Processing Systems , volume=

    Time: A multi-level benchmark for temporal reasoning of llms in real-world scenarios , author=. Advances in Neural Information Processing Systems , volume=

  16. [27]

    2024 IEEE International Conference on Big Data (BigData) , pages=

    Comparative Analysis of LLM-Generated Event Timeline Summarization for Legal Investigations , author=. 2024 IEEE International Conference on Big Data (BigData) , pages=. 2024 , organization=

    2024

  17. [28]

    Proceedings of the 16th International Workshop on Spoken Dialogue System Technology , pages=

    Benchmarking Multilingual Temporal Reasoning in LLMs: The Temporal Reasoning Dataset , author=. Proceedings of the 16th International Workshop on Spoken Dialogue System Technology , pages=

  18. [30]

    CL aw: Benchmarking C hinese Legal Knowledge in Large Language Models - A Fine-grained Corpus and Reasoning Analysis

    Xu, Xinzhe and Zhao, Liang and Xu, Hongshen and Chenchenc. CL aw: Benchmarking C hinese Legal Knowledge in Large Language Models - A Fine-grained Corpus and Reasoning Analysis. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025

    2025

  19. [32]

    2026 , howpublished=

    2026

  20. [33]

    2025 , howpublished =

    Gemini 3 Flash , author =. 2025 , howpublished =

    2025

  21. [34]

    Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Vaughan, Alex and others , journal=

  22. [35]

    Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Zhang, Ruoyu and Ma, Shirong and Bi, Xiao and Zhang, Xiaokang and Yu, Xingkai and Wu, Yu and Wu, Z. F. and Gou, Zhibin and Shao, Zhihong and Li, Zhuoshu and Gao, Zi...

    2025

  23. [38]

    Qwen3.5: Accelerating Productivity with Native Multimodal Agents , url =

  24. [40]

    Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

    Neural legal judgment prediction in English , author=. Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

  25. [41]

    2: Pushing the frontier of open large language models , author=

    Deepseek-v3. 2: Pushing the frontier of open large language models , author=. arXiv preprint arXiv:2512.02556 , year=

    arXiv

  26. [42]

    James F. Allen. 1983. https://doi.org/10.1145/182.358434 Maintaining knowledge about temporal intervals . Commun. ACM, 26(11):832–843

    1983 doi

  27. [43]

    Claire Barale, Leslie Barrett, Vikram Sunil Bajaj, and Michael Rovatsos. 2025. L ex T ime: A benchmark for temporal ordering of legal events. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 5220--5236, Suzhou, China. Association for Computationa...

    2025

  28. [44]

    Yida Cai, Ranjuexiao Hu, Huiyuan Xie, Chenyang Li, Yun Liu, Yuxiao Ye, Zhenghao Liu, Weixing Shen, and Zhiyuan Liu. 2025. Lexrel: Benchmarking legal relation extraction for chinese civil cases. arXiv preprint arXiv:2512.12643

    2025 arXiv

  29. [45]

    Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural legal judgment prediction in english. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 4317--4323

    2019

  30. [46]

    China Judgments Online . 2013. China Judgments Online . https://wenshu.court.gov.cn. (Accessed on July 29, 2025)

    2013

  31. [47]

    Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. 2024. Timebench: A comprehensive evaluation of temporal reasoning abilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Li...

    2024

  32. [48]

    Yongfu Dai, Duanyu Feng, Jimin Huang, Haochen Jia, Qianqian Xie, Yifang Zhang, Weiguang Han, Wei Tian, and Hao Wang. 2025. Laiw: A chinese legal large language models benchmark. In Proceedings of the 31st International conference on computational linguistics, pages 10738--10766

    2025

  33. [49]

    DeepSeek-AI. 2026. DeepSeek-V4 : Towards highly efficient million-token context intelligence. https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf

    2026

  34. [50]

    Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, and 1 others. 2024. Lawbench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 conference on empirical methods in natura...

    2024

  35. [51]

    Google . 2025. Gemini 3 flash. https://blog.google/products-and-platforms/products/gemini/gemini-3/#gemini-3-deep-think

    2025

  36. [52]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The Llama 3 Herd of Models . arXiv preprint arXiv:2407.21783

    2024 arXiv

  37. [53]

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher R \'e , Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, and 1 others. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. A...

    2023

  38. [54]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. https://doi.org/10.1038/s41586-0...

    2025 doi

  39. [55]

    David Herel, Vojtech Bartek, Jiri Jirak, and Tomas Mikolov. 2024. Time awareness in large language models: benchmarking fact recall across time. arXiv preprint arXiv:2409.13338

    2024 arXiv

  40. [56]

    Cong Jiang and Xiaolei Yang. 2023. Legal syllogism prompting: Teaching large language models for legal judgment prediction. In Proceedings of the nineteenth international conference on artificial intelligence and law, pages 417--421

    2023

  41. [57]

    Soyeon Kim, Jindong Wang, Xing Xie, and Steven Euijong Whang. 2026. Harnessing temporal databases for systematic evaluation of factual time-sensitive question-answering in llms. In The Fourteenth International Conference on Learning Representations

    2026

  42. [58]

    Haitao Li, Yifan Chen, Shuo Miao, Qian Dong, Jia Chen, Yiran Hu, Junjie Chen, Minghao Qin, Yueyue Wu, Yujia Zhou, and 1 others. 2026. Legalone: a family of foundation models for reliable legal reasoning. arXiv preprint arXiv:2602.00642

    2026

  43. [59]

    Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang, and Yiqun Liu. 2024. Lexeval: A comprehensive chinese legal benchmark for evaluating large language models. Advances in Neural Information Processing Systems, 37:25061--25094

    2024

  44. [60]

    Wenjin Liu, Haoran Luo, Xin Feng, Xiang Ji, Lijuan Zhou, Rui Mao, Jiapu Wang, Shirui Pan, and Erik Cambria. 2025. Lexgenius: An expert-level benchmark for large language models in legal general intelligence. arXiv preprint arXiv:2512.04578

    2025 arXiv

  45. [61]

    Qiang Mao, Adam Dabrowski, Fusheng Wei, Eric Olson, Robert Neary, Jingchao Yang, Han Qin, and Nathaniel Huber-Fliflet. 2024. Comparative analysis of llm-generated event timeline summarization for legal investigations. In 2024 IEEE International Conference on Big Data (BigData)...

    2024

  46. [62]

    Vittorio Mazzia, Sandro Pollastrini, Davide Bernardi, Chiara Rubagotti, and Daniele Amberti. 2026. Benchmarking multilingual temporal reasoning in llms: The temporal reasoning dataset. In Proceedings of the 16th International Workshop on Spoken Dialogue System Technology, page...

    2026

  47. [63]

    OpenAI . 2026. Introducing GPT‑5.4 . https://openai.com/index/introducing-gpt-5-4/

    2026

  48. [64]

    Qwen Team . 2026. https://qwen.ai/blog?id=qwen3.5 Qwen3.5: Accelerating productivity with native multimodal agents

    2026

  49. [65]

    TYSS Santosh and Tuan-Quang Vuong. 2025. Lextempus: Enhancing temporal generalizability of legal language models through dynamic mixture of experts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6608--6624

    2025

  50. [66]

    Weihang Su, Qingyao Ai, Yueyue Wu, Anzhe Xie, Changyue Wang, Yixiao Ma, Haitao Li, Zhijing Wu, Yiqun Liu, and Min Zhang. 2025. Pre-training for legal case retrieval based on inter-case distinctions. ACM Trans. Inf. Syst., 43(5)

    2025

  51. [67]

    Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023. Towards benchmarking and improving the temporal reasoning capability of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14820--14835

    2023

  52. [68]

    Anka Chandrahas Tummepalli and Preethu Rose Anish. 2026. Lexchronos: An agentic framework for structured event timeline extraction in indian jurisprudence. arXiv preprint arXiv:2603.01651

    2026

  53. [69]

    Yuqing Wang and Yun Zhao. 2024. Tram: Benchmarking temporal reasoning for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6389--6415

    2024

  54. [70]

    Shaohang Wei, Wei Li, Feifan Song, Wen Luo, Tianyi Zhuang, Haochen Tan, Zhijiang Guo, and Houfeng Wang. 2026. Time: A multi-level benchmark for temporal reasoning of llms in real-world scenarios. Advances in Neural Information Processing Systems, 38

    2026

  55. [71]

    Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, and 1 others. 2018. Cail2018: A large-scale legal dataset for judgment prediction. arXiv preprint arXiv:1807.02478

    2018 arXiv

  56. [72]

    Huiyuan Xie, Chenyang Li, Huining Zhu, Chubin Zhang, Yuxiao Ye, Zhenghao Liu, and Zhiyuan Liu. 2026. Lexchain: Modeling legal reasoning chains for chinese tort case analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 35913--35921

    2026

  57. [73]

    Huiyuan Xie, Felix Steffek, Joana De Faria, Christine Carter, and Jonathan Rutherford. 2024. The CLC - UKET dataset: Benchmarking case outcome prediction for the UK employment tribunal. In Proceedings of the Natural Legal Language Processing Workshop 2024, pages 81--96, Miami,...

    2024

  58. [74]

    Xinzhe Xu, Liang Zhao, Hongshen Xu, and Chenchenc. 2025. CL aw: Benchmarking C hinese legal knowledge in large language models - a fine-grained corpus and reasoning analysis. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 12071--12103, Suzhou, ...

    2025

  59. [75]

    Fangyi Yu, Lee Quartey, and Frank Schilder. 2023. Exploring the effectiveness of prompt engineering for legal reasoning tasks. In Findings of the association for computational linguistics: ACL 2023, pages 13582--13596

    2023

  60. [76]

    Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang, and 1 others. 2023. Disc-lawllm: Fine-tuning large language models for intelligent legal services. arXiv preprint arXiv:2309.11325

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.