{"id":"63e3141a-068c-45c7-bbd8-4a36aac1d4d0","arxiv_id":"2412.06272","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper releases the AusLaw Citation Benchmark and shows that instruction-tuned 7B-8B LLMs plus retrieval re-ranking outperform general and law-specific pretrained LLMs for legal citation prediction, reaching about 52% accuracy with a large gap remaining.","lead":"An Australian legal citation benchmark with 55k instances and 18,677 unique citations is introduced to test how well LLMs and retrieval systems can predict which case or law a sentence should cite. The study finds that fine-tuning a small open-source model beats much larger pretrained legal models, and that a trained re-ranker over a retrieval database gives the best accuracy of about 52%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test queries' own LLM-generated Reasons-of-Citation appear to be included in the RoC Aggregations retrieval index, leaking the gold citation into the database and inflating the central Table 1 results.","rationale":"The reader's weakest assumption concerned the accuracy and comprehensiveness of the automatic RoCs. My concern is different and more specific: the RoC Aggregations index appears to contain, for each test query, a description generated from that exact query, creating a form of test-set leakage. This is supported by the text: RoCs are generated for all 82,530 extracted citations (Section 3), the final 55,005-instance set is then split, and the RoC Aggregations are built from all M references per citation (Section 4.2). No holdout step is described. If true, this invalidates the head-to-head comparison of database granularities and the conclusion that hybrid re-ranking is best, while also making the reported 'near 50% gap' an underestimate of true difficulty. The proposed check is straightforward because the data and code are released. I therefore adjust the verdict from the reader's CONDITIONAL to REJECT pending this verification; if the check shows no leakage, the original conditional acceptance would be appropriate.","tokens_in":15534,"tokens_out":6668,"duration_ms":64947,"concrete_test":"Inspect the released code and data (auslawbench.github.io) to check whether each test instance's own generated RoC appears in its gold citation's RoC Aggregation. If it does, rebuild the vector database with that RoC removed (leave-one-out), re-run the retrieval-only, query-expansion, RAG, and re-ranker experiments from Table 1, and compare ACC@1/ACC@5 for RoC Aggregations. A material drop (e.g., >5 points) would confirm that the reported superiority of RoC Aggregations and the 52.1% re-ranker result are partly artifacts of test-set leakage.","verdict_should_be":"REJECT","load_bearing_attack":"The most load-bearing issue is a likely train/test leakage in the RoC Aggregations index. In Section 3, the authors generate a Reason-of-Citation (RoC) for every citation instance from the exact sentence containing the citation, its preceding sentence, and the full text of the cited case. They then aggregate all M RoCs per cited case into the RoC Aggregations database entry (Section 4.2). The test set of 1k citations is extracted from the same 55,005 instances after these RoCs are generated. Consequently, the gold citation's RoC Aggregation for a test query includes the RoC generated from that very test query's text, so the query is semantically present inside the index that retrieval, RAG, and re-ranking operate on. This inflates the RoC Aggregations granularity (Table 1) relative to Full Cases and Catchwords, which contain no query-specific text, and inflates the re-ranker's 52.1% ACC@1. The paper never states that test-instance RoCs were excluded from the index. Because the central claim of optimal database granularity and the re-ranker's superiority depend on these numbers, the evaluation protocol, as written, is compromised.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AusLaw Citation Benchmark, a dataset of 55,005 Australian legal citation instances with 18,677 unique citations, and systematically evaluates LLM-based approaches to legal citation prediction. The methods compared include zero-shot prompting of general and law-specialised LLMs, retrieval-only pipelines with different embeddings and index granularities, instruction fine-tuning, and hybrid retrieval-augmented strategies (query expansion, voting ensembles, RAG, and trained re-rankers). The main empirical claims are that zero-shot LLMs perform near 0% accuracy, instruction-tuned domain models (Cite-SaulLM-7B) reach 51.7% ACC@1, and a re-ranker over an aggregated Reason-of-Citation (RoC) index reaches 52.1% ACC@1, leaving a roughly 50% gap. The paper argues that index granularity and domain-specialised embeddings are critical, and that jurisdictional pre-training on a small Australian corpus can match or beat much larger mixed-law pre-training.","tokens_in":15843,"tokens_out":4004,"duration_ms":41380,"significance":"If the empirical findings are reliable, the benchmark would be a valuable resource: it is, to the authors' knowledge, the first large-scale citation prediction benchmark for Australian law, and the systematic comparison across prompting, fine-tuning, and retrieval is useful for the community. The release of data, code, and trained models is a concrete strength, and the paper makes a clear distinction between open-world and closed-world settings. However, the central quantitative claims depend on the correctness of the evaluation protocol, and the current protocol has a load-bearing leakage problem in the RoC Aggregations index that likely inflates the reported retrieval, RAG, and re-ranker numbers. The fine-tuning results are less directly affected but still lack variance reporting and test/train overlap analysis. The benchmark's value is therefore conditional on a corrected and re-run evaluation.","major_comments":[{"comment":"The RoC Aggregations retrieval index leaks test-query information. Section 3 states that for each of the 55,005 citation instances an LLM generates a RoC from the exact sentence containing the citation, the preceding sentence, and the full text of the cited case. The test set of 1k instances is then extracted from these same 55,005 instances. Section 4.2 defines RoC Aggregations as the concatenation of all RoCs for a cited case. Consequently, for each test instance, the RoC generated from that very test instance's text is included in the database entry of the gold citation. Because retrieval queries are the same (Si-1, Mask(Si)) text from which the test RoC was generated, the gold entry contains a near-duplicate of the query. This inflates all Table 1 results that index RoC Aggregations, including retrieval-only (ACC@1 27.1/29.5), query expansion (up to 35.1), voting ensemble (up to 48.2), RAG (42.2/42.9), and the re-ranker (up to 52.1), relative to Full Cases and Catchwords granularities that contain no test-query text. The paper never states that test-instance RoCs were excluded from the index. This compromises the central claims that RoC Aggregations is the best index granularity and that the re-ranker is the best method; the experiments must be re-run with a sanitised index.","section":"Sec. 3 and Sec. 4.2"},{"comment":"No variance or significance testing is reported for any of the main results. All ACC@1 values in Tables 1 and 2 appear to come from single runs, yet the paper draws fine-grained conclusions such as 'Cite-AusLawLLM-7B slightly outperforms Cite-SaulLM-7B' (Sec. 5.2) when the difference is 52.0 vs. 51.7, and the best re-ranker is 52.1 vs. 51.7 for the fine-tuned model. With a test set of 1,000 instances, a 0.4-point difference is well within typical run-to-run variance for LoRA fine-tuning and for stochastic retrieval. The authors should report multiple seeds with means and standard deviations, or otherwise perform a significance test, before making claims about the ranking of the best methods.","section":"Sec. 5 and Table 1"},{"comment":"The relationship between training and test instances at the cited-case level is not addressed. The paper says 'we extracted 1k citations as test set' without specifying whether these are 1,000 unique citation cases or 1,000 instances, and without stating whether the remaining training instances share cited cases with the test set. If the same case appears in both training and test sets (with different citing contexts), the fine-tuned models may memorise case-specific patterns, inflating their accuracy. If, instead, the test set consists of unique cited cases, the frequency distribution of the test set would be very different from the overall benchmark (since 54% of citations appear only once), which affects the interpretation of Figure 1 and the difficulty of the task. The paper should clarify the sampling procedure and report overlap statistics.","section":"Sec. 3, Appendix A.2"},{"comment":"The quality of the LLM-generated RoCs is not adequately validated. The paper states only that 'we manually checked a subset of the LLM-generated descriptions for quality assurance and optimizing the prompt wording' and gives no size of the subset, no annotation protocol, no inter-annotator agreement, and no error rate. Since RoCs serve three roles — retrieval index entries (Sec. 4.2), supervision targets for instruction-tuned models (Sec. 4.1), and inputs to re-ranker training (Sec. 4.3) — noisy or biased RoCs can change method rankings and the benchmark's difficulty. The authors should report quantitative QA statistics and, ideally, release the QA annotations along with the benchmark.","section":"Sec. 3, Sec. 4.3"}],"minor_comments":[{"comment":"There are typographical errors: 'Lawps-specialised' should be 'law-specialised', 'emeddings' should be 'embeddings', and the phrase 'pre-training pre-training' in Sec. 5.2 is duplicated.","section":"Sec. 5.1"},{"comment":"Some cells in Table 1 omit ACC@5 values without explanation (e.g., the LLM-only rows and the re-ranker rows). Either fill in all values or add a note that ACC@5 is not applicable for those settings.","section":"Table 1"},{"comment":"The comparison between text-embedding-3-large (3072 dimensions) and AusLaw-embedding (384 dimensions) is acknowledged as unfair, but the paper does not report retrieval latency or storage cost. A brief note on computational overhead would make the comparison more actionable.","section":"Sec. 4.2"},{"comment":"Hyperparameters such as learning rate, LoRA rank, and number of epochs are fixed without any ablation or sensitivity analysis. A short discussion of how these were chosen (e.g., a small validation set) would strengthen reproducibility.","section":"Appendix A.3"}],"recommendation":"major_revision","confidential_remarks":"The leakage issue in the RoC Aggregations index is the most serious problem and is fixable by re-running experiments with test-instance RoCs excluded from the index. I would encourage the editor to ask for that re-run, along with variance reporting, before considering publication. The benchmark itself has value and the authors have made a good-faith effort to release resources, but the current numbers are not trustworthy as presented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the note. I read the paper and agree with the stress-test: the RoC Aggregations index almost certainly contains the test instances' own RoCs, so the retrieval, RAG, and re-ranker numbers are inflated. This is the main thing you should know.\n\nThe benchmark itself is a genuine contribution. AusLaw is the first large-scale citation prediction dataset for Australian law, and the release includes the data, code, and trained models. The paper also does a thorough job of comparing zero-shot prompting, retrieval, fine-tuning, and hybrids. The finding that task-specific instruction tuning jumps a near-zero baseline to ~50% is credible and useful, and the result that domain-specific pre-training alone doesn't help is worth reporting. Those points stand even after the leakage is accounted for because the instruction-tuned models never see the RoC Aggregations index.\n\nThe soft spot is load-bearing though. Section 3 describes generating an RoC for every citation instance, then Section 4.2 aggregates all RoCs per cited case into the retrieval index. The 1k test set is extracted from the same 55k instances after RoCs are generated. So for a test query, its own RoC is sitting in the index under the gold citation. That makes the similarity search trivially easier for that granularity. Full Cases and Catchwords don't contain query-specific text, so the granularity comparison in Table 1 is confounded, and every hybrid that retrieves from RoC Aggregations—query expansion, voting ensemble, RAG, re-ranker—gets an unfair boost. The paper never states that test RoCs were removed from the index. That's an omission, not an intentional cheat, but it undermines the central claim that RoC Aggregations is the best index and that the re-ranker's 52.1% is the top result.\n\nThe fix is straightforward: exclude test-instance RoCs from the database and recompute. I'd also like to see variance across runs, since the numbers jump around a lot between settings, and a clearer statement about how many RoCs were manually checked. Those are minor compared with the leakage.\n\nWho should read it? Anyone building legal retrieval or evaluation benchmarks. The dataset is useful; the methodology needs a major revision before the results can be trusted. I'd send it to review—the benchmark deserves referee time—but the authors need to redo the retrieval experiments and be explicit about the index construction.","headline":"Useful benchmark and credible instruction-tuning results, but the retrieval numbers are inflated by test-query RoCs sitting in the RoC Aggregations index.","tokens_in":16268,"tokens_out":3759,"would_cite":false,"duration_ms":36166,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Legal citation prediction is far from solved: the best approaches, instruction-tuned LLMs and re-rankers over reason-of-citation retrieval, reach only about 52% accuracy.","keywords":["legal citation prediction","Australian law","benchmark dataset","instruction tuning","retrieval-augmented generation","reason-of-citation","domain-specific pre-training","re-ranking"],"falsifier":"Take a random sample of, say, 500 test instances, replace the automatically generated RoCs with versions written or verified by practising lawyers, and re-run the retrieval, re-ranker, and instruction-tuned pipelines; if ACC@1 and the ranking of methods shift materially, the reported benchmark difficulty and method comparison are artifacts of RoC quality rather than of the underlying task.","tokens_in":15391,"feed_emoji":"⚖️","tokens_out":9948,"duration_ms":85086,"temperature":0.7,"pith_summary":"This paper sets out to establish how to predict the case citation a legal passage should reference, a task at the heart of precedent-based legal reasoning. It introduces AusLaw Citation Benchmark, a 55,005-instance dataset built from NSW Caselaw with 18,677 unique citations, and benchmarks four families of solutions: prompting general and law-specialised LLMs, retrieval-only pipelines, instruction fine-tuning, and hybrid retrieval-generation methods. The paper's central finding is that neither general nor law-specialised pre-training is remotely sufficient for this fine-grained task: zero-shot prompting gives near-zero accuracy even for a 54B law model, while instruction tuning on task-specific data lifts accuracy to 51.7% (Cite-SaulLM-7B) and 46.2% (Cite-LLaMA-3.1-8B). The best result, 52.1% ACC@1, comes from a trained re-ranker operating over top-5 candidates retrieved from databases indexed by aggregated 'Reasons-of-Citation'. The paper concludes that a gap of nearly 50% remains, making the benchmark a rigorous test-bed for future legal-domain research.","feed_headline":"Best legal-citation models still miss nearly half","feed_subtitle":"A 55k-case Australian benchmark: zero-shot LLMs fail; task fine-tuning and re-ranking reach about 52%.","key_machinery":"The load-bearing object is the Reason-of-Citation (RoC): an LLM-generated prose description, produced for every citation instance from the citing sentence, the preceding sentence, and the full text of the cited case, stating why the case is being cited (e.g., 'The cited case is referenced to establish the standard required to demonstrate a likelihood of injury under the Trees Act'). RoCs do three jobs in the paper: (1) concatenated per citation as 'RoC Aggregations', they form the best database granularity for dense retrieval, outperforming full case text and catchwords; (2) they serve as the generation target for instruction-tuned models, which are trained to emit both the RoC and the citation; and (3) they provide the candidate description input for training the re-rankers that achieve the best reported ACC@1 of 52.1%.","core_discovery":"On the paper's own terms, the discovery is that legal citation prediction is not solved by domain pre-training, general LLM scale, or off-the-shelf retrieval, but is substantially advanced by two ingredients: task-specific instruction tuning and the right index granularity. The authors show that SaulLM-54B, despite being pre-trained on 94B tokens of legal text including the very NSW corpus used here, scores 2% or less when prompted zero-shot, while Claude Sonnet 3.5 with the leaked citation reason reaches only 15.5%. Instruction tuning the same or smaller backbones on the benchmark's training set (using LoRA) produces a jump to 51.7% for Cite-SaulLM-7B and 46.2% for Cite-LLaMA-3.1-8B. On the retrieval side, indexing each citation by concatenating all its LLM-generated Reasons-of-Citation ('RoC Aggregations') outperforms full-case text and catchwords by a wide margin, and a re-ranker trained on RoC-based candidates raises retrieval-only accuracy from 35.1% to 52.1% ACC@1. A companion experiment shows that pre-training on just 0.5B tokens of Australian law (Cite-AusLawLLM-7B, 52.0%) slightly beats pre-training on 94B tokens of mixed-jurisdiction law (Cite-SaulLM-7B, 51.7%), suggesting jurisdiction-specific data is more valuable than sheer legal-corpus size for this task.","pith_inferences":["A sensitivity analysis that re-runs all pipelines with human-verified RoCs would quantify how much of the benchmark's difficulty depends on the automatically generated RoC descriptions.","Extending the RoC approach to statutes and to passages citing multiple authorities would test whether the same components, RoC aggregation and re-ranking, dominate across citation types.","The jurisdiction pre-training result suggests a testable conjecture: a model pre-trained only on a given jurisdiction's caselaw should beat a mixed-jurisdiction model by a larger margin as the target task becomes more jurisdiction-specific.","The near-50% gap suggests citation prediction may require explicit modeling of precedent networks (overruling, distinguishing, following) rather than only local passage-to-case similarity; enriching the benchmark with such relational metadata would probe this."],"forward_implications":["Task-specific instruction tuning, not domain pre-training or model scale, is the main performance lever for fine-grained legal tasks like citation prediction; even a generic 8B model beats a 70B general instruct model and a 54B law model.","Legal retrieval systems should index citations by aggregated reasons-for-citation rather than by full case text or catchwords; the paper reports up to 70% variation in accuracy across index granularities.","Trained re-rankers over top-5 retrieval candidates convert a substantial share of the top-5 accuracy (up to about 60%) into top-1 accuracy, making re-ranking the strongest component of the hybrid pipelines.","Jurisdiction-specific pre-training on a small corpus (0.5B tokens) can match or exceed much larger mixed-jurisdiction legal pre-training, pointing toward a cheaper path for building sovereign legal models.","The 52% ceiling on this benchmark implies that legal citation prediction needs mechanisms beyond semantic similarity and parametric memory, such as reasoning over precedent structure."],"supporting_citations":[{"why":"Supplies SaulLM-7B, the law-specialised backbone that instruction tuning converts into the best single model (Cite-SaulLM-7B, 51.7%).","marker":"(Colombo et al., 2024b)"},{"why":"Supplies SaulLM-54B/141B, the larger law-specialised LLMs whose near-zero zero-shot accuracy establishes that domain pre-training alone is insufficient.","marker":"(Colombo et al., 2024a)"},{"why":"Supplies the LLaMA-3.1-8B and 70B backbones used for prompting, fine-tuning, and the Australian pre-training experiments.","marker":"(Grattafiori et al., 2024)"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method used for all instruction-tuned models and re-rankers.","marker":"(Hu et al., 2022)"},{"why":"Provides GPT-4o and GPT-4o-mini, the general LLMs used for prompting, RAG re-ranking, RoC generation, and reason merging.","marker":"(Achiam et al., 2023)"}],"fun_headline_variants":["Fine-tuning and re-ranking top legal citation benchmark at 52%","Zero-shot LLMs score near zero on legal citation benchmark","Instruction tuning beats domain pre-training for legal citations","Re-ranking retrieval matches fine-tuning on legal citations","Australian legal citation benchmark: best models still miss half"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire pipeline leans on the automatically generated Reason-of-Citation descriptions being accurate and complete enough to serve both as the retrieval index and as the supervision signal; only a subset of these descriptions was manually checked.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning and re-ranking top legal citation benchmark at 52%","Zero-shot LLMs score near zero on legal citation benchmark","Instruction tuning beats domain pre-training for legal citations","Re-ranking retrieval matches fine-tuning on legal citations","Australian legal citation benchmark: best models still miss half"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000996,"raw_usage":{"total_tokens":4308,"prompt_tokens":1128,"completion_tokens":3180,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":744,"completion_tokens_details":{"reasoning_tokens":3102}},"tokens_in":744,"tokens_out":3180,"duration_ms":23059,"temperature":1.0,"reasoning_tokens":3102,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:49:59.525312+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 500 test instances, replace the automatically generated RoCs with versions written or verified by practising lawyers, and re-run the retrieval, re-ranker, and instruction-tuned pipelines; if ACC@1 and the ranking of methods shift materially, the reported benchmark difficulty and method comparison are artifacts of RoC quality rather than of the underlying task.","supporting_citations":[],"review_version":1}