{"id":"a6ac3dba-38ea-4b66-9683-0876c09b6f8a","arxiv_id":"2505.02172","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Zero-shot GPT-4o and other LLMs match fine-tuned legal models on CaseHOLD, performance scales with model size, and results persist after case names are anonymized.","lead":"This paper tests eight large language models on the CaseHOLD legal benchmark and finds that zero-shot GPT-4o reaches a macro F1 of 0.744, competing with fine-tuned legal models. It also introduces a citation anonymization test to check whether the strong results come from memorizing case names.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The citation anonymization control replaces only case names and citations, leaving ~90% of each prompt and all answer choices unchanged, so it cannot rule out passage-level or answer-option memorization; the 'not rote memorization' claim needs a stronger control.","rationale":"The reader's weakest assumption identified the same general risk: memorization might not be mediated by citations, and semantic preservation by GPT-4o-mini is unvalidated. I agree with that assessment and add a sharper mechanism: the answer choices themselves are not anonymized, and 90% of the prompt's surface tokens survive the intervention, so the control cannot separate reasoning from either passage-level memorization or shallow lexical matching to the correct holding. This is the most load-bearing concern because the paper's novel contribution and its abstract-level claim of 'not due to rote memorization' depend entirely on the validity of this control. The measured F1 scores on the original CaseHOLD test set remain useful empirical findings, and the paper explicitly labels the test inconclusive, so rejection would be too harsh. However, the current evidence does not support the unqualified no-memorization claim in the abstract; a conditional acceptance requiring the paraphrase control and/or explicit validation of semantic preservation is appropriate. Minor reporting inconsistencies (e.g., GPT-4o F1 stated as 0.742 in the introduction and 0.744 elsewhere) are secondary and do not change this assessment.","tokens_in":8859,"tokens_out":6670,"duration_ms":93949,"concrete_test":"Run a full-paraphrase control on a random subset of at least 300 CaseHOLD items: have a separate LLM (or human annotators) rewrite each citing prompt so that both the citations and the non-citation legal content are paraphrased while preserving legal meaning, and also paraphrase the five answer options to remove verbatim overlap with the original holdings. Then apply the same zero-shot protocol. If macro F1 remains near 0.73, the no-memorization conclusion is supported; if F1 collapses toward chance, the original anonymization test was too weak because the unchanged 90% of the prompt was carrying the information.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central no-memorization inference in Section 3 rests on the assumption that 'rote memorization' of CaseHOLD would be mediated by the case names, years, courts, and page numbers that Prompt 6 replaces. That assumption is too narrow. CaseHOLD is a multiple-choice task, and the five candidate holdings are not anonymized; a model that memorized the test item, the correct holding sentence, or a lexical mapping from key legal phrases to answer options would be completely unaffected by citation changes. The paper reports a median Levenshtein distance of 91 edits, changing roughly 10% of the prompt, meaning 90% of the surface tokens—including distinctive legal content words—remain identical to the training-time text. A model that memorized whole passages could still recognize the prompt and retrieve the memorized answer. The test also depends on GPT-4o-mini preserving semantic difficulty, which is asserted but not validated; if the rewrite makes questions easier or harder, the aggregate F1 comparison (0.744 to 0.728) is not diagnostic. The paper itself concedes the test is 'not conclusive,' yet the abstract states the results are 'suggesting the performance is not due to rote memorization,' which overstates what the control can show.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates eight zero-shot large language models (roughly 3B to 90B+ parameters) on the CaseHOLD legal holding identification benchmark. It reports macro F1 scores, with GPT-4o at 0.744 and Amazon Nova Pro at 0.720, and argues these are competitive with published fine-tuned legal models. The paper also observes within-family scaling of performance with model size and introduces a citation anonymization procedure that replaces case names and citations with fictitious equivalents; under this perturbation the macro F1 remains 0.728, which the authors interpret as evidence against rote memorization. Additional analyses cover answer agreement across models and question difficulty measured by the number of unique model choices.","tokens_in":9163,"tokens_out":5071,"duration_ms":66181,"significance":"If the core measurements are correct, the paper provides a useful reference point for legal NLP: a clean zero-shot evaluation over the full CaseHOLD test set with temperature 0, a transparent regex parsing scheme, a documented random tie-break for the under-1% unparseable responses, and public code and data. The comparison against published fine-tuned models is a valuable contribution, and the scaling observation across three model families is suggestive. The main weakness is the memorization claim: the anonymization test is too narrow to support the abstract's statement that performance is 'not due to rote memorization.' The paper's strongest contribution is the systematic benchmark measurement, not the no-memorization conclusion, which needs to be either substantially strengthened or appropriately qualified.","major_comments":[{"comment":"The citation anonymization test in Section 3 cannot support the abstract's claim that performance is 'not due to rote memorization.' The intervention replaces only case names, years, courts, and page numbers, affecting roughly 10% of the prompt by the paper's own median Levenshtein estimate of 91 edits, while leaving the other 90% of the context and, critically, all five answer choices untouched. A model that memorized the test item, the correct holding sentence, or a lexical mapping from legal content words to an answer option would be completely unaffected. The drop from 0.744 to 0.728 is therefore not diagnostic of memorization. The paper itself concedes the test is 'not conclusive' (Section 3), so the abstract and Key Findings should be softened, or a stronger control (e.g., paraphrasing the full prompt and answer choices, or evaluating on a non-public set of holdings) should be added.","section":"3 (Prompt 6, Figure 7)"},{"comment":"Prompt 6 uses GPT-4o-mini to generate the anonymized prompts, and the paper asserts that these rewrites preserve the semantic content and difficulty of the originals without validating this assumption. Since the same class of models is being evaluated, this introduces a self-referential component: any systematic simplification or obfuscation by the anonymizer changes the difficulty in a way that the aggregate F1 comparison cannot detect. The paper should either validate difficulty preservation (e.g., human annotation, a second independent anonymizer, or per-item difficulty checks) or explicitly restrict the claim to 'robust to citation replacement' rather than 'not rote memorization.'","section":"3 (Prompt 6)"},{"comment":"The scaling claim rests on a single deterministic run per model with no confidence intervals, bootstrapping, or significance testing. With three to four points per family, the observed monotonicity is suggestive but not quantified, and the comparison of GPT-4o (0.744) with published models (e.g., 0.770 in [13]) would be more informative with uncertainty bounds. I recommend adding bootstrap confidence intervals over the 5,314 test items and, if feasible, multiple API repetitions or random seeds, and reporting effect sizes.","section":"2.3, Figure 5"}],"minor_comments":[{"comment":"The reported GPT-4o macro F1 is 0.744 in the Abstract and Section 2.3 but 0.742 in the first bullet of Section 1; please reconcile these numbers.","section":"Abstract, Section 1, Section 2.3"},{"comment":"Model naming is inconsistent (e.g., 'GPT4o' vs. 'GPT-4o' and 'AmazonNovaPro' vs. 'NovaPro'); standardize the names, especially in Figure 5 axis labels and Table 1 captions.","section":"Throughout"},{"comment":"The parsing rule is described as extracting 'ANSWER: X' with regex \\bANSWER:\\s*([A-E]); please clarify whether the letter is case-sensitive and how lower-case answers (e.g., 'a') are handled.","section":"2.2"},{"comment":"The 'changes in red' are difficult to see in black-and-white print; consider using a different annotation (e.g., underline or bold) or a table of changed tokens.","section":"Figure 7"},{"comment":"The accuracy column appears to aggregate across all models; a per-model version, or an explicit note that accuracy is averaged over models, would make the table self-explanatory.","section":"Table 1"},{"comment":"The conclusion says 'suggesting the models are doing more than mere memorization of case names,' which is a more defensible claim than the abstract's 'not due to rote memorization' and should be used consistently throughout the paper.","section":"4"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of ICAIL and the benchmark evaluation is solid and reproducible. The main revision risk is that the authors might only soften the abstract while leaving the weak Section 3 experiment unchanged; the no-memorization claim should either be replaced with a narrower 'robust to citation replacement' claim or supported by a stronger control. No concerns about novelty or disclosure: the anonymization idea is a useful addition even though it is not conclusive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take: this is a solid, small empirical paper that updates CaseHOLD numbers for current LLMs and tries something genuinely new with the citation anonymization control. It deserves a serious referee, but the no-memorization conclusion is weaker than the abstract suggests.\n\nWhat's actually new: zero-shot scores for GPT-4o, Amazon Nova, Llama 3.2 on the full CaseHOLD test set, with clear methodology (temperature 0, regex parsing, random tie-break for under 1% unparseable). The scaling observation across three model families is plausible and consistent with broader trends. The citation anonymization test is a real contribution — no one else has done exactly that on CaseHOLD, and shipping code and data is commendable. The comparison to published fine-tuned models is fair; GPT-4o's 0.744 is competitive with several specialist models.\n\nWhere it softens: the memorization control only swaps case names, years, courts, and page numbers. That leaves roughly 90% of the prompt tokens untouched, including the five answer options. A model that memorized the test item, the correct holding sentence, or a lexical mapping from key phrases to answer choices would sail through the anonymized prompts. The paper acknowledges the test is \"not conclusive\" in Section 3, but the abstract's \"suggesting the performance is not due to rote memorization\" oversells what the control can actually show. To support that claim you'd need a stronger perturbation — e.g., paraphrasing the whole passage or altering the answer options — or a membership-inference-style test. Also, the scaling claim is presented without error bars or significance tests; on a 5,314-item test set, the differences between some model pairs could be noise. Minor inconsistency: the abstract and Figure 5 say GPT-4o at 0.744, while the intro bullet says 0.742.\n\nBottom line: the main empirical results are useful and reproducible, and the anonymization diagnostic is a worthwhile first step even if it doesn't nail the memorization question. I'd send it to peer review with a request to temper the abstract and add uncertainty measures. Not a groundbreaking paper, but a legitimately careful within-subfield contribution.\n\nRecommendation: accept for review, expect revision.","headline":"Useful zero-shot results on CaseHOLD with a clever but limited memorization control; worth a look, but don't lean on the anti-memorization claim.","tokens_in":9612,"tokens_out":1168,"would_cite":true,"duration_ms":16127,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Modern LLMs identify legal holdings as well as specialized legal models without any fine-tuning.","keywords":["Large Language Models","CaseHOLD","legal holdings","zero-shot evaluation","model scaling","memorization","citation anonymization","legal NLP"],"falsifier":"A direct way to test the claim: fully paraphrase the entire prompt text of all 5,314 questions, not just the citations, and see whether macro F1 stays near 0.728; a large drop would show the earlier robustness was tied to the citation-swap procedure rather than to true non-memorization.","tokens_in":8683,"feed_emoji":"⚖️","tokens_out":6298,"duration_ms":70280,"temperature":0.7,"pith_summary":"The paper asks whether general-purpose large language models can do a core legal-reading task—identifying the holding a citation stands for—without any legal training, and whether their scores reflect real understanding or memorized case law. It evaluates eight off-the-shelf models, from 3B to 90B+ parameters, on the full 5,314-question CaseHOLD test set and finds that the best, GPT-4o, reaches a macro F1 of 0.744, outperforming several published models built specifically for legal text. Performance rises with model size within every family tested. To address the memorization worry, the paper builds a citation anonymization test that swaps every case name and citation for a plausible fake; scores stay high (0.728), with 88% of answers unchanged, which the paper reads as evidence against rote memorization. The practical stakes are that legal analytics may no longer require expensive domain-specific training, while benchmark designers gain a reusable check for contamination.","feed_headline":"Zero-shot LLMs match fine-tuned legal models on CaseHOLD","feed_subtitle":"GPT-4o hits 0.744 macro F1 and holds near it when case names are replaced with fakes.","key_machinery":"The carrying mechanism is the citation anonymization test, a two-step procedure: first a language model rewrites each citing passage so that all case names, jurisdictions, years, and page numbers become invented but legally formatted equivalents, while the <HOLDING> tag, punctuation, and legal reasoning stay exactly in place; then the evaluated model receives the rewritten passage under the same zero-shot chain-of-thought prompt and must pick the completion. The test assumes that a model leaning on memorized identifiers should stumble once those identifiers are faked, whereas a model using the legal content should not. The rewrite is deliberately large—a median of 91 Levenshtein edits, roughly 10% of the prompt—so that the change is far beyond the small perturbations known to shift LLM answers.","core_discovery":"The central claim is that a modern general-purpose LLM, prompted zero-shot with chain-of-thought reasoning, identifies legal holdings on CaseHOLD at a level competitive with custom legal models: GPT-4o scores 0.744 macro F1, ahead of the best result reported in the original CaseHOLD paper and of several published specialized legal models, while Nova Pro reaches 0.720. The same capability scales with model size across the Llama, Nova, and GPT-4o families. The paper's second claim is that this performance is not explained by memorization of case names or citations: after a cheaper model rewrites every prompt with fictitious but consistent case names, courts, years, and page numbers—median Levenshtein distance 91 edits, about 10% of the prompt—the macro F1 remains 0.728 and 88% of individual answers are unchanged. The paper presents these results as evidence against rote memorization, while explicitly noting the test is not conclusive.","pith_inferences":["Beyond the paper, if the anonymization robustness generalizes to other legal benchmarks, contamination from publicly available judicial opinions may be less crippling than feared, because the models seem to use the legal reasoning in the prompt rather than the case identifier.","A testable extension would be to apply the same rewriting procedure to the answer choices rather than only the prompt text, since memorized choice wording is another possible shortcut.","Another extension: measure memorization directly by asking models to complete masked passages of well-known cases and checking whether outputs match CaseHOLD test text.","The scaling effect implies that performance on legal holdings may keep improving as frontier models grow, which would change the cost-benefit of building domain-specific legal models."],"forward_implications":["Zero-shot general-purpose LLMs can now reach the published performance bar of custom legal models on CaseHOLD, so a large legal corpus and fine-tuning pipeline is no longer required to reach that benchmark.","Within each model family, larger models score higher, so continued general-purpose scaling should push legal-holding identification higher without legal-specific training.","The citation anonymization test gives legal NLP a reusable way to check whether benchmark scores reflect memorized case details rather than reasoning from the prompt text.","When models disagree on a question, accuracy is lower for every model, so ensemble or mixture-of-expert aggregation is a natural next step for these benchmarks."],"supporting_citations":[{"why":"Supplies the CaseHOLD benchmark itself: the dataset of 5,314 citing prompts and holding options that the paper evaluates all models on.","marker":"[25]"},{"why":"Supplies the published custom legal models whose macro F1 scores (0.717-0.770) serve as the baselines GPT-4o is compared against.","marker":"[13]"},{"why":"Earlier benchmark paper that incorporates CaseHOLD and provides additional published results (0.708-0.754) used for comparison.","marker":"[2]"},{"why":"Provides the chain-of-thought prompting method the paper uses to elicit reasoning before the final answer choice.","marker":"[24]"},{"why":"Documents that even tiny prompt changes can shift LLM outputs, supporting the claim that the anonymization edits are large enough to matter.","marker":"[16]"},{"why":"Establishes the broader memorization risk in general-purpose LLMs that motivates the anonymization experiment.","marker":"[5]"},{"why":"Shows that legal text appears in public training corpora, making test-set contamination a live possibility to rule out.","marker":"[6]"}],"fun_headline_variants":["Zero-shot LLMs rival fine-tuned legal models on CaseHOLD","Scaling boosts LLM accuracy on legal holdings","Fake case names don't break LLM legal reasoning","GPT-4o zero-shot hits 0.744 F1 on legal benchmark","LLMs generalise legal judgment, not memorise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion stands only if rote memorization of the test set would have to pass through the case names and citations that the anonymization replaces; if a model instead memorized whole test passages, the fake citation swap would not erase the trigger, and the paper concedes that its test is not conclusive.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot LLMs rival fine-tuned legal models on CaseHOLD","Scaling boosts LLM accuracy on legal holdings","Fake case names don't break LLM legal reasoning","GPT-4o zero-shot hits 0.744 F1 on legal benchmark","LLMs generalise legal judgment, not memorise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000342,"raw_usage":{"total_tokens":1898,"prompt_tokens":977,"completion_tokens":921,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":835}},"tokens_in":593,"tokens_out":921,"duration_ms":11682,"temperature":1.0,"reasoning_tokens":835,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:58:49.644260+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct way to test the claim: fully paraphrase the entire prompt text of all 5,314 questions, not just the citations, and see whether macro F1 stays near 0.728; a large drop would show the earlier robustness was tied to the citation-swap procedure rather than to true non-memorization.","supporting_citations":[],"review_version":1}