{"id":"8deea5f7-5735-49b8-a3d1-740d75c89b72","arxiv_id":"2509.06356","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"PL-CA applies parametric RAG with LoRA to Chinese legal tasks and presents a 2,580-instance expert-annotated benchmark, claiming improved performance and lower context overhead than vanilla RAG.","lead":"PL-CA stores legal knowledge as small trainable modules inside a language model instead of pasting long legal documents into the prompt. The authors also introduce a new expert-annotated Chinese legal benchmark and report results that beat conventional retrieval-augmented generation on several tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central overhead-reduction claim is unmeasured and likely false: the online stage trains a LoRA adapter per query, which is more expensive than long-context inference, yet no runtime or cost data are reported.","rationale":"The reader's weakest assumption identifies exactly the load-bearing soft spot in the central claim. The abstract promises overhead reduction, but the method's online stage performs per-query LoRA optimization, which is a training procedure, not a cheap parameter merge. The paper reports no runtime or cost measurements, so the claim is purely speculative. This is not a matter of disagreeing with the P-RAG paradigm; it is a missing measurement that is directly required by the stated contribution. I agree with the reader's assessment. I also note the low BM25 Recall@5 for articles (3.6%, Table 3) compounds the concern: the online stage may train on a retrieved case whose statutes are not the test case's relevant articles, making the cost-benefit even worse. The absence of error bars and the promised-but-missing code/dataset are additional issues, but the overhead claim is the most central. If the authors supplied a runtime table, the paper could become conditionally acceptable; as it stands, the central claim is unsupported.","tokens_in":11867,"tokens_out":4436,"duration_ms":36893,"concrete_test":"Measure end-to-end wall-clock time and peak GPU memory for 100 Legal-CA test queries under the full PL-CA pipeline (online retrieval, LoRA training on the top-1 retrieved case, merge, inference) and under vanilla RAG with the same retrieved documents. If the mean per-query PL-CA time exceeds vanilla RAG inference time, the abstract's overhead-reduction claim is falsified. Report both numbers in a table.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that PL-CA 'reduces the overhead associated with excessively long contexts while maintaining competitive performance.' The performance half is at best only partially supported (no error bars, no significance tests). The overhead half is not supported at all. The online P-RAG stage (§Online P-RAG) requires, for each test query, retrieving the top-1 case, reformatting it, and 'the parameterized update is carried out according to Equation 2' — i.e., optimizing LoRA parameters with a next-token prediction loss on the retrieved case. This is per-query gradient training through the LLM. No wall-clock time, FLOPs, GPU memory, or monetary cost is reported anywhere in the paper. A single forward pass through a long context is typically far cheaper than a gradient update on a small batch, even with rank-2 LoRA, because the backward pass and optimizer state dominate. The paper's own Table 3 shows BM25 Recall@5 for legal articles is 3.6%, meaning the retrieved top-5 cases usually lack the relevant statutes; the online stage thus spends expensive per-query training to inject frequently irrelevant knowledge. The offline one-time training cost is also not amortized in the per-query comparison. Without any cost measurement, the headline motivation for P-RAG collapses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PL-CA, a parametric retrieval-augmented generation framework for Chinese legal tasks. It combines an offline stage (LoRA fine-tuning on an augmented corpus of 1,990 legal cases) with an online stage (per-query retrieval of a similar case and LoRA fine-tuning on that retrieved case before answering) to 'encode' legal knowledge into model parameters instead of placing it in the context. The authors also introduce Legal-CA, a small expert-annotated multi-task benchmark (590 test / 1,990 training instances), and claim that PL-CA reduces the overhead of long contexts while matching or exceeding conventional RAG, and even surpasses GPT-4o on several legal subtasks. The paper's central efficiency claim is never measured, the online stage has a serious answer-leakage problem, and the reported results lack error bars and significance testing.","tokens_in":12293,"tokens_out":6169,"duration_ms":53842,"significance":"If validated, the idea of parameterizing retrieved legal knowledge could be of interest to the legal NLP community, and the Legal-CA benchmark would be a resource for multi-task evaluation. The authors have made a genuine effort to construct an expert-annotated Chinese legal dataset with structured components (fact, reason, articles, judgment). However, the current manuscript does not support its headline claims: the overhead-reduction claim is not backed by any efficiency measurement and is, as described, likely false because the online stage performs gradient-based LoRA training per query. The performance improvements are confounded by the online stage training on the retrieved case's known judgment and articles. The benchmark is not publicly accessible (no appendix or URL), so its community value cannot currently be realized. The paper is not in a publishable state.","major_comments":[{"comment":"The abstract claims that PL-CA 'reduces the overhead associated with excessively long contexts,' but no runtime, FLOPs, GPU memory, or monetary cost is reported anywhere. The online stage trains a LoRA adapter on the retrieved case using next-token prediction (Eq. (2)), which requires at least one forward and backward pass through the LLM for each test query. This is generally more expensive than a single forward pass over a long context. Without any efficiency measurement, the central overhead-reduction claim is unsubstantiated and, as described, very likely false.","section":"Online P-RAG, Eq. (2)-(3)"},{"comment":"The online parametric injection trains the LoRA adapter on the full retrieved case, including its `Reason`, `Articles`, and `Judgment` components. The test query is then answered after this injection, meaning the model has been explicitly supervised on the answer (judgment and applicable articles) of a retrieved similar case before making its prediction. This is answer leakage and confounds the improvement of PL-CA over vanilla RAG in Table 4: the gain could come from memorizing the retrieved answer rather than from encoding legal knowledge. No control experiment removes the judgment/articles fields from the online training signal.","section":"Online P-RAG, Figure 1"},{"comment":"The experimental comparisons lack error bars, statistical significance tests, and multiple random seeds. The test set has only 590 instances, and many numerical differences are small (for example, Table 4: Charge 92.39 vs. 88.62; Probation 40.52 vs. 40.79). The claim of consistent and significant improvement over baselines is therefore not established by the reported numbers.","section":"Results, Tables 4-7"},{"comment":"The newly introduced evaluation metric d(r,h) = 1 - 1/(1 + exp(-|r-h|/(|r|+|h|+eps))) is not validated against human judgments or existing metrics. The LDG semantic-similarity metric based on chinese-roberta-wwm-ext is also used without any correlation analysis. Since the paper's central quality claims depend on these metrics, an unvalidated metric is a load-bearing issue.","section":"Metric, Eq. (4)"},{"comment":"BM25's article Recall@5 is only 3.6% (Table 3), meaning that the top-5 retrieved legal articles almost never contain the relevant statutes. The online stage nonetheless 'extracts the corresponding legal statutes from the top-5 retrieved cases' and uses them for parametric injection. The paper does not analyze how such low retrieval quality affects the injected knowledge, and it does not report the impact of retrieval errors on the final downstream performance.","section":"Table 3 and Online P-RAG"},{"comment":"The comparison with GPT-4o is not matched: GPT-4o is evaluated as a zero-shot API baseline, while PL-CA is fine-tuned on 1,990 training instances and also uses online per-query adaptation. The claim of 'surpassing GPT-4o' is therefore not an apples-to-apples comparison and overstates the result.","section":"Experiment Setting, Table 4"}],"minor_comments":[{"comment":"The abstract states 'Our code and dataset are provided in the appendix,' but the submission contains no appendix and no code or dataset link. This must be corrected.","section":"Abstract"},{"comment":"The 'avg token length' for Legal-CA is listed as 415, which conflicts with the text claim that Legal-CA 'ranks among the top in terms of average token count.' The column definition or the number appears to be inconsistent.","section":"Table 1"},{"comment":"The entry for gpt-4o's Fine column appears as '43.6325.75' (a missing space), making the table difficult to read.","section":"Table 4"},{"comment":"The paper introduces the acronym 'SAG' for statute article generation but later uses 'SAR' in related work and 'LA' in the results. The terminology should be unified and defined consistently.","section":"Preliminary"},{"comment":"The variable d is used both for documents (e.g., d_i in the offline stage) and for the metric function d(r,h). Renaming one of these would avoid confusion.","section":"Equation (4)"}],"recommendation":"reject","confidential_remarks":"The online-stage answer leakage is the most serious issue: the method is essentially test-time fine-tuning on the retrieved case's known judgment and statutes, so the reported gains over vanilla RAG are confounded. Even if the authors added error bars, the headline efficiency claim is untenable without a baseline cost comparison, and the GPT-4o comparison is not matched. The missing appendix also means the dataset is not actually available. This would require a fundamental reworking of the claims and experiments, so I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The benchmark is the real contribution here. Legal-CA, with 590 expert-annotated test cases spanning criminal, administrative, and civil law, is a solid addition to the legal NLP landscape, and the first P-RAG application to legal tasks is a legitimate extension of existing work. The consistent gains over vanilla RAG on several subtasks, including the headline LA-F1 jump from 16.56 to 24.01, are worth taking seriously. The authors also did the right thing by selecting test cases published after the training cutoff to mitigate leakage.\n\nBut the central claim that PL-CA reduces computational overhead is unmeasured and, as described, probably false. The online stage requires training a LoRA adapter on the retrieved case for every query. That means a backward pass and optimizer update per query, which is typically far more expensive than a single forward pass with a long context. No wall-clock time, FLOPs, memory, or cost appears anywhere in the paper. That is not a minor omission; it is the paper's stated motivation.\n\nThe circularity burden is also real. In the online stage, the model is fine-tuned on the retrieved case's known judgment and statutes before being asked to predict that same case's outcome. The performance gain could simply be memorization, not parametric knowledge integration. The poor retriever recall (BM25 Recall@5 of 3.6% for articles) makes this worse, because the model is often training on irrelevant or misleading cases.\n\nOther issues: no error bars or significance tests, an unvalidated custom metric for sentencing differences, and the promised code and dataset are absent despite the appendix claim. The 'Combine Both' results, where vanilla RAG plus P-RAG often underperforms P-RAG alone, are unexplained and undercut the story.\n\nWho is this paper for? Researchers working on RAG variants and legal NLP will find the benchmark useful and the questions worth discussing. The paper deserves a serious referee because the empirical phenomenon—parametric injection beating context injection on legal tasks—is nontrivial even if the overhead claim collapses. But it needs major revision: measure actual costs, address the circularity with a proper held-out answer setup, report variance, and validate the metric.\n\nMy recommendation: send it to peer review, but the reviewers should push hard on the efficiency and memorization issues. This is not a rejection-worthy desk reject; it is a paper with a real contribution and a load-bearing unsupported claim.","headline":"A genuinely useful new legal benchmark wrapped around an efficiency claim that the paper never measures and that the per-query LoRA training procedure probably contradicts.","tokens_in":12665,"tokens_out":2017,"would_cite":false,"duration_ms":19689,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that packing retrieved legal cases into an LLM's weights, rather than its prompt, lifts a 7B model's legal-article F1 to 24.01 against 15.43 for GPT-4o.","keywords":["parametric RAG","LoRA injection","legal judgment prediction","statute article generation","multi-task legal benchmark","Chinese legal AI","long-context degradation","data augmentation"],"falsifier":"Measure wall-clock time and FLOPs for one online PL-CA query—retrieve, train one LoRA adapter on the retrieved case, merge, generate—and compare it with vanilla RAG on the same query; if PL-CA is not cheaper, the overhead-reduction claim fails regardless of accuracy. A second check is to see whether correctly generated statute articles actually appear in the top-1 retrieved case, or whether they come from the offline parametric corpus, which would reveal what the online retrieval path is contributing.","tokens_in":11686,"feed_emoji":"⚖️","tokens_out":9041,"duration_ms":73985,"temperature":0.7,"pith_summary":"The paper is trying to establish that retrieval-augmented generation for law does not have to work by stuffing retrieved case text into the LLM's prompt. Its PL-CA framework instead turns each legal case into a small LoRA weight update, so the model absorbs legal knowledge parametrically and keeps its context window short. If the claim holds, legal assistants could handle multi-step, mixed legal tasks with compact inputs and still beat both conventional RAG and much larger closed models on statute and judgment prediction. The paper also contributes Legal-CA, an expert-annotated multi-task benchmark covering criminal, administrative, and civil law, plus a 1,990-case parametric training corpus.","feed_headline":"Injecting case law into model weights beats stuffing it into prompts","feed_subtitle":"Packing retrieved case law into LoRA weights lifts a 7B model past GPT-4o on legal-article F1.","key_machinery":"The central mechanism is parametric RAG (P-RAG), adapted to legal cases and implemented through Low-Rank Adaptation (LoRA). Offline, each of 1,990 professional cases is rewritten by GPT-4o-mini into four variants of four components, multiplying each case into sixteen training pairs used to learn low-rank matrices $A_i$ and $B_i$; online, BM25 retrieves a similar case, the same parameterization is applied, and the trained adapter is merged into the frozen model. This turns retrieval output from prompt text into a weight update, which is what removes the retrieved document from the context and lets the model carry legal knowledge internally.","core_discovery":"PL-CA shows that a legal case can be decomposed into fact, dispute focus, reasoning, applicable statutes, and judgment; augmented into paraphrased variants; and encoded as a case-specific LoRA adapter that is merged into a frozen LLM. An offline stage parameterizes 1,990 professionally annotated cases, and an online stage retrieves one similar case from the Legal-KD corpus and parameterizes it on the spot before generation. On Legal-CA, Qwen1.5-7B-Chat with PL-CA reaches an LA-F1 of 24.01 on statute-article tasks versus 15.43 for GPT-4o-2024-11-20, and it outperforms vanilla RAG on most subtasks. This is taken as evidence that parametric knowledge injection is more effective than contextual injection for long legal documents, while also reducing the context-length overhead.","pith_inferences":["If the per-query cost of LoRA training plus merging turns out acceptable, the same weight-injection idea transfers to other long-document, attention-diluted domains such as medical records, patents, or contracts, with no architectural change.","Because BM25 finds the correct statute in the top 5 only 3.6% of the time, the current online pipeline is often parameterizing a neighboring case rather than the decisive authority; a better retriever would likely raise the ceiling of P-RAG.","The offline stage is in effect a 1,990-case fine-tuning corpus, so the comparison against vanilla RAG does not isolate retrieval value from extra training signal; an offline-only, retrieval-free fine-tuned baseline at equal data would separate these two contributions.","Storing one LoRA adapter per case turns the retrieval index into a large parameter store, raising practical questions of storage cost, adapter routing, and model update hygiene that PL-CA does not address."],"forward_implications":["Legal LLMs can operate with short, roughly fixed-length prompts, because retrieved knowledge is carried in LoRA weights rather than in the input.","On the Legal-CA benchmark, PL-CA beats vanilla RAG on most subtasks and surpasses GPT-4o on statute-article F1 by about 8.5 points, all from a 7B open-weight base model.","The approach scales better with model size: Qwen1.5-7B-Chat shows clear gains from parametric injection, while the 1.8B model benefits more from plain context RAG.","Online and offline parametric injection both help, and online injection contributes more at equal data scale because the retrieved case is semantically closer to the query.","The Legal-CA dataset itself provides a multi-task, expert-annotated testbed that single-task benchmarks cannot offer, making the measured gains visible in the first place."],"supporting_citations":[{"why":"Defines the parametric RAG paradigm this paper adapts and transfers to the legal domain.","marker":"Su et al. 2025a"},{"why":"Supplies the claim that LLMs use internalized parametric knowledge more effectively than context-injected knowledge.","marker":"Yu and Ananiadou 2024"},{"why":"Supports the motivation that long contexts disrupt attention and degrade downstream task performance.","marker":"Li et al. 2024e"},{"why":"Co-support for the long-context degradation claim that motivates replacing context injection with parameter injection.","marker":"Qian et al. 2024"},{"why":"Identifies the sentencing-recommendation shortcut that Legal-CA removes from case facts to make evaluation fair.","marker":"He et al. 2024"},{"why":"Co-identifies the same shortcut in legal judgment prediction and motivates the evaluation metric change.","marker":"Chang et al. 2025"},{"why":"Supplies chinese-roberta-wwm-ext, the semantic encoder used to score legal document generation quality.","marker":"Cui et al. 2019"}],"fun_headline_variants":["Parametric RAG: Pack legal knowledge into LoRA, not prompts","LoRA-encoded case law beats long prompts for legal AI","Store legal rules in weights: PL-CA slashes context without losing accuracy","Case law as adapters: PL-CA beats GPT-4o on legal F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the online stage's per-query process—training a LoRA adapter on the retrieved case and merging it into the model—costs no more than injecting the retrieved text as context; the paper reports no runtime or cost measurement, so if that premise fails the overhead-reduction claim collapses even if accuracy stays high.","fun_headline_variants_meta":{"raw":{"variants":["Parametric RAG: Pack legal knowledge into LoRA, not prompts","LoRA-encoded case law beats long prompts for legal AI","Store legal rules in weights: PL-CA slashes context without losing accuracy","Case law as adapters: PL-CA beats GPT-4o on legal F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000933,"raw_usage":{"total_tokens":4007,"prompt_tokens":975,"completion_tokens":3032,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":2950}},"tokens_in":591,"tokens_out":3032,"duration_ms":19701,"temperature":1.0,"reasoning_tokens":2950,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:17:04.717690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure wall-clock time and FLOPs for one online PL-CA query—retrieve, train one LoRA adapter on the retrieved case, merge, generate—and compare it with vanilla RAG on the same query; if PL-CA is not cheaper, the overhead-reduction claim fails regardless of accuracy. A second check is to see whether correctly generated statute articles actually appear in the top-1 retrieved case, or whether they come from the offline parametric corpus, which would reveal what the online retrieval path is contributing.","supporting_citations":[],"review_version":2}