{"id":"6fca588e-5246-4598-9f4f-f7d3f7c02425","arxiv_id":"2504.13603","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Continued pre-training on Taiwanese legal text plus instruction tuning did not consistently improve legal reasoning over base models or LoRA routes, and DPO and ORPO alignment degraded accuracy.","lead":"This paper trained legal language models for Taiwan, including an expensive round of continued pre-training on legal text, and found that the extra training did not clearly beat simpler approaches, while preference tuning made scores worse. It also builds a small Taiwanese legal reasoning benchmark, with multiple choice, symposium, and essay tasks, that is not yet released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that legal DACP is 'not what you need' rests on Table 3 comparisons that never isolate the CPT stage from a different base model and a different tuning recipe; a same-base, same-recipe no-CPT control is the missing experiment.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing problem: the experimental design does not isolate the legal continual pre-training stage. I agree with that assessment after checking Sections 3.1-3.4 and Table 3. The claim in the abstract and conclusion is essentially counterfactual - it says adding legal DACP does not help uniformly - so the evidence needs a control that differs only by the presence of the CPT stage. No row in Table 3 provides that control: LLAWA-TCxYZL-Instruct differs from TAIDE base in both CPT and subsequent instruction tuning, and differs from BLLAWA/BLAWSTRAL in base model, tuning method, and CPT simultaneously. This is not a disagreement with the consensus about DACP; it is an internal design gap that makes the title claim unverifiable from the reported data. The DPO/ORPO degradation and the Task C/D results may be real, and the authors are candid about many limitations, but those descriptive results do not establish the causal statement about DACP specifically. A same-base, same-recipe no-CPT run would settle it. Because the reader's CONDITIONAL verdict is already grounded in this same concern, my stress-test does not move the verdict; it reinforces the condition.","tokens_in":11434,"tokens_out":3314,"duration_ms":31536,"concrete_test":"Run LLAWA-noCPT: starting from Llama-3-TAIDE-LX-8B-Chat-Alpha1, skip the Section 3.2 legal continual pre-training and directly apply the exact LLAWA-TCxYZL-Instruct instruction-tuning recipe (one-stage Taiwan Chat + legal instruction dataset, shuffled, two epochs, same hyperparameters). Evaluate on Tasks A, B, and C with identical prompts and greedy decoding, and on Task D if the authors can extend the protocol. If LLAWA-noCPT matches LLAWA-TCxYZL-Instruct within noise, the null result for DACP is reproduced with an isolated ablation; if LLAWA-noCPT is materially worse on Task C, DACP has a positive effect and the title claim fails. A second arm that applies the same full-parameter legal recipe to Meta-Llama-3-8B-Instruct would further separate the base-model effect from the CPT effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 3 compares LLAWA-TCxYZL-Instruct (legal CPT applied to TAIDE, then full-parameter one-stage instruction tuning) against Llama-3-TAIDE-LX-8B-Chat-Alpha1 (no legal CPT, no legal SFT), Meta-Llama-3-8B-Instruct (different base), and LoRA-tuned models. Every row differs from LLAWA-TCxYZL-Instruct in at least two of: base model, presence of the legal CPT stage, and tuning method/recipe. The paper's central conclusion, that DACP 'does not uniformly improve performance across all legal tasks' and the title's implication that continual pre-training is 'not what you need,' require a comparison where only the legal CPT stage is removed while base model and instruction-tuning recipe are held fixed. Without that control, the flat Task A-C numbers could be caused by the choice of TAIDE as base, by full-parameter versus LoRA tuning, or by the interaction with Taiwan Chat and the legal instruction data, and the causal attribution to DACP is ungrounded. The paper itself never acknowledges this missing ablation; Section 6 lists contamination and corpus-mixture limitations but not this confound. Additionally, the first half of the claimed trade-off, that DACP 'enhances domain-specific knowledge,' is not directly measured by any knowledge probe, only inferred from task accuracy, making the lack of an isolated CPT comparison even more consequential.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper examines whether Domain-Adaptive Continual Pre-Training (DACP) improves legal reasoning for Taiwanese-Mandarin legal tasks. The authors train LLAWA by continually pre-training the TAIDE Llama-3-8B base on legal corpora, then instruction-tuning it; they additionally train LoRA-based baselines (BLLAWA from Meta-Llama-3-8B-Instruct and BLAWSTRAL from Mistral-Nemo-Instruct-2407) and experiment with DPO and ORPO. They construct four evaluation tasks from Taiwanese bar and judicial exam materials and a legal symposium dataset, reporting accuracy on Tasks A–C and LLM-judged essay scores on Task D. The main conclusion is that while DACP may improve some domain-specific abilities, it does not uniformly improve all legal tasks, and preference optimization actually degrades performance on the targeted tasks. The manuscript argues that continual pre-training is therefore 'not what you need' for all legal tasks, and it discusses trade-offs between domain knowledge and generalization.","tokens_in":11623,"tokens_out":3498,"duration_ms":33939,"significance":"If the central claim were fully supported, the paper would be a valuable counterpoint to the prevailing assumption that domain-adaptive continual pre-training reliably helps legal LLMs, particularly in a non-Anglo-American legal system. The authors contribute a new benchmark for Taiwanese legal reasoning, transparent reporting of training hyperparameters and data sources, and a candid discussion of evaluation pitfalls such as LLM-judge bias and data contamination. The paper also makes a useful empirical observation that DPO and ORPO, in their setup, reduced accuracy on Tasks A–C. However, the headline causal claim about DACP is not yet established because the experimental design never isolates the DACP stage from other variables, so the significance is conditional on a missing control experiment. The work is publishable as a contribution to the empirical literature only after that design gap is addressed.","major_comments":[{"comment":"The paper's central claim that DACP 'does not uniformly improve performance across all legal tasks' is not supported by the current comparisons. LLAWA-TCxYZL-Instruct differs from every other row in Table 3 in at least two of three variables: base model, presence of the legal CPT stage, and instruction-tuning recipe. Compared with Llama-3-TAIDE-LX-8B-Chat-Alpha1, it changes both the legal CPT stage and the one-stage instruction tuning; compared with BLLAWA or Blawstral, it changes the base model and the full-parameter versus LoRA recipe. Without a same-base, same-recipe ablation that removes only the legal CPT stage, the flat or slightly improved Task A–C numbers cannot be causally attributed to DACP; they could equally reflect the choice of the TAIDE base or the full-parameter tuning recipe. The authors should add a control model trained from the TAIDE base with the identical instruction-tuning recipe (Taiwan Chat plus legal data, two epochs) but without the legal pre-training stage, and report it in Table 3.","section":"Sections 3.2–3.4, Table 3"},{"comment":"The Task D evaluation is described inconsistently and its validity is load-bearing for the paper's generalization trade-off claim. Section 4.3 states that GPT-4-Turbo segments answers into four chunks and that 'these segmented answers are then evaluated by legal experts from law school,' but the following paragraph and the Table 3 footnote say that GPT-4o evaluates similarity and legal rationality against the golden segmented answers. Moreover, the reference answers were produced by GPT-4-Turbo, one of the evaluated models, and Section 6 acknowledges that using LLMs to evaluate other LLMs 'introduces potential biases.' This makes the Task D scores self-referential in part and uninterpretable as a measure of legal quality. The authors should clarify the exact pipeline (who segments, who grades, and with what prompt) and either provide human-expert scores or explicitly label Task D as an exploratory LLM-as-judge experiment rather than a claim about legal reasoning ability.","section":"Section 4.3, Table 3 footnote, Figure 2"},{"comment":"The abstract and conclusion assert that DACP 'enhances domain-specific knowledge,' but the paper never directly measures domain knowledge. No knowledge probe, fact-retrieval test, or legal-terminology assessment is reported; the only evidence is task accuracy, which is also the variable used to show the lack of uniform improvement. Since the paper's takeaway is a trade-off between domain-specific gains and generalization or prompt-based ability, the positive side of that trade-off needs an independent measurement. Without a direct knowledge probe, the claim that DACP improved legal knowledge while hurting other skills is an inference from a design that, as noted above, does not isolate DACP.","section":"Abstract, Sections 4 and 5"},{"comment":"The limitation section acknowledges potential data contamination but does not attempt to measure overlap between the evaluation data and the training corpora. This matters especially for Tasks C and D, whose public sources (Taiwan High Court website, bar and judicial exams) are exactly the kind of web-scraped legal text that plausibly appears in pretraining or in the legal CPT corpus. The authors should report a contamination analysis (e.g., n-gram overlap between the evaluation instances and the pretraining data) or, if overlap is found, re-evaluate on a fresh or time-shifted sample. This is not a reason to reject the paper, but it is a required check for the benchmark claims.","section":"Section 6, Tasks C and D"}],"minor_comments":[{"comment":"The title uses 'Domain Adaption' while the text uses 'Adaptation,' and the abstract contains a stray space in 'Y ou'; these should be corrected.","section":"Title and Abstract"},{"comment":"The model name is written inconsistently as 'LLawa,' 'Llawa,' and 'LLAWA'; the authors should pick one spelling and use it throughout.","section":"Section 3.3"},{"comment":"There is a typo in 'Eergebnis' (should be 'Ergebnis'), and the four German chunk labels should be given uniformly with consistent capitalization.","section":"Section 4.3"},{"comment":"Task D entries for LLAWA-TC-YZL-Instruct and LLAWA-TCxYZL-Instruct are missing without explanation; a footnote should state why these models were not evaluated on the essay task.","section":"Table 3"},{"comment":"Task C outputs are free-form final arguments, but the paper writes that accuracy 'focuses solely on comparing the final conclusion' without specifying how the conclusion was extracted from each model's output or how duplicate or partial conclusions were scored; this should be operationalized.","section":"Section 4.2"},{"comment":"The preference optimization data are described only as 'the ground truth as the preferred output and the model's generated output as the rejected one'; the authors should report the number of preference pairs, the source of the generated outputs, and any filtering or format normalization applied.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is not ready in its current form because the headline causal claim about DACP rests on comparisons that never isolate the CPT stage from the base model and tuning recipe. The missing same-base, same-recipe no-CPT control is the key experiment; the Task D evaluation inconsistency and the unmeasured contamination are secondary but also need attention. These issues are fixable within the scope of a revision, which is why I recommend major revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one if you care about the empirical question of whether legal continual pre-training is worth the compute. The descriptive results are probably real: an 8-day, 16-H100 legal CPT run on top of TAIDE, plus instruction tuning, did not beat the base model on their Taiwanese legal tasks, and DPO/ORPO actively hurt. That is a useful data point, especially since the authors are candid about contamination risk, unknown corpus mixture, and LLM-judge bias.\n\nWhat is genuinely new is the Taiwanese legal benchmark itself: Tasks A-D, including the legal symposium task and the essay task with the Gutachten-style chunked grading. The headline trade-off claim is not new; the paper cites Cheng et al. (2023) and Xie et al. (2023) for exactly that pattern. What is new is the assembly and the preference-optimization negative result, which is stark (Task C drops from 53.07 to 43.94 with DPO).\n\nThe soft spots are real and they sit on the central claim. Table 3 never isolates the legal CPT stage: every comparison row differs in base model, tuning method, and presence of CPT simultaneously. The paper's own Section 6 lists limitations but not this confound. So the title's causal reading—'continual pre-training is not what you need'—is not supported by the design. A same-base, same-recipe no-CPT control would fix it. Also, Task D is described three different ways in the text (GPT-4-Turbo segments, legal experts evaluate, then GPT-4o grades), and the grading is self-referential: GPT-4-Turbo gold answers graded by GPT-4o. The paper concedes LLM-judge bias but not this specific circularity. Task C's accuracy metric is undefined for free-form answers, and there are no error bars anywhere. These are fixable, not fatal.\n\nThe paper deserves a serious referee. The negative results are plausible, the benchmark could be reused, and the missing control is an addressable design flaw, not a dishonest one. The authors clearly thought about what they did. I would send it to review with a request for the missing ablation and a consistent Task D protocol.\n\nFor you: worth a skim if you work on domain adaptation or legal NLP. I would not cite the causal claim, but I might cite the benchmark or the preference-optimization negative result.\n\nBest,\n[You]","headline":"Honest, expensively produced negative-result paper whose title overclaims: no ablation isolates the legal CPT stage, so the causal claim outruns the design, but the descriptive results and benchmark are worth a serious look.","tokens_in":12361,"tokens_out":632,"would_cite":false,"duration_ms":6999,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Domain-adaptive continual pre-training of legal LLMs improves some tasks, hurts others.","keywords":["domain-adaptive continual pre-training","legal large language models","Taiwanese law","instruction tuning","preference optimization","DPO","ORPO","legal reasoning benchmark"],"falsifier":"Train the same base model (Llama-3-TAIDE-LX-8B-Chat-Alpha1) with and without the legal continual pre-training stage, then apply the identical instruction-tuning recipe to both; if the version without legal CPT matches or beats the version with it on Tasks A–C, the paper's central claim about DACP's trade-off would be falsified. A weaker check: run DPO with alternative preference data and hyperparameters to see whether the accuracy drop on Task C persists.","tokens_in":11097,"feed_emoji":"⚖️","tokens_out":4047,"duration_ms":32227,"temperature":0.7,"pith_summary":"This paper asks whether Domain-Adaptive Continual Pre-Training (DACP) — continued pre-training of a general LLM on legal text — improves legal reasoning. Testing on a new Traditional-Chinese legal benchmark with four task types from Taiwan's legal framework, the authors find that DACP increases some domain-specific knowledge but does not uniformly beat the base model, and can lower performance on prompt-based and general tasks. They also report that preference optimization via DPO and ORPO failed to improve their target tasks, with Task C accuracy falling from 53.07 to 43.94 and 45.67 respectively. The paper's contribution is empirical evidence that the common assumption that more domain pre-training is unconditionally beneficial does not hold for legal LLMs.","feed_headline":"Legal domain pre-training is no uniform win","feed_subtitle":"Taiwanese legal benchmark shows DACP helps some tasks but hurts prompt-based accuracy.","key_machinery":"The engine of the study is a three-stage training pipeline: first, domain-adaptive continual pre-training on Taiwanese legal texts, German legal texts (reflecting the Continental-law influence on Taiwan), and self-curated reasoning data; second, full-parameter instruction tuning on general and legal instruction data; third, preference alignment via DPO or ORPO. The evaluation is a new benchmark with four tasks: single/multiple-choice legal questions (Tasks A and B), argument-based decision-making in legal symposia (Task C), and essay questions graded by GPT-4o with a Gutachten-style rubric (Task D). The load-bearing comparison is Table 3, which pits the LLAWA models against their base models and two LoRA-tuned baselines.","core_discovery":"The central claim is that DACP is a trade-off, not a free lunch: it may improve certain types of domain-specific reasoning while simultaneously diminishing performance on prompt-based tasks and other generalization capabilities. In the reported experiments, the instruction-tuned LLAWA-TCxYZL-Instruct model performs only slightly better than its TAIDE base on Task A (28.55 vs 24.92) and equally on Task B (33.47 vs 33.47), and preference alignment makes things worse: DPO drops Task C accuracy from 53.07 to 43.94 and ORPO to 45.67. The paper concludes that hybrid approaches combining DACP with task-specific fine-tuning or meta-learning may be needed, and that evaluation benchmarks should capture task-specific trade-offs rather than a single aggregate score.","pith_inferences":["The paper's title claim is broader than what the experiment can prove: no run isolates the legal continual-pre-training stage, because the base model, tuning method, and dataset all differ between the LLAWA row and the comparison rows. An ablation that removes only the legal-CPT step from an otherwise identical recipe would be needed to confirm that DACP itself causes the observed pattern.","A plausible reading of the DPO/ORPO failure is that the preference dataset construction (ground truth as chosen, model output as rejected) and hyperparameters rather than preference optimization per se; a different recipe might show gains.","The GPT-4o-based essay grading, with its acknowledged LLM-judge bias, could be masking true legal-quality differences; human expert scoring on a subset would be a cheaper robustness check.","If the trade-off is real, a testable extension is to vary the mixture ratio of legal to general corpus during DACP, as the paper itself notes is unexplored."],"forward_implications":["If the claim holds, organizations cannot assume that adding legal-domain pre-training will improve legal AI; task-specific evaluation is required before deployment.","Preference optimization methods like DPO and ORPO need careful validation on legal tasks, because in these experiments they actively reduced multiple-choice accuracy.","The choice of base model matters as much as the domain pre-training: TAIDE's prior Traditional-Chinese pre-training may already supply much of the benefit.","The observed trade-off between domain-specific reasoning and prompt-based generalization should be a first-class consideration in any domain-adaptation pipeline.","Benchmarks that include argumentation and essay tasks, not just multiple choice, reveal effects that simpler benchmarks miss."],"supporting_citations":[{"why":"Establishes domain-adaptive continual pre-training as a method the paper applies to legal text.","marker":"Gururangan et al., 2020"},{"why":"Prior observation that DACP can reduce performance on prompting tasks, which the paper's results echo.","marker":"Cheng et al. (2023)"},{"why":"Comparative study of FinPythia vs Pythia showing task-dependent DACP effects, a key precedent.","marker":"Xie et al. (2023)"},{"why":"Provides the DPO preference-optimization method that the paper applies after instruction tuning.","marker":"Rafailov et al. (2024)"},{"why":"Provides the ORPO method the paper uses for monolithic preference optimization.","marker":"Hong et al. (2024)"},{"why":"LawBench is a reference legal benchmark the paper's task design builds on.","marker":"Fei et al., 2023"},{"why":"DISC-LawLLM-Eval offers the LLM-based subjective evaluation template adapted for Task D.","marker":"Yue et al., 2023"},{"why":"Taiwan Chat is the Traditional-Chinese instruction dataset used in instruction tuning.","marker":"Lin and Chen, 2023"}],"fun_headline_variants":["DACP: legal pre-training is a trade-off, not a win","Continual pre-training: not the universal legal fix","Legal DACP: boosts some tasks, sinks prompt accuracy","Legal domain tuning: sometimes worse than skipping it","Continual pre-training helps legal tasks selectively"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper treats Table 3 as evidence about the effect of legal continual pre-training, but the design never isolates that stage: the main model differs from every comparison in base model, tuning method, and presence of the legal-CPT stage at once, so something else could be driving the flat results.","fun_headline_variants_meta":{"raw":{"variants":["DACP: legal pre-training is a trade-off, not a win","Continual pre-training: not the universal legal fix","Legal DACP: boosts some tasks, sinks prompt accuracy","Legal domain tuning: sometimes worse than skipping it","Continual pre-training helps legal tasks selectively"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001474,"raw_usage":{"total_tokens":5877,"prompt_tokens":846,"completion_tokens":5031,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":4953}},"tokens_in":462,"tokens_out":5031,"duration_ms":32640,"temperature":1.0,"reasoning_tokens":4953,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:05:09.855878+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same base model (Llama-3-TAIDE-LX-8B-Chat-Alpha1) with and without the legal continual pre-training stage, then apply the identical instruction-tuning recipe to both; if the version without legal CPT matches or beats the version with it on Tasks A–C, the paper's central claim about DACP's trade-off would be falsified. A weaker check: run DPO with alternative preference data and hyperparameters to see whether the accuracy drop on Task C persists.","supporting_citations":[],"review_version":1}