{"id":"1075c975-43cc-43dc-8231-dfb69ee2b63b","arxiv_id":"2412.04119","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"The authors release the first Romanian legal MCQA dataset, a law corpus, a legal knowledge graph, and a graph retrieval method that beats standard baselines on most exam settings.","lead":"This paper introduces JuRO, the first openly available Romanian legal multiple-choice question answering dataset, along with a legal corpus and a knowledge graph. It also presents GRAF, a graph retrieval method that outperforms existing baselines on this dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4's GRAF row is a per-branch best-of-many mosaic; no single GRAF model hits 60.09%, so the 'exceeds SOTA' claim is not supported by a fixed configuration.","rationale":"The reader's conditional acceptance is driven by the test-time count leak for encoder models. That is a real issue, but it only affects entrance/bar exams and only the comparison against LLMs; it does not affect GRAF vs. encoder baselines or promotion exams. A more general problem is the construction of the headline result tables. Tables 4–6 appear to select the best-performing backbone or prompt per legal branch, producing a mosaic that no single model realizes. On promotion exams, the reported GRAF average of 60.09% is not achieved by any of the five GRAF backbones in Table 12; the best single GRAF model (jurBERT-large) obtains 55.81%. The best reported baseline row (LLM LFT 55.75%) is likewise a per-branch mix; the best single baseline (FLAN-T5 XXL LFT) obtains 55.31%. The headline gap collapses from 4.34 to 0.50 points, within single-run noise. Since the paper reports a single run and no variance, 'exceeds SOTA in most settings' is not established. The same mosaic pattern appears in Tables 5 and 6. This is a load-bearing concern because the central claim is the GRAF method's superiority. The paper's resource contributions (JuRO, CROL, Law-RoG) remain valuable independently. The count leak flagged by the reader is valid but secondary; the mosaic issue is broader and also affects the single-answer promotion setting. I therefore recommend keeping the conditional verdict but adding the requirement that all headline comparisons be recomputed with a single fixed configuration per method, selected on validation, with multiple seeds.","tokens_in":33741,"tokens_out":12197,"duration_ms":102432,"concrete_test":"Write a script that reconstructs Tables 4–6 from Tables 12–14, computing (a) each fixed model's per-branch and average accuracy (e.g., GRAF+jurBERT-large, FLAN-T5 XXL LFT) and (b) the per-branch best-of-many row. If the fixed-config average gap on promotion is ≤1 point or changes sign across seeds, the headline 'exceeds SOTA' should be downgraded to 'competitive'; also report standard deviations over ≥5 seeds for the fixed-config comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main comparison tables (Tables 4–6) report 'GRAF' and baseline rows as single methods, but each row is a per-branch best-of-many selection. On promotion exams, the GRAF row (60.09%) matches different backbone encoders in different branches: Civil 49.46 is GRAF+jurBERT-base, Penal 52.70 is GRAF+RoBERT-large, Civil Procedure 78.46 is GRAF+jurBERT-base, etc. No fixed GRAF configuration achieves the row; GRAF+jurBERT-large, the best single configuration, averages 55.81%. Similarly, the best baseline row (LLM LFT 55.75%) mixes FLAN-T5 XL/XXL and Llama results; the best single baseline (FLAN-T5 XXL LFT) averages 55.31%. The fixed-config gap is therefore 0.50 points on promotion exams, not 4.34 points, and with single-run results this is within noise. This mosaic selection also affects entrance and bar tables. The central claim that GRAF exceeds SOTA in most settings is thus not established by the reported numbers; a corrected comparison may show competitiveness rather than superiority.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces three Romanian legal NLP resources and a retrieval-augmented MCQA method: JuRO, a public dataset of 10,836 multiple-choice exam questions; CROL, a corpus of Romanian laws; and Law-RoG, a knowledge graph extracted from CROL. The proposed method, GRAF, builds a claim graph from a question-choice pair with an LLM, retrieves a subgraph from Law-RoG, encodes both with a GAT over a language-model backbone, and scores choices with self-attention. The authors report accuracy on promotion, entrance, and bar exams, comparing GRAF with BERT-family encoders, LLM zero-shot baselines, RAG, and LoRA fine-tuning, and include ablations and topic-level analyses.","tokens_in":34033,"tokens_out":6045,"duration_ms":61915,"significance":"If the empirical claims held, the paper would be a useful contribution to low-resource legal NLP: it is the first open Romanian legal MCQA dataset, provides a structured legal corpus and a Romanian legal knowledge graph, and releases code and resources. The method is described in sufficient detail to reproduce, and the ablation isolates the contribution of the claim graph and the KG. However, the headline claim of exceeding state-of-the-art methods is not currently established because of evaluation asymmetries and reporting choices; the resource contribution is real and likely valuable, but the comparative evaluation needs repair.","major_comments":[{"comment":"Encoder-based models and GRAF are given the exact number of correct choices |T_i| at inference time because they select the top |T_i| scores, while the LLM baselines are only instructed that 'at most 2 answers are correct' (Appendices I/J). On the entrance and bar exams in Tables 5 and 6, where |T_i| is either 1 or 2, this is an information asymmetry: knowing the exact answer count is a substantial extra signal for multi-answer scoring. The superiority of GRAF in these settings is therefore not established unless all systems are evaluated with the same answer-count information, for example by learning to predict the count or by using a protocol that does not rely on |T_i| at inference.","section":"Appendix D, Eq. (16)"},{"comment":"The GRAF rows in the main tables are per-branch best-of-many mosaics, not a single fixed model. In Table 4, the 60.09% promotion average combines different backbone models in different branches, e.g., Civil 49.46 is GRAF+jurBERT-base, Penal 52.70 is GRAF+RoBERT-large, and Civil Procedure 78.46 is GRAF+jurBERT-base; the fixed GRAF+jurBERT-large configuration averages 55.81% in Table 12. The LLM LFT baseline row similarly mixes configurations: the best single fixed baseline, FLAN-T5 XXL LFT, averages 55.31%. The fixed-configuration gap on promotion exams is therefore about 0.50 points rather than the 4.34 points implied by comparing the mosaic rows, and the statement in Section 5.3 that GRAF outperforms baselines in 6 of 9 branches is a property of the per-branch best selection. Please report fixed-configuration comparisons and clearly separate them from any per-branch upper-bound analysis.","section":"Tables 4-6 and Appendix H"},{"comment":"The appendix states 'We report the results of a single run.' After aligning configurations, the central advantage on promotion exams is about 0.50 points, and the test sets are small (approximately 50-400 examples per branch in Table 11). Without multiple seeds or paired statistical tests, the reported differences cannot be distinguished from noise. The paper should provide variance estimates across seeds or significance testing, and the abstract and conclusions should be worded to match the strength of the evidence.","section":"Appendix E"},{"comment":"The ablation reports GRAF at 55.61% on promotion exams, but no backbone or configuration is named, and this value matches neither the mosaic row of 60.09% in Table 4 nor any fixed configuration in Table 12 (e.g., GRAF+jurBERT-large at 55.81%, GRAF+RoBERT-large at 55.13%). Since this ablation is the main evidence that both the claim graph and the KG contribute, it must be run with the same configuration used in the main comparison and reported with that configuration; otherwise the effect sizes cannot be interpreted.","section":"Section 5.4, Table 8"}],"minor_comments":[{"comment":"The abstract says CROL contains 93 documents with modifications from 763 time spans, while Section 3.2 states 768 different versions; please make these numbers consistent.","section":"Abstract vs. Section 3.2"},{"comment":"The text refers to 'Algorithm 1' several times, but no algorithm block appears in the manuscript; either include the pseudocode or remove the references.","section":"Section 4.2"},{"comment":"The translated FLAN-T5 RAG prompt contains the typo 'documets' instead of 'documents'.","section":"Appendix J"},{"comment":"The appendix contains the typo 'arameter fine-tuning' in the LLM LFT paragraph; it should read 'parameter fine-tuning'.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The resource contribution is genuine and likely useful to the community, and the method is clearly described. The main obstacle is comparability of the headline numbers: the exact-count oracle for encoders, the per-branch best-of-many reporting, and the single-run results together mean the claimed superiority is not yet demonstrated. These issues are fixable within the scope of the manuscript by reporting fixed-configuration comparisons and a fair evaluation protocol; if after that the fixed-configuration gaps largely disappear, the paper should still be judged on its resources, but the 'exceeds SOTA' claim would need to be substantially softened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the dataset and resources are a genuine contribution; the performance claim for GRAF is not. The headline 60.09% on promotion exams is a per-branch best-of-many mosaic. No single GRAF configuration achieves it. The best fixed GRAF config (jurBERT-large) averages 55.81%, vs 55.31% for the best fixed baseline (FLAN-T5 XXL LFT). That's 0.5 points, from a single run, which is noise. So the abstract's 'exceeds SOTA in most settings' should be read as 'competitive', not 'superior'.\n\nWhat's actually new: JuRO, CROL, Law-RoG. These fill a real gap - there was no public Romanian legal MCQA data. The corpus with temporal versions is useful for IR. The KG is a reasonable attempt. GRAF itself is a combination of known pieces (LLM claim extraction, BM25, BFS subgraph sampling, GAT, self-attention), applied to a new domain. That's fine, but it's not a new learning principle. The ablation shows both claim graph and KG contribute, which is good evidence the pieces matter.\n\nSoft spots: (1) the mosaic reporting, as above; it also affects the entrance and bar tables. (2) Single run, no error bars; given the small gaps, significance is not established. (3) In multi-answer exams, encoder models (including GRAF) get the true number of correct choices at inference (Eq 16), while LLMs must infer it. That's an asymmetry; the paper splits the evaluation but doesn't remove the advantage. (4) The KG validation is light: 5 experts on 10 documents per domain. It's a sanity check, not a thorough audit.\n\nWho's this for: people working on low-resource legal NLP, dataset builders, and anyone interested in the pitfalls of best-of-many reporting. The resources deserve to be cited.\n\nRecommendation: send to peer review, but with a request to report fixed-config results and multiple seeds, and to address the K-inference asymmetry. Without those, the method comparison is not reliable.","headline":"Genuinely useful new resources for Romanian legal NLP, but the headline performance claim is a per-branch best-of-many artifact and should not be taken as evidence that GRAF beats SOTA.","tokens_in":34568,"tokens_out":4593,"would_cite":true,"duration_ms":42713,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A graph-retrieval model, GRAF, beats LLM baselines on Romanian legal multiple-choice exams by grounding each answer in a knowledge graph.","keywords":["Romanian legal question answering","multiple-choice QA","knowledge graph","retrieval-augmented generation","graph attention network","low-resource NLP","legal dataset","information retrieval"],"falsifier":"Rerun the entrance and bar exam evaluations with the number of correct choices withheld from GRAF at inference, or included in the LLM prompt, and separate scores by whether the question has one or two correct answers. If GRAF's margin over the best LLM baseline disappears or reverses when the answer count is not supplied, then the headline advantage comes from that information rather than from graph retrieval.","tokens_in":33592,"feed_emoji":"⚖️","tokens_out":6421,"duration_ms":60462,"temperature":0.7,"pith_summary":"This paper claims that a retrieval-augmented multiple-choice QA system for Romanian law can beat strong retrieval and fine-tuned baselines by grounding answers in a knowledge graph. To test this, the authors build and release the first open Romanian legal exam dataset (JuRO), a curated law corpus (CROL), and a legal knowledge graph (Law-RoG). Their method, GRAF, turns each question-choice pair into a small claim graph, retrieves the most relevant law-graph neighbourhood, and aligns the two before scoring. On promotion exams GRAF reaches 60.09% average accuracy against 55.75% for the strongest baseline and leads in six of nine legal branches; on entrance and bar exams it also improves over encoder and LLM baselines. The wider point is that structured legal knowledge can be a viable alternative to larger language models in low-resource settings.","feed_headline":"Graph retrieval beats LLM baselines on Romanian legal exams.","feed_subtitle":"New open dataset JuRO and knowledge graph Law-RoG help GRAF reach 60% accuracy on promotion exams.","key_machinery":"The load-bearing mechanism is the GRAF pipeline built around Law-RoG, a knowledge graph of 160k nodes and 320k edges extracted from the CROL law corpus. For each candidate choice, an LLM produces a claim graph of (entity; relation; entity) triplets; a BM25-based sampler retrieves the top entities and their one-hop neighbourhoods from Law-RoG; and a Graph Attention Network with separate node and edge transformations encodes both graphs. The paper's central identity is the relevance alignment between encoded claim nodes and the sampled knowledge-graph relations --- cosine similarity $R_{ij} = \\cos(h_i^c, h_j)$ --- which carries the extracted facts into the attention head that scores the choice.","core_discovery":"GRAF's central claim is that claim-aware graph retrieval improves multiple-choice legal QA: instead of retrieving raw legal text, the model first asks an LLM to extract entity-relation-entity claims from the question and each candidate answer, samples a domain-specific subgraph from Law-RoG using BM25 and breadth-first search, then encodes nodes and edges with a relational Graph Attention Network. A cosine-similarity alignment selects the knowledge-graph relations most relevant to the claims, and self-attention combines this evidence with the question-choice encoding to produce a score. On the promotion-exam portion of JuRO, this reaches 60.09% accuracy versus 55.75% for the best LLM baseline, and the paper reports gains in six of nine legal branches, with the largest margins on civil and commercial law. The paper also reports that removing either the claim graph or the knowledge graph drops accuracy, and that simple summation of graph embeddings damages performance further.","pith_inferences":["An implication the paper leaves implicit: the claim-graph plus KG-alignment recipe could transfer to other low-resource legal systems, since the knowledge graph itself is built with an LLM rather than hand-crafted resources.","A testable extension is to withhold the correct-answer count from GRAF at inference, or give that count to the LLM baselines, to isolate whether the multi-answer advantage comes from graph retrieval or from the evaluation protocol.","The paper's LLM-based triplet extraction was only spot-checked by human experts, so a natural stress test is measuring how GRAF behaves when Law-RoG is built from a different seed model or deliberately corrupted.","The same alignment machinery could be applied to other fact-verification tasks where a candidate claim must be checked against a structured repository, not just to legal exam questions."],"forward_implications":["If GRAF's results hold, Romanian legal MCQA now has an open benchmark and a strong baseline that future methods must match.","Legal-domain pre-training is the best backbone choice, since GRAF's largest gains come from jurBERT rather than general-domain Romanian encoders.","Both the claim graph and the knowledge graph are load-bearing: ablating either one costs about two accuracy points, and losing both plus collapsing embeddings costs 6.10 points.","The released corpus and graph let other researchers do information retrieval and knowledge-graph work on Romanian law without re-scraping official portals.","GRAF's performance improves with stronger backbone encoders, so the approach should continue to improve as better Romanian language models appear."],"supporting_citations":[{"why":"Provides the LLM few-shot entity-relation extraction procedure used to build Law-RoG and the per-choice claim graphs.","marker":"(Edge et al., 2024)"},{"why":"Supplies the Graph Attention Network component that GRAF adapts with separate node and edge transformations.","marker":"(Veličković et al., 2018)"},{"why":"Supplies the BM25 retriever used to sample the most relevant entities and one-hop neighbourhoods from Law-RoG.","marker":"(Robertson and Jones, 1976)"},{"why":"Provides jurBERT, the legal-domain encoder backbone on which GRAF achieves its best results.","marker":"(Masala et al., 2021)"},{"why":"Defines the retrieval-augmented generation paradigm that serves as the strongest LLM baseline GRAF is compared against.","marker":"(Lewis et al., 2020)"},{"why":"Provides the LoRA fine-tuning method used by the best fine-tuned LLM baseline.","marker":"(Hu et al., 2022)"}],"fun_headline_variants":["GRAF graph retrieval beats LLMs on Romanian legal MCQA","Claim-aware graph retrieval tops LLMs in Romanian legal QA","Open dataset JuRO and KG Law-RoG power GRAF's legal QA win","GRAF beats LLMs on Romanian legal exams with graph retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the entrance and bar exam comparisons are fair even though GRAF and the other encoder models are told exactly how many choices are correct while the LLM baselines have to infer that number from the prompt; if that comparison is unfair, the claimed advantage on multi-answer exams is not established.","fun_headline_variants_meta":{"raw":{"variants":["GRAF graph retrieval beats LLMs on Romanian legal MCQA","Claim-aware graph retrieval tops LLMs in Romanian legal QA","Open dataset JuRO and KG Law-RoG power GRAF's legal QA win","GRAF beats LLMs on Romanian legal exams with graph retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000709,"raw_usage":{"total_tokens":3200,"prompt_tokens":956,"completion_tokens":2244,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":2178}},"tokens_in":572,"tokens_out":2244,"duration_ms":15211,"temperature":1.0,"reasoning_tokens":2178,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:43:46.664351+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the entrance and bar exam evaluations with the number of correct choices withheld from GRAF at inference, or included in the LLM prompt, and separate scores by whether the question has one or two correct answers. If GRAF's margin over the best LLM baseline disappears or reverses when the answer count is not supplied, then the headline advantage comes from that information rather than from graph retrieval.","supporting_citations":[],"review_version":1}