{"id":"01f8f6e1-6a59-48fe-a3ea-77b5bd61df0c","arxiv_id":"2606.21676","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CRAwLeR creates and validates Danish and Polish benchmarks for cross-reference-aware legal chunk retrieval, where current methods reach only 55-59% Recall@10 and the gap is attributed to the contextualizing LLM.","lead":"The paper introduces CRAwLeR, a benchmark and pipeline for testing chunk retrieval in legal documents that requires understanding cross-references. A smart generalist might read it to learn how to build more reliable search tools for specialized texts like laws and regulations.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Manual analysis protocol details missing, weakening support for 80% validity claim","rationale":"Matches the reader's weakest assumption exactly; the abstract-only limitation directly blocks verification of the manual analysis that underpins the 'first to consider construct validity' claim. No other internal inconsistency appears in the reported pipeline or results.","tokens_in":1745,"tokens_out":260,"duration_ms":20462,"concrete_test":"Release the manual analysis protocol including sampled query count, full annotation rubric, and Cohen's/Fleiss' kappa; recompute validity rate on a fresh blinded sample of 100 queries using the rubric—if agreement <0.65 or validity drops below 70%, the construct validity support fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that LLM-generated queries (post-adversarial filtering) genuinely demand the labelled context chunks for the narrow cross-reference phenomenon. This rests on the manual analysis result of ~80% validity. The abstract provides no information on sample size, exact annotation criteria for 'requiring context', blinding procedures, or inter-annotator agreement. Without these, the 80% figure cannot be evaluated for reliability, and systematic failure patterns cannot be assessed for bias in the validation itself.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces CRAwLeR, a pipeline for constructing datasets that operationalize context-aware chunk retrieval for legal cross-references. It detects cross-references, generates LLM queries intended to require context, applies adversarial filtering and assurance prompts, and releases CRAwLeR-DK and CRAwLeR-PL (Danish and Polish). Manual analysis is reported to find ~80% validity; benchmarks show Recall@10 of 55% (DK) and 59% (PL), with ablations attributing the gap to the contextualizing LLM rather than the retriever. The work positions itself as the first to carefully address construct validity for this narrow phenomenon.","tokens_in":1850,"tokens_out":391,"duration_ms":16055,"significance":"If the manual validation holds, the paper contributes new datasets and empirical measurements on a well-defined legal IR task, including concrete Recall@10 figures, a strong baseline, and ablations that isolate the contextualizer as the bottleneck. The emphasis on construct validity and systematic failure patterns is a positive step beyond repurposed benchmarks.","major_comments":[{"comment":"Abstract: the central claim that the queries 'genuinely require context' (and thus that the datasets demonstrate careful construct validity) rests on the ~80% manual validity rate, yet no sample size, annotation criteria for 'requiring context', blinding procedures, or inter-annotator agreement are provided. Without these, the reliability of the 80% figure cannot be assessed and systematic biases in the validation cannot be ruled out.","section":"Abstract"}],"minor_comments":[{"comment":"The abstract mentions 'failures following systematic and named patterns' but does not name them; expanding this in the main text would aid interpretability of the failure analysis.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comment on the transparency of our manual validation. We address it point by point below and will revise the manuscript to incorporate the requested details.","responses":[{"response":"We agree that the manuscript does not currently report the sample size, annotation criteria, blinding procedures, or inter-annotator agreement for the manual validation. The abstract summarizes the ~80% figure and notes systematic failure patterns, but the full methodological details are absent. We will add a dedicated subsection in the revised manuscript that specifies: the exact sample size drawn for manual review, the annotation guidelines used to determine whether a query 'genuinely requires context' (i.e., cannot be answered from the target chunk alone), whether annotators were blinded to query provenance, and any inter-annotator agreement metrics. This will allow readers to evaluate reliability and potential biases directly.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the central claim that the queries 'genuinely require context' (and thus that the datasets demonstrate careful construct validity) rests on the ~80% manual validity rate, yet no sample size, annotation criteria for 'requiring context', blinding procedures, or inter-annotator agreement are provided. Without these, the reliability of the 80% figure cannot be assessed and systematic biases in the validation cannot be ruled out."}],"tokens_in":1395,"tokens_out":301,"duration_ms":21514,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that this paper releases CRAwLeR-DK and CRAwLeR-PL, two datasets built around legal cross-references as a concrete case of context dependence in chunk retrieval. They use a pipeline that detects references, generates LLM queries, applies adversarial filtering against a non-contextual baseline, and then does manual checks claiming roughly 80% of the queries genuinely require the labelled context chunk.\n\nWhat the work does well is address a real weakness in existing benchmarks: many repurposed queries do not actually need the extra context, which makes scores hard to interpret. By narrowing to cross-references in legal documents and releasing the data with a strong contextualization baseline, they give the field something concrete to test against. The reported Recall@10 numbers (55% DK, 59% PL) and the ablation blaming the contextualizing LLM rather than the retriever are useful signals. Even when the target is retrieved, the context chunks often rank higher, which matches practical experience in legal search.\n\nThe soft spot is the manual validation step that supports the central claim. The abstract states the 80% figure but supplies no sample size, annotation criteria for \"requires context,\" blinding method, or agreement stats. Without those, it is difficult to judge how reliable the validity rate is or whether the failures follow the patterns they name. That gap directly affects how much weight the new datasets should carry.\n\nThis is for researchers building or evaluating context-aware legal IR systems and for anyone trying to improve benchmark design around genuine context needs. The new resources and the narrow focus make it worth a serious referee's time so the validation protocol can be examined and the datasets can be stress-tested by others.","headline":"CRAwLeR ships two new legal datasets that try to fix construct validity in context-aware retrieval by focusing on cross-references, but the 80% manual validation lacks the protocol details needed to assess it.","tokens_in":2322,"tokens_out":433,"would_cite":false,"duration_ms":17134,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A pipeline builds legal retrieval benchmarks where queries require cross-reference context, exposing that current models top out at 55-59% recall.","keywords":["context-aware retrieval","legal documents","cross-references","chunk retrieval","construct validity","information retrieval benchmark","Danish legal dataset","Polish legal dataset"],"falsifier":"A fresh manual review of the released queries finding substantially below 80% validity, or a retrieval system that reaches high recall without using the provided context chunks.","tokens_in":2642,"feed_emoji":"⚖️","tokens_out":705,"duration_ms":19568,"temperature":0.7,"pith_summary":"The paper establishes a focused benchmark for context-aware chunk retrieval in legal documents by operationalizing the specific phenomenon of cross-references. Existing benchmarks rarely confirm that their queries actually need surrounding context, which makes performance numbers hard to interpret. The authors create a pipeline that detects cross-references, generates demanding queries via LLM, and filters them with adversarial checks plus manual validation showing roughly 80% genuine cases. They release Danish and Polish datasets plus a contextualization baseline, demonstrating that the remaining performance gap traces to the contextualizing LLM rather than the retriever itself.","feed_headline":"Legal benchmark requires context for cross-references","feed_subtitle":"CRAwLeR datasets for Danish and Polish law reach only 55-59% recall at 10 because models fail to use provided cross-reference context","key_machinery":"The CRAwLeR pipeline that detects legal cross-references, generates LLM queries, and applies adversarial non-contextual filtering plus assurance prompts to ensure queries require the labelled context.","core_discovery":"CRAwLeR operationalizes cross-reference-aware context utilization for chunk retrieval in legal documents through a pipeline that detects cross-references, identifies query candidates, links target chunks to relevant context, generates context-demanding queries with an LLM, and filters via adversarial non-contextual baseline and assurance prompts. The resulting CRAwLeR-DK and CRAwLeR-PL datasets achieve approximately 80% validity on manual review, yet best Recall@10 reaches only 55% and 59% respectively. Ablation shows the gap stems from the contextualising LLM, not the retriever, and labelled context chunks routinely outrank the target even when it is retrieved.","pith_inferences":["The same cross-reference detection and adversarial filtering steps could be reused to create analogous benchmarks in other document domains that rely on internal references.","Future work could test whether replacing the contextualising LLM with a more capable model closes the observed gap without changes to the retriever.","The named failure patterns provide a concrete starting point for improving query generation or context handling in legal retrieval systems."],"forward_implications":["The benchmarks remain unsolved at 55-59% Recall@10 despite a strong contextualization baseline.","Performance gaps are attributable to the contextualising LLM rather than the underlying retriever.","Even when the target chunk is retrieved in the top ten, labelled context chunks routinely rank higher.","Failures follow systematic and named patterns that can be inspected for targeted fixes."],"fun_headline_variants":["CRAwLeR for cross-reference aware legal retrieval","Context demanding queries in legal retrieval datasets","Danish and Polish legal cross-reference benchmarks","Chunk retrieval requires cross-reference context awareness","Legal benchmarks test cross-reference context use"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The filtered LLM-generated queries genuinely require the labelled context chunks rather than being answerable from the target alone.","fun_headline_variants_meta":{"raw":{"variants":["CRAwLeR for cross-reference aware legal retrieval","Context demanding queries in legal retrieval datasets","Danish and Polish legal cross-reference benchmarks","Chunk retrieval requires cross-reference context awareness","Legal benchmarks test cross-reference context use"]},"model":"grok-4.3","cost_usd":0.00572,"raw_usage":{"total_tokens":2782,"prompt_tokens":773,"num_sources_used":0,"completion_tokens":64,"cost_in_usd_ticks":57199500,"prompt_tokens_details":{"text_tokens":773,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1945,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":773,"tokens_out":64,"duration_ms":14497,"temperature":1.0,"reasoning_tokens":1945,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T12:35:35.897668+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A fresh manual review of the released queries finding substantially below 80% validity, or a retrieval system that reaches high recall without using the provided context chunks.","supporting_citations":[],"review_version":1}