{"id":"9c0a4e31-00e6-494e-a404-8e0dc2728226","arxiv_id":"2607.27792","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"LeDA is a legal annotation tool with dynamic tag creation and adjudication, used to tag 200 Indian Supreme Court cases with thematic concepts like 'Murder on parole.'","lead":"A team built a web tool called LeDA for labeling legal case documents with thematic concept tags, such as 'murder during parole.' It is designed to capture the gist of a court case in a structured way that keyword searches miss, and was tested on 200 Supreme Court of India rulings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IAA computation is load-bearing for dataset reliability but is left unspecified and unreported, so the claimed reliable annotation is unsupported.","rationale":"The strongest claim is that LeDA provides a usable workflow and an annotated dataset for thematic legal concepts. For that claim to hold, the annotations must be reliable. The paper's only evidence for reliability is the IAA computation, which is not formally specified and for which no scores are reported. The reader's weakest assumption identifies exactly this issue. I agree: without a precise definition of the IAA measure and actual computed values, the dataset quality cannot be assessed. The presence of a GitHub link is a positive sign, but the paper does not state that the IAA algorithm is included in the repository. Secondary issues such as the novelty of dynamic tags and the unreleased dataset reinforce the need for conditions, but they are less central than the missing IAA specification. Therefore the appropriate verdict remains conditional: the paper should be accepted only if the authors provide the IAA algorithm and the empirical agreement scores, or an equivalent standard metric, and ideally release the dataset or a sample for verification.","tokens_in":4807,"tokens_out":3697,"duration_ms":34308,"concrete_test":"Clone the linked GitHub repository (github.com/subinayadhikary/LeDA), locate the IAA computation module, and execute it on the raw JSON annotations produced by the two legal experts for the 200 documents. Require that the output includes per-document and aggregate IAA scores. Independently compare these scores against a standard multi-label agreement measure (e.g., Cohen's kappa or Krippendorff's alpha on span–tag units). If the module is absent, cannot be run, or yields scores below 0.7, the claim of reliable annotation is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central value is the claimed 200-document thematic annotation dataset. The only quantitative quality check described for that dataset is the IAA computation in Section 3. However, the method is described only in prose: 'three key components: i) the annotated span, ii) the tags used to label that span, and iii) the sentence number of the span.' No formula, no pseudocode, no threshold, and no reported scores are given. The paper acknowledges that 'It posed a considerable challenge to devise a method that would yield an accurate IAA measure,' yet it never specifies how that challenge was resolved. Because the dataset is the basis for the proposed bag-of-concepts representation, an unformulated and unverified agreement measure leaves the central contribution unsupported. Secondary issues—overstated novelty of dynamic tags in light of Table 2, the dataset not being released, and user satisfaction asserted without an instrument—also matter, but the IAA gap is the most load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LeDA, a web-based legal data annotation system designed to annotate thematic concepts in legal case proceedings. Its main claimed features are the dynamic creation of new tags during annotation, a super-annotator adjudication workflow, and the computation of inter-annotator agreement (IAA). The authors report using LeDA with two legal experts and one senior expert to annotate 200 Indian Supreme Court case proceedings with concept tags such as 'Murder on parole' and 'Second murder', producing a bag-of-concepts representation intended for downstream tasks like prior case retrieval and judgment prediction. The paper includes a feature comparison with existing tools, a workflow description with screenshots, a case study, and a plan for future work.","tokens_in":5043,"tokens_out":2218,"duration_ms":21979,"significance":"If the claims hold, the paper demonstrates a functioning annotation tool with useful workflow features for legal concept annotation, and it provides a descriptively curated set of thematic tags for Indian Supreme Court proceedings. The dynamic tag mechanism addresses a genuine need in settings where no predefined ontology exists. The tool is publicly available on GitHub, which supports reproducibility of the tool itself. However, the paper's central dataset-quality claim rests on an IAA computation that is not specified, and no quantitative annotation-quality results are reported. The paper is best viewed as a system demonstration; its contribution would be strengthened substantially by formalizing and validating the IAA method and reporting actual agreement scores.","major_comments":[{"comment":"The IAA computation is load-bearing for the claim that the 200-document dataset is reliably annotated, but it is described only verbally. The manuscript states that IAA uses 'i) the annotated span, ii) the tags used to label that span, and iii) the sentence number of the span,' but it gives no formula, pseudocode, threshold, or reported IAA values. It also acknowledges that 'devising a method that would yield an accurate IAA measure' was challenging, yet it does not explain how the challenge was resolved. Without a precise definition and numerical results, the reader cannot verify that the agreement measure is meaningful or that the subsequent adjudication was based on valid scores. Please specify the algorithm, justify its correctness (e.g., by comparing it with established metrics like Cohen's kappa or Krippendorff's alpha), and report the computed IAA scores for the 200 documents.","section":"Section 3, 'Analysis of annotated data'"},{"comment":"The text repeatedly emphasizes that dynamic tag creation is a 'central and innovative feature' and that existing tools lack critical features for legal annotation. Yet Table 2 marks 'Dynamic tag' as present in BRAT, GATE, Label Studio, and UBIAI as well as LeDA. This apparent contradiction undercuts the novelty claim. Please clarify what specifically distinguishes LeDA's dynamic-tag mechanism from those tools (e.g., request/approval workflow, multi-level permissions, integration with adjudication) and revise the table or the text to be consistent.","section":"Table 2 and Section 2"},{"comment":"The paper asserts that 'the feedback we received on LeDA's features was highly satisfactory' and that 'no new feature suggestions were made, affirming the tool's effectiveness.' No details of the feedback collection are provided: no questionnaire, no number of respondents, no summary of responses. As stated, this is an unsupported claim of user satisfaction. Either present the evaluation instrument and results or remove the claim and limit the discussion to the system's functionality.","section":"Section 2, 'Key features of LeDA'"}],"minor_comments":[{"comment":"Typo: 'allows to dynamic create' should be 'allows dynamic creation' or 'allows annotators to dynamically create.' Also, 'put forward a set of diverse concepts' is vague; consider listing examples or referencing Table 1.","section":"Abstract"},{"comment":"Tag names such as 'Murderon parole' and 'Secondmurder' lack spaces, which harms readability. Consider formatting them with spaces or adding separators to match the prose descriptions.","section":"Table 1"},{"comment":"Reference [1] is incomplete: it lacks a publication venue and year. Reference [10], cited for IAA, appears to describe rhetorical role identification rather than an annotation agreement measure; please verify that this citation supports the IAA claim.","section":"References"},{"comment":"The figure is dense and the workflow labels (A–N) are not all explained in a single list. A step-by-step walkthrough of the figure, or a clearer schematic, would aid comprehension.","section":"Figure 2"},{"comment":"The description of the dataset's scope is narrow ('murder-related case proceedings'), yet the abstract claims a general 'set of diverse concepts for legal case proceedings.' Please clarify whether the concept set is intended to be domain-specific or generalizable.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper falls within the scope of a system/demo venue, and the tool itself appears usable. The main concern is the unformalized IAA measure and the absence of any reported agreement numbers; because the dataset is a central contribution, this is a load-bearing gap. The feature-comparison inconsistency also needs attention. With these fixed, the paper could be an acceptable system demonstration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper describes LeDA, a web-based annotation tool for legal concepts, and a 200-document annotation of Indian Supreme Court case proceedings. As a system demo it's usable, but the load-bearing piece—the inter-annotator agreement computation—is left as a verbal sketch with no formula and no results, so the dataset's reliability is unverified.\n\nWhat's genuinely new: the specific tag set for murder-related Indian cases (Murder on parole, Evidence insufficiency, etc.) and the descriptive observation that annotated spans often cross sentence boundaries and carry multiple tags. That last point is actually useful for anyone doing sequence labeling on legal text. The tool appears functional, with dynamic tag creation, a dual-view adjudication interface, and remote access; screenshots and a GitHub link support that. The workflow—two annotators plus a senior adjudicator—is standard but sound.\n\nThe soft spots are in proportion. The IAA method is the big one. Section 3 names three components (span, tag, sentence number) but gives no algorithm, no threshold, no reported scores. The paper says devising an accurate IAA measure posed a considerable challenge, then never says how the challenge was resolved. This matters because the only quantitative quality check for the 200-document dataset is IAA, and without numbers or even a precise method, the reliability claim is unsupported. Second, the dataset is not released or linked anywhere, so the main artifact is not verifiable. Third, the novelty claims for dynamic tags are overstated—their own Table 2 shows Label Studio and UBIAI already have that feature. User satisfaction is also asserted without any instrument, despite the oddly convenient claim that no new feature suggestions were made.\n\nNone of this means the tool doesn't work or the tag set is worthless. The descriptive analysis is plausible and the tool likely does what the screenshots show. But the paper as written is a demo in need of real revision, not a finished research contribution.\n\nI'd still send it to peer review rather than desk reject. The annotation workflow and the tag set have value for the legal NLP community, and a competent referee could push the authors to formulate the IAA method and release at least a sample of the data. If those fixes land, this becomes a solid demo paper. Without them, it's a screenshot deck with an unverifiable dataset.","headline":"The tool is probably real and the tag set is a useful small artifact, but the paper's central quality claim—the IAA measure—is never specified or reported, and the dataset isn't released.","tokens_in":5477,"tokens_out":2538,"would_cite":false,"duration_ms":24178,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Legal case proceedings can be captured as bags of thematic concepts rather than raw text, and the LeDA annotation system makes this practical by letting annotators create new concept tags while they read.","keywords":["legal data annotation","thematic concepts","dynamic tag creation","inter-annotator agreement","Indian Supreme Court case proceedings","bag-of-concepts representation","adjudication workflow","prior case retrieval"],"falsifier":"Compute a standard agreement coefficient (e.g., Cohen's kappa) on a sample of the 200 annotated judgments, comparing both annotators' span-level labels span by span. If those scores come out low (say below 0.5) for documents where LeDA's own IAA appears high, the novel IAA measure is not tracking agreement; and since no IAA scores are reported, the dataset's reliability claim currently has no observable check.","tokens_in":4743,"feed_emoji":"⚖️","tokens_out":4423,"duration_ms":39318,"temperature":0.7,"pith_summary":"This paper is trying to establish that the 'aboutness' of a legal case—what the case is thematically about, such as a murder committed during parole—can be captured through human annotation with concept tags, and that this structured representation is more useful than a flat bag of words for downstream legal tasks. To make this feasible, the paper presents LeDA, a web-based annotation system whose defining feature is that annotators can dynamically add new tags when existing ones do not cover a concept; a super-annotator then adjudicates conflicts and computes inter-annotator agreement. The system was used by legal experts to annotate 200 Indian Supreme Court case proceedings, building a dataset of concept-labeled spans. A sympathetic reader would care because this is a concrete workflow for producing structured legal datasets in a setting where no ontology exists in advance, and because such datasets are prerequisites for prior-case retrieval and judgment prediction.","feed_headline":"Dynamic tags let annotators capture legal case themes on the fly","feed_subtitle":"A web tool and 200 annotated Supreme Court judgments replace bag-of-words with bag-of-concepts for retrieval and prediction.","key_machinery":"The load-bearing artifact is LeDA, an open web annotation tool whose novel mechanism is dynamic tag creation: an annotator encountering a concept not in the initial tag list can request a new tag from the super-annotator, so the concept vocabulary evolves as documents are read. Around this, the workflow includes independent annotation by multiple annotators, a dual-view adjudication step where a senior annotator reconciles conflicting labels, and an inter-annotator agreement measure built from three components: the text span, the tags labeling it, and the sentence number of the span. This machinery is what converts raw case text into a bag-of-concepts representation.","core_discovery":"The paper's central claim is that a small team of legal annotators, working with an evolving tag set, can label the fine-grained topics of case proceedings well enough to represent documents as bags of concepts. The authors argue that static tag lists fail for legal texts because the relevant concepts are not atomic entities but thematic happenings—'Murder on parole,' 'Second murder,' 'Testimony challenged'—that surface only during reading. LeDA therefore supports dynamic tag creation, a dual-view adjudication interface for the super-annotator, and a bespoke inter-annotator agreement computation that compares the annotated span, the tags attached to it, and the sentence number where the span","pith_inferences":["A quantitative test of the paper's thesis would be to compare prior-case retrieval using bag-of-concepts representations against a bag-of-words baseline on the same corpus; the paper does not report such an experiment.","The inter-annotator agreement metric is described only by its three inputs; without a formula or reported IAA scores, the reliability of the dataset is not yet independently checkable. Computing a standard coefficient like Cohen's kappa on the same spans would settle whether the measure captures genuine agreement.","The dynamic tag process suggests that after enough documents the evolving ontology may converge to a stable taxonomy, which could be reused as a seed ontology for automated concept extraction—an extension the authors mention as future work.","The same annotation workflow could transfer to other legal systems, but the concept tags derived from Indian Supreme Court murder jurisprudence would likely need to be re-derived for different jurisdictions or domains."],"forward_implications":["Annotated case documents can be represented as bags of thematic concepts, capturing the aboutness that phrase extraction misses.","The dynamic-tag feature makes annotation feasible in domains without a predefined ontology, since the concept set can grow during annotation.","The 200-document Indian Supreme Court dataset, with spans labeled by legal experts and adjudicated by a senior expert, becomes a resource for training and evaluating downstream legal NLP tasks.","The workflow's inter-annotator agreement and adjudication steps provide a mechanism to detect and resolve disagreements, supporting dataset quality.","If the representation is adopted, prior-case retrieval and judgment prediction can operate on concepts rather than on flat text, which the authors argue should improve processing."],"fun_headline_variants":["Legal case topics emerge as annotators tag on the fly","LeDA lets annotators craft tags while reading case law","From bag-of-words to bag-of-concepts for legal docs","Dynamic tagging uncovers legal case themes in court proceedings","Annotate and adjudicate legal concepts with ever-evolving tags"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the paper's inter-annotator agreement measure—based on the annotated span, its tags, and its sentence number—actually reflects annotation quality; the paper neither specifies the measure's formula nor reports any IAA scores, so the claimed reliability of the dataset rests on an unformalized metric.","fun_headline_variants_meta":{"raw":{"variants":["Legal case topics emerge as annotators tag on the fly","LeDA lets annotators craft tags while reading case law","From bag-of-words to bag-of-concepts for legal docs","Dynamic tagging uncovers legal case themes in court proceedings","Annotate and adjudicate legal concepts with ever-evolving tags"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000821,"raw_usage":{"total_tokens":3430,"prompt_tokens":747,"completion_tokens":2683,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":2600}},"tokens_in":491,"tokens_out":2683,"duration_ms":16794,"temperature":1.0,"reasoning_tokens":2600,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T01:16:35.830534+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute a standard agreement coefficient (e.g., Cohen's kappa) on a sample of the 200 annotated judgments, comparing both annotators' span-level labels span by span. If those scores come out low (say below 0.5) for documents where LeDA's own IAA appears high, the novel IAA measure is not tracking agreement; and since no IAA scores are reported, the dataset's reliability claim currently has no observable check.","supporting_citations":[],"review_version":1}