{"id":"43b08af0-fed8-4438-9d7d-c9524ba24366","arxiv_id":"2506.10715","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new dataset and fine-tuned language models improve adjective hypernymy prediction, extending taxonomy learning beyond nouns and verbs.","lead":"The authors create the first gold-standard dataset of English adjective hypernymy, with 302 validated pairs, and show that fine-tuning language models on it improves their ability to predict adjective hypernymy. The work could help connect the Open English Wordnet's adjective graph and support research on lexical semantics beyond nouns and verbs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4's scoring rule for multi-output models is unspecified: TaxoLLaMa-ft-multi and zero-shot return lists, yet a single P/R value is reported, so the headline improvement may be a list-length artifact.","rationale":"The dataset and theoretical discussion are useful, and the paper is transparent about limitations. However, the paper's central quantitative claim is an evaluation claim, and Table 4's protocol is internally inconsistent: a model that returns multiple hypernyms cannot be assigned a single precision/recall value without specifying the matching rule. The reader's dataset-validity concern is real, but it is secondary: even a perfect gold standard would not validate the improvement scores until the multi-output evaluation is defined. The fix is analytic and inexpensive, so conditional acceptance with required re-scoring is appropriate.","tokens_in":12614,"tokens_out":5643,"duration_ms":61477,"concrete_test":"Recompute Table 4 with one rank-aware protocol for all models: parse the full output of TaxoLLaMa-zero-shot and TaxoLLaMa-ft-multi, compute Hits@1, Hits@k and MRR against the single gold hypernym, and compare with single-output models at Hits@1. If TaxoLLaMa-ft-multi no longer outperforms TaxoLLaMa-ft-single under Hits@1 or MRR, the quantitative support for the central claim collapses. Release raw model outputs so output lengths can be checked.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 7 compares models on exact correct hypernym prediction (Table 4) and then on lists (Table 5). The text says that apart from TaxoLLaMa-Zero-shot all models output a single hypernym, but Table 5 shows TaxoLLaMa-ft-multi with P=0.15 and R=0.20, so it outputs multiple hypernyms. For Table 4, no scoring rule is given for multi-answer outputs against the single gold hypernym. If any output item counts as a hit, multi-output models are advantaged by output length; if only the first item counts, the single reported value for those rows is misleading. The headline result, that fine-tuning on the multiple dataset yields the best exact-hypernym scores (0.35/0.44 vs 0.32/0.31), therefore depends on an unstated evaluation convention. This is more immediately load-bearing than dataset noise because it affects every reported improvement, including the adaptation claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the lack of adjective hypernymy relations in the Open English Wordnet (OEWN). It proposes a substitution-based theoretical definition of adjective hypernymy, constructs a gold-standard dataset of 302 English adjective hypernym pairs by translating and validating pairs from Polish and Dutch wordnets, and fine-tunes two language models (TaxoLLaMa and SmolLM-360M-Instruct) on two versions of this dataset (a 'single' version with one exact hypernym and a 'multiple' version augmented with hypernym synonyms). The authors report improvements in part-of-speech prediction and hypernym prediction after fine-tuning, concluding that the TaxoLLaMa methodology can be adapted to adjective hypernymy. The paper also releases the dataset in RDF and the fine-tuned models on HuggingFace.","tokens_in":12826,"tokens_out":9825,"duration_ms":93858,"significance":"If the results hold, the paper contributes a novel, openly released gold-standard dataset for English adjective hypernymy, a relation absent from Princeton WordNet and OEWN, and demonstrates that large language models can be adapted to this task, with potential to extend hypernymy structure to the adjective portion of OEWN. The theoretical discussion distinguishing adjective hypernymy from noun/verb hypernymy is a useful starting point. The paper also makes its code and models publicly available, which supports reproducibility. However, the evaluation is preliminary: it relies on a small test set (91 pairs), lacks significance testing, and the scoring protocol for multi-output models is underspecified, so the reported improvements must be interpreted with caution.","major_comments":[{"comment":"Table 4's scoring rule for multi-output models is not specified. The caption states that 'apart from TaxoLLaMa-Zero-shot all the other models output a single hypernym', but this is contradicted by Table 5, where TaxoLLaMa-ft-multi has precision 0.15 and recall 0.20, indicating multiple outputs. Moreover, the text in Section 7 reports that evaluating only the first-ranked hypernym from base TaxoLLaMa's output list drops its correct predictions from 14 to 7, which implies Table 4 uses an any-match convention. If any output item is counted as a hit, TaxoLLaMa-ft-multi is advantaged by generating a longer list; if only the first item is counted, the single reported value for that row is misleading. Because the headline claim that TaxoLLaMa-ft-multi achieves the best exact-hypernym scores (0.35/0.44 vs 0.32/0.31) rests on this unstated convention, please specify the exact evaluation protocol (e.g., top-1, any-match, or set-based) and report results under both conventions for all model variants.","section":"Section 7, Table 4"},{"comment":"The evaluation is based on a test set of only 91 pairs, yet no confidence intervals, error bars, or significance tests are reported. For instance, the difference between TaxoLLaMa-ft-multi and TaxoLLaMa-ft-single in the no-definition column of Table 4 is 0.35 vs 0.32, which is well within the range of random variation for a sample of this size. The central claim that fine-tuning on the synonym-augmented 'multiple' dataset improves exact-hypernym prediction would be substantially strengthened by a bootstrap or exact binomial test, or at least by reporting the number of correct predictions per condition. Without such analysis, the observed improvements (and the apparent lack of improvement from definitions for TaxoLLaMa-ft-single) cannot be distinguished from noise.","section":"Section 7, Tables 4-5"},{"comment":"The gold-standard dataset is constructed by translating hypernym pairs from Polish and Dutch wordnets and validating them with annotators, with Cohen's kappa of 0.61-0.65 across samples and a Fleiss kappa of 0.48 for the three-annotator subsample. The paper does not report how the 'yes-maybe' cases that were retained after discussion were resolved, nor does it provide any external validation of the translated pairs against an independent English resource (e.g., dictionary definitions or a separate substitution test). Since this dataset is the basis for both fine-tuning and evaluation, a more detailed reliability analysis (e.g., per-source agreement, examples of pairs discarded at each stage, and the resolution protocol for disagreements) is needed to support the claim that the resource is a reliable gold standard for English adjective hypernymy.","section":"Section 5, Dataset creation"}],"minor_comments":[{"comment":"The MRR values reported in the text (9.4, 25.8, 23.6, 33.3) are not shown in any table, and the scale is inconsistent with the earlier mention of 54.39 (likely a percentage). Please either report MRR in a table or remove it from the text for consistency.","section":"Section 7"},{"comment":"The phrase 'before the fine-tuning' is ambiguous; it should read 'zero-shot' for the corresponding rows.","section":"Table 4 caption"},{"comment":"Use 'Cohen's kappa' (with an apostrophe) consistently instead of 'Cohen’s kappa'.","section":"Section 5"},{"comment":"'Unsloth method' should be 'Unsloth library' or 'using Unsloth'.","section":"Section 6.1"},{"comment":"The model name 'TaxoLLama' should be 'TaxoLLaMa'.","section":"Table 5 caption"},{"comment":"The future-work list (a-d) is inline and hard to read; consider formatting it as a proper list.","section":"Section 8"},{"comment":"The references McCrae et al. (2019a) and (2019b) appear to cite the same paper; please verify and merge if duplicate.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a resource/benchmark paper, but the evaluation protocol needs substantial clarification and the small test set with no significance testing weakens the central adaptation claim. The title somewhat overstates the contribution, as no hypernymy links are actually added to OEWN in this work. Please ask the authors to address the scoring-rule inconsistency and add a basic significance analysis before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this is the first English adjective hypernymy dataset I'm aware of, and the authors make a real theoretical case for why adjective hypernymy is not just noun hypernymy with different words. The resource itself is small but carefully built—302 pairs translated from Polish and Dutch wordnets, filtered through OEWN synsets, with two-annotator agreement around kappa 0.61–0.65 and a clear description of the disagreements. They release it in RDF, and the fine-tuning code and model weights are on GitHub/HuggingFace, so the empirical part is reproducible.\n\nWhat the paper does well: it identifies a concrete gap—English WordNet has no adjective hypernymy structure—and shows that fine-tuning TaxoLLaMa and SmolLM on this small dataset changes behavior. The POS prediction gains are dramatic and believable: zero-shot TaxoLLaMa often outputs nouns, after fine-tuning it outputs adjectives almost always. The definition-conditioned prompting analysis is also useful.\n\nThe soft spot is more specific than dataset noise, and the stress-test note is right: Table 4 has an undefined scoring rule. The caption says that apart from zero-shot TaxoLLaMa all models output a single hypernym, so P=R. But TaxoLLaMa-ft-multi was trained to output multiple hypernyms, and Table 5 shows P≠R for it. So the single value reported for that row is uninterpretable as either precision or recall. If any list item counts as a hit, multi-output models get a length advantage; if only the first item counts, the value needs recomputation. This directly affects the headline claim that ft-multi is best at exact hypernym prediction. The paper needs to state the scoring rule and report both any-match and first-item results.\n\nSecond, the test set is 91 pairs and the gaps between fine-tuned variants are small (0.35 vs 0.32). No error bars or significance tests. Bootstrap confidence intervals would settle whether those differences are real. The moderate kappa is a genuine limitation but the authors already acknowledge it; I don't think it undermines the resource, only the precision of the claims.\n\nBottom line: this is a solid, honest contribution for wordnet maintainers and people working on lexical relation induction. It deserves serious peer review, but the authors should fix the Table 4 evaluation before I'd trust the fine-tuning gains. A revise-and-resubmit with the scoring clarified and error bars added would be the right outcome.","headline":"A genuinely new English adjective hypernymy dataset and a thoughtful theoretical framing, but the headline fine-tuning gains rest on an unclear scoring rule for multi-output models in Table 4.","tokens_in":13318,"tokens_out":3513,"would_cite":false,"duration_ms":36861,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on a new 302-pair gold standard improves language models' adjective hypernym prediction, extending a wordnet-link method to adjectives.","keywords":["adjective hypernymy","hypernym discovery","wordnet","Open English Wordnet","language model fine-tuning","TaxoLLaMa","lexical semantics","linked data"],"falsifier":"Have a fresh set of annotators re-judge the 302 pairs from scratch in English, or test the fine-tuned models on an independently written English adjective-hypernym dataset not derived from Dutch or Polish wordnets; if agreement on the gold standard collapses or the models' advantage over zero-shot disappears, the reported gains are artifacts of the annotation and translation process.","tokens_in":12435,"feed_emoji":"🔗","tokens_out":4878,"duration_ms":50428,"temperature":0.7,"pith_summary":"This paper argues that adjective hypernymy, the 'is a kind of' relation between adjectives, can be defined, annotated, and learned, even though standard wordnets omit it. The authors build a gold-standard English dataset of 302 hyponym-hypernym adjective pairs by translating and validating relations from Dutch and Polish wordnets against the Open English Wordnet, using definitions for sense disambiguation. They fine-tune two language models on this dataset and report that the models, especially when trained with multiple synonymous hypernyms, improve at outputting adjective hypernyms and at producing adjectives rather than nouns. The intended payoff is a method for adding missing hypernym links to the adjective portion of the Open English Wordnet.","feed_headline":"302 adjective pairs boost language-model hypernym links","feed_subtitle":"Fine-tuning on a new gold standard lifts adjective hypernym prediction and fixes the noun bias in outputs.","key_machinery":"The load-bearing mechanism is the substitution-based inclusion test for adjective hypernymy, paired with sense disambiguation through Open English Wordnet definitions. A candidate hypernym is accepted when it can replace the hyponym in a sentence while preserving meaning at a general level and without contradiction; the definition of each adjective pins down the intended sense. Training turns this criterion into data: hyponym-hypernym pairs are fed to the models in a chat format with and without definitions, and the 'multiple' version of the dataset expands each hypernym to its synset synonyms so the model learns one-to-many valid answers.","core_discovery":"The central claim is that the TaxoLLaMa hypernym-discovery methodology, previously applied to nouns and verbs, transfers to adjectives when supported by a purpose-built gold standard. On the paper's own terms, adjective hypernymy is real but requires sense-level grounding: a hypernym is an adjective whose meaning includes the hyponym's meaning and that can be substituted for the hyponym in context without contradiction. The paper's 302-pair dataset operationalizes this through Open English Wordnet definitions and synset membership, and fine-tuning on it, particularly the version enriched with hypernym synonyms, raises adjective hypernym prediction from an MRR of 9.4 to 23.6 without definitions and from 25.8 to 33.3 with definitions for TaxoLLaMa, while also shifting the predicted part of speech from mostly nouns to almost always adjectives.","pith_inferences":["The small size and translation-based origin of the dataset mean the reported gains may partly reflect learning the annotation style rather than general adjective semantics; an independent English-only test set would separate the two.","The substitution criterion could be turned into a direct model probe: ask an entailment model whether replacing the hyponym with the hypernym in a sentence preserves truth, and compare those judgements with the annotated pairs.","Because best F1 on the multiple-answer evaluation reaches only 0.26 for TaxoLLaMa-ft-multi with definitions, using the model to propose new links would need human review; the immediate practical gain is the benchmark and the part-of-speech correction, not turnkey link completion.","The findings suggest adjective hypernymy may be better represented as a graded, context-dependent scale than as a strict noun-style hierarchy, which would align with how the dataset's 'maybe' disagreements cluster."],"forward_implications":["Fine-tuning on the 302-pair dataset lifts TaxoLLaMa's adjective-hypernym MRR from 9.4 to 23.6 without definitions and from 25.8 to 33.3 with definitions, and makes its outputs adjectives 95-100% of the time.","Training with multiple synonymous hypernyms consistently beats single-hypernym training, indicating that adjective hypernymy is naturally one-to-many.","Providing the hyponym's definition improves hypernym prediction in most settings, confirming word-sense disambiguation as a core ingredient.","The same fine-tuning recipe works for a much smaller 360M-parameter model, suggesting the method is accessible without large compute.","The released gold standard gives the community a first benchmark for adjective hypernymy and a basis for adding hypernym links to unlinked adjective synsets in the Open English Wordnet."],"supporting_citations":[{"why":"Supplies TaxoLLaMa, the model whose methodology the paper adapts to adjectives.","marker":"Moskvoretskii et al., 2024a"},{"why":"Defines the Open English Wordnet, the target resource and source of synsets and definitions.","marker":"McCrae et al., 2019a"},{"why":"Provides Open Dutch WordNet adjective hypernym pairs used as source data.","marker":"Postma et al., 2016"},{"why":"Provides Polish WordNet adjective hypernym pairs used as source data.","marker":"Maziarz et al., 2016"},{"why":"Provides the lexical substitution principle the annotation uses to accept or reject hypernym candidates.","marker":"McCarthy and Navigli, 2007"},{"why":"Establishes Princeton WordNet's structure and the absence of adjective hypernymy, the gap being addressed.","marker":"Fellbaum, 1998"}],"fun_headline_variants":["302 adjective pairs improve hypernym prediction in wordnets","Fine-tuned LLMs give adjectives their own hypernym links","New gold standard lifts adjective hypernym MRR to 33","TaxoLLaMa adapted for adjective hypernymy with 302 pairs","Adjective hypernyms: fine-tuning fixes noun bias in LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset built by translating Dutch and Polish wordnet pairs and manually validating them in English accurately represents English adjective hypernymy, despite annotators agreeing only moderately (Cohen's kappa about 0.61-0.65).","fun_headline_variants_meta":{"raw":{"variants":["302 adjective pairs improve hypernym prediction in wordnets","Fine-tuned LLMs give adjectives their own hypernym links","New gold standard lifts adjective hypernym MRR to 33","TaxoLLaMa adapted for adjective hypernymy with 302 pairs","Adjective hypernyms: fine-tuning fixes noun bias in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000688,"raw_usage":{"total_tokens":3051,"prompt_tokens":810,"completion_tokens":2241,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":2152}},"tokens_in":426,"tokens_out":2241,"duration_ms":16138,"temperature":1.0,"reasoning_tokens":2152,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:18:50.319185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a fresh set of annotators re-judge the 302 pairs from scratch in English, or test the fine-tuned models on an independently written English adjective-hypernym dataset not derived from Dutch or Polish wordnets; if agreement on the gold standard collapses or the models' advantage over zero-shot disappears, the reported gains are artifacts of the annotation and translation process.","supporting_citations":[],"review_version":1}