{"id":"3377f4c2-7bab-4b06-9b5d-3881936fdd8b","arxiv_id":"2607.29553","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Metric-derived context improves relation-type prediction and data-property assignment for ontology extension on CertGraph, but degrades parent-class prediction.","lead":"COntExt suggests how new concepts and properties found in operational metric definitions (like YAML security rules) should be added to an existing ontology, by predicting parent classes, relation types, and data-property owners. On the CertGraph cybersecurity ontology, adding metric description and comment text improved relation-type prediction (MRR 0.21→0.27) and data-property assignment accuracy (0.71→0.82), but hurt parent-class prediction (MRR 0.49→0.34).","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ2's headline gains rest on selected metric fields and 10/28 test cases with no significance test; the claim of improvement over ontology context is not yet established.","rationale":"The reader identified the single-corpus generalizability of RQ2 as the weakest assumption. I agree that this is a limitation, but I find a more immediate and load-bearing problem: the RQ2 result is not shown to be statistically reliable even on CertGraph. The test sets are tiny, no case-level permutation test is applied, and the reported gains select different metric fields per task. The framework itself is honest, reproducible, and useful, and the paper explicitly discloses the single-corpus limitation in Section 5.2, so this is not an objection to the authors' integrity. The issue is evidenciary: the abstract's 'Our results show that metric-derived context improves...' overstates what the measurements support.\n\nBecause the paper already has a CONDITIONAL verdict, my recommendation is to keep that verdict rather than move to ACCEPT or REJECT. The condition should be sharpened: before the central claim can be accepted, the authors should run paired significance tests on RQ2 and either pre-specify the metric-context condition or correct for multiple comparisons. If those tests fail, the central claim would need to be significantly weakened. If they pass, the remaining single-corpus concern still justifies CONDITIONAL pending replication on a second ontology/metric-corpus pair. Thus 'UNCHANGED' reflects that the reader's CONDITIONAL verdict remains appropriate, though with a more precise rationale than the one stated.","tokens_in":15769,"tokens_out":6525,"duration_ms":67211,"concrete_test":"Re-run RQ2 from the released repository, extracting per-case reciprocal ranks (Task 2, n=10) and per-case accuracies (Task 3, n=28) for all context variants. Apply the same paired two-sided approximate-randomization test defined in Section 4.4 to the pre-registered comparison 'all metric-derived context vs. ontology comments', and also to each individual metric field with Holm-Bonferroni correction across the field variants. If the all-context comparison yields p ≥ 0.05, or if the apparent gain is driven only by the best selected field, the central claim that metric-derived context improves over ontology-context baselines is unsupported even on CertGraph.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that metric-derived context improves suggestions over ontology-context baselines for relation type prediction and data property assignment—rests entirely on RQ2 on CertGraph (Tables 6 and 7). Task 2 improves from MRR 0.21 to 0.27 with metric description; Task 3 improves from 0.71 to 0.82 with metric comment. These are the two numbers cited in Section 5.1. But the evidence is internally fragile in three ways.\n\nFirst, the test sets are tiny: 10 relation cases and 28 property cases (Section 4.6.2). A 6 pp MRR gain over 10 cases can be driven by one or two cases; an 11 pp accuracy gain over 28 cases can be driven by about three cases. Second, no significance testing is reported for RQ2. Section 4.4 defines a paired approximate-randomization test at the level of masked cases, but it is applied only to RQ1. The paper's justification—'no setup for multiple random seeds'—is not a valid reason to omit significance testing, since the defined test is paired at the case level and does not require training seeds.\n\nThird, the comparison is not pre-specified. For each task, three metric-context variants are evaluated (description, comment, description+comment, plus all context), and the conclusion selects the best-performing field per task. Under the more natural 'all context' condition, the gains shrink to +0.01 MRR for Task 2 (0.21→0.22) and +0.04 accuracy for Task 3 relative to ontology comments (0.64→0.68). Moreover, ontology comments beat or tie metric-only variants on several rows: Task 2 Hits@5/Hits@10 scores with ontology comments are 0.40/0.50 versus 0.30/0.40 with metric description. Thus the headline effect may be a multiple-comparisons artifact on a very small sample. The single-corpus generalization concern noted in Section 5.2 is real, but it is secondary: the effect on CertGraph itself is not yet statistically established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents COntExt, an algorithm-agnostic framework that uses structured operational metric definitions (e.g., YAML metric catalogues) as contextual input for ontology extension. The extension problem is decomposed into three sub-tasks: parent class prediction, relation type prediction, and data property assignment. The authors evaluate several algorithms for each task on seven ontologies spanning four domains (RQ1), and then evaluate whether adding ontology- or metric-derived context improves suggestion quality on the CertGraph ontology and its associated security-metric corpus (RQ2). The paper's central claim is that metric-derived context improves suggestions over ontology-context baselines for relation type prediction and data property assignment, while it degrades parent class prediction.","tokens_in":16159,"tokens_out":3685,"duration_ms":38292,"significance":"If the central claim is validated, the paper demonstrates a practical, low-friction source of context for ontology maintenance: organizations that already maintain structured metric catalogues could reuse them to semi-automatically extend their ontologies. The paper has several strengths: the framework decomposition is clear; the source code is publicly available; RQ1 uses paired approximate-randomization tests and honestly reports that most algorithm differences are statistical ties rather than overclaiming; and the single-pair limitation for RQ2 is explicitly acknowledged in Section 5.2. However, the load-bearing RQ2 evidence is thin, is not significance-tested, and is limited to one (ontology, metric corpus) pair developed by the authors' group. The significance of the work therefore depends on strengthening the statistical support and on appropriately scaling the claims.","major_comments":[{"comment":"RQ2's central claim is not supported by significance testing. The paper defines a paired approximate-randomization test in §4.4 that is paired at the level of masked cases and does not require multiple training seeds. The statement in §4.6.2 that there is 'no setup for multiple random seeds' therefore does not justify omitting the test. With only 10 Task 2 cases and 28 Task 3 cases, the reported 0.06 MRR gain (0.21→0.27) and 0.11 accuracy gain (0.71→0.82) can be driven by one to three cases. The authors should run the defined permutation test on RQ2, report per-case differences, and provide confidence intervals, or explicitly label these results as exploratory.","section":"§4.6.2, Tables 6–7"},{"comment":"The headline gains are selected post hoc from four metric-context variants. The conclusion chooses '+ Metric description' for Task 2 and '+ Metric comment' for Task 3. Under the framework's own 'All context (metric + ontology)' condition, the gains essentially disappear: Task 2 MRR goes from 0.21 to 0.22 (+0.01), and Task 3 accuracy is 0.68, below the 0.71 no-context baseline and only +0.04 over ontology comments. Because multiple context variants are compared without correction, the abstract's claim that 'metric-derived context improves the suggestions' is not established by the selected best-field comparison. The authors should report all variants with the §4.4 significance test, or treat the per-field gains as hypothesis-generating.","section":"§4.6.2, Tables 6–7; §5.1"},{"comment":"The RQ2 evidence rests on a single (ontology, metric corpus) pair: CertGraph and the authors' own security-metric catalogue (Schöberl et al., 2024). The test cases are exactly the concepts mentioned in those metric files. This creates a real risk that the measured benefit reflects the particular YAML schema, prose style, and author-aligned terminology of that one corpus, rather than a general property of operational metric catalogues. Section 5.2 acknowledges the limitation, but the Abstract and Conclusion nevertheless state the general claim that 'operational metric catalogues are a practical and underexploited source for ontology extension.' The authors should either add at least one independent (ontology, metric corpus) pair or restrict the central claim to a proof-of-concept for CertGraph-like catalogues.","section":"§4.3, §4.6.2, §5.2; Abstract"}],"minor_comments":[{"comment":"The abstract in the submitted text says 'across seven ontologies spanning four domains,' while the shorter version in the provided metadata says 'across four cybersecurity ontologies.' The abstract should match the body of the paper.","section":"Abstract"},{"comment":"The sentence 'there is no setup for multiple random seeds' is misleading because the heuristic methods (ChildAgg and MLM prompting) are deterministic in inference mode; the issue is not seed variance but missing case-level significance testing.","section":"§4.6.2"},{"comment":"Typo: 'efficiently' and 'efficiency' use a non-standard 'ff' ligature and should be 'efficiently' and 'efficiency'.","section":"§5.2"},{"comment":"The table lists 'Classes with Attributes' counts but the body of §4.6.1 explains exclusions only in prose; a short note under Table 1 or Table 4 would clarify why Pizza, CSOsec, and FIBO_BE are absent from Task 3.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a coherent framework and an honest RQ1 evaluation, but the central metric-context claim needs substantially stronger statistical support and a more careful scope. I would be willing to accept after the authors run the existing paired permutation test on RQ2, report all context variants rather than selected best fields, and adjust the abstract/conclusion to reflect the single-corpus limitation. The single-corpus issue is not by itself fatal if framed as a proof of concept, but the current general wording overstates the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one-sentence take: COntExt is a useful, clearly-written framework and the RQ1 evaluation is more honest than most, but the headline — that metric context improves relation and property suggestions — is not established by the evidence given.\n\nWhat's new: using structured operational metric catalogues as context for ontology extension is a genuine gap, and COntExt decomposes it cleanly into parent, relation, and property tasks. The code is public. RQ1 deserves real credit: paired approximate-randomization tests with Holm correction, and most comparisons are reported as statistical ties rather than overclaimed wins. That is rare, and it makes the algorithm comparison useful even where the scores are modest.\n\nThe soft spot is RQ2, which carries the abstract's main claim. It is run only on CertGraph and its companion metric catalogue — the limitation is stated in Section 5.2, so at least it is transparent. The test sets are 10 relation cases and 28 property cases. The reported gains of +6pp MRR and +11pp accuracy are single numbers with no significance testing, and the stated reason (no multi-seed setup) misses the point: their own permutation test is paired per case and does not require different training seeds. The stress-test note is right that the comparison is not pre-specified. The authors select the best metric field per task, and the 'all context' variant shrinks the gains to +0.01 MRR and +0.04 accuracy. Ontology comments also beat metric-only variants on several rows. So the central claim is not statistically established even on CertGraph, let alone across domains.\n\nThere is still a decent paper here. The negative parent-class result is an honest finding, the framework is modular, and the limitations are stated rather than hidden. With significance testing, a pre-specified context condition, and RQ2 repositioned as an exploratory case study, the claim could become defensible.\n\nWho should read it: people working on ontology completion, knowledge-graph maintenance, or regulated-domain metric catalogues will get value from the framework comparison and the public implementation. I would send it to a serious referee, but with the expectation that RQ2 needs major rework before the headline is accepted.","headline":"Useful framework and an honestly-written RQ1, but the headline claim rests on 10/28 test cases, no significance test, and one self-referential corpus.","tokens_in":16677,"tokens_out":2465,"would_cite":false,"duration_ms":60359,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structured metric definitions can be reused as context to extend an ontology, and the paper shows that metric-derived text beats ontology-only context for relation type prediction and data property assignment on a cybersecurity use case.","keywords":["ontology extension","context enrichment","operational metrics","metric catalogues","parent class prediction","relation type prediction","data property assignment","cybersecurity ontology"],"falsifier":"Run the same leave-one-out protocol on a second (ontology, metric catalogue) pair in a different domain—for example an IoT ontology paired with its metric definitions—and check whether metric description/comment still beats ontology-only context on relation-type MRR and data-property accuracy. If the gains disappear, the central claim is specific to CertGraph; alternatively, remove the configuration block from metric context and see whether the data-property accuracy gain from 0.71 to 0.82 collapses.","tokens_in":15685,"feed_emoji":"🧩","tokens_out":6525,"duration_ms":62273,"temperature":0.7,"pith_summary":"The paper tries to establish that the operational metrics an organization already writes down—YAML files describing how to measure security, compliance, or performance—can be reused to keep its formal ontology up to date. COntExt splits ontology extension into three prediction tasks and injects metric text (descriptions, comments, configuration values) as context into language-model scorers. On the CertGraph cybersecurity ontology, metric-derived context outperformed ontology-only context for relation type prediction (MRR 0.21 → 0.27) and data property assignment (accuracy 0.71 → 0.82), but it degraded parent class prediction (MRR 0.49 → 0.34). The broader claim is that operational metric catalogues are an underexploited bridge between operational knowledge and formal ontologies, allowing lower-cost, semi-automated ontology maintenance. A sympathetic reader would note that the context-enrichment evidence rests on a single (ontology, metric) pair with very small test sets, but the direction is new and the framework itself is algorithm-agnostic.","feed_headline":"Metric files give ontology updates a context boost","feed_subtitle":"COntExt shows YAML metric comments lift data-property accuracy from 71 to 82 percent on a cybersecurity ontology.","key_machinery":"The carrying mechanism is context enrichment: before scoring, the framework concatenates a concept's bare name with text from the metric definition (description, comments, configuration values) or from ontology annotations, and feeds the resulting prompt into a language-model scorer such as ChildAgg or masked-language-model prompting. The framework itself is algorithm-agnostic; its decomposition of ontology extension into parent class prediction, relation type prediction, and data property assignment is what lets the same metric corpus serve different extension needs. The prompts act as the bridge that turns prose in YAML metric files into ranking signals.","core_discovery":"COntExt formalizes ontology extension as three sub-tasks—parent class prediction, relation type prediction, and data property assignment—and treats metric definitions as a first-class input. Its key empirical claim is that enriching suggestions with metric-derived context (descriptions, comments, configuration values) improves ranked suggestions over both no-context and ontology-comment baselines for relation type prediction and data property assignment, while the same context hurts parent class prediction. The evidence comes from a leave-one-out protocol on CertGraph: metric description raises relation-type MRR from 0.21 to 0.27, and metric comment raises data-property accuracy from 0.71 to","pith_inferences":["Editorial inference: the parent-class degradation is probably a context-type mismatch—metric prose describes relations and configurations, not subclass placement—so a production system could route each task to the context source that best matches its semantics.","Editorial inference: the data-property gains may be partly driven by configuration values (booleans, thresholds) leaking datatype information into the prompt; ablating the configuration block would separate prose effects from typing clues.","Editorial inference: since only template-based scorers were context-enriched, adapting structural methods (e.g., translating-embedding models) to ingest metric prose is a natural next test; if they also gain, the result is not an artifact of masked-language-model prompting.","Editorial inference: the hand-designed prompt templates could be replaced by automatically constructed prompts from metric fields, which would let the framework transfer to new domains without manual template tuning."],"forward_implications":["Organizations that already maintain metric catalogues can feed them through COntExt to get ranked suggestions for where a new class belongs, how it relates to existing classes, and which class owns a property—starting ontology updates from suggestions instead of from scratch.","The framework separates the extension problem into three tasks, so teams can adopt it selectively where metric context helps (relations and properties) and rely on structure-only methods where it hurts (parent placement).","Because COntExt is algorithm-agnostic and accepts any formal ontology, the same metric-to-context enrichment can be reused with better scorers or LLM-based components as they mature.","If the effect replicates beyond CertGraph, metric catalogues become a first-class input to ontology evolution, removing the need for engineers to re-transcribe operational knowledge into formal axioms."],"fun_headline_variants":["Metric context lifts ontology relation and property predictions","Operational metrics as a goldmine for ontology extension","Metric comments raise ontology data-property accuracy to 82%","Context from metrics helps two of three ontology tasks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is that the metric-context benefit—measured on a single pair, the CertGraph ontology and its 64-metric cybersecurity catalogue, with only 10 relation cases and 28 property cases—generalizes to other ontologies and metric corpora; if CertGraph's YAML schema and prose style are atypical, the claimed improvement could be an artifact of that one corpus.","fun_headline_variants_meta":{"raw":{"variants":["Metric context lifts ontology relation and property predictions","Operational metrics as a goldmine for ontology extension","Metric comments raise ontology data-property accuracy to 82%","Context from metrics helps two of three ontology tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000646,"raw_usage":{"total_tokens":2788,"prompt_tokens":710,"completion_tokens":2078,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":2016}},"tokens_in":454,"tokens_out":2078,"duration_ms":14004,"temperature":1.0,"reasoning_tokens":2016,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:43:38.341162+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same leave-one-out protocol on a second (ontology, metric catalogue) pair in a different domain—for example an IoT ontology paired with its metric definitions—and check whether metric description/comment still beats ontology-only context on relation-type MRR and data-property accuracy. If the gains disappear, the central claim is specific to CertGraph; alternatively, remove the configuration block from metric context and see whether the data-property accuracy gain from 0.71 to 0.82 collapses.","supporting_citations":[],"review_version":1}