{"id":"e18d266f-8364-4a65-8818-52c543605746","arxiv_id":"1908.06676","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"EDAM, an ontology-driven and expert-refined pipeline, classifies primary studies for systematic reviews with agreement comparable to senior human annotators in a 25-paper sample.","lead":"This paper proposes EDAM, a methodology that automatically builds and refines a domain ontology to classify papers for systematic reviews, replacing manual keywording. It tests EDAM on software architecture and reports that its paper classifications are statistically indistinguishable from senior researchers, although the evaluation sample is small and selected for easy cases.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation only samples EDAM's unambiguous single-topic assignments, so the claim of equivalence to senior researchers is untested on the multi-label, ambiguous cases that arise in real systematic reviews.","rationale":"The paper is a serious, reproducible proposal: the pipeline, data, and evaluation materials are released, and Section 3's earlier manual-vs-automatic comparison provides complementary qualitative evidence that automatic SRs can find missed studies. The central claim, however, is specifically about classification equivalence, and the evaluation in Section 5.1 is built around EDAM's best cases. The reader's weakest assumption identified this selection bias; I agree and would add that the statistical treatment (chi-square p=0.77) is not an equivalence test, so even within the selected sample the paper does not show interchangeability, only lack of detected difference. Neither issue is fatal to the methodology as a proposal; both are addressable with a representative evaluation and a pre-specified equivalence margin. The paper's own 'simple not-ambiguous papers' remark and 'preliminary' framing support keeping a conditional verdict rather than rejecting the work. Since the reader already judged this CONDITIONAL, my read does not change the verdict.","tokens_in":21620,"tokens_out":5852,"duration_ms":62472,"concrete_test":"Re-run the Section 5.1 evaluation on a random sample of 100 papers from DSA without conditioning on EDAM's output, using the full 46-topic SA ontology and allowing multiple labels per paper. Have the six original annotators and EDAM label the sample; compute per-item F1 and Fleiss' kappa for EDAM versus the senior group, and run a two one-sided equivalence test (TOST) with margin equal to the observed senior-human inter-annotator variability (e.g., the pairwise agreement or kappa among senior annotators). If the EDAM-senior confidence interval falls outside that margin, or if performance is significantly lower on multi-label or ambiguous papers than on single-label papers, the central claim must be restricted to the easy-case subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 restricts the evaluation to five 'unambiguous categories' and to papers 'classified by EDAM exclusively under that topic'. This conditions the sample on exactly the cases where EDAM's exact-term mapping is at its best, and it silently drops ambiguous, multi-topic, and previously misclassified papers. Yet EDAM's real output in step 6 of Section 4.2 is explicitly multi-label ('it allows us to associate multiple categories to the same paper'), over a 46-topic ontology. The reported p=0.77 therefore supports 'not statistically different from the senior group' only for an easy subset, not for the actual classification workload. The authors appear to concede this when they write that EDAM 'seems to perform well in handling simple not-ambiguous papers' (Section 5.1). A compounding problem is that a chi-square 'not significant' on n=25 is not an equivalence test: failing to reject the null hypothesis does not establish that EDAM is interchangeable with senior experts. Both issues are external-validity and methodological, not internal contradictions, but together they mean the abstract's unqualified conclusion overstates what the evaluation demonstrates.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EDAM, an Expert-Driven Automatic Methodology for assisting systematic reviews in software engineering. EDAM replaces the manual keywording and data-extraction steps of a mapping study with (i) automatic ontology learning from a scholarly dataset, (ii) human refinement of that ontology, and (iii) automatic classification of primary studies against the refined ontology. The methodology is instantiated for the Software Architecture area using the Klink-2 algorithm on a Scopus dump, producing a 46-topic Software Architecture sub-ontology. The authors evaluate the classification step by comparing EDAM's annotations on 25 papers with those of six human experts, report that EDAM is not statistically different from the senior expert group (p=0.77), compare several unsupervised classifiers on a separate 70-paper gold standard, and discuss limitations, reuse scenarios, and implications for systematic mappings. The paper concludes that EDAM allows researchers to skip tedious keywording and classification tasks, freeing effort for analysis and discussion.","tokens_in":21806,"tokens_out":4611,"duration_ms":47932,"significance":"If the central equivalence claim were fully supported, the paper would make a useful contribution: it provides a concrete, reproducible pipeline for ontology-driven systematic mapping studies, releases its data and ontology, and includes a human-annotation comparison that is rare in this literature. The classifier comparison in Section 5.2 is also a valuable empirical baseline, and the authors are explicit about several limitations. However, the evaluation as reported does not establish the advertised equivalence for the realistic multi-label classification workload, and the effort-reduction motivation is not measured. The contribution is therefore promising but currently preliminary: the methodology is plausible and well described, but the load-bearing evaluation needs to be strengthened before the abstract-level claims can be accepted.","major_comments":[{"comment":"The 25-paper evaluation sample is not representative of the actual classification task. The authors write that for each of five hand-picked categories they \"randomly selected from the DSA dataset five primary studies that were classified by EDAM exclusively under that topic.\" This conditions the sample on exactly the cases where EDAM's exact-term mapping is strongest, and it excludes the ambiguous and multi-topic papers that arise in real systematic reviews. This is especially problematic because Section 4.2 (step 6) explicitly states that the mapping \"allows us to associate multiple categories to the same paper\" over a 46-topic ontology. The reported p=0.77 and the 84% majority-agreement figure therefore support equivalence only for single-topic, unambiguous cases, not for the multi-label classification workload that EDAM is designed to automate. The authors should re-run the evaluation on a random sample of DSA or DSA-MV without filtering on EDAM's exclusivity, and should report multi-label agreement measures (e.g., per-label precision/recall or multi-label kappa) rather than only forced single-label agreement.","section":"Section 5.1, evaluation sample"},{"comment":"The claim that EDAM is \"not statistically different\" from the senior group rests on a p-value of 0.77, but the paper does not provide the details of the chi-square test (the contingency table, the categories included, or the expected cell counts). More importantly, with n=25, failing to reject the null hypothesis is not evidence of equivalence. The appropriate analysis would be an equivalence test (e.g., a two one-sided test with a pre-specified agreement margin) or a confidence interval for the difference in agreement rates. As it stands, the statement \"not statistically different\" overstates what the experiment can show, and the abstract's unqualified conclusion is not justified by the statistical evidence.","section":"Section 5.1, statistical inference"},{"comment":"There is a circular structure in the evaluation. The ontology is learned from the same Scopus dump that is later used to construct DSA and DSA-MV (steps 2-5 of Section 4.2), and the evaluation sample is drawn from papers that EDAM itself classified unambiguously using the same term-matching mapping function (step 6). This means the evaluation is effectively testing EDAM on cases generated by EDAM's own decision rule. An independent evaluation, using papers that were not part of the ontology-learning corpus or at least not filtered by EDAM's exclusivity, is needed to support the claim that the classification generalizes to the broader population of primary studies.","section":"Sections 4.2 and 5.1, circularity"},{"comment":"The paper's stated goal is to reduce the effort of systematic reviews, and the abstract and conclusions claim that EDAM lets researchers \"skip the tedious tasks\" of keywording and manual classification. However, no effort measurement is reported. The only quantitative time-related datum is the approximately 20 minutes experts spent reviewing a 46-topic ontology in Section 4.1; there is no comparison of total time or workload between a manual mapping study and an EDAM-assisted one. The effort-reduction claim is load-bearing for the paper's motivation, so either an effort evaluation (even a rough time-and-motion comparison) should be added, or the claims should be softened to say that EDAM automates classification rather than demonstrating a reduction in effort.","section":"Abstract, Section 1, and Section 5.3"}],"minor_comments":[{"comment":"In the description of the LDA baselines, the third model is listed as \"LDA100\" but appears to be the 1,000-topic model; it should be labeled LDA1000 for consistency with the preceding two sentences.","section":"Section 5.2, LDA description"},{"comment":"The paragraph contains spelling errors such as \"automaticaly\", \"affectiveness\", and \"avaliable\"; these should be corrected before publication.","section":"Section 5.2, typos"},{"comment":"The classifier comparison in Section 5.2 uses a different gold standard (70 papers from Semantic Web, NLP, and Data Mining) than the human-annotator evaluation in Section 5.1; this should be stated explicitly in the text so that readers do not conflate the two evaluations when interpreting the reported F-measures.","section":"Section 5.2, classifier comparison"},{"comment":"The text in Section 4.2 says Figure 6 shows the \"percentage of papers published over time,\" while the figure caption says \"Number of publications of the top ten main topics in DSA over time.\" The y-axis metric should be defined consistently in the text and captions.","section":"Figures 6 and 8"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest about many limitations in Section 5.3, and the authors explicitly call the application \"preliminary.\" My main concern is that the abstract and the Section 5.1 conclusion outrun the evidence: the evaluation sample is filtered by EDAM's own unambiguous output, and no equivalence testing is performed. These issues are fixable within the scope of the paper, so I recommend major revision rather than rejection. The paper fits the journal's scope and the released artifacts are a genuine strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it so you know the state of play: EDAM is a named methodology that stitches together the authors' own Klink-2/CSO ontology learning and a direct-mapping classifier to automate keywording and classification in systematic reviews. The idea is not crazy—it targets a real bottleneck—and the paper gives a concrete application to Software Architecture with public data. But the headline claim, that EDAM's classifications are statistically indistinguishable from senior researchers, rests on an evaluation that samples only the easiest cases. The abstract overstates what is shown.\n\nWhere credit is due: the integrated pipeline is a reasonable synthesis of prior work, with a clear role for experts (refine the ontology rather than read thousands of abstracts). The authors provide the ontology (OWL) and datasets on Zenodo, and the classifier comparison in Section 5.2 uses a separate 70-paper gold standard with three annotators per paper. That is reproducible evidence. The limitations section is honest about several threats, including the fact that they explicitly note EDAM 'seems to perform well in handling simple not-ambiguous papers.'\n\nNow the soft spots. The main one is the evaluation design in Section 5.1. Twenty-five papers were drawn from five categories, and only papers that EDAM had assigned unambiguously to a single topic were eligible. Human annotators then made a forced single choice among those five categories or 'none.' That is not how EDAM is used in practice—its own step 6 outputs multi-label assignments over 46 topics. So the 'not statistically different from the senior group' (p=0.77) is a statement about an easy subset, not the real classification workload. Moreover, a non-significant chi-square on n=25 is not an equivalence test; the power to detect a meaningful difference is low. The authors concede the 'simple not-ambiguous' point, but the abstract and conclusions do not carry that caveat.\n\nSecond, the ontology was learned from the same Scopus dump used for the trend analysis, making the Software Architecture 'trends' a re-description of the ontology's term matches. That is fine as a methodology demonstration, but it is not an independent finding. Third, the paper is motivated by effort reduction but never measures effort; there is no time comparison versus manual keywording.\n\nMinor: the classifier comparison grid-searches LDA thresholds on the gold standard, which is tuning on the test set. The paper reports it transparently, but it levels the field against simpler baselines.\n\nVerdict: this is a plausible feasibility study that deserves a proper peer review, with the expectation of a major revision. The right fix is to reframe the claims, add a harder evaluation—random multi-label cases, an equivalence bound, and ideally some effort measurement—and lean on the public data, which is a genuine asset.","headline":"A plausible feasibility study whose headline claim overreaches: the evaluation only samples EDAM's unambiguous single-topic cases, so 'statistically indistinguishable from senior researchers' is not established, but the methodology and public data are worth engaging.","tokens_in":22355,"tokens_out":3080,"would_cite":false,"duration_ms":31980,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an automated ontology-driven pipeline, EDAM, classifies primary studies for software-engineering systematic reviews as accurately as senior human researchers, reporting statistical evidence of equivalence (p=0.77).","keywords":["systematic reviews","software engineering","ontology learning","semantic web","software architecture","primary study classification","expert-driven methodology","digital libraries"],"falsifier":"Sample papers from the same domain at random, without pre-selecting only those EDAM classifies unambiguously, then compare EDAM's labels against the majority of several senior researchers; if EDAM's agreement with the majority falls below the average human-human agreement, the claimed equivalence does not generalise to the real classification task.","tokens_in":21378,"feed_emoji":"📚","tokens_out":7123,"duration_ms":69320,"temperature":0.7,"pith_summary":"The paper aims to establish that the least creative steps of a systematic review, keywording primary studies and assigning them to a classification schema, can be handed to an automated pipeline, EDAM, without losing accuracy. EDAM first learns a domain ontology (a hierarchy of research topics) from scholarly metadata, has experts refine that ontology, and then automatically classifies each paper under every ontology topic whose label or a subcategory label appears in its title, abstract, or keywords. On a 25-paper evaluation in the Software Architecture area, the authors report that EDAM's annotations agreed with the majority of six human annotators and were not statistically distinguishable from the annotations of the senior researchers (p=0.77). If that result holds, systematic reviews could scale to datasets of hundreds of thousands of papers while freeing researchers for analysis and synthesis.","feed_headline":"Automated paper classification matches senior experts","feed_subtitle":"Machine-built domain ontology agreed with senior researchers, cutting manual keywording.","key_machinery":"The load-bearing mechanism is the EDAM three-stage pipeline. First, the Klink-2 algorithm, an ontology-learning method that infers hierarchical topic relationships from co-occurrence statistics, temporal signals, string similarity, and external sources, builds a multi-level ontology of the field from a large scholarly dataset (the Software Engineering ontology produced here has 956 topics and 5,461 relationships). Second, domain experts refine the ontology by adding or deleting categories and relationships, a step reported to take about 20 minutes for 46 topics. Third, a direct-mapping function assigns each paper to every topic whose label, equivalent label, or subcategory label appears in the paper's title, abstract, or keywords; this mapping is what lets the classification run at scale and be reproduced exactly.","core_discovery":"On the paper's own terms, the discovery is that an ontology-driven automatic classifier can stand in for a human expert during primary-study classification. EDAM classifies a paper by checking its title, abstract, and keywords against a hierarchy of research topics: the paper is assigned to every category whose label, an equivalent label, or a label of a subcategory appears in the metadata. Comparing EDAM with six human annotators on 25 papers drawn from five unambiguous Software Architecture topics, the authors find that EDAM has the highest average pairwise agreement, agrees with the relative majority of annotators in 84% of the cases, and its behaviour is not statistically different from the senior group (p=0.77). The authors conclude that EDAM can replace manual keywording and classification in a systematic review.","pith_inferences":["The paper only evaluates 25 papers that EDAM itself assigned unambiguously to one of five topics, so the headline equivalence is not directly established for multi-topic, ambiguous, or previously misclassified papers; a deployment would likely need a confidence threshold or expert adjudication for such cases, which the paper does not test.","Because the ontology-learning step detects mature topics more readily than emerging ones, an EDAM review aimed at detecting brand-new research fronts would probably miss them unless experts add the new topics manually; the paper acknowledges this but does not quantify the effect.","The reuse scenarios imply a future shared ecosystem of validated domain ontologies; a natural test would be to have two independent expert panels refine the same automatically learned ontology and measure how much their refinements diverge.","The same classification machinery could be pointed at non-English metadata or other scholarly corpora, but the current evidence is limited to Computer Science metadata from 2005-2013, so transfer should be verified per domain."],"forward_implications":["Researchers can drop the manual keywording and classification phases of a systematic review and instead refine an ontology, select inclusion criteria, and run a scripted classification.","Mapping studies can cover complete corpora rather than samples, because the automatic pipeline has no human time bottleneck.","A validated domain ontology can be reused for later updates and replications, so the same review can be refreshed with new publications without repeating the classification work.","The machine-readable ontology plus mapping function makes the classification process reproducible and auditable, since the rules are explicit rather than hidden in an annotator's judgment.","The paper's comparison of classifiers shows that when recall matters more than precision, the same EDAM pipeline can swap in a different unsupervised classifier that yields a higher F-measure at lower precision."],"supporting_citations":[{"why":"Supplies the Klink-2 algorithm that learns the domain ontology used to select and classify primary studies.","marker":"[37]"},{"why":"Defines the classic systematic mapping study steps that EDAM replaces.","marker":"[42]"},{"why":"Presents the ontology-driven classifier with the best F-measure in the paper's comparison of classification approaches.","marker":"[55]"},{"why":"Provides the ontology used as the target taxonomy in the comparison of classifiers.","marker":"[52]"},{"why":"Introduced the direct-mapping approach that EDAM reuses to map ontology topics to papers.","marker":"[39]"},{"why":"LDA serves as one of the baseline classifiers in the comparison, showing the limits of topic-model clustering for this task.","marker":"[8]"},{"why":"TF-IDF serves as another baseline classifier in the comparison.","marker":"[44]"},{"why":"Defines the purpose and procedure of systematic reviews that motivate the reduction of manual effort.","marker":"[20]"}],"fun_headline_variants":["AI classification matches senior experts in systematic reviews","Automated ontology reduces systematic review effort","EDAM: expert-level paper classification with less manual work","Ontology-based classification rivals human annotators","Cut systematic review effort with automated classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That the 25 evaluation papers, all chosen because EDAM assigned them to one of five categories without ambiguity, fairly represent the full mix of papers a systematic review must classify.","fun_headline_variants_meta":{"raw":{"variants":["AI classification matches senior experts in systematic reviews","Automated ontology reduces systematic review effort","EDAM: expert-level paper classification with less manual work","Ontology-based classification rivals human annotators","Cut systematic review effort with automated classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1546,"prompt_tokens":977,"completion_tokens":569,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":502}},"tokens_in":593,"tokens_out":569,"duration_ms":6625,"temperature":1.0,"reasoning_tokens":502,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:37:18.913646+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample papers from the same domain at random, without pre-selecting only those EDAM classifies unambiguously, then compare EDAM's labels against the majority of several senior researchers; if EDAM's agreement with the majority falls below the average human-human agreement, the claimed equivalence does not generalise to the real classification task.","supporting_citations":[{"cited_title":"Klink-2: integrating multiple web sources to generate semantic topic networks","cited_arxiv_id":null,"evidence_quote":"Supplies the Klink-2 algorithm that learns the domain ontology used to select and classify primary studies."},{"cited_title":"Guidelines for conducting systematic mapping studies in soft- ware engineering: An update","cited_arxiv_id":null,"evidence_quote":"Defines the classic systematic mapping study steps that EDAM replaces."},{"cited_title":"The cso classifier: Ontology- driven detection of research topics in scholarly articles","cited_arxiv_id":null,"evidence_quote":"Presents the ontology-driven classifier with the best F-measure in the paper's comparison of classification approaches."},{"cited_title":"The computer science ontology: a large-scale taxonomy of research areas","cited_arxiv_id":null,"evidence_quote":"Provides the ontology used as the target taxonomy in the comparison of classifiers."},{"cited_title":"Automatic classification of springer nature proceedings with smart topic miner","cited_arxiv_id":null,"evidence_quote":"Introduced the direct-mapping approach that EDAM reuses to map ontology topics to papers."},{"cited_title":"Latent dirichlet allocation","cited_arxiv_id":null,"evidence_quote":"LDA serves as one of the baseline classifiers in the comparison, showing the limits of topic-model clustering for this task."},{"cited_title":"Using tf-idf to determine word relevance in document queries","cited_arxiv_id":null,"evidence_quote":"TF-IDF serves as another baseline classifier in the comparison."},{"cited_title":"Procedures for performing systematic reviews","cited_arxiv_id":null,"evidence_quote":"Defines the purpose and procedure of systematic reviews that motivate the reduction of manual effort."}],"review_version":1}