Pith. sign in

REVIEW 4 major objections 4 minor 69 references

Reducing the Effort for Systematic Reviews in Software Engineering

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that an automated ontology-driven pipeline, EDAM, classifies primary studies for software-engineering systematic reviews as accurately as senior human researchers, reporting statistical evidence of equivalence (p=0.77).

desk verdict A plausible feasibility study whose headline claim overreaches: the evaluation only samples EDAM's unambiguous single-topic cases, so 'statistically indistinguishable from senior researchers' is not established, but the methodology and public data are worth engaging. read the letter →

arxiv 1908.06676 v1 pith:EIOEZKWP submitted 2019-08-19 cs.SE cs.DL

Francesco Osborne , Henry Muccini , Patricia Lago , Enrico Motta This is my paper · ORCID
classification cs.SEcs.DL
keywords systematicreviewssoftwareengineeringontologylearningsemanticwebarchitectureprimarystudyclassificationexpert-drivenmethodologydigitallibraries
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that the least creative steps of a systematic review, keywording primary studies and assigning them to a classification schema, can be handed to an automated pipeline, EDAM, without losing accuracy. EDAM first learns a domain ontology (a hierarchy of research topics) from scholarly metadata, has experts refine that ontology, and then automatically classifies each paper under every ontology topic whose label or a subcategory label appears in its title, abstract, or keywords. On a 25-paper evaluation in the Software Architecture area, the authors report that EDAM's annotations agreed with the majority of six human annotators and were not statistically distinguishable from the annotations of the senior researchers (p=0.77). If that result holds, systematic reviews could scale to datasets of hundreds of thousands of papers while freeing researchers for analysis and synthesis.

What carries the argument

The load-bearing mechanism is the EDAM three-stage pipeline. First, the Klink-2 algorithm, an ontology-learning method that infers hierarchical topic relationships from co-occurrence statistics, temporal signals, string similarity, and external sources, builds a multi-level ontology of the field from a large scholarly dataset (the Software Engineering ontology produced here has 956 topics and 5,461 relationships). Second, domain experts refine the ontology by adding or deleting categories and relationships, a step reported to take about 20 minutes for 46 topics. Third, a direct-mapping function assigns each paper to every topic whose label, equivalent label, or subcategory label appears in the paper's title, abstract, or keywords; this mapping is what lets the classification run at scale and be reproduced exactly.

What would settle it

Sample papers from the same domain at random, without pre-selecting only those EDAM classifies unambiguously, then compare EDAM's labels against the majority of several senior researchers; if EDAM's agreement with the majority falls below the average human-human agreement, the claimed equivalence does not generalise to the real classification task.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that an ontology-driven automatic classifier can stand in for a human expert during primary-study classification. EDAM classifies a paper by checking its title, abstract, and keywords against a hierarchy of research topics: the paper is assigned to every category whose label, an equivalent label, or a label of a subcategory appears in the metadata. Comparing EDAM with six human annotators on 25 papers drawn from five unambiguous Software Architecture topics, the authors find that EDAM has the highest average pairwise agreement, agrees with the relative majority of annotators in 84% of the cases, and its behaviour is not statistically different from the senior group (p=0.77). The authors conclude that EDAM can replace manual keywording and classification in a systematic review.

Load-bearing premise

That the 25 evaluation papers, all chosen because EDAM assigned them to one of five categories without ambiguity, fairly represent the full mix of papers a systematic review must classify.

Editorial extensions

If this is right

  • Researchers can drop the manual keywording and classification phases of a systematic review and instead refine an ontology, select inclusion criteria, and run a scripted classification.
  • Mapping studies can cover complete corpora rather than samples, because the automatic pipeline has no human time bottleneck.
  • A validated domain ontology can be reused for later updates and replications, so the same review can be refreshed with new publications without repeating the classification work.
  • The machine-readable ontology plus mapping function makes the classification process reproducible and auditable, since the rules are explicit rather than hidden in an annotator's judgment.
  • The paper's comparison of classifiers shows that when recall matters more than precision, the same EDAM pipeline can swap in a different unsupervised classifier that yields a higher F-measure at lower precision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper only evaluates 25 papers that EDAM itself assigned unambiguously to one of five topics, so the headline equivalence is not directly established for multi-topic, ambiguous, or previously misclassified papers; a deployment would likely need a confidence threshold or expert adjudication for such cases, which the paper does not test.
  • Because the ontology-learning step detects mature topics more readily than emerging ones, an EDAM review aimed at detecting brand-new research fronts would probably miss them unless experts add the new topics manually; the paper acknowledges this but does not quantify the effect.
  • The reuse scenarios imply a future shared ecosystem of validated domain ontologies; a natural test would be to have two independent expert panels refine the same automatically learned ontology and measure how much their refinements diverge.
  • The same classification machinery could be pointed at non-English metadata or other scholarly corpora, but the current evidence is limited to Computer Science metadata from 2005-2013, so transfer should be verified per domain.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes EDAM, an Expert-Driven Automatic Methodology for assisting systematic reviews in software engineering. EDAM replaces the manual keywording and data-extraction steps of a mapping study with (i) automatic ontology learning from a scholarly dataset, (ii) human refinement of that ontology, and (iii) automatic classification of primary studies against the refined ontology. The methodology is instantiated for the Software Architecture area using the Klink-2 algorithm on a Scopus dump, producing a 46-topic Software Architecture sub-ontology. The authors evaluate the classification step by comparing EDAM's annotations on 25 papers with those of six human experts, report that EDAM is not statistically different from the senior expert group (p=0.77), compare several unsupervised classifiers on a separate 70-paper gold standard, and discuss limitations, reuse scenarios, and implications for systematic mappings. The paper concludes that EDAM allows researchers to skip tedious keywording and classification tasks, freeing effort for analysis and discussion.

Significance. If the central equivalence claim were fully supported, the paper would make a useful contribution: it provides a concrete, reproducible pipeline for ontology-driven systematic mapping studies, releases its data and ontology, and includes a human-annotation comparison that is rare in this literature. The classifier comparison in Section 5.2 is also a valuable empirical baseline, and the authors are explicit about several limitations. However, the evaluation as reported does not establish the advertised equivalence for the realistic multi-label classification workload, and the effort-reduction motivation is not measured. The contribution is therefore promising but currently preliminary: the methodology is plausible and well described, but the load-bearing evaluation needs to be strengthened before the abstract-level claims can be accepted.

major comments (4)
  1. [Section 5.1, evaluation sample] The 25-paper evaluation sample is not representative of the actual classification task. The authors write that for each of five hand-picked categories they "randomly selected from the DSA dataset five primary studies that were classified by EDAM exclusively under that topic." This conditions the sample on exactly the cases where EDAM's exact-term mapping is strongest, and it excludes the ambiguous and multi-topic papers that arise in real systematic reviews. This is especially problematic because Section 4.2 (step 6) explicitly states that the mapping "allows us to associate multiple categories to the same paper" over a 46-topic ontology. The reported p=0.77 and the 84% majority-agreement figure therefore support equivalence only for single-topic, unambiguous cases, not for the multi-label classification workload that EDAM is designed to automate. The authors should re-run the evaluation on a random sample of DSA or DSA-MV without filtering on EDAM's exclusivity, and should report multi-label agreement measures (e.g., per-label precision/recall or multi-label kappa) rather than only forced single-label agreement.
  2. [Section 5.1, statistical inference] The claim that EDAM is "not statistically different" from the senior group rests on a p-value of 0.77, but the paper does not provide the details of the chi-square test (the contingency table, the categories included, or the expected cell counts). More importantly, with n=25, failing to reject the null hypothesis is not evidence of equivalence. The appropriate analysis would be an equivalence test (e.g., a two one-sided test with a pre-specified agreement margin) or a confidence interval for the difference in agreement rates. As it stands, the statement "not statistically different" overstates what the experiment can show, and the abstract's unqualified conclusion is not justified by the statistical evidence.
  3. [Sections 4.2 and 5.1, circularity] There is a circular structure in the evaluation. The ontology is learned from the same Scopus dump that is later used to construct DSA and DSA-MV (steps 2-5 of Section 4.2), and the evaluation sample is drawn from papers that EDAM itself classified unambiguously using the same term-matching mapping function (step 6). This means the evaluation is effectively testing EDAM on cases generated by EDAM's own decision rule. An independent evaluation, using papers that were not part of the ontology-learning corpus or at least not filtered by EDAM's exclusivity, is needed to support the claim that the classification generalizes to the broader population of primary studies.
  4. [Abstract, Section 1, and Section 5.3] The paper's stated goal is to reduce the effort of systematic reviews, and the abstract and conclusions claim that EDAM lets researchers "skip the tedious tasks" of keywording and manual classification. However, no effort measurement is reported. The only quantitative time-related datum is the approximately 20 minutes experts spent reviewing a 46-topic ontology in Section 4.1; there is no comparison of total time or workload between a manual mapping study and an EDAM-assisted one. The effort-reduction claim is load-bearing for the paper's motivation, so either an effort evaluation (even a rough time-and-motion comparison) should be added, or the claims should be softened to say that EDAM automates classification rather than demonstrating a reduction in effort.
minor comments (4)
  1. [Section 5.2, LDA description] In the description of the LDA baselines, the third model is listed as "LDA100" but appears to be the 1,000-topic model; it should be labeled LDA1000 for consistency with the preceding two sentences.
  2. [Section 5.2, typos] The paragraph contains spelling errors such as "automaticaly", "affectiveness", and "avaliable"; these should be corrected before publication.
  3. [Section 5.2, classifier comparison] The classifier comparison in Section 5.2 uses a different gold standard (70 papers from Semantic Web, NLP, and Data Mining) than the human-annotator evaluation in Section 5.1; this should be stated explicitly in the text so that readers do not conflate the two evaluations when interpreting the reported F-measures.
  4. [Figures 6 and 8] The text in Section 4.2 says Figure 6 shows the "percentage of papers published over time," while the figure caption says "Number of publications of the top ten main topics in DSA over time." The y-axis metric should be defined consistently in the text and captions.

Circularity Check

2 steps flagged · score 4.0 of 10

Evaluation samples EDAM's own unambiguous single-topic assignments, so the expert-equivalence claim is only demonstrated on an easy, self-selected subset; the trend analysis also re-describes the ontology-learning corpus.

  1. self definitional [Section 5.1, 'Evaluation of the primary study classification']
    "For simplifying the task and allowing to compare the annotation algorithmically, we first selected five unambiguous categories from the main topics of SA: Design Decisions, Service-oriented Architectures, Model-driven Architectures, Architecture Description Languages, and Views. For each category, we randomly selected from the DSA dataset five primary studies that were classified by EDAM exclusively under that topic, for a total of 25 papers."

    EDAM's mapping function (Section 4.2, step 6) assigns a paper to a category exactly when the title, abstract, or keywords contain the category label, a relatedEquivalent, a broader category, or an equivalent of a broader category. The evaluation sample is therefore selected on EDAM's own outcome: 'classified by EDAM exclusively under that topic' guarantees, by definition of the mapping function, that each sampled paper contains the term EDAM matches. EDAM's labels in the test set are fixed by the inclusion criterion, so the test cannot observe misclassification on ambiguous or multi-topic papers, and the reported p=0.77 compares EDAM with senior experts only on this easy subset. The paper itself concedes the point: 'It thus seems to perform well in handling simple not-ambiguous papers.'

  2. renaming known result [Section 4.2, steps 2-3 and 7]
    "We selected all papers in a dump of the Scopus dataset about Computer Science in the period 2005-2013. ... We applied the Klink-2 algorithm [37] on the Scopus dump for learning an ontology representing the main 'Software Architecture' research area in SE. ... We identified the main trends by running a script to count the number of studies about each sub-topic in each year."

    Klink-2 learns the ontology from term co-occurrence statistics over the same Scopus dump that is later classified and counted. The trend analysis (e.g., the reported rise of Model-driven Architectures) counts papers by the very labels and equivalent terms that the ontology-learning step had already extracted as frequent in that corpus. The 'findings' are thus a re-description of the corpus statistics that generated the ontology, not an independent confirmation. This is descriptive data synthesis rather than a falsifiable prediction, so it is less damaging than the evaluation sampling issue, but it is still a partial circularity in the presentation of the illustrative trend results.

full rationale

The central equivalence claim is not fully circular: EDAM's classifications are compared with six independent human annotators, and the agreement statistics (p=0.77 with the senior group; Cohen's kappa 0.58 on average) are genuine observations about those annotators. However, the evaluation set is constructed from papers that EDAM had already classified exclusively into one of five categories under an exact-term matching function, which fixes EDAM's labels on the test set by construction and excludes the multi-label, ambiguous cases that the paper itself says EDAM's step 6 handles ('it allows us to associate multiple categories to the same paper'). The ontology is also learned from the same Scopus dump that is later classified and counted, so the illustrative trend analysis re-describes the corpus rather than testing a hypothesis. The citations to Klink-2 [37] and CSO [52] are self-citations by the same group, but the paper specifies Algorithm 1 and the mapping function, and the human evaluation is external to those prior papers, so those self-citations are not the main load-bearing element. The chi-square 'not significant' result is not an equivalence test either, though that is a statistical-validity concern rather than a circularity. Overall the core claim has independent human-judgment content, but its scope is overstated because the sample design selects exactly the cases where EDAM's term matching succeeds.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

EDAM does not introduce new physical entities or forces. The pipeline depends on inherited numeric parameters from Klink-2, on the assumption that term matching is a faithful proxy for topic content, and on the assumption that the Scopus dataset and small expert refinement are unbiased enough for ontology construction. No independent falsifiable handle is provided for the correctness of the generated ontology beyond the downstream evaluation.

free parameters (4)
  • Klink-2 subsumption threshold = 0.25
    Hierarchical relationships between topics are inferred whenever HR(x,y) or TR(x,y) exceeds 0.25, which shapes the ontology's granularity and therefore the classification of papers into topics.
  • Klink-2 temporal weight gamma = 2
    The temporal relationship metric TR weighs yearly co-occurrences by (year - debut + 1)^gamma with gamma=2, affecting which topic relationships are inferred and thus the ontology structure.
  • LDA grid-search thresholds (j, k) = Best values found on the gold standard; individual values not reported
    The LDA baseline classifiers were tuned by grid search on the same 70-paper gold standard used for evaluation, so their reported F-measures are optimistic.
  • Levenshtein similarity thresholds for term mapping = 0.8 for TF-IDF and LDA mapping; 0.94 for CSO-C1
    These thresholds control which terms are mapped to CSO topics in the classifier comparison, directly affecting precision and recall.
assumptions (4)
  • domain assumption Exact matching of ontology labels, equivalents, and broader terms in title, abstract, or keywords is a sufficient proxy for the topical content of a paper.
    This is the mapping function defined in Section 4.2 step 6. If term presence does not reflect topic content, the entire classification step fails.
  • domain assumption The Scopus Computer Science dump for 2005-2013 is an unbiased representation of the Software Architecture literature.
    Section 4.2 step 2 selects this dataset for ontology learning and asserts it is unbiased with respect to the study focus. Any bias propagates into the ontology and the selection of primary studies.
  • domain assumption Three senior researchers refining the ontology through a spreadsheet, with majority vote for disagreements, yields a taxonomy consistent with the research community.
    Section 4.1 step 4 relies on this expert refinement as the bridge between the data-driven ontology and the community's mental model. The classification evaluation then compares against other senior researchers, assuming this refined ontology is a fair basis.
  • domain assumption The Klink-2 algorithm performs correctly, and its published evaluation on Semantic Web topics transfers to the Software Architecture subtopic.
    The paper uses Klink-2 from prior work [37] without re-evaluating it on the Software Architecture ontology. The quality of the generated ontology is load-bearing for EDAM.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reducing the Effort for Systematic Reviews in Software Engineering." pith.science (2026). https://pith.science/paper/EIOEZKWP

@misc{pith2026190806676,
  author       = {Pith},
  title        = {Pith review of: Reducing the Effort for Systematic Reviews in Software Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EIOEZKWP}},
  note         = {Machine review of arXiv:1908.06676}
}
read the original abstract

Context. Systematic Reviews (SRs) are means for collecting and synthesizing evidence from the identification and analysis of relevant studies from multiple sources. To this aim, they use a well-defined methodology meant to mitigate the risks of biases and ensure repeatability for later updates. SRs, however, involve significant effort. Goal. The goal of this paper is to introduce a novel methodology that reduces the amount of manual tedious tasks involved in SRs while taking advantage of the value provided by human expertise. Method. Starting from current methodologies for SRs, we replaced the steps of keywording and data extraction with an automatic methodology for generating a domain ontology and classifying the primary studies. This methodology has been applied in the Software Engineering sub-area of Software Architecture and evaluated by human annotators. Results. The result is a novel Expert-Driven Automatic Methodology, EDAM, for assisting researchers in performing SRs. EDAM combines ontology-learning techniques and semantic technologies with the human-in-the-loop. The first (thanks to automation) fosters scalability, objectivity, reproducibility and granularity of the studies; the second allows tailoring to the specific focus of the study at hand and knowledge reuse from domain experts. We evaluated EDAM on the field of Software Architecture against six senior researchers. As a result, we found that the performance of the senior researchers in classifying papers was not statistically significantly different from EDAM. Conclusions. Thanks to automation of the less-creative steps in SRs, our methodology allows researchers to skip the tedious tasks of keywording and manually classifying primary studies, thus freeing effort for the analysis and the discussion.

Figures

Figures reproduced from arXiv: 1908.06676 by the authors.

Figure 1
Figure 1. Some evidence on the Benefits of Automated SRs [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Steps of a systematic mappings adopting the EDAM methodology. The gray-shaded elements refer to the alternative [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Classic steps of systematic mappings (inspired by [42]) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Number of publications in DSA and DSA-MV over the years. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Number of publications and citations of the main topics in DSA and DSA-MV. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Number of publications of the top ten main topics in DSA over time. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Comparison DSA and DSA-MV in terms of topic distribution. The percentage value refers to the ratio between the [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Number of publications of the top ten main topics in DSA-MV over time. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Percentage of annotations that agree with other [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: (S2a)). The list of (updated) primary studies can be automatically re-calculated (in step [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 10
Figure 10. Figure 10: Possible EDAM applications [PITH_FULL_IMAGE:figures/full_fig_p030_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 67 canonical work pages

  1. [1]

    Vision for SLR tooling infrastructure: Prioritizing value-added requirements

    Ahmed Al-Zubidy, Jeffrey C Carver, David P Hale, and Edgar E Hassler. Vision for SLR tooling infrastructure: Prioritizing value-added requirements. Information and Software Technology, 91:72–81, November 2017

    work page 2017

  2. [2]

    A survey of topic modeling in text mining

    Rubayyi Alghamdi and Khalid Alfalqi. A survey of topic modeling in text mining. I. J. ACSA, 6(1):147–153, 2015

    work page 2015

  3. [3]

    Constructing a search strategy and searching for evidence

    Edoardo Aromataris and Dagmara Riitano. Constructing a search strategy and searching for evidence. American Journal of Nursing, 114(5):49–56, 2014

    work page 2014

  4. [4]

    Lodifier: Generating linked data from unstructured text

    Isabelle Augenstein, Sebastian Padó, and Sebastian Rudolph. Lodifier: Generating linked data from unstructured text. In Extended Semantic Web Conference, pages 210–224. Springer, 2012

    work page 2012

  5. [5]

    Bio2rdf: towards a mashup to build bioinformatics knowledge systems

    François Belleau, Marc-Alexandre Nolin, Nicole Tourigny, Philippe Rigault, and Jean Morissette. Bio2rdf: towards a mashup to build bioinformatics knowledge systems. Journal of biomedical informatics, 41(5):706–716, 2008

    work page 2008

  6. [6]

    Exploring the educational potential of robotics in schools: A systematic review

    Fabiane Barreto Vavassori Benitti. Exploring the educational potential of robotics in schools: A systematic review. Com- puters & Education, 58(3):978–988, 2012

    work page 2012

  7. [7]

    Automatic labelling of topics with neural embeddings

    Shraey Bhatia, Jey Han Lau, and Timothy Baldwin. Automatic labelling of topics with neural embeddings. arXiv preprint arXiv:1612.05340, 2016

    arXiv 2016

  8. [8]

    Latent dirichlet allocation

    David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3 (Jan):993–1022, 2003

    work page 2003

Show all 69 references
  1. [9]

    Ontology forecasting in scientific literature: Semantic concepts prediction based on innovation-adoption priors

    Amparo Elizabeth Cano-Basave, Francesco Osborne, and Angelo Antonio Salatino. Ontology forecasting in scientific literature: Semantic concepts prediction based on innovation-adoption priors. In Knowledge Engineering and Knowledge Management: 20th International Conference, EKAW ...

    2016

  2. [10]

    Relationship between young peoples’ sedentary behaviour and biomedical health indicators: a systematic review of prospective studies.Obesity reviews, 12(7):e621–e632, 2011

    MJM Chinapaw, KI Proper, J Brug, W Van Mechelen, and AS Singh. Relationship between young peoples’ sedentary behaviour and biomedical health indicators: a systematic review of prospective studies.Obesity reviews, 12(7):e621–e632, 2011

    2011

  3. [11]

    Text2onto

    Philipp Cimiano and Johanna Völker. Text2onto. In International Conference on Application of Natural Language to Information Systems, pages 227–238. Springer, 2005

    2005

  4. [12]

    Replication of empirical studies in software engineering research: a systematic mapping study

    Fabio Q B da Silva, Marcos Suassuna, A César C França, Alicia M Grubb, Tatiana B Gouveia, Cleviton V F Monteiro, and Igor Ebrahim dos Santos. Replication of empirical studies in software engineering research: a systematic mapping study. Empirical Software Engineer, 19(3):501–5...

    2014

  5. [13]

    Scientific research ontology to support systematic review in software engineering

    Jorge Calmon de Almeida Biolchini, Paula Gomes Mian, Ana Candida Cruz Natali, Tayana Uchôa Conte, and Guil- herme Horta Travassos. Scientific research ontology to support systematic review in software engineering. Advanced Engineering Informatics, 21(2):133–151, 2007

    2007

  6. [14]

    Surveys in software engineering: Identifying representative samples

    Rafael Maiani De Mello and Guilherme Horta Travassos. Surveys in software engineering: Identifying representative samples. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Mea- surement, ESEM ’16, pages 55:1–55:6, New York, NY ,...

    2016

  7. [15]

    Linked open data to support content-based recommender systems

    Tommaso Di Noia, Roberto Mirizzi, Vito Claudio Ostuni, Davide Romito, and Markus Zanker. Linked open data to support content-based recommender systems. In Proceedings of the 8th International Conference on Semantic Systems , pages 1–8. ACM, 2012

    2012

  8. [16]

    Us- ing forward snowballing to update systematic reviews in software engineering

    Katia Romero Felizardo, Emilia Mendes, Marcos Kalinowski, Érica Ferreira Souza, and Nandamudi L Vijaykumar. Us- ing forward snowballing to update systematic reviews in software engineering. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engin...

    2016

  9. [17]

    Systematic Literature Study on Sus- tainable Software

    Bojan Filipovic, Boris Van Lindschoten, Giuseppe Procaccianti, and Patricia Lago. Systematic Literature Study on Sus- tainable Software. VU Technical Report, 2 2017. F . Osborne et al. / Reducing the Effort for Systematic Reviews in Software Engineering 27

    2017

  10. [18]

    Semantic web machine reading with fred

    Aldo Gangemi, Valentina Presutti, Diego Reforgiato Recupero, Andrea Giovanni Nuzzolese, Francesco Draicchio, and Misael Mongiovì. Semantic web machine reading with fred. Semantic Web, 8(6):873–893, 2017

    2017

  11. [19]

    Identification of SLR tool needs – results of a community workshop

    Edgar Hassler, Jeffrey C Carver, David Hale, and Ahmed Al-Zubidy. Identification of SLR tool needs – results of a community workshop. Information and Software Technology, 70:122–129, 2016

    2016

  12. [20]

    Procedures for performing systematic reviews

    Barbara Kitchenham. Procedures for performing systematic reviews. Keele, UK, Keele University, 33(2004):1–26, 2004

    2004

  13. [21]

    A systematic review of systematic review process research in software engineer- ing

    Barbara Kitchenham and Pearl Brereton. A systematic review of systematic review process research in software engineer- ing. Information and software technology, 55(12):2049–2075, 2013

    2013

  14. [22]

    Guidelines for performing systematic literature reviews in software engineer- ing, 2007

    Barbara A Kitchenham and Stuart Charters. Guidelines for performing systematic literature reviews in software engineer- ing, 2007

    2007

  15. [23]

    Core: three access levels to underpin open access

    Petr Knoth and Zdenek Zdrahal. Core: three access levels to underpin open access. D-Lib Magazine, 18(11/12), 2012

    2012

  16. [24]

    Broadening the scope of nanopub- lications

    Tobias Kuhn, Paolo Emilio Barbano, Mate Levente Nagy, and Michael Krauthammer. Broadening the scope of nanopub- lications. In Extended Semantic Web Conference, pages 487–501. Springer, 2013

    2013

  17. [25]

    On the pragmatic design of literature studies in software engineering: an experience-based guideline

    Marco Kuhrmann, Daniel Méndez Fernández, and Maya Daneva. On the pragmatic design of literature studies in software engineering: an experience-based guideline. Empirical Software Engineer, pages 1–40, 6 January 2017

    2017

  18. [26]

    The measurement of observer agreement for categorical data

    J Richard Landis and Gary G Koch. The measurement of observer agreement for categorical data. biometrics, pages 159–174, 1977

    1977

  19. [27]

    Dblp: some lessons learned

    Michael Ley. Dblp: some lessons learned. Proceedings of the VLDB Endowment, 2(2):1493–1500, 2009

    2009

  20. [28]

    Automatic taxonomy construction from keywords

    Xueqing Liu, Yangqiu Song, Shixia Liu, and Haixun Wang. Automatic taxonomy construction from keywords. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1433–

  21. [29]

    The evolution of ijhcs and chi: A quantitative analysis

    Andrea Mannocci, Francesco Osborne, and Enrico Motta. The evolution of ijhcs and chi: A quantitative analysis. Inter- national Journal of Human-Computer Studies, 2019

    2019

  22. [30]

    Tools to support systematic reviews in software engi- neering: a cross-domain survey using semi-structured interviews

    Christopher Marshall, Pearl Brereton, and Barbara Kitchenham. Tools to support systematic reviews in software engi- neering: a cross-domain survey using semi-structured interviews. In Proceedings of the 19th International Conference on Evaluation and Assessment in Software Eng...

    2015

  23. [31]

    Dbpedia spotlight: shedding light on the web of documents

    Pablo N Mendes, Max Jakob, Andrés García-Silva, and Christian Bizer. Dbpedia spotlight: shedding light on the web of documents. In Proceedings of the 7th international conference on semantic systems, pages 1–8. ACM, 2011

    2011

  24. [32]

    Survey guidelines in software engineering: An annotated review

    Jefferson Seide Molleri, Kai Petersen, and Emilia Mendes. Survey guidelines in software engineering: An annotated review. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Mea- surement - ESEM ’16, pages 1–6. ACM Press, 2016

    2016

  25. [33]

    Investigating the use of a hybrid search strategy for systematic reviews

    Erica Mourão, Marcos Kalinowski, Leonardo Murta, Emilia Mendes, and Claes Wohlin. Investigating the use of a hybrid search strategy for systematic reviews. In Proceedings of the 11th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM ’17, ...

    2017

  26. [34]

    Conference linked data: the scholarlydata project

    Andrea Giovanni Nuzzolese, Anna Lisa Gentile, Valentina Presutti, and Aldo Gangemi. Conference linked data: the scholarlydata project. In International Semantic Web Conference, pages 150–158. Springer, 2016

    2016

  27. [35]

    Semi-automatic selection of primary studies in systematic literature reviews: is it reasonable? Empirical Software Engineer, 20(6):1898–1917, 2015

    Fábio R Octaviano, Katia R Felizardo, José C Maldonado, and Sandra C P. Semi-automatic selection of primary studies in systematic literature reviews: is it reasonable? Empirical Software Engineer, 20(6):1898–1917, 2015

    1917

  28. [36]

    Mining semantic relations between research areas

    Francesco Osborne and Enrico Motta. Mining semantic relations between research areas. In International Semantic Web Conference 2012, pages 410–426. Springer, 2012

    2012

  29. [37]

    Klink-2: integrating multiple web sources to generate semantic topic networks

    Francesco Osborne and Enrico Motta. Klink-2: integrating multiple web sources to generate semantic topic networks. In International Semantic Web Conference 2015, pages 408–424. Springer, 2015

    2015

  30. [38]

    Exploring scholarly data with rexplore

    Francesco Osborne, Enrico Motta, and Paul Mulholland. Exploring scholarly data with rexplore. InInternational semantic web conference 2013, pages 460–477. Springer, 2013

    2013

  31. [39]

    Automatic classification of springer nature proceedings with smart topic miner

    Francesco Osborne, Angelo Salatino, Aliaksandr Birukou, and Enrico Motta. Automatic classification of springer nature proceedings with smart topic miner. In International Semantic Web Conference 2016, pages 383–399. Springer, 2016

    2016

  32. [40]

    Setting our bibliographic references free: towards open citation data

    Silvio Peroni, Alexander Dutton, Tanya Gray, and David Shotton. Setting our bibliographic references free: towards open citation data. Journal of Documentation, 71(2):253–277, 2015

    2015

  33. [41]

    Systematic mapping studies in software engineering

    Kai Petersen, Robert Feldt, Shahid Mujtaba, and Michael Mattsson. Systematic mapping studies in software engineering. In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering , EASE, pages 68–77, Swinton, UK, UK, 2008. British Co...

    2008

  34. [42]

    Guidelines for conducting systematic mapping studies in soft- ware engineering: An update

    Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. Guidelines for conducting systematic mapping studies in soft- ware engineering: An update. Information and Software Technology, 64:1–18, 2015

    2015

  35. [43]

    Ontology learning in the deep

    Giulio Petrucci, Chiara Ghidini, and Marco Rospocher. Ontology learning in the deep. In Knowledge Engineering and Knowledge Management: 20th International Conference, EKAW 2016, Bologna, Italy, November 19-23, 2016, Proceed- ings 20, pages 480–495. Springer, 2016

    2016

  36. [44]

    Using tf-idf to determine word relevance in document queries

    Juan Ramos et al. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, volume 242, pages 133–142. Piscataway, NJ, 2003. 28 F . Osborne et al. / Reducing the Effort for Systematic Reviews in Softw...

    2003

  37. [45]

    A review on artificial intelligence based load demand forecasting tech- niques for smart grid and buildings

    Muhammad Qamar Raza and Abbas Khosravi. A review on artificial intelligence based load demand forecasting tech- niques for smart grid and buildings. Renewable and Sustainable Energy Reviews, 50:1352–1372, 2015

    2015

  38. [46]

    Computer-based psychological treatments for depression: a systematic review and meta-analysis

    Derek Richards and Thomas Richardson. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clinical psychology review, 32(4):329–342, 2012

    2012

  39. [47]

    Nerd: a framework for unifying named entity recognition and disambiguation extraction tools

    Giuseppe Rizzo and Raphaël Troncy. Nerd: a framework for unifying named entity recognition and disambiguation extraction tools. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 73–76. As...

    2012

  40. [48]

    A machine learning approach for semi-automated search and selec- tion in literature studies

    Rasmus Ros, Elizabeth Bjarnason, and Per Runeson. A machine learning approach for semi-automated search and selec- tion in literature studies. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, EASE’17, pages 118–127. ACM,...

    2017

  41. [49]

    Verifying conceptual domain models with human computation: A case study in software engineering

    Marta Sabou, Dietmar Winkler, Peter Penzerstadler, and Stefan Biffl. Verifying conceptual domain models with human computation: A case study in software engineering. In Sixth AAAI Conference on Human Computation and Crowdsourc- ing, 2018

    2018

  42. [50]

    Semantic sentiment analysis of twitter

    Hassan Saif, Yulan He, and Harith Alani. Semantic sentiment analysis of twitter. The Semantic Web–ISWC 2012, pages 508–524, 2012

    2012

  43. [51]

    How are topics born? understanding the research dynamics preceding the emergence of new areas

    Angelo A Salatino, Francesco Osborne, and Enrico Motta. How are topics born? understanding the research dynamics preceding the emergence of new areas. PeerJ Computer Science, 3:e119, 2017

    2017

  44. [52]

    The computer science ontology: a large-scale taxonomy of research areas

    Angelo A Salatino, Thiviyan Thanapalasingam, Andrea Mannocci, Francesco Osborne, and Enrico Motta. The computer science ontology: a large-scale taxonomy of research areas. In International Semantic Web Conference, pages 187–205. Springer, 2018

    2018

  45. [53]

    Classifying research papers with the computer science ontology

    Angelo A Salatino, Thiviyan Thanapalasingam, Andrea Mannocci, Francesco Osborne, and Enrico Motta. Classifying research papers with the computer science ontology. InInternational Semantic Web Conference (P&D/Industry/BlueSky). CEUR Workshop Proceedings, volume 2180, 2018

    2018

  46. [54]

    Improving editorial workflow and meta- data quality at springer nature

    Angelo A Salatino, Francesco Osborne, Aliaksandr Birukou, and Enrico Motta. Improving editorial workflow and meta- data quality at springer nature. In International Semantic Web Conference 2019, 2019

    2019

  47. [55]

    The cso classifier: Ontology- driven detection of research topics in scholarly articles

    Angelo A Salatino, Francesco Osborne, Thiviyan Thanapalasingam, and Enrico Motta. The cso classifier: Ontology- driven detection of research topics in scholarly articles. In TPDL 2019: 23rd International Conference on Theory and Practice of Digital Libraries, 2019

    2019

  48. [56]

    Deriving concept hierarchies from text

    Mark Sanderson and Bruce Croft. Deriving concept hierarchies from text. InProceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pages 206–213. ACM, 1999

    1999

  49. [57]

    Topic detection and tracking using idf-weighted cosine coefficient

    J Michael Schultz and Mark Liberman. Topic detection and tracking using idf-weighted cosine coefficient. InProceedings of the DARPA broadcast news workshop, pages 189–192. San Francisco: Morgan Kaufmann, 1999

    1999

  50. [58]

    An overview of microsoft academic service (mas) and applications

    Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-june Paul Hsu, and Kuansan Wang. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web, pages 243–246. ACM, 2015

    2015

  51. [59]

    Towards evidence-based ontology for supporting sys- tematic literature review

    Yueming Sun, Ye Yang, He Zhang, Wen Zhang, and Qing Wang. Towards evidence-based ontology for supporting sys- tematic literature review. In International Conference on Evaluation and Assessment in Software Engineering (EASE) . IET, 2012

    2012

  52. [60]

    Twenty-eight years of component-based software engineering

    Tassio Vale, Ivica Crnkovic, Eduardo Santana de Almeida, Paulo Anselmo da Mota Silveira Neto, Yguarata Cerqueira Cavalcanti, and Silvio Romero de Lemos Meira. Twenty-eight years of component-based software engineering. Journal of Systems and Software, 111(1):128 – 148, 2016. I...

    2016

  53. [61]

    Design Science Methodology for Information Systems and Software Engineering:

    Roel J Wieringa. Design Science Methodology for Information Systems and Software Engineering: . Springer Berlin Heidelberg, 2014

    2014

  54. [62]

    The fair guiding principles for scientific data management and stewardship

    Mark D Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E Bourne, et al. The fair guiding principles for scientific data management and stewardship. Scienti...

    2016

  55. [63]

    Dynamic integration of multiple evidence sources for ontology learning

    Gerhard Wohlgenannt, Albert Weichselbraun, Arno Scharl, and Marta Sabou. Dynamic integration of multiple evidence sources for ontology learning. Journal of Information and Data Management, 3(3):243, 2012

    2012

  56. [64]

    Wohlin, P

    C. Wohlin, P. Runeson, M. Höst, M.C. Ohlsson, B. Regnell, and A. Wesslén. Experimentation in Software Engineering. Computer Science. Springer, 2012

    2012

  57. [65]

    Systematic literature reviews in software engineering

    Claes Wohlin and Rafael Prikladnicki. Systematic literature reviews in software engineering. Information and Software Technology, 55(6):919–920, 2013

    2013

  58. [66]

    On the reliability of mapping studies in software engineering

    Claes Wohlin, Per Runeson, Paulo Anselmo da Mota Silveira Neto, Emelie Engström, Ivan do Carmo Machado, and Eduardo Santana de Almeida. On the reliability of mapping studies in software engineering. The Journal of systems and software, 86(10):2594–2610, October 2013

    2013

  59. [67]

    Wolfram, Patricia Lago, and Francesco Osborne

    Nina J.E. Wolfram, Patricia Lago, and Francesco Osborne. Sustainability in software engineering. In IFIP Conference on Sustainable Internet and ICT for Sustainability (SustainIT), December 2017. F . Osborne et al. / Reducing the Effort for Systematic Reviews in Software Engineering 29

    2017

  60. [68]

    Systematic reviews in software engineering: An empirical investigation

    He Zhang and Muhammad Ali Babar. Systematic reviews in software engineering: An empirical investigation. Informa- tion and Software Technology, 55(7), 2013. ISSN 0164-1212

    2013

  61. [69]

    Identifying relevant studies in software engineering

    He Zhang, Muhammad Ali Babar, and Paolo Tell. Identifying relevant studies in software engineering. Information and Software Technology, 53(6):625–637, 2011. 30 F . Osborne et al. / Reducing the Effort for Systematic Reviews in Software Engineering Fig. 10. Possible EDAM applications

    2011

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.