Knowledge engineering for open science: Building and deploying knowledge bases for metadata standards

Josef Hardi; Marcos Martinez-Romero; Mark A. Musen; Martin J. O'Connor

REVIEW 3 major objections 5 minor 14 references

Knowledge engineering for open science: Building and deploying knowledge bases for metadata standards

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Metadata standards encoded as templates can drive FAIR data tools

desk verdict Useful system overview of CEDAR, not a new research contribution; the knowledge-base framing is rhetorical, and the FAIRness claims outrun the evidence on conditional constraints. read the letter →

arxiv 2507.22391 v1 pith:EPAC33PG submitted 2025-07-30 cs.DL

Mark A. Musen , Martin J. O'Connor , Josef Hardi , Marcos Martinez-Romero This is my paper · ORCID

classification cs.DL

keywords openscienceFAIRguidingprinciplesmetadatastandardsknowledgeengineeringtemplatesCEDARWorkbenchbasesdataannotation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the reason research data are rarely FAIR is that metadata standards are scarce and hard to apply. It presents CEDAR, a system that encodes a scientific community's reporting guidelines as machine-readable templates—lists of typed fields linked to controlled vocabularies—and treats each template as a declarative knowledge base. Because the same template can render web forms, generate spreadsheets, validate entries, and guide artificial-intelligence tools that convert legacy metadata, the standard is not a document but a reusable computational object. The paper concludes that knowledge-engineering methods from early expert systems are still a practical route to open science, because communities can author a standard once and deploy it everywhere.

What carries the argument

The load-bearing object is the CEDAR template model: a JSON Schema-based representation in which a template is composed of reusable fields and elements. A field declares a datatype (string, integer, date, controlled term, or a specialized identifier), may draw values from an ontology or a curated value set, and carries a machine-resolvable IRI plus human-readable labels; instances of a template are encoded in JSON-LD so values retain their connection to the template and to source terms. This one representation is what makes a single standard portable across systems: the template itself generates the user interfaces, the spreadsheet layout, the validation logic, and the context for AI-assisted metadata correction.

What would settle it

Take a collection of datasets annotated through a CEDAR template for a given experiment type and ask three independent curators to extract the experimental conditions; if their agreement is no better than for the same datasets annotated with free text, the template has not carried the community's knowledge.

Watch

Extended reading notes

Core claim

The central claim is that a community's metadata preferences are a form of knowledge, and that this knowledge can be captured symbolically in a CEDAR template, which formally defines the attributes of a reporting guideline, including names, datatypes, cardinalities, and links to ontologies or value sets. A template is not merely a form; it is a knowledge base that makes the standard inspectable, editable, and reusable. Once encoded, the same template can drive a web form for data entry, generate a column-per-field spreadsheet for bulk annotation, supply the schema for a validator that finds and suggests fixes for non-adherent entries, and improve a large language model's ability to convert legacy metadata toward the community standard. The paper presents deployments in several research consortia and data repositories as evidence that templates created once in this model can be shared across independent systems.

Load-bearing premise

The approach assumes that a scientific community's metadata needs can be completely and usefully captured as a fixed set of typed fields with controlled values, so that instantiating a template guarantees FAIRness.

Editorial extensions

If this is right

A metadata standard authored once as a template can be deployed in web forms, spreadsheets, and external data repositories without re-encoding.
Research consortia can use templates as a shared reference and training artifact, so dataset annotations are consistent from the moment they are created.
Spreadsheet-based metadata entry, which many scientists prefer, can be made standards-adherent when the same template drives both the spreadsheet and its validator.
Large language models asked to standardize legacy metadata perform better when given a template's structure and controlled values as context.
Templates give fields without existing reporting guidelines a concrete path to FAIR data, because they turn standard development into a structured knowledge-engineering exercise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The template-as-knowledge-base pattern is not limited to biomedicine; any discipline with an ontology or naming authority could follow the same recipe, so the approach's generality depends more on community organization than on the technology.
A testable extension would compare retrieval precision and recall for a repository annotated with template-generated metadata against the same repository annotated with free text; the paper's argument predicts template annotations win.
Because templates encode current consensus, they will need versioning and governance as experimental methods evolve; template maintenance, not creation, may become the long-term bottleneck that the paper does not address.
The claim that templates improve large-language-model metadata conversion suggests a further experiment: measure conversion quality with and without template context on identical legacy metadata, holding the model fixed.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. This is a competent overview of the CEDAR platform, written by the people who built it. It is not a new research contribution: the template model, the Workbench, and the deployments have all been described in earlier papers (Musen et al. 2022, O'Connor et al. 2025a/b). What's new is the consolidated narrative and the explicit framing of templates as 'knowledge bases' in the 1980s expert-systems tradition. That framing is a nice rhetorical hook, but it doesn't introduce any new machinery.

What the paper does well: it gives an accurate, readable picture of how CEDAR works—templates in JSON Schema, instances in JSON-LD, controlled vocabularies from BioPortal, and the recent integrations (OSF, Dryad, spreadsheet generation, validator, and the LLM-based metadata conversion work). The deployment list (HEAL, IDG, Health-RI, HuBMAP) is real evidence of adoption, and the authors are appropriately modest in places: they admit that knowledge engineering is labor-intensive, that templates are 'at best crude models,' and that templates alone don't solve tasks.

The soft spots are mostly about scope. The abstract and conclusion claim that templates 'capture the knowledge of a community of practice' and make datasets FAIR. That's asserted, not shown. There are no outcome measures, no comparisons with other metadata approaches, and no evidence that the templates express the full logical content of a reporting guideline. The stress-test is right that the template model has no explicit mechanism for conditional cross-field constraints (e.g., 'if assay type is X, then fields Y and Z are required'). Many real guidelines have such rules. So the strong reading of 'captures the knowledge' outruns the evidence. The authors do not claim to support conditional constraints, and the validator only checks missing values and datatype/value-set deviations. This is a limitation of the paper's rhetoric more than of the software: CEDAR is still useful for what it does, which is to standardize the attribute list and value domains. But the paper should either soften the language or show a guideline with inter-field rules being enforced.

The citation pattern is heavy on self-citation, but that's appropriate for a system-description paper; the prior work is the evidence. No data or code is shipped, but the software is deployed and available.

Bottom line: this is a useful reference for someone who wants to understand CEDAR's architecture and ecosystem, not a paper that advances the field. I'd send it to peer review only if the venue values system descriptions and the authors agree to reframe the claims. For a top research venue, I'd desk reject. If I were refereeing, I'd ask for a sharper statement of what templates can and cannot represent, plus at least one concrete example of a conditional guideline being handled (or an explicit statement that conditional rules are out of scope).

Referee Report

3 major / 5 minor

Summary. The manuscript argues that metadata standards for scientific data can be encoded as CEDAR templates, which are declarative knowledge bases that enumerate typed attributes with ontology links, and that these templates can be reused to drive Web forms, spreadsheets, validators, embedded editors, and LLM-based metadata conversion. It describes the CEDAR template model, reports deployments in HEAL, IDG, Health-RI, OSF, Dryad, and HuBMAP, and concludes that such templates are the path to making datasets FAIR. The paper also draws an analogy to classical expert-system knowledge bases and describes community-authoring processes such as GO FAIR Metadata for Machines Workshops.

Significance. Strengths: the paper reports on real, deployed systems and gives concrete examples (RNAseq template, Psych-DS on OSF, HuBMAP spreadsheets); it makes the knowledge-engineering process explicit; and it offers a checkable claim—that a community's reporting guidelines can be captured as attribute-value templates—that others could test. If the central claim holds, the work provides a practical infrastructure for machine-actionable metadata. The primary limitation is evidentiary: the paper contains no quantitative outcome measures, no comparisons with alternative metadata approaches, and no assessment of the expressiveness of the template model relative to the reporting guidelines it claims to represent.

major comments (3)

[Materials and Methods (template model)] The paper says a CEDAR template 'formally defines the attributes that constitute a reporting guideline,' but the described model supports only local field-level constraints: datatypes, cardinalities, and controlled-vocabulary links. It does not describe any mechanism for conditional or cross-field constraints (e.g., 'if assay type is X, fields Y and Z are required' or 'if organism is human, ethics approval is required'). Many reporting guidelines, including MIAME and MINSEQE and likely several HuBMAP assay guidelines, contain such conditional requirements. Because the paper claims that templates 'ensure adherence to standards,' the absence of any described mechanism for inter-field dependencies is load-bearing. Please either show that the CEDAR model can express such constraints, provide examples where it has, or explicitly state that templates provide a local approximation and adjust the adherence/FAIRness claims accordingly.
[Results and Discussion] The manuscript asserts that CEDAR designs make datasets 'guaranteed to be FAIR' (Discussion) and that the systems ensure standards-adherent metadata. However, no quantitative or comparative evidence is provided: no counts of metadata records validated, no error rates before/after, no user studies, no comparison of FAIRness scores or search/interoperability outcomes against an alternative standard-encoding approach. As an experience report, the deployments are informative, but the conclusion that templates are the mechanism for FAIR data outruns the evidence presented. Please either add relevant outcome data or reframe the paper explicitly as an experience/position paper whose claims are about feasibility rather than demonstrated effectiveness.
[Discussion] The sentence 'If a research community is serious about open science, data sharing, and data reuse, it is not clear that there is any alternative to the approach offered by CEDAR' is an unsupported exclusivity claim. The paper does not compare CEDAR with other metadata standardization and annotation approaches (e.g., ISA-Tab/ISA-JSON, Schema.org/DCAT, RO-Crate, or JSON-LD-based profiles), nor does it discuss conditions under which simpler or different representations might suffice. Please soften this claim or provide a comparative analysis.

minor comments (5)

[Throughout] The manuscript contains rendering errors such as 'diYicult' and 'oYers'; these should be corrected.
[Figure 3] The JSON-LD example uses the field name 'Acquition_instrument_vendor,' which appears to be a typo for 'Acquisition_instrument_vendor.'
[Figure 2 caption] The Figure 2 caption contains extraneous editing artifacts: 'FileEditViewInsertFormatToolsExtensionsHelp 100% Image optionsReplace imageSuggesting1 2 3 4 5 6 7 8 9 10 ⼝11234 AI Magazine - Figure 2 Share.' These should be removed.
[References and text] Author names are inconsistent: 'Martínez-Romano' in the Discussion vs 'Martínez-Romero' in the References, and 'Vidrović' vs 'Vidović'; also 'Tollefsin' should be 'Tollefsen.'
[References] The Hayes-Roth et al. reference appears after 'Vidrović,' breaking alphabetical order; please reorder the reference list.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a descriptive deployment report whose self-citations document implemented tools rather than serving as load-bearing derivation.

full rationale

The paper is a descriptive systems and position report, not a derivation: there is no chain of equations or fitted parameters in which an output reduces to an input. The central claim, that CEDAR templates encode community metadata preferences and can drive Web forms, spreadsheets, validators, and LLM-based metadata conversion, is supported by descriptions of deployed infrastructure and by adoption in external consortia (HEAL, IDG, Health-RI, OSF, Dryad, HuBMAP). Citations to the authors' prior work (O'Connor et al. 2016, 2025a, 2025b; Sundaram et al. 2025) document tool components and evaluation studies; they do not function as an unverified uniqueness theorem or as a fitted input later renamed a prediction. The statement that a template 'formally defines the attributes that constitute a reporting guideline' is a modeling convention, not a self-definitional derivation of an empirical result. The skeptic's concern about conditional cross-field constraints is an expressiveness or correctness limitation, not a circularity, because the paper does not use that modeling convention to deduce the empirical success claims. No circular step can be exhibited by quotation and reduction, so the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No numerical fits or new postulated entities appear in the paper. The argument rests on representational assumptions about metadata standards, not on derived quantities.

assumptions (3)

domain assumption FAIR data require rich, discipline-specific, standardized metadata.
Stated in the Abstract and Introduction as the overarching requirement. The entire argument depends on this premise without defending it.
domain assumption A community's metadata preferences can be captured as finite, structured attribute-value templates.
The CEDAR template model is presented as the representation, but the paper offers no proof or empirical test that all scientific metadata needs are expressible this way.
domain assumption Standardized metadata adhering to community templates is sufficient to make datasets FAIR.
The conclusion asserts that templates can make datasets FAIR, but the paper does not analyze other FAIR dimensions such as persistent identifiers, accessibility protocols, or licensing.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Knowledge engineering for open science: Building and deploying knowledge bases for metadata standards." pith.science (2026). https://pith.science/paper/EPAC33PG

@misc{pith2026250722391,
  author       = {Pith},
  title        = {Pith review of: Knowledge engineering for open science: Building and deploying knowledge bases for metadata standards},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EPAC33PG}},
  note         = {Machine review of arXiv:2507.22391}
}

read the original abstract

Scientists strive to make their datasets available in open repositories, with the goal that they be findable, accessible, interoperable, and reusable (FAIR). Although it is hard for most investigators to remember all the guiding principles associated with FAIR data, there is one overarching requirement: The data need to be annotated with rich, discipline-specific, standardized metadata. The Center for Expanded Data Annotation and Retrieval (CEDAR) builds technology that enables scientists to encode metadata standards as templates that enumerate the attributes of different kinds of experiments. These metadata templates capture preferences regarding how data should be described and what a third party needs to know to make sense of the datasets. CEDAR templates describing community metadata preferences have been used to standardize metadata for a variety of scientific consortia. They have been used as the basis for data-annotation systems that acquire metadata through Web forms or through spreadsheets, and they can help correct metadata to ensure adherence to standards. Like the declarative knowledge bases that underpinned intelligent systems decades ago, CEDAR templates capture the knowledge in symbolic form, and they allow that knowledge to be applied in a variety of settings. They provide a mechanism for scientific communities to create shared metadata standards and to encode their preferences for the application of those standards, and for deploying those standards in a range of intelligent systems to promote open science.

Figures

Figures reproduced from arXiv: 2507.22391 by the authors.

**Figure 1.** Web form for creating metadata instances. The fields in the Web form are generated dynamically from a designated metadata template, in this case for a biological assay known as RNAseq. The user has provided values for the first four fields in the forms. The template indicates that values for the field “Analyte class” come from a predefined value set stored in the BioPortal repository, and the Web form thus displays … view at source ↗

**Figure 2.** Relationships between the CEDAR template model, metadata templates, and metadata instances. Each CEDAR template is an instantiation of the template model and denotes a reporting guideline for some class of experiment. A CEDAR template consists of a set of fields (attributes) that may be grouped into template elements. Each metadata instance is a set of attribute–value pairs that represents an instantiation of some C… view at source ↗

**Figure 4.** The CEDAR Workbench as used by the HEAL consortium. The figure shows CEDAR as used to acquire structured metadata describing a clinical trial, in this case a study of transcutaneous electrical nerve stimulation (TENS) for management of fibromyalgia. investigators to help them to annotate experimental datasets in compliance with standards developed by the community. The HEAL investigators have become adept at metadat… view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: OSF acquires standardized metadata using the CEDAR Embeddable Editor. The figure shows the template for acquiring Psych-DS metadata, a specification for sharing data in the social and behavioral sciences. Here the user is entering structural metadata about a dataset de…

**Figure 6.** Figure 6: A metadata-entry spreadsheet generated from a CEDAR template. The spreadsheet was created from the same template for RNAseq metadata used to create the Web form shown in [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

**Figure 7.** Figure 7: The CEDAR Metadata Validator to ensure the correctness of spreadsheet data. The Validator takes as input a metadata spreadsheet such as the one in [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages

[1]

guiding principles

Knowledge Engineering for Open Science: Building and Deploying Knowledge Bases for Metadata Standards Mark A Musen, Martin J. O’Connor, Josef Hardi, Marcos Martínez-Romero Stanford Center for Biomedical Informatics Research Stanford University School of Medicine Stanford, CA 94305 musen@stanford.edu Abstract Background: For more than a decade, scientists ...

work page 1984
[3]

Illuminating the Druggable Genome

Acquition_instrument_vendor and acquisition_instrument_model have values that are speciﬁc research resource identiﬁers (RRIDs). 9 the advantages of using a core model to represent metadata standards in a reusable, machine-actionable manner. This diversity of implementations underscores how a reusable metadata model—by abstracting community standards into ...

work page 2025
[4]

analyte class

Like the Web form, the spreadsheet uses a predeﬁned value set to ensure that the selected value for “analyte class” adheres to the given standard. Scientists, of course, enjoy this ﬂexibility, but the creation of nonstandard metadata entries risks the generation of datasets that are not FAIR. When metadata do not adhere to standard reporting guidelines, d...

work page 2022
[6]

and Musen, M.A

Gonçalves, R.S. and Musen, M.A. (2019). The variable quality of metadata about biological samples used in biomedical experiments. Scientiﬁc Data 6:Article 190021. Health-RI (2025). Health-RI: Enabling data-driven health and life sciences. Online: https://www.health-ri.nl Accessed July, 14

work page 2019
[7]

HuBMAP Metadata Speciﬁcations

21 HuBMAP Consortium (2025). HuBMAP Metadata Speciﬁcations. Online: https://docs.hubmapconsortium.org/metadata Accessed July 25,

work page 2025
[8]

Jain, S., Pei, L., Spraggins, J.M., et al. (2023). Advances and prospects for the Human Biomolecular Atlas Program. Nature Cell Biology 25:1089–1100. Jonquet, C., Graybeal, J., Bouazzouni, S., et al. (2023). Ontology repositories and semantic artefact catalogues with the OntoPortal technology. In: Payne, T.R., et al. eds. The Semantic Web – ISWC

work page 2023
[10]

Martínez-Romero, M., Horridge, M., Mistry, N., et al. (2025). A Cloud-based platform for harmonized COVID-19 data: design and implementation of the RADx Data Hub. JMIR Public Health and Surveillance, in press. Musen, M.A. (1993). An overview of knowledge acquisition. In: David, J.M., Krivine, J.P. , and Simmons, R., eds. Second Generation Expert Systems. ...

work page 2025
[13]

Sansone, S.-A., McQuilton, P ., Rocca-Serra, P ., et al. (2019). FAIRsharing as a community approach to standards, repositories and policies. Nature Biotechnology 37:358–367. Sawyer-Morris, G., Kushmerick-McCune, B., Clark, K.J., et al. (2025). Advancing research translation in addiction and pain: a portfolio analysis of the NIH HEAL initiative. Journal o...

work page Pith review arXiv 2019

Show all 14 references

[14]

Tollefsin, J., and Van Noorden, R. (2022). US government reveals big changes to open-access policy. Nature 609:234–235. Vendetti, J., Harris, N.L., Dorf, M.V ., et al. (2025). BioPortal: an open community resource for sharing, searching, and utilizing biomedical ontologies. Nu...

2022
[265]

22 Olsen, E., and Corker, K. (2024). Making It easy to make specialized research FAIR: Introducing ﬁeld-speciﬁc metadata templates for OSF . Center for Open Science (April 18, 2024), Online: https://www.cos.io/blog/cedar-embeddable-editor, Accessed July 14,

2024
[696]

O’Connor, M.J., Martínez-Romero, M., Egyedi, A.L., et al. (2016). An open repository model for acquiring knowledge about scientiﬁc experiments. In: Proceedings of the 20th International Conference on Knowledge Engineering and Knowledge Management (EKAW 2016), Lecture Notes in ...

2016
[2019]

AI ready,

provides details for more than 300 such reporting guidelines for standardizing metadata across many branches of science. When investigators create a metadata record, they assign a value to each attribute in the relevant reporting guideline. They thus may say, informally, that ...

2019
[2023]

Lecture Notes in Computer Science 14266, pp. 38–58. Heidelberg: Springer. Lippincott, S. (2024). New at Dryad: Empowering reuse of cognitive neuroscience data. Dryad News, (April 9, 2024), Online: https://blog.datadryad.org/2024/04/09/new-at-dryad-empowering-reuse-of-cognitive...

2024
[2025]

learnable

and that they can markedly improve the performance of large language models asked to convert legacy metadata to a form that is closer to a given community standard (Sundaram et al., 2025). CEDAR templates thus oYer a canonical form in which a community can store and reuse its ...

2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

guiding principles

Knowledge Engineering for Open Science: Building and Deploying Knowledge Bases for Metadata Standards Mark A Musen, Martin J. O’Connor, Josef Hardi, Marcos Martínez-Romero Stanford Center for Biomedical Informatics Research Stanford University School of Medicine Stanford, CA 94305 musen@stanford.edu Abstract Background: For more than a decade, scientists ...

work page 1984

[2] [3]

Illuminating the Druggable Genome

Acquition_instrument_vendor and acquisition_instrument_model have values that are speciﬁc research resource identiﬁers (RRIDs). 9 the advantages of using a core model to represent metadata standards in a reusable, machine-actionable manner. This diversity of implementations underscores how a reusable metadata model—by abstracting community standards into ...

work page 2025

[3] [4]

analyte class

Like the Web form, the spreadsheet uses a predeﬁned value set to ensure that the selected value for “analyte class” adheres to the given standard. Scientists, of course, enjoy this ﬂexibility, but the creation of nonstandard metadata entries risks the generation of datasets that are not FAIR. When metadata do not adhere to standard reporting guidelines, d...

work page 2022

[4] [6]

and Musen, M.A

Gonçalves, R.S. and Musen, M.A. (2019). The variable quality of metadata about biological samples used in biomedical experiments. Scientiﬁc Data 6:Article 190021. Health-RI (2025). Health-RI: Enabling data-driven health and life sciences. Online: https://www.health-ri.nl Accessed July, 14

work page 2019

[5] [7]

HuBMAP Metadata Speciﬁcations

21 HuBMAP Consortium (2025). HuBMAP Metadata Speciﬁcations. Online: https://docs.hubmapconsortium.org/metadata Accessed July 25,

work page 2025

[6] [8]

Jain, S., Pei, L., Spraggins, J.M., et al. (2023). Advances and prospects for the Human Biomolecular Atlas Program. Nature Cell Biology 25:1089–1100. Jonquet, C., Graybeal, J., Bouazzouni, S., et al. (2023). Ontology repositories and semantic artefact catalogues with the OntoPortal technology. In: Payne, T.R., et al. eds. The Semantic Web – ISWC

work page 2023

[7] [10]

Martínez-Romero, M., Horridge, M., Mistry, N., et al. (2025). A Cloud-based platform for harmonized COVID-19 data: design and implementation of the RADx Data Hub. JMIR Public Health and Surveillance, in press. Musen, M.A. (1993). An overview of knowledge acquisition. In: David, J.M., Krivine, J.P. , and Simmons, R., eds. Second Generation Expert Systems. ...

work page 2025

[8] [13]

Sansone, S.-A., McQuilton, P ., Rocca-Serra, P ., et al. (2019). FAIRsharing as a community approach to standards, repositories and policies. Nature Biotechnology 37:358–367. Sawyer-Morris, G., Kushmerick-McCune, B., Clark, K.J., et al. (2025). Advancing research translation in addiction and pain: a portfolio analysis of the NIH HEAL initiative. Journal o...

work page Pith review arXiv 2019