Datasheets for AI and medical datasets (DAIMS): a data validation and documentation framework before machine learning analysis in medical research

Anne Svane Frahm (1); Maja Milojevic (1); Ramtin Zargari Marandi (1)

REVIEW 4 major objections 4 minor 28 references

Datasheets for AI and medical datasets (DAIMS): a data validation and documentation framework before machine learning analysis in medical research

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DAIMS is a new framework that extends Datasheets for Datasets to medical ML, adding a 24-item standardization checklist, an automated validation tool, a data dictionary, and a flowchart that maps research questions to suggested ML methods.

desk verdict DAIMS is a practical, incremental extension of Datasheets for Datasets for medical ML, but its automated-validation claim is over-stated and it lacks empirical evaluation. read the letter →

arxiv 2501.14094 v1 pith:ASVQFVXX submitted 2025-01-23 cs.LG

Ramtin Zargari Marandi (1) , Anne Svane Frahm (1) , Maja Milojevic (1) This is my paper · ORCID

classification cs.LG

keywords datavalidationdocumentationmedicaldatasetsmachinelearningstandardizationdatasheetsfordictionarychecklist

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes DAIMS (Datasheets for AI and Medical Datasets), a data-validation and documentation framework meant to sit before machine learning analysis on medical tabular datasets. It claims that medical datasets prepared for ML need a dedicated, domain-specific standardization process, and that existing frameworks like Datasheets for Datasets or CDISC do not cover this fully. DAIMS supplies a 24-item data-cleaning checklist, an open-source tool that automatically checks the first 15 items, an extended documentation form with a data dictionary, and a flowchart that maps research questions to suggested ML methods. If DAIMS is taken up, it would give medical researchers a common reference for what counts as a clean, documented, ML-ready dataset and a clearer route from question to analysis.

What carries the argument

The load-bearing object is the DAIMS checklist of 24 data-standardization requirements, where each item is marked as done, not done, or unsure, and the first 15 items are automatically checked by an open-source validation tool that verifies wide format, unique patient IDs, absence of Unicode characters and stray separators, consistent missing-value encoding, and data-dictionary consistency. The checklist's score, #done/(#done+#not done), quantifies documentation completeness. The other components—the extended datasheet form, the data dictionary template, and the ML-method flowchart—support this by documenting the dataset, defining variables and their measurement error, and guiding model choice; together they operationalize what 'ML-ready medical dataset' means.

What would settle it

Run the DAIMS tool and checklist on a corpus of real medical datasets alongside a manual expert audit, and count data defects that escape the 24 items; if a common defect (e.g., label leakage, unit mix-ups inside a single variable, or date-formats not listed in the dictionary) appears that no checklist item flags, the completeness claim is refuted. Similarly, a randomized comparison showing that datasets passing all 24 items do not yield fewer downstream data-related errors than datasets failing several items would refute the framework's central usefulness claim.

Watch

Extended reading notes

Core claim

DAIMS is a comprehensive framework that standardizes the preparation of medical tabular datasets for machine learning. The paper's contribution is the framework itself: a 24-item checklist of data standardization requirements, a software tool that validates 15 of those items automatically, a datasheet form that extends the Datasheets for Datasets questionnaire with medical-domain concerns (privacy, GDPR, ethics, terminology), a data dictionary template that adds 'typical observation error' to standard variable descriptions, and a flowchart suggesting ML methods conditional on data type and task. The checklist is scored as #done/(#done+#not done), giving a quick measure of documentation completeness. The framework is intended as a reference and roadmap for researchers, and the authors argue it addresses gaps left by previous standards for clinical data.

Load-bearing premise

The framework assumes its 24 checklist items are a complete and sufficient set of data-standardization requirements for medical ML datasets, but the paper offers no empirical evidence that these 24 items cover every necessary check or that completing them improves downstream machine-learning outcomes.

Editorial extensions

If this is right

If DAIMS is followed, medical datasets entering ML pipelines will share a common structural baseline: wide format, unique IDs, no stray separators, consistent missing entries, and a data dictionary that defines every variable.
The 24-item checklist gives reviewers, journals, and consortia a concrete scale (e.g., 15/24) for reporting how complete a dataset's documentation is, making data-preparation work visible and auditable.
The flowchart lets a researcher start from a research question and arrive at a suggested model family (e.g., logistic regression for binary tabular prediction, tree ensembles for performance, RNN/graphical models for text or time series), narrowing the model-selection space before any code is written.
The data-dictionary template's 'typical observation error' column formalizes measurement-error reporting, which is rarely included but needed for computing uncertainty in per-patient predicted risks.
DAIMS also functions as a living document for dataset versioning and warehousing, so a single dataset can be reused across studies with a record of what changed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not empirically validate that using DAIMS improves downstream ML results; a natural test would compare data-defect rates and model reproducibility in cohorts prepared with and without DAIMS.
The 24-item checklist may be necessary but not sufficient: many ML failure modes (label leakage, sampling bias, distribution shift, small sample size) are not covered, so DAIMS is best read as a hygiene check before deeper design decisions.
Because the first 15 items are automatable, the checklist could be turned into a continuous-integration step in data-warehousing pipelines, flagging regressions whenever a dataset is updated.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

DAIMS is a practical extension of Gebru et al.'s Datasheets for Datasets, aimed at medical tabular datasets. What's actually new is the combination: a 24-item standardization checklist, a data dictionary template that includes 'typical observation error' (a useful addition), a flowchart for picking ML methods by research question, and a Streamlit tool that automates part of the checklist. The code and app are public and easy to run. For a medical research team without dedicated data engineers, that is a genuinely useful starting point.

The main soft spot is the automated-validation claim. The paper states the first 15 checklist items are checked by the tool, but items 6, 10, 11, 12, 13 and 15 depend on knowing the target variable, having a data dictionary, or making clinical judgments. The manuscript never specifies how the tool learns the target column, whether it accepts a data dictionary as input, or how it decides that rare categories were 'grouped' or outliers are 'irrelevant.' So the tool's objective checks are likely limited to the syntactic items (1-5, 7-9). The DAIMS score, as presented, is not an objective ML-readiness measure. That should be rewritten as 'the tool checks a subset of the syntactic items; the rest require manual review.'

The larger limitation is the absence of any evaluation. There is no demonstration on a real dataset, no comparison with ydata-profiling or Great Expectations, and no evidence that following the checklist or scoring 24/24 improves downstream ML outcomes. That's common for framework proposals and not fatal, but it means the value rests on plausibility rather than evidence.

What the paper does well: it is clearly written, the checklist is sensible, and it honestly acknowledges that manual checks and additional pipelines are still needed. The flowchart is very coarse and the paper admits it is not exhaustive. The self-citations are background references and not load-bearing.

Who is this for? Medical researchers who want a checklist and tool to prepare and document tabular datasets before ML analysis, and anyone teaching data preparation. It won't change the world, but it's a usable process improvement.

I'd send it to peer review, with the request that the authors clarify exactly which items the tool can check automatically and show the tool running on a sample dataset, even synthetic, to make the claims reproducible.

Referee Report

4 major / 4 minor

Summary. The manuscript introduces DAIMS (Datasheets for AI and medical datasets), a framework intended to standardize data preparation and documentation for machine learning in medical research. The framework consists of four components: a 24-item data-cleaning checklist (Table 1), an open-source data-validation tool implemented with Streamlit and Python, an extended datasheet form adapted from Gebru et al., and a flowchart that maps research questions to suggested ML methods. The paper reports that the first 15 checklist items can be checked by the DAIMS tool, with the remaining items checked manually, and that a binary score (#done/(#done+#not done)) measures documentation completeness. The GitHub repository and online app are provided.

Significance. If the framework is taken as a proposal rather than a validated instrument, the contribution is of moderate practical value: it packages existing data-hygiene practices into a concrete, medically oriented checklist, provides a readily usable open-source tool, and makes an explicit documentation template publicly available. A particularly useful element is the inclusion of 'Typical Observation Error' in the data dictionary, which is rarely reported in medical ML papers. The strengths are the availability of the code and online app, the clear presentation of the checklist, and the attention to medical-data-specific issues such as GDPR, informed consent, and outcome-variable definition. However, the paper provides no empirical evidence that the checklist is complete, that the tool reliably automates the claimed checks, or that following DAIMS improves downstream ML outcomes; these gaps currently limit the significance to that of a proposal rather than a demonstrated standard.

major comments (4)

[Overview, Table 1] The central claim that the first 15 checklist items can be checked by the DAIMS validation tool is underspecified and, as written, not reproducible. Items 6, 10, 11, 12, 13, and 15 of Table 1 all require information external to the raw data: the target variable, a data dictionary, or clinical judgment about grouping rare categories and removing outliers. The manuscript does not state how the tool obtains the outcome-variable position, whether it accepts a data dictionary as input, how it maps dictionary ranges and categories onto observed values, or how it adjudicates subjective items such as 'rare categories are grouped' and 'irrelevant observations are removed'. Without this specification, the claim that the tool automates these checks cannot be verified, and the resulting DAIMS score cannot be interpreted as an objective measure of ML-readiness.
[Discussion] The paper provides no empirical evaluation of the framework's utility. There is no application of the checklist and tool to even one representative medical dataset, no comparison with existing profilers such as ydata-profiling and Great Expectations (which are cited in the Discussion), and no measurement of inter-rater agreement or of whether achieving a high DAIMS score is associated with improved data quality or ML performance. Consequently, statements that DAIMS 'enhances consistency', 'facilitates efficient preparation', and 'serves as a reference for standardizing datasets' are asserted rather than demonstrated. For a framework paper this is not fatal, but the absence of any worked case study makes it difficult to assess whether the checklist is complete, whether the tool's checks are correct, and whether the flowchart's recommendations would actually guide a user to a suitable model.
[Overview, paragraph on the checklist] The manuscript contains an internal inconsistency about which items are automated. It states that 'The first 15 items can be checked by the DAIMS data validation tool' and that 'the rest of the items must be checked manually', but the very next sentence says 'Some items are left to be checked manually because of the technical complexities around them.' The abstract also says only that the tool checks 'a subset' of the 24 items. The authors should state precisely which items the tool checks, which it does not, and why; this distinction is load-bearing because the abstract and highlights emphasize automated data quality checks as a key contribution.
[Flowchart (Figure 1)] The flowchart is described as a roadmap for selecting ML analyses, but the manuscript provides no evidence that its recommendations are sensible or useful in practice. Items such as 'If model performance needs to be improved, more complex models are suggested' are generic, and no examples trace a realistic research question through the flowchart to a concrete method. A single worked example, or at least a discussion of how the flowchart was derived from common practices, would strengthen the claim that it serves as a practical guide rather than an arbitrary decision tree.

minor comments (4)

[Abstract] The abstract says the tool 'checks and validate' a subset of the checklist; the grammar should be corrected ('checks and validates'), and the abstract should state explicitly which subset is automated to avoid the inconsistency noted above.
[Table 2] The data dictionary example lists 'Diagnosis' with categories 'Negative; Positive' but the example value is 'Diabetes'; these are inconsistent, as a diabetes diagnosis is not well represented as negative/positive. Either the categories or the example should be changed.
[Table 2] The 'Role' column contains 'Other' for 'Notes', but the text says all items except 'Typical Observation Error' are necessary for ML studies; the paper should clarify whether free-text 'Notes' variables are intended to be used as predictors or excluded from the analysis.
[Discussion, first paragraph] The phrase 'notably the Datasheet for datasets' appears in the summary paragraph and should be italicized or quoted for consistency with the rest of the manuscript, which uses 'datasheets for datasets'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DAIMS is a proposed checklist and documentation framework whose validity is asserted rather than derived, and no prediction or fitted result reduces to its own inputs.

full rationale

DAIMS is a framework proposal, not a derivation chain, so the circularity patterns of fitted-inputs-called-predictions and self-citational forcing do not apply. The paper's central claim is that the framework provides a 24-item checklist, a software tool, a documentation form, a data dictionary, and a flowchart for selecting ML methods. Nothing in this claim is obtained by fitting data or by invoking a uniqueness theorem. The first 15 checklist items are said to be checkable by the tool, but several of them require an external data dictionary or clinical context (e.g., items 6, 10, 11, 12, 13, 15); this is an underspecification or reproducibility limitation, not circularity, because checking a column against an external dictionary does not define the checklist's validity. The self-citations (refs. 12, 16, 27) are background references for model fusion, SHAP, and a downstream ML toolbox, respectively, and are not load-bearing for the framework's completeness or correctness. The lack of empirical evidence that achieving 24/24 improves ML outcomes is a correctness and validation gap, which the paper itself acknowledges by stating that researchers remain responsible for confirming dataset suitability and that additional checks are needed. Therefore the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No free parameters because no quantitative model is proposed. The checklist and flowchart are domain assumptions without empirical backing. No invented physical or formal entities are introduced.

assumptions (2)

domain assumption The 24 checklist items represent the common data standardization requirements for medical ML datasets.
The authors state this without empirical validation or a formal derivation; the list is asserted as the set of requirements (Table 1).
domain assumption The flowchart correctly maps research questions to appropriate ML methods.
The flowchart design is based on common ML tasks, but no evidence or benchmark supports the completeness or correctness of the mapping (Figure 1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Datasheets for AI and medical datasets (DAIMS): a data validation and documentation framework before machine learning analysis in medical research." pith.science (2026). https://pith.science/paper/ASVQFVXX

@misc{pith2026250114094,
  author       = {Pith},
  title        = {Pith review of: Datasheets for AI and medical datasets (DAIMS): a data validation and documentation framework before machine learning analysis in medical research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ASVQFVXX}},
  note         = {Machine review of arXiv:2501.14094}
}

read the original abstract

Despite progresses in data engineering, there are areas with limited consistencies across data validation and documentation procedures causing confusions and technical problems in research involving machine learning. There have been progresses by introducing frameworks like "Datasheets for Datasets", however there are areas for improvements to prepare datasets, ready for ML pipelines. Here, we extend the framework to "Datasheets for AI and medical datasets - DAIMS." Our publicly available solution, DAIMS, provides a checklist including data standardization requirements, a software tool to assist the process of the data preparation, an extended form for data documentation and pose research questions, a table as data dictionary, and a flowchart to suggest ML analyses to address the research questions. The checklist consists of 24 common data standardization requirements, where the tool checks and validate a subset of them. In addition, we provided a flowchart mapping research questions to suggested ML methods. DAIMS can serve as a reference for standardizing datasets and a roadmap for researchers aiming to apply effective ML techniques in their medical research endeavors. DAIMS is available on GitHub and as an online app to automate key aspects of dataset evaluation, facilitating efficient preparation of datasets for ML studies.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 24 canonical work pages

[1]

Gebru, T. et al. Datasheets for datasets. Commun. ACM 64, 86–92 (2021)

work page 2021
[2]

https://www.cdisc.org/ (2024)

Clinical Data Interchange Standards Consortium. https://www.cdisc.org/ (2024)

work page 2024
[3]

& Huser, V

Hume, S., Aerts, J., Sarnikar, S. & Huser, V. Current applications and future directions for the CDISC Operational Data Model standard: A methodological review. J. Biomed. Inform. 60, 352–362 (2016)

work page 2016
[4]

K., Nossent, J., Woldegiorgis, B

Zadeh, F. K., Nossent, J., Woldegiorgis, B. T., Bauwens, W. & van Griensven, A. Impact of measurement error and limited data frequency on parameter estimation and uncertainty quantification. Environ. Model. \& Softw. 118, 35–47 (2019)

work page 2019
[5]

M., Lee, J

Jennings, J., Kim, J. M., Lee, J. & Taylor, D. Measurement error, fixed effects, and false positives in accounting research. Rev. Account. Stud. 29, 959–995 (2024)

work page 2024
[6]

& Sun, J

He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. in Proceedings of the IEEE conference on computer vision and pattern recognition 770–778 (2016)

work page 2016
[7]

& Weinberger, K

Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. in Proceedings of the IEEE conference on computer vision and pattern recognition 4700– 4708 (2017)

work page 2017
[8]

& Brox, T

Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. in Medical image computing and computer-assisted intervention--MICCAI 2015: 18th 9 international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 234–241 (2015)

work page 2015

Show all 28 references

[9]

Ke, G. et al. LightGBM: A highly efficient gradient boosting decision tree. in Advances in Neural Information Processing Systems (2017)

2017
[10]

V., Ershov, V

Dorogush, A. V., Ershov, V. & Gulin, A. CatBoost: gradient boosting with categorical features support. (2018)

2018
[11]

Wang, Y. et al. Deep time series models: A comprehensive survey and benchmark. arXiv Prepr. arXiv2407.13278 (2024)

2024 arXiv
[12]

Zargari Marandi, R. et al. Pre-Transplant Prediction of Acute Graft-versus-Host Disease Using the Gut Microbiome. Cells 11, 4089 (2022)

2022
[13]

Naveed, H. et al. A comprehensive overview of large language models. arXiv Prepr. arXiv2307.06435 (2023)

2023 arXiv
[14]

& Wager, S

Nie, X. & Wager, S. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika 108, 299–319 (2021)

2021
[15]

Lundberg, S. M. & Lee, S.-I. A unified approach to interpreting model predictions. in Advances in neural information processing systems 4765–4774 (2017)

2017
[16]

ExplaineR: an R package to explain machine learning models

Zargari Marandi, R. ExplaineR: an R package to explain machine learning models. Bioinforma. Adv. 4, vbae049 (2024)

2024
[17]

& others

McKinney, W. & others. pandas: a foundational Python library for data analysis and statistics. Python high Perform. Sci. Comput. 14, 1–9 (2011)

2011
[18]

Python data science handbook: Essential tools for working with data

VanderPlas, J. Python data science handbook: Essential tools for working with data. (‘ O’Reilly Media, Inc.’, 2016)

2016
[19]

Virtanen, P. et al. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat. Methods 17, 261–272 (2020)

2020
[20]

Hunter, J. D. Matplotlib: A 2D graphics environment. Comput. Sci. \& Eng. 9, 90–95 (2007)

2007
[21]

Waskom, M. L. Seaborn: statistical data visualization. J. Open Source Softw. 6, 3021 (2021)

2021
[22]

Clemente, F. et al. ydata-profiling: Accelerating data-centric AI with high-quality data. Neurocomputing 554, 126585 (2023)

2023
[23]

Narayanan, P. K. Data Validation using Great Expectations. in Data Engineering for Machine Learning Pipelines: From Python Libraries to ML Pipelines and Cloud Platforms 197–223 (Springer, 2024)

2024
[24]

Wickham, H. et al. Welcome to the Tidyverse. J. open source Softw. 4, 1686 (2019)

2019
[25]

Lee, J. et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 1234–1240 (2020)

2020
[26]

& Liu, Q

Yang, Y., Wang, Y., Sen, S., Li, L. & Liu, Q. Unleashing the Potential of Large Language Models for Predictive Tabular Tasks in Data Science. arXiv Prepr. arXiv2403.20208 (2024)

2024 arXiv
[27]

Z., Frahm, A

Marandi, R. Z., Frahm, A. S., Lundgren, J., Murray, D. D. & Milojevic, M. Medical artificial intelligence toolbox (MAIT): an explainable machine learning framework for binary classification, survival modelling, and regression analyses. arXiv Prepr. arXiv2501.04547 (2025)

2025 arXiv
[28]

Paetzold, J. C. et al. Whole Brain Vessel Graphs: A Dataset and Benchmark for Graph Learning and 10 Neuroscience (VesselGraph). (2021) doi:10.5281/zenodo.5301621

2021 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Gebru, T. et al. Datasheets for datasets. Commun. ACM 64, 86–92 (2021)

work page 2021

[2] [2]

https://www.cdisc.org/ (2024)

Clinical Data Interchange Standards Consortium. https://www.cdisc.org/ (2024)

work page 2024

[3] [3]

& Huser, V

Hume, S., Aerts, J., Sarnikar, S. & Huser, V. Current applications and future directions for the CDISC Operational Data Model standard: A methodological review. J. Biomed. Inform. 60, 352–362 (2016)

work page 2016

[4] [4]

K., Nossent, J., Woldegiorgis, B

Zadeh, F. K., Nossent, J., Woldegiorgis, B. T., Bauwens, W. & van Griensven, A. Impact of measurement error and limited data frequency on parameter estimation and uncertainty quantification. Environ. Model. \& Softw. 118, 35–47 (2019)

work page 2019

[5] [5]

M., Lee, J

Jennings, J., Kim, J. M., Lee, J. & Taylor, D. Measurement error, fixed effects, and false positives in accounting research. Rev. Account. Stud. 29, 959–995 (2024)

work page 2024

[6] [6]

& Sun, J

He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. in Proceedings of the IEEE conference on computer vision and pattern recognition 770–778 (2016)

work page 2016

[7] [7]

& Weinberger, K

Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. in Proceedings of the IEEE conference on computer vision and pattern recognition 4700– 4708 (2017)

work page 2017

[8] [8]

& Brox, T

Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. in Medical image computing and computer-assisted intervention--MICCAI 2015: 18th 9 international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 234–241 (2015)

work page 2015