Pith. sign in

REVIEW 4 major objections 6 minor 49 references

A Generative Approach for Semantic Auditing of Electronic Health Records

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-generated semantic unit tests find EHR divergences from epidemiological knowledge that pass syntactic checks; on three datasets 90-97% of such tests failed.

desk verdict A fresh demonstration of LLM-generated semantic tests for EHRs, but the 90–97% failure rate is uncalibrated without a negative control. read the letter →

arxiv 2507.02628 v2 pith:OTDRPO6C submitted 2025-07-03 cs.LG

Irena Girshovitz , Atai Ambus , Moni Shahar , Ran Gilad-Bachrach This is my paper
classification cs.LG
keywords electronichealthrecordssemanticdataqualityauditinglargelanguagemodelsunittestingretrieval-augmentedgenerationepidemiologicalpriorsgapMedicalPecking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Electronic health records routinely pass structural checks while remaining inconsistent with established medical knowledge, and today's quality tools either validate syntax or demand hand-written rules that do not scale. This paper proposes Medical Data Pecking, a software unit-testing approach in which a large language model generates context-aware semantic tests from the literature — expected comorbidity rates, demographic shares, lab ranges — and executes them against a target cohort. The reference tool generated between 55 and 73 tests per cohort across three datasets and reported that 90-97% of semantic tests failed, catching both genuine data problems and expected cohort-selection effects. If the approach works as claimed, it gives data teams a scalable first-pass semantic audit layer that runs alongside syntax checks without manual rule authoring.

What carries the argument

The carrying object is the semantic unit test: an executable assertion that compares an observed EHR statistic with an LLM-derived epidemiological expectation. The machinery around it has four parts: a three-level validation taxonomy (metadata, distributional, and subpopulation or contextual); a retrieval-augmented generation pipeline that turns a study specification into a test matrix using web-searched regional statistics and vector-search mapping into standardized vocabularies such as OMOP, ICD-9, and SNOMED; an Auditor Agent double-pass verification loop, inspired by self-refinement, that re-searches each reference value and fixes or discards unsupported anchors; and the DFtest execution engine, which flags a test when Welch's t-test gives p < 0.05, the standardized mean difference reaches 0.2, or the observed-to-expected ratio falls outside 0.85–1.15. The design's key move is decoupling generation from execution: the LLM only ever sees the study specification and data dictionary, the test matrix is frozen as a versioned asset, and raw patient data stays inside the user's secure environment.

What would settle it

Run MDPT on a dataset whose population is known to match the reference priors almost exactly — for instance, a national survey sample with verified coding — and restrict the audit to tests whose references are independently confirmed. If failure rates stay near 90%, the signal is coming from the anchors, thresholds, or code-mapping pipeline rather than from genuine gaps in the data; if failures fall toward noise, the method is separating true semantic gaps from ordinary cohort variation.

Watch

Extended reading notes

Core claim

The paper's central claim is that syntactic validity and semantic plausibility are distinct, measurable properties of a dataset, and that the gap between them can be probed automatically. It defines Semantic Data Coverage as the share of fields validated against epidemiological ground truth and a Semantic Gap as a statistically significant divergence in data that is structurally intact. The reference implementation, MDPT, grounds every expected value in a retrieval pipeline that searches regional statistics, maps concepts to standardized vocabularies, and runs a second LLM pass — the Auditor Agent — that independently re-searches and corrects or discards unsupported values before tests are persisted as a versioned test matrix. Across four cohorts (T2D and CKD in All of Us, CHF in MIMIC-III, HTN in SyntheticMass), between 90% and 97% of all semantic tests failed, and 87% to 100% of tests whose references were verified as correct also failed. The paper reads these failures as a measure of the distance between the dataset and literature baselines, not as proof that the data is corrupt: it explicitly flags the 'reverse ecological fallacy' of applying general-population priors to specialized cohorts, and it shows one concrete data error the approach caught — MIMIC-III ICD-9 codes stored without decimal points, which made valid diagnoses invisible to standard phenotype algorithms.

Load-bearing premise

The load-bearing premise is that the expected values the language model retrieves are accurate and genuinely apply to the cohort, coding system, and time period at hand, so that a failed test indicates a semantic gap in the data rather than a bad benchmark; the paper's own validation found correct references in only 35% to 93% of generated tests (Table S5), with drug-prevalence anchors the least reliable.

Editorial extensions

If this is right

  • If the approach is correct, a one-line study specification can yield dozens of context-specific semantic tests without a human writing any rules, making semantic auditing practical at the scale of real EHR pipelines.
  • Failure lists double as cohort characterization: an ICU dataset such as MIMIC-III will legitimately fail many general-population tests, and that failure pattern quantifies how the study population differs from the literature baseline.
  • Formatting artifacts that silently break downstream phenotype algorithms, such as ICD-9 codes stored without decimal points, become detectable through implausible prevalence rather than requiring inspection of the raw strings.
  • Persisting the validated test matrix as a versioned asset gives audits a reproducible trail, and reporting Passed / Failed / Inconclusive as three distinct states prevents an absence of warnings from being mistaken for a certificate of data quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because Hypertension in SyntheticMass had the highest reference validity (93%) yet the highest failure rate (97.3%), while CHF in MIMIC-III had the lowest validity (35.2%) yet the lowest failure rate (92.7%), the reported failures do not simply track anchor quality; that pattern supports the paper's reading of a genuine divergence signal, but it deserves a direct, controlled test.
  • A natural extension the paper leaves unrun is an injection study: corrupt a clean cohort's codes, formats, and recorded values in known ways, then measure whether the generated tests fire on exactly those fields — a sensitivity-and-specificity profile that the current positive control (demographic skews) only partially provides.
  • The paper notes that internal institutional baselines could eventually replace general-population literature as ground truth; carried further, the same frozen test matrix could run continuously on each new data load as a live institutional data monitor, turning one-time audits into ongoing governance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript introduces Medical Data Pecking (MDP), a framework for semantic auditing of electronic health records (EHRs) that adapts software unit-testing principles to clinical data. The reference implementation (MDPT) uses a retrieval-augmented generation pipeline with GPT-4o and Bing Search to generate executable unit tests from epidemiological literature, an 'Auditor Agent' to verify expected values, and a custom execution engine (DFtest) that flags deviations using Welch's t-test, standardized mean difference, and a proportional tolerance interval. The authors apply MDPT to four cohorts (Type 2 Diabetes and Chronic Kidney Disease from All of Us, Congestive Heart Failure from MIMIC-III, and Hypertension from SyntheticMass) and report that 90–97% of generated semantic tests failed. They also report a positive control (detection of known demographic skew in All of Us), a sensitivity analysis over thresholds, and a detailed failure taxonomy, concluding that MDPT can complement syntax-only data-quality tools by identifying 'semantic gaps' where data are syntactically valid but epidemiologically implausible. The central claim is that LLM-generated semantic unit tests can automatically surface discrepancies that escape syntactic checks.

Significance. If validated, the proposed framework would address a real and growing need for scalable, context-aware EHR data-quality assessment, and the paper has several strengths: the code is publicly available, the architecture cleanly separates generative synthesis from local execution, the positive control provides a minimal sensitivity check, and the Discussion candidly acknowledges the 'reverse ecological fallacy' and the implied-truth effect. The sensitivity analysis over statistical thresholds is a useful robustness check, and the tri-state reporting proposal is a constructive step. However, the quantitative headline (90–97% failure) is not yet calibrated: the manuscript does not show that the test suite passes when data actually match the anchors, and the low reference validity reported in Table S5 means that a large fraction of tests are anchored on values that even the authors could not confirm. The significance of the work therefore depends entirely on completing the calibration and anchor-validity experiments described below.

major comments (4)
  1. [Methods, 'Execution and Statistical Validation'; Figure 2D] The failure rule is permissive (Welch p<0.05, SMD≥0.2, or observed/expected ratio outside 0.85–1.15), but no negative control demonstrates that tests pass when the data are actually consistent with the anchors. The Results report that 90–97% of tests fail in all cohorts, yet this number has no meaning unless the machinery is shown to have a low failure rate on data that match the epidemiological priors. A load-bearing missing experiment is to run the same pipeline on synthetic data generated to satisfy the anchor values (or on a real cohort segment known to align with the anchors) and report the pass/fail rate; without such a control, the system is indistinguishable from a permissive flag generator that fails almost everything.
  2. [Table S5 and Results, 'Validation of Automated Test Generation'] Reference validity ranges from 35.2% (CHF) to 93.0% (HTN), and the 'correct' label is assigned through a single-reviewer curation process. Since a failed test is only informative when the anchor is trustworthy, the high failure rates among 'correct' tests (Figure 2C) inherit the anchor-quality problem. The paper should provide an inter-rater reliability assessment (e.g., two independent reviewers on a sample of tests) and report outcomes restricted to anchors that are independently confirmed against primary sources. It should also report how often the Auditor Agent changed a value and validate the Auditor's corrections against a gold standard, rather than relying on the same LLM family's self-assessment.
  3. [Methods, 'Automated Grounding and Evidence Verification'] The Auditor Agent and the Generator both use GPT-4o, so the 'independent' verification loop does not actually provide an independent source of ground truth. This is a mild circularity that weakens the claim that 'every test is anchored in verifiable sources' (Discussion). The authors should test whether the Auditor's corrections are corroborated by a different model family (e.g., an open-weight biomedical LM or a second commercial model) or by human verification on a random sample. At minimum, report the agreement rate between the Auditor and human review for the tests where the Auditor changed the expected value.
  4. [Discussion, 'reverse ecological fallacy'; Results, 'Uncovering Semantic Divergence'] The Discussion explicitly concedes that a specialized cohort (e.g., MIMIC-III ICU) can fail many tests even when the data are perfectly clean, and the AoU positive control itself shows that known recruitment skew produces failures. Yet the Results present the 90–97% failure rate as the primary finding without stratifying failures by root cause. The manuscript should classify each failed test (formatting artifact, cohort-selection effect, anchor mismatch, genuine data error) and report how many flags survive this classification. Without that stratification, the headline failure rate conflates the framework's detection signal with the noise floor created by the admitted reverse ecological fallacy.
minor comments (6)
  1. [Figure 2 caption] The caption refers to 'Section S5' for metric definitions, but the terms 'correct' and 'non-correct' are used in the main text before they are formally defined; consider moving the definitions of 'accurate', 'inaccurate', 'incorrect', 'qualitative', and 'irrelevant' into the main text or a dedicated section.
  2. [Table S5] The percentages in Table S5 do not sum cleanly (e.g., T2D: 59.3+16.9+20.3+0+3.9 = 100.4), and the relationship between 'Valid References' and the listed categories is unclear; please clarify whether 'valid' includes only 'accurate' or also 'inaccurate', and fix rounding so the column sums are transparent.
  3. [Results, 'Uncovering Semantic Divergence'] The sentence describing CHF states 'failure rates within this verified subset were 87.1% ... (36.4% failed vs. 5.5% passed)', but the percentages in parentheses appear to be relative to all tests, not to the correct subset; specify the denominator explicitly for each cohort.
  4. [Supplementary Table S11] The table title says 'when tests that evaluate prevalence of drug usage are disregarded', but the criterion for identifying drug-prevalence tests is not stated; please define it in the caption or methods.
  5. [Results, 'Failure and Robustness Analysis'] The zero-prevalence analysis is informative, but the text does not state how 'No Reference' tests are mapped to pass/fail in the execution framework; given the Discussion's call for tri-state reporting, clarify whether 'No Reference' is currently scored as a failure, a pass, or excluded.
  6. [Abstract and Introduction] The abstract's phrase 'These discrepancies encompass both genuine data inconsistencies and expected cohort-selection effects' is an important qualification that appears only at the end; consider placing a similar caveat alongside the 90–97% failure-rate claims in the Results so readers are not misled by the raw number.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the core comparison is observed data versus externally retrieved epidemiological priors, not versus fitted parameters; one minor self-referential validation element (Auditor Agent from the same LLM family) is noted but is not load-bearing.

full rationale

The load-bearing comparison in MDPT is between observed EHR distributions and expected values retrieved from web search and literature via RAG (Methods, 'Automated Grounding and Evidence Verification'; Figure 5). The expected values are not fitted to the audited data, so the 90-97% failure rates are not forced by construction. The positive control (AoU demographic skew) is an external, known recruitment bias, and the paper's Discussion explicitly concedes the 'reverse ecological fallacy' and the absence of a negative control, which are calibration and correctness risks rather than circularity. Two mild self-referential elements exist but do not reduce the central claim to its inputs: (1) the Auditor Agent uses GPT-4o, the same model family as the Generator, so the 'correct' anchor label is partially self-assessed; the paper states 'both the Generator and Auditor utilize GPT-4o; however, independence is maintained through information decoupling'; this is a validation-reliability concern, not an equation-level reduction. (2) The reference-value correctness is judged by a 'single-reviewer curation process' (Table S5 caption), again a self-assessment caveat. Neither element defines the prediction in terms of the outcome, and neither is a self-citation. No circular step can be exhibited by quoting the paper's own equations; therefore the score is 1, reflecting only the minor self-referential validation element.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework introduces procedural concepts (Semantic Gap, Semantic Data Coverage) but no new physical or mechanistic entities. The free parameters are the statistical thresholds used to declare test failure. The axioms are the domain assumptions that the expected values are correct, the concept mapping is correct, the auditor is independent, and the statistical tests are appropriate.

free parameters (3)
  • Proportional tolerance interval = 0.85 to 1.15 (15% deviation)
    Hand-chosen threshold for categorical prevalence tests; sensitivity analysis (Figure 3) shows failure rates vary with this value. Cited to prior EHR validation work (ref 36), but not derived from the current data.
  • SMD threshold = 0.2
    Hand-chosen cutoff for continuous variable deviations, meant to distinguish statistical from clinical significance (ref 35); sensitivity analysis shows modest effects.
  • Welch t-test alpha = 0.05
    Standard significance level used for distributional equivalence tests; acts as a fixed calibration setting.
assumptions (4)
  • domain assumption Web-retrieved epidemiological statistics are accurate and apply to the cohort being audited.
    The whole test suite is built on these expected values; Methods 'Automated Grounding and Evidence Verification'. Observed reference validity as low as 35% (Table S5) puts this in doubt.
  • domain assumption LLM concept mapping to standardized vocabularies (OMOP, ICD-9, SNOMED) is correct.
    Tests rely on concept IDs; mismapping leads to zero-prevalence false failures (Table S14).
  • domain assumption The Auditor Agent's separate search provides independent verification.
    Both Generator and Auditor use GPT-4o; independence is only asserted through separate search threads (Methods 'The Auditor Agent (Double-Pass Verification)').
  • domain assumption Statistical tests are appropriate for comparing EHR distributions to literature priors.
    Welch t-test, SMD, and proportional tolerance are applied uniformly; calibration is acknowledged as reference settings (Methods 'Execution and Statistical Validation').

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Generative Approach for Semantic Auditing of Electronic Health Records." pith.science (2026). https://pith.science/paper/OTDRPO6C

@misc{pith2026250702628,
  author       = {Pith},
  title        = {Pith review of: A Generative Approach for Semantic Auditing of Electronic Health Records},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OTDRPO6C}},
  note         = {Machine review of arXiv:2507.02628}
}
read the original abstract

The reliability of clinical artificial intelligence (AI) depends on high-quality data, yet Electronic Health Records are often inconsistent with existing scientific knowledge. Current quality assessments are limited: they either focus on syntax or rely on labor-intensive manual rules to capture semantic nuances. To overcome these scalability barriers, we propose Medical Data Pecking, a methodology that adopts software unit testing principles for medical data validation. It introduces Semantic Data Coverage, employing Large Language Models to generate context-aware tests that "peck" for inconsistencies between observed data and epidemiological evidence. To demonstrate this methodology, we implemented a reference tool using a Retrieval-Augmented Generation architecture that synthesizes medical literature into executable code. When applied to three datasets, this implementation generated dozens of tests per cohort, identifying discrepancies between observed distributions and epidemiological priors. These discrepancies encompass both genuine data inconsistencies and expected cohort-selection effects. This work provides an initial framework for scalable semantic auditing, shifting assurance from manual rules to the generative and context-sensitive verification required for trustworthy AI.

Figures

Figures reproduced from arXiv: 2507.02628 by the authors.

Figure 1
Figure 1. Semantic Data Coverage Heatmap. This representative example illustrates the density of semantic validation across clinical variables. The dense red clusters (see annotations) highlight systemic Semantic Gaps, areas where data is syntactically present (gray in standard tools) but fails validation against epidemiological priors (red in MDPT). 4 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The Semantic Unit Testing Paradigm and MDPT Architecture. (a) The Semantic Gap in data quality. Standard quality assessment tools (red) typically provide only syntactic cov￾erage (metadata and formatting). The proposed framework extends validation to the distributional…
Figure 5
Figure 5. Figure 5: A schematic overview of the MDPT workflow. User-defined study parameters are input into an LLM module that retrieves regional disease rates, contextual epidemiology, and generates lists of diagnosis, drug, measurement, and procedure codes via vector-database search. Th…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 37 canonical work pages

  1. [1]

    Perspectives for medical informatics

    Prokosch HU and Ganslandt T. Perspectives for medical informatics. Reusing the electronic medical record for clinical research. Methods Inf. Med. 2009;48:38–44

    work page 2009

  2. [2]

    Validity of The Health Improvement Network (THIN) for the study of psoriasis

    Seminara N et al. Validity of The Health Improvement Network (THIN) for the study of psoriasis. Br. J. Dermatol. 2011;164:602–9. DOI: 10.1111/j.1365-2133.2010.10134.x

    work page arXiv 2011

  3. [3]

    Electronic health records: new opportunities for clinical research

    Coorevits P et al. Electronic health records: new opportunities for clinical research. J. Intern. Med. 2013;274. eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/joim.12119:547–60. DOI: 10.1111/joim.12119

    work page doi:10.1111/joim.12119:547 2013

  4. [4]

    Research data warehouse: using electronic health records to conduct population- based observational studies

    Chen W et al. Research data warehouse: using electronic health records to conduct population- based observational studies. JAMIA Open 2023;6:ooad039. DOI: 10 . 1093 / jamiaopen / ooad039

    work page 2023

  5. [5]

    Availability of Evidence for Predictive Machine Learning Algorithms in Primary Care: A Systematic Review

    Rakers MM et al. Availability of Evidence for Predictive Machine Learning Algorithms in Primary Care: A Systematic Review. JAMA Netw. Open 2024;7:e2432990. DOI: 10.1001/ jamanetworkopen.2024.32990. 18

    work page arXiv 2024

  6. [6]

    Use of Artificial Intelligence in the Search for New Information Through Routine Laboratory Tests: Systematic Review

    Cardozo G, Tirloni SF, Pereira Moro AR, and Marques JLB. Use of Artificial Intelligence in the Search for New Information Through Routine Laboratory Tests: Systematic Review. JMIR Bioinform. Biotechnol. 2022;3:e40473. DOI: 10.2196/40473

    work page doi:10.2196/40473 2022

  7. [7]

    Artificial Intelligence Algo- rithm for Subclinical Breast Cancer Detection

    Gjesvik J, Moshina N, Lee CI, Miglioretti DL, and Hofvind S. Artificial Intelligence Algo- rithm for Subclinical Breast Cancer Detection. JAMA Network Open 2024;7:e2437402.DOI: 10.1001/jamanetworkopen.2024.37402

    work page arXiv 2024

  8. [8]

    Digital Health Data Quality Issues: Systematic Review

    Syed R et al. Digital Health Data Quality Issues: Systematic Review. J Med Internet Res 2023;25:e42615. DOI: 10.2196/42615

    doi:10.2196/42615 2023

Show all 49 references
  1. [9]

    Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research

    Weiskopf NG and Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J. Am. Med. Inform. Assoc. 2013;20:144– 51

    2013

  2. [10]

    Everyone wants to do the model work, not the data work

    Sambasivan N, Kapania S, Highfill H, Akrong D, Paritosh P, and Aroyo LM. “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. In: Pro- ceedings of the 2021 CHI Conference on Human Factors in Computing Systems . CHI ’21. New York, NY , USA:...

    2021

  3. [11]

    Garbage in–garbage out

    Kilkenny MF and Robinson KM. Data quality:“Garbage in–garbage out”. Health Inf Manag. 2018;47:103–5

    2018

  4. [12]

    Transparent reporting of data quality in distributed data networks

    Kahn MG et al. Transparent reporting of data quality in distributed data networks. Egems 2015;3. 19

    2015

  5. [13]

    Fundamentals of quality control and improvement

    Mitra A. Fundamentals of quality control and improvement. John Wiley & Sons, 2016

    2016

  6. [14]

    The economics of unit testing

    Ellims M, Bridges J, and Ince DC. The economics of unit testing. Empirical Software Engi- neering 2006;11:5–31

    2006

  7. [15]

    Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices

    Shahin M, Ali Babar M, and Zhu L. Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices. IEEE Access 2017;5:3909–

    2017

  8. [16]

    Veracity in big data: How good is good enough

    Reimer AP and Madigan EA. Veracity in big data: How good is good enough. Health Infor- matics J. 2019;25:1290–8

    2019

  9. [17]

    Electronic health record data quality assessment and tools: a systematic re- view

    Lewis AE et al. Electronic health record data quality assessment and tools: a systematic re- view. J. Am. Med. Inform. Assoc. 2023;30:1730–40

    2023

  10. [18]

    Large language models for data extraction from unstructured and semi- structured electronic health records: a multiple model performance evaluation

    Ntinopoulos V et al. Large language models for data extraction from unstructured and semi- structured electronic health records: a multiple model performance evaluation. BMJ Health & Care Informatics 2025;32. Publisher: BMJ Publishing Group Ltd. DOI: 10.1136/bmjhci- 2024-101139

    2025 doi

  11. [19]

    Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models

    Alsentzer E et al. Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models. npj Digital Medicine 2023;6. Number: 1 Publisher: Nature Publishing Group:1–10. DOI: 10.1038/s41746-023-00957-x

    2023 doi

  12. [20]

    Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT

    Yang J et al. Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT. Patterns 2023:100887. DOI: 10.1016/j.patter.2023. 100887. 20

    2023 doi

  13. [21]

    Towards automated phenotype definition extraction using large language models

    Tekumalla R and Banda JM. Towards automated phenotype definition extraction using large language models. Genomics & Informatics 2024;22:21. DOI: 10 . 1186 / s44342 - 024 - 00023-2

    2024

  14. [22]

    Utility of Large Language Models for Concept Set Curation

    Anand A, Ostropolets A, Ryan P, and Hripcsak G. Utility of Large Language Models for Concept Set Curation. Tech. rep. 2025

    2025

  15. [23]

    GenSpectrum Chat: Data Exploration in Public Health Using Large Language Models

    Chen C and Stadler T. GenSpectrum Chat: Data Exploration in Public Health Using Large Language Models. arXiv:2305.13821 [cs, q-bio]. 2023. DOI: 10 . 48550 / arXiv . 2305 . 13821. arXiv: 2305.13821

    2023 doi arXiv

  16. [24]

    Evaluation of GPT-4 for 10-year cardiovascular risk prediction: Insights from the UK Biobank and KoGES data

    Han C et al. Evaluation of GPT-4 for 10-year cardiovascular risk prediction: Insights from the UK Biobank and KoGES data. iScience 2024;27:109022. DOI: https://doi.org/10. 1016/j.isci.2024.109022

    2024

  17. [25]

    Event Stream GPT: A Data Pre- processing and Modeling Library for Generative, Pre-trained Transformers over Continuous- time Sequences of Complex Events

    McDermott MBA, Nestor B, Argaw P, and Kohane I. Event Stream GPT: A Data Pre- processing and Modeling Library for Generative, Pre-trained Transformers over Continuous- time Sequences of Complex Events. arXiv:2306.11547 [cs]. 2023. DOI: 10.48550/arXiv. 2306.11547. arXiv: 2306.11547

    2023 doi arXiv

  18. [26]

    Can Foundation Models Wrangle Your Data? arXiv:2205.09911 [cs]

    Narayan A, Chami I, Orr L, Arora S, and R ´e C. Can Foundation Models Wrangle Your Data? arXiv:2205.09911 [cs]. 2022. DOI: 10.48550/arXiv.2205.09911. arXiv: 2205.09911

    2022 doi arXiv

  19. [27]

    Represen- tation of race and ethnicity in the contemporary US health cohort all of US research program

    Kathiresan N, Cho SMJ, Bhattacharya R, Truong B, Hornsby W, and Natarajan P. Represen- tation of race and ethnicity in the contemporary US health cohort all of US research program. JAMA Cardiol. 2023;8:859–64. 21

    2023

  20. [28]

    Back to the Building Blocks: A Path Toward Secure and Measurable Software. Tech. rep. The White House, 2024

    2024

  21. [29]

    Bipartisan House Task Force on Artificial Intelligence. Tech. rep. U.S. House of Representa- tives, 2024

    2024

  22. [30]

    Tackling algorithmic bias and promoting transparency in health datasets: the STANDING Together consensus recommendations

    Alderman JE et al. Tackling algorithmic bias and promoting transparency in health datasets: the STANDING Together consensus recommendations. The Lancet Digital Health 2025;7:e64– e88. DOI: 10.1016/S2589-7500(24)00224-3

    2025 doi

  23. [31]

    Structured programming

    Dahl OJ, Dijkstra EW, and Hoare CAR. Structured programming. Academic Press Ltd., 1972

    1972

  24. [32]

    The implied truth effect: Attaching warn- ings to a subset of fake news headlines increases perceived accuracy of headlines without warnings

    Pennycook G, Bear A, Collins ET, and Rand DG. The implied truth effect: Attaching warn- ings to a subset of fake news headlines increases perceived accuracy of headlines without warnings. Manag. Sci. 2020;66:4944–57

    2020

  25. [33]

    Code coverage and test suite effectiveness: Empirical study with real bugs in large systems

    Kochhar PS, Thung F, and Lo D. Code coverage and test suite effectiveness: Empirical study with real bugs in large systems. In: 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER) . ISSN: 1534-5351. 2015:560–4. DOI: 10. 1109/SANER...

    2015

  26. [34]

    Self-refine: Iterative refinement with self-feedback

    Madaan A et al. Self-refine: Iterative refinement with self-feedback. Adv. Neural Inf. Process. Syst. 2023;36:46534–94

    2023

  27. [35]

    Statistical significance versus clinical relevance

    Rijn MH van, Bech A, Bouyer J, and Brand JA van den. Statistical significance versus clinical relevance. Nephrol. Dial. Transplant. 2017;32. eprint: https://academic.oup.com/ndt/article- pdf/32/suppl 2/ii6/11145781/gfw385.pdf:ii6–ii12. DOI: 10.1093/ndt/gfw385. 22

    2017 doi

  28. [36]

    Comparing Prevalence Estimates From Population-Based Surveys to Inform Surveil- lance Using Electronic Health Records

    Tatem KS. Comparing Prevalence Estimates From Population-Based Surveys to Inform Surveil- lance Using Electronic Health Records. Prev. Chronic Dis. 2017;14. DOI: 10.5888/pcd14. 160516

    2017 doi

  29. [37]

    MIMIC-III, a freely accessible critical care database

    Johnson AEW et al. MIMIC-III, a freely accessible critical care database. Scientific Data 2016;3. Publisher: Nature Publishing Group:160035. DOI: 10.1038/sdata.2016.35

    2016 doi

  30. [38]

    Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record

    Walonoski J et al. Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record. J. Am. Med. Inform. Assoc. 2017;25. eprint: https://academic.oup.com/jamia/article-pdf/25/3/230/34150150/ocx079.pdf:230–

    2017

  31. [39]

    error" explanation += ' ' + str(e).replace('

    Mirzadeh I, Alizadeh K, Shahrokhi H, Tuzel O, Bengio S, and Farajtabar M. GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. arXiv:2410.05229 [cs] version: 1. 2024. DOI: 10.48550/arXiv.2410.05229. arXiv: 2410.05229. 23 Type 2 diabet...

    2024 doi arXiv

  32. [40]

    DOI: 10.1093/jamia/ocx079

    doi

  33. [42]

    Test description: [your answer]

  34. [43]

    DOI: 10.1109/ACCESS.2017.2685629

    2017

  35. [47]

    use a vertical line | as separator

    Fuction recommendation: [your answer - Fix (if incorrect), No fix (if correct), Remove (if unsure)] 76 If according to your feedback the test suggestion needs fixing due to reference value correction, please provide it betweeen triple backticks (‘‘‘csv). use a vertical line | ...

  36. [52]

    use a vertical line | as separator

    Recommendation: [your answer - Fix (if incorrect), No fix (if correct), Remove (if unsure)] Please provide the modified statistics in a similar format as the original, betweeen triple backticks (‘‘‘csv). use a vertical line | as separator. If according to your feedback the sta...

  37. [53]

    Statistic: [your answer]

  38. [54]

    Correct reference value: [your answer]

  39. [55]

    Provided reference value: [your answer]

  40. [56]

    Feedback: [your answer - Correct or Incorrect or Unsure]

  41. [57]

    use a vertical line | as separator

    Recommendation: [your answer - Fix (if incorrect), No fix (if correct), Remove (if unsure)] Please provide the modified statistics in a similar format as the original, betweeen triple backticks (‘‘‘csv). use a vertical line | as separator. If according to your feedback the sta...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.