Pith. sign in

REVIEW 3 major objections 4 minor 32 references

The NordDRG AI Benchmark for Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that the full rule graph behind the NordDRG hospital-funding system can be released as an open, machine-readable benchmark, and reports that GPT-5 Thinking is the first LLM to partially emulate the complete grouper…

desk verdict Useful benchmark, honestly reported, but the rule-completeness and gold keys are asserted rather than verified; referee it and ask for the missing checks. read the letter →

arxiv 2506.13790 v3 pith:WPKRTIOS submitted 2025-06-11 cs.AI

Tapio Pitkäranta This is my paper · ORCID
classification cs.AI
keywords Diagnosis-RelatedGroupsNordDRGLLMbenchmarkgrouperemulationhospitalreimbursementexact-matchscoringclinicalcodinghealthcareAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the full rule graph behind NordDRG—the Diagnosis-Related Group system that routes hospital payments across the Nordic countries—can be packaged as an open, machine-readable benchmark, and that today's LLMs can be scored against it exactly. It releases the roughly twenty-sheet definition tables, expert manuals, prompts, and gold answer keys, and defines two suites: 13 logic tasks that require reading and joining the rule tables, and 13 grouper tasks that require emulating the complete grouping control flow. Under an artefact-only, no-web setting, the best model (GPT-5 Thinking) solves all 13 logic tasks but only 7 of 13 grouper tasks, with other models far lower. If the rule-completeness claim holds, the benchmark gives hospital-funding AI a reproducible yardstick for head-to-head and longitudinal comparison, something generic LLM leaderboards do not provide.

What carries the argument

The carrying object is the drg_logic table, the ordered rule sheet at the heart of the NordDRG specification: each row maps diagnosis and procedure properties, age/sex bounds, complication status, and national activation flags to a DRG, and the ORD column fixes the order in which rows must be tried. The benchmark's strict scoring rule—an answer is correct only when both the DRG and the exact drg_logic.id match the gold key—makes the grouper's control flow, not just its output, the thing being measured. The accompanying agents enforce artefact-only operation, feeding the model the tables and manuals with no web access, so performance isolates rule-graph competence.

What would settle it

Run the thirteen Grouper test cases through a reference implementation of the official production NordDRG grouper; if any case produces a different drg_nat or drg_logic.id than the published gold keys, then the rule-completeness claim fails and the benchmark scores are not measuring the true grouper.

Watch

Extended reading notes

Core claim

The core discovery is that a rule-complete release is sufficient to turn DRG grouper logic into an LLM benchmark: the definition tables encode deterministic control flow, including execution order, age/sex bounds, MDC entry, surgical evidence, complication handling with exclusions, and national activation flags, and exact-match scoring can demand both the final DRG and the triggering rule row. On that basis the paper reports that GPT-5 Thinking is the first model to partially emulate the complete NordDRG grouper, matching both fields on 7 of 13 cases, with o3 at 6/13, o4-mini at 3/13, and all other tested endpoints at 1/13 or below. This is the first public report, to the authors' knowledge, of an LLM reproducing complete grouper logic with governance-grade traceability rather than merely predicting a DRG label.

Load-bearing premise

The public definition tables contain every rule the production grouper uses, with no behaviour living only in compiled grouper binaries or in manuals outside the bundle.

Editorial extensions

If this is right

  • Researchers get a reproducible, exact-match yardstick for comparing LLM reasoning over hospital-funding rules, independent of proprietary datasets.
  • Annual NordDRG releases can be slotted into the same schema-stable prompts, so model progress can be tracked across versions without rewriting the benchmark.
  • The logic suite isolates specific weak spots, such as cross-table joins, property tracing, and CC/MCC exclusions, that generic leaderboards do not expose.
  • The grouper suite sets a current ceiling: the best artefact-only model solves 7 of 13 cases, so full governed emulation remains an open problem.
  • Because answers must include the triggering rule row, model outputs become auditable in the same terms as production groupers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the tables truly are rule-complete, the same exact-match pair design could be carried to other public DRG systems such as MS-DRG, AR-DRG, or APR-DRG to test whether the results generalise across different rule graphs.
  • The sharp drop from 13/13 logic to 7/13 grouper suggests that exact emulation requires executing a priority-ordered control flow, not merely retrieving facts; a testable extension would give models a step-by-step trace from a deterministic reference grouper and measure how much accuracy recovers.
  • Because the benchmark scores a rule row rather than only the final group, partial-credit metrics based on trace overlap could expose where failures occur, although the paper itself reports only strict exact-match results.
  • A hybrid setup that lets the LLM call deterministic table operations instead of reading all sheets from context might push grouper accuracy higher; the released agents deliberately avoid such tools, so this remains untested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces NordDRG-AI-Benchmark, a public benchmark for evaluating large language models on Diagnosis-Related Group (DRG) grouping logic. The release bundles roughly twenty NordDRG definition-table sheets, expert governance manuals, two 13-task suites (Logic and Grouper), reference agents, and gold-answer keys. Under an artefact-only, no-web evaluation protocol, the authors report that GPT-5 Thinking and Opus 4.1 solve 13/13 Logic tasks, while GPT-5 Thinking achieves 7/13 on strict Grouper emulation requiring exact matches for both the DRG and the triggering drg_logic.id. The paper claims that this is the first public, rule-complete benchmark for DRG reasoning and the first report of an LLM partially emulating the complete NordDRG grouper logic.

Significance. If the released tables are indeed rule-complete and the gold keys are verified outputs of the production NordDRG grouper, this benchmark fills a genuine gap in LLM evaluation for hospital-funding logic. The exact-match scoring on both the DRG and the triggering rule identifier is a well-designed, governance-oriented metric, and grounding the gold keys in an external specification avoids the circularity that plagues many model-generated benchmarks. The open release of tables, prompts, and scoring scripts is a concrete contribution that should enable head-to-head and longitudinal comparison.

major comments (3)
  1. [Section 4.4 / Table 4] The Grouper tasks are not reproducible from the manuscript because the test-case inputs are referenced only by opaque IDs (e.g., '49483') and are said to be 'supplied in the accompanying test-case sheet,' which is not included in the paper. A reader cannot check whether the gold answers in Table 5 follow from the stated inputs, nor can they rerun the benchmark without guessing the case data. Please include the full structured test cases (principal and secondary diagnoses, procedures, age, sex, care setting, and any other fields used by the grouper) or, failing that, provide the exact repository file path and commit hash in the text.
  2. [Section 5.1 / Section 5.2] The baseline evaluation omits sampling parameters, API model versions, run counts, and response-parsing details. The paper reports single per-model correctness values for 13-item suites without stating the temperature, top-p, repetition penalty, or the number of stochastic runs per task, even though exact-match scoring is sensitive to sampling noise. This is load-bearing for the central comparison claim and for the paper's stated goal of 'fully reproducible' baselines; please add a complete evaluation-protocol subsection and, ideally, release the raw model outputs.
  3. [Section 4.1 / Section 5.4] The central 'rule-complete' claim is asserted rather than demonstrated. Section 4.1 states that 'all rule-bearing tables and columns needed for deterministic control flow are included,' and Section 5.4 repeats that 'no lossy transformations were applied,' but the paper provides no sheet/column manifest, no version pin with a commit hash for the exact tables used, and no independent check against the production Finnish NordDRG grouper. Because the gold keys in Tables 3 and 5, the model scores, and the reported 54% ceiling all rest on this assertion, the authors should supply a verification artifact: a machine-readable manifest of sheets and columns, and a script that compares every gold key against the official grouper output for the same release, or an independent reimplementation of the control flow.
minor comments (4)
  1. [Section 3] There is an orphaned author note, 'Author note: LLM use 1,' that appears mid-section and interrupts the text; this should be removed or completed as a proper statement.
  2. [Section 4.8] The GrouperAgent system prompt uses the field name drg_logic_id, while Section 4.4 and Table 5 use drg_logic.id; please standardize the notation to avoid ambiguity in the output format.
  3. [Section 5.3] The consumer-subscription throughput observations are time- and product-dependent and are presented with appropriate caveats, but they are tangential to the benchmark's validity; consider moving this material to an appendix or a clearly labeled operational note.
  4. [Tables 7 and 8] The per-task checkmarks indicate correctness but do not show the model outputs, so failure modes must be inferred from the summary text; releasing or citing the raw output logs would strengthen the diagnostic value of the reported results.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: gold keys are external, fixed, and not fitted to the models; the only self-citations are contextual and non-load-bearing.

full rationale

The benchmark's central chain is: official NordDRG definition tables and manuals (external, third-party artifacts) are packaged; tasks are defined against those artifacts; gold answer keys (Tables 3 and 5) encode the official grouping outcomes; and LLM outputs are scored by exact match to those fixed keys. Nothing in this chain is defined in terms of the model outputs or fitted to them. No parameter is estimated from the benchmark data, and no 'prediction' is derived from the models and then reported as ground truth. The only self-citations (Pitkäranta 2024a,b, Section 2.8) are explicitly contextual, describing preliminary proof-of-concept work that the present article extends; they are not used to justify the rule-completeness claim, the gold keys, or the scoring protocol. The 'rule-complete' assertion (Sections 4.1 and 5.4) is an empirical fidelity claim about the released workbook relative to the production NordDRG specification; if the claim were false, the benchmark would be invalid or incomplete, but that is a correctness/verification risk, not circularity. The paper also includes a transparent author note about LLM use for grammar and formatting, which does not affect the scientific derivation. Overall, the derivation chain is self-contained against external specification data, so no circular step is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the reliability of the released definition tables and gold keys. There are no fitted constants in a numerical derivation, but there are two hand-selected task sets and several unverified domain assumptions about rule-completeness, gold-answer correctness, artifact-only evaluation, and single-run stability.

free parameters (2)
  • Grouper task set composition = 13 hand-selected cases (IDs 49483, 50184, 50166, 50629, 66704, 53273, 49486, 49571, 49597, 49606, 49968, 54051, 54096)
    The aggregate pass rates depend on which cases were selected for the benchmark. The paper does not describe a random sampling or coverage procedure, so the scores are conditional on this hand-picked set.
  • Logic task set composition = 13 hand-selected prompts (Logic-1 through Logic-13)
    The benchmark's logic scores depend on the specific prompts chosen. The paper assigns difficulty tags by hand and does not sample from a larger task population, so the 13/13 ceiling may not generalize to all DRG reasoning queries.
assumptions (4)
  • domain assumption The NordDRG definition tables included in the release are rule-complete and match production grouper behavior.
    Section 4.1 asserts that all rule-bearing tables and columns needed for deterministic control flow are included, but the authors provide no external audit or cross-check against compiled grouper outputs.
  • domain assumption The gold answers for Logic and Grouper tasks are correct.
    Sections 4.3 and 4.4 provide answer keys, but the paper does not describe how they were generated or independently validated against the official grouper.
  • domain assumption Models were constrained to the provided artifacts and did not use web search or internal memorized DRG data.
    Section 5.1 reports prompt instructions forbidding web use, but no technical enforcement mechanism is described.
  • domain assumption Single-run exact-match scores are stable enough to report without repetition or temperature settings.
    The paper does not state temperature, number of runs, or sampling parameters, so stochastic variation in LLM outputs is unquantified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The NordDRG AI Benchmark for Large Language Models." pith.science (2026). https://pith.science/paper/WPKRTIOS

@misc{pith2026250613790,
  author       = {Pith},
  title        = {Pith review of: The NordDRG AI Benchmark for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WPKRTIOS}},
  note         = {Machine review of arXiv:2506.13790}
}
read the original abstract

Large language models (LLMs) are being piloted for clinical coding and decision support, yet no open benchmark targets the hospital-funding layer where Diagnosis-Related Groups (DRGs) determine reimbursement. In most OECD systems, DRGs route a substantial share of multi-trillion-dollar health spending through governed grouper software, making transparency and auditability first-order concerns. We release NordDRG-AI-Benchmark, the first public, rule-complete test bed for DRG reasoning. The package includes (i) machine-readable approximately 20-sheet NordDRG definition tables and (ii) expert manuals and change-log templates that capture governance workflows. It exposes two suites: a 13-task Logic benchmark (code lookup, cross-table inference, grouping features, multilingual terminology, and CC/MCC validity checks) and a 13-task Grouper benchmark that requires full DRG grouper emulation with strict exact-match scoring on both the DRG and the triggering drg_logic.id. Lightweight reference agents (LogicAgent, GrouperAgent) enable artefact-only evaluation. Under an artefact-only (no web) setting, on the 13 Logic tasks GPT-5 Thinking and Opus 4.1 score 13/13, o3 scores 12/13; mid-tier models (GPT-5 Thinking Mini, o4-mini, GPT-5 Fast) achieve 6-8/13, and remaining models score 5/13 or below. On full grouper emulation across 13 tasks, GPT-5 Thinking solves 7/13, o3 6/13, o4-mini 3/13; GPT-5 Thinking Mini solves 1/13, and all other tested endpoints score 0/13. To our knowledge, this is the first public report of an LLM partially emulating the complete NordDRG grouper logic with governance-grade traceability. Coupling a rule-complete release with exact-match tasks and open scoring provides a reproducible yardstick for head-to-head and longitudinal evaluation in hospital funding. Benchmark materials available in Github.

Figures

Figures reproduced from arXiv: 2506.13790 by the authors.

Figure 1
Figure 1. LLM Leaderboard illustration: Vellum leaderboard (Vellum.ai, 2025) [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Materials used: NordDRG definition tables (NordicCasemixCentreForum, 2025) [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Materials used: NordDRG definition table expert documentation (NordicCasemixCentreForum, 2025) [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Combined results. Percentage accuracy on the NordDRG-AI-Benchmark Logic (rule-level) and Grouper (full emulation) suites. Logic saturates for top models, while exact-match emulation—requiring the correct drg_nat and triggering drg_logic.id—remains challenging; only a f…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 15 canonical work pages

  1. [1]

    bench Collaboration, B. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615

    arXiv 2022

  2. [2]

    Boukhers, Z., Khan, A., Ramadan, Q., and Yang, C. (2024). Large language model in medical informatics: Direct classification and enhanced text representations for automatic icd coding. arXiv preprint arXiv:2411.06823

    work page Pith review arXiv 2024

  3. [3]

    S., Kascenas, A., Lok, P., Liakata, M., and O'Neil, A

    Boyle, J. S., Kascenas, A., Lok, P., Liakata, M., and O'Neil, A. Q. (2023). Automated clinical coding using off-the-shelf large language models. arXiv preprint arXiv:2310.06552

    arXiv 2023

  4. [4]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems , 33:1877--1901

    2020

  5. [5]

    S., and Sun, L

    Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., and Sun, L. (2023). A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt. arXiv preprint arXiv:2303.04226

    arXiv 2023

  6. [6]

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., et al. (2022). Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311

    arXiv 2022

  7. [7]

    Christian, B. (2020). The alignment problem: Machine learning and human values . WW Norton & Company. ISBN: 9781786494306

    work page 2020

  8. [8]

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2018

Show all 32 references
  1. [9]

    Hajialigol, D., Kaknes, D., Barbour, T., Yao, D., North, C., Sun, J., Liem, D., and Wang, X. (2023). Drgcoder: Explainable clinical coding for the early prediction of diagnostic-related groups. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proc...

    2023

  2. [10]

    He, Y., Wang, C., Zhang, S., Li, N., Li, Z., and Zeng, Z. (2022). Kg-mtt-bert: Knowledge graph enhanced BERT for multi-type medical text classification. arXiv preprint arXiv:2210.03970

    2022 arXiv

  3. [11]

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., et al. (2021). Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2021 arXiv

  4. [12]

    N., et al

    Hoffmann, J., Botev, A., Penedo, G., Alvi, A. N., et al. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556

    2022 arXiv

  5. [13]

    Jones, C. R. and Bergen, B. K. (2025). Large language models pass the turing test

    2025

  6. [14]

    Kwan, K. (2024). Large language models are good medical coders, if provided with tools. arXiv preprint arXiv:2407.12849

    2024 arXiv

  7. [15]

    Liang, P., Bommasani, R., Lee, T., Tsipras, D., et al. (2022). Holistic evaluation of language models. arXiv preprint arXiv:2211.09110

    2022 arXiv

  8. [16]

    U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A

    Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A. (2023). A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435

    2023 arXiv

  9. [17]

    Norddrg specification, url: https://nordcase.org/

    NordicCasemixCentre (2025). Norddrg specification, url: https://nordcase.org/

    2025

  10. [18]

    E., Rossi, M., Hui, W., Virtanen, V., and Bragge, J

    Peffers, K., Tuunanen, T., Gengler, C. E., Rossi, M., Hui, W., Virtanen, V., and Bragge, J. (2020). Design science research process: A model for producing and presenting information systems research

    2020

  11. [19]

    Pitk \"a ranta, T. (2024a). Discussing with your casemix specification with natural language: how to make large language models (llm) understand norddrg logic? In PCSI 2024: Proceedings of the 36th International Conference on Patient Classification Systems, May 2024, Bled, Slovenia

    2024

  12. [20]

    Pitk \"a ranta, T. (2024b). Teaching llms the nuances of hospital funding instruments. In Proceedings of the International Conference on Computer-Human Interaction Research and Applications (CHIRA 2024) . SCITEPRESS

    2024

  13. [21]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1--67

    2020

  14. [22]

    E., Was, J., Li, Q., Bates, D

    Renc, P., Jia, Y., Samir, A. E., Was, J., Li, Q., Bates, D. W., and Sitek, A. (2024). Zero shot health trajectory prediction using transformer. arXiv preprint arXiv:2407.21124

    2024 arXiv

  15. [23]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  16. [24]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems , 30

    2017

  17. [25]

    Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2019). Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems (NeurIPS)

    2019

  18. [26]

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2018). Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP , pages 353--355

    2018

  19. [27]

    Wang, H., Wu, Z., Kolar, G., Korsapati, H., Bartlett, B., Hull, B., and Sun, J. (2025). Reinforcement learning for out-of-distribution reasoning in llms: An empirical study on diagnosis-related group coding. arXiv preprint arXiv:2505.21908

    2025

  20. [28]

    Wang, X., Zhang, L., Yan, M., et al. (2023). Drg‐llama: An open‐source model for diagnosis‐related group coding. arXiv preprint arXiv:2310.01234

    2023 arXiv

  21. [29]

    Warstadt, A., Liu, A., Kirov, C., and et al. (2023). The babylm challenge: Sample-efficient pretraining on 10m tokens. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  22. [30]

    Wilson, E. O. (1999). Consilience: The unity of knowledge. Vintage; Reprint edition . ISBN: 978-0679768678

    1999

  23. [31]

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., et al. (2022). Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  24. [32]

    Zheng, S., Zhao, Z., Li, E., and et al. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.