Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Evaluating Structured Output Robustness of Small Language Models for Open Attribute-Value Extraction from Clinical Notes

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Small language models asked to pull attributes from clinical notes produce parseable JSON significantly more often than YAML or XML, and targeted prompts plus larger models push parseability higher.

desk verdict A useful and honest benchmark paper; JSON's parseability advantage over YAML and XML for small clinical LLMs is real, and the methodological caveats are fixable. read the letter →

arxiv 2507.01810 v1 pith:3S6OC6K7 submitted 2025-07-02 cs.CL cs.IR

Nikita Neveditsin , Pawan Lingras , Vijay Mago This is my paper · ORCID
classification cs.CLcs.IR
keywords structuredoutputparseabilitysmalllanguagemodelsclinicalnotesJSONYAMLXMLattribute-valueextraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks which serialization format—JSON, YAML, or XML—small language models can actually produce in a form a standard parser accepts, in the task of open attribute-value extraction from clinical notes. On 105 de-identified notes from the EHRCon/MIMIC-III collection, tested with seven open-weight instruction-tuned models ranging from 3 billion to 14 billion parameters, the paper finds that JSON is consistently the most parseable format, with every pairwise difference statistically significant. It also finds that targeted prompts (asking for medications, symptoms, or demographics) improve parseability in every format, that larger models are more structurally robust, and that longer documents and physician notes are more likely to fail. The practical point is that format choice and prompt design can be decided from evidence rather than habit when deploying small models in privacy-sensitive clinical settings.

What carries the argument

The load-bearing object is the parseability rate $\rho(D) = n_v / |D|$, defined as the fraction of documents whose model outputs a standard parser accepts without manual correction. The argument works by generating outputs with seven small instruction-tuned models under a fixed greedy decoding configuration, extracting structured content with format-specific regular expressions when direct parsing fails, and then comparing parseability across formats with paired McNemar tests and chi-squared tests. The error analysis supplies the mechanism behind the ranking: JSON's failures concentrate in a few quotable-string violations, whereas YAML and XML accumulate more varied and harder-to-repair structural errors, and infinite repetition blocks break the extraction step itself.

What would settle it

A concrete check: rerun the seven models on the same 105 notes at temperature 0.7 with a repetition penalty, or with grammar-constrained decoding, and compare parseability rates; if XML or YAML matches or beats JSON under either condition, the claim that JSON is consistently the most parseable format does not generalize. A second check: repeat the protocol on a different clinical corpus of comparable size and see whether YAML's open-setting weakness and JSON's lead both reproduce.

Watch

Extended reading notes

Core claim

The central claim is that JSON outperforms both YAML and XML on parseability—the fraction of outputs a standard parser accepts without manual correction—across every tested model and both extraction scenarios. The strongest single result is the paired McNemar tests: JSON versus YAML and JSON versus XML are both significant at $p \ll 0.05$, with JSON ahead in all 14 model-by-setting rows of Table 3. The paper also establishes that parseability is not a fixed property of a format: targeted prompts raised average parseability substantially, especially for YAML (open YAML parseability across models ranged 23.4–52.3%, while targeted YAML ranged 75.1–97.8%), and larger models failed less often, though the effect sizes were modest (Cramér's $V = 0.11$ for model size, $0.42$ for prompt type). The failure analysis attributes most extraction-related failures to infinite repetition and most malformed outputs to format-specific syntax violations, such as unquoted numerals with slashes in JSON, unescaped reserved characters in XML, and indentation or alias problems in YAML.

Load-bearing premise

The format ranking could shift if any of the fixed test conditions change—the exact prompt wording, greedy decoding with no sampling, the specific 105-note corpus, or the standard Python parsers—so the paper's 'JSON always wins' claim is about this setup, not about all structured generation.

Editorial extensions

If this is right

  • Default to JSON when asking a small model to return structured clinical extractions; it gives the highest chance of parseable output in both open and targeted settings.
  • If YAML must be used, pair it with targeted prompts and a larger model; open-ended YAML extraction is the least reliable configuration measured.
  • For longer and denser notes, such as physician notes with vitals and units, expect lower parseability and plan stricter prompting or post-processing that quotes values.
  • Address infinite repetition explicitly, for example with repetition penalties or constrained decoding, since it is the dominant extraction-level failure mode for medium and small models under targeted prompts.
  • Report parseability separately from semantic accuracy in structured extraction evaluations, because a syntactically invalid output fails downstream use regardless of content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper fixes greedy decoding and free-form generation; a natural extension is grammar-constrained decoding, where XML's rigid nesting might close much of the gap with JSON, a direction the paper itself suggests but does not test.
  • Parseability treats every failure as equal, but downstream cost differs: a JSON unquoted number is often repaired with one regex, while a YAML indentation collapse can corrupt a whole block, so a cost-weighted robustness metric might shift the ranking.
  • The interaction between prompt type and error type (extraction errors made up 45.5% of targeted failures but only 2.4% of open failures) suggests repetition arises when the model is given a short, category-specific list task, which could be tested by varying list length or instruction phrasing.
  • Because only 105 notes from one corpus were used, the paper implicitly assumes EHRCon represents the range of clinical narrative styles; testing on a second corpus with different note templates would show whether the format ranking survives.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper compares the parseability of JSON, YAML, and XML outputs generated by seven small instruction-tuned language models (3–14B) for open and targeted attribute-value extraction from 105 clinical notes in the EHRCon/MIMIC-III dataset. The central empirical claim is that JSON achieves the highest parseability across all models and settings, that targeted prompting and larger models improve parseability, and that longer documents and physician notes are associated with lower parseability. The authors use McNemar tests for format comparisons, chi-square tests for model size, prompt type, and note type, a point-biserial correlation for length, a logistic regression for note type and length, and a hand-coded error analysis that distinguishes extraction-related failures (mostly infinite repetition) from malformed-output syntax errors.

Significance. If the results hold, the paper offers practically useful guidance for selecting serialization formats and prompts in privacy-sensitive clinical NLP deployments using small local models. The reproducible experimental setup, the open models and dataset, and the detailed, format-specific error taxonomy in Appendix B are notable strengths. The paper also addresses a real gap, since most prior clinical extraction work focuses on semantic accuracy rather than syntactic robustness. However, the significance is limited by the current analysis's failure to separate decoding degeneration (infinite repetition) from true serialization-syntax robustness, and by statistical tests that ignore the repeated-measures structure of the data.

major comments (3)
  1. [Section 5, Table 8] The parseability metric conflates two distinct failure modes. Table 8 shows that all 31 JSON and 78 XML 'extraction-related' errors are infinite repetitions, and 109 of 112 YAML such errors are too; these are decoding-degeneration phenomena, not serialization-syntax failures. Because parseability (Section 3.3) counts every unparsed output as a failure, the format ranking in Table 3 and the McNemar comparisons in Table 4 may reflect differential propensity of the prompts/formats to trigger repetition loops rather than the syntactic robustness of JSON/YAML/XML. Please report parseability conditional on non-repetitive output (or otherwise remove repetition failures) and state explicitly whether the JSON advantage persists on that subset.
  2. [Section 4] The chi-square tests and logistic regression pool observations across models, prompt types, and formats, but the same 105 documents are measured repeatedly in all 42 conditions (7 models × 2 prompts × 3 formats). This violates the independence assumption of the tests; for example, the chi-square for prompt type (χ² = 1579.41) treats each of the thousands of outputs as independent. A mixed-effects logistic model with a random intercept for document, or cluster-robust standard errors, is needed for the document-level claims; as written, the p-values are anti-conservative. Also, 'p ≪ 0.05' is not a precise report; please provide exact p-values.
  3. [Section 4, logistic regression] The sentence 'discharge notes, though longer on average, are more parseable than nursing notes' is ambiguous relative to Figure 3, which shows nursing notes with the highest raw parseability; presumably the regression is adjusted for both length and note type, but the model specification, covariates, reference categories, and standard errors are not given. Please clarify and report the full model, including whether length is treated linearly and how repeated measures are handled.
minor comments (4)
  1. [Throughout] The name 'YAML' is inconsistently typeset as 'Y AML' in several places, including Tables 4, 8, and in the introduction; please standardize.
  2. [Table 3] Report raw counts (n/N) alongside percentages; with only 105 documents, differences such as 97.8% vs. 96.9% are small and need exact counts for interpretation.
  3. [Section 2] Since Elnashar et al. (2025) already compared JSON and YAML (and hybrid CSV) output formats, the claim of being the 'first comparative analysis' should be qualified to 'first comparative analysis of parseability across JSON, YAML, and XML for small models in clinical note extraction' to avoid overclaiming.
  4. [Appendix A] The prompt templates contain literal ' ' in the displayed text; ensure the appendix clearly shows the actual line breaks and that the fenced-code instructions match the regular expressions in Table 7.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's parseability results are direct empirical measurements, not derivations from fitted inputs or self-cited constraints.

full rationale

The paper's central claim, that JSON yields the highest parseability among JSON, YAML, and XML for small-language-model structured outputs from clinical notes, is supported by directly measured parseability rates (Section 3.3, Table 3) and paired McNemar tests on those observed outcomes (Table 4). No parameter is fitted to a subset of the data and then renamed as a prediction; the parseability rate rho(D) = n_v / |D| is a raw count of outputs accepted by standard parsers. The error analysis in Section 5 and Appendix B further decomposes failures into extraction-related and malformed-output categories, and these categories are themselves reported counts, not constructed outputs. The paper does invoke prior work (e.g., Holtzman et al. 2020 for infinite repetition), but that citation is contextual, not load-bearing: the empirical counts stand independently of it. The self-citation to Neveditsin et al. (2025) appears only as motivation that SLMs can be deployed locally and does not justify any result. The limitation about a single greedy decoding configuration is a generalizability caveat, not a circularity: it weakens external validity but does not make the measured outcomes equal to their inputs. Similarly, the concern that infinite-repetition failures are pooled with syntactic failures is a construct-validity or interpretation issue, not a circular derivation, because the parseability metric is defined and measured exactly as stated. On the criteria of self-definition, fitted-input prediction, load-bearing self-citation, imported uniqueness, ansatz smuggling, and renaming, the paper is self-contained and empirically grounded, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities; the evaluation is direct measurement. The main assumptions are representativeness of the dataset, the choice of standard parsers, and the fixed decoding configuration.

assumptions (4)
  • domain assumption The EHRCon subset of MIMIC-III (105 notes) is representative of clinical narratives for evaluating format parseability.
    The paper uses only these 105 notes (Section 3.2) and claims general practical guidance; representativeness is assumed.
  • domain assumption Python's json, xml.etree, and PyYAML parsers correctly represent 'standard' parsing behavior for each format.
    Parseability is defined by these parsers (Appendix A).
  • domain assumption The regex extraction of fenced code blocks (Appendix A, Table 7) does not bias one format over another.
    The same regex pattern is used for each format, but YAML and XML may be more sensitive to missing fences.
  • domain assumption Greedy decoding (do_sample=False) is a representative or at least meaningful configuration for small model deployment.
    The authors acknowledge this in Limitations and never test other settings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Structured Output Robustness of Small Language Models for Open Attribute-Value Extraction from Clinical Notes." pith.science (2026). https://pith.science/paper/3S6OC6K7

@misc{pith2026250701810,
  author       = {Pith},
  title        = {Pith review of: Evaluating Structured Output Robustness of Small Language Models for Open Attribute-Value Extraction from Clinical Notes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3S6OC6K7}},
  note         = {Machine review of arXiv:2507.01810}
}
read the original abstract

We present a comparative analysis of the parseability of structured outputs generated by small language models for open attribute-value extraction from clinical notes. We evaluate three widely used serialization formats: JSON, YAML, and XML, and find that JSON consistently yields the highest parseability. Structural robustness improves with targeted prompting and larger models, but declines for longer documents and certain note types. Our error analysis identifies recurring format-specific failure patterns. These findings offer practical guidance for selecting serialization formats and designing prompts when deploying language models in privacy-sensitive clinical settings.

Figures

Figures reproduced from arXiv: 2507.01810 by the authors.

Figure 1
Figure 1. Workflow for evaluating structured output [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Parseability rates by document type for open [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Breakdown of parse errors across JSON, XML, and YAML formats. Bars show the number of extraction-related and malformed output errors per for￾mat. Second, malformed output errors, which arise when the output is syntactically invalid and can￾not be parsed after successful extraction [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. XML Prompting as Grammar-Constrained Interaction: Fixed-Point Semantics, Convergence Guarantees, and Human-AI Protocols

    cs.PL 2025-09 reject novelty 4.0 of 10

    XML prompting is formalized as fixed-point iteration over an XML-tree lattice, with convergence claimed via Knaster-Tarski and Banach theorems, plus example XML recipe templates.

Reference graph

Works this paper leans on

24 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 others. 2024 a . https://arxiv.org/abs/2404.14219 Phi-3 technical report: A...

    arXiv 2024

  2. [2]

    Hewett, Mojan Javaheripi, Piero Kauffmann, James R

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 others. 2024 b . https://arxiv.org/abs/2412.08905 Phi-4 tech...

    arXiv 2024

  3. [3]

    Jeya Balaji Balasubramanian, Daniel Adams, Ioannis Roxanis, Amy Berrington de Gonzalez, Penny Coulson, Jonas S Almeida, and Montserrat Garc \' a-Closas. 2025. Leveraging large language models for structured information extraction from pathology reports. arXiv preprint arXiv:2502.12183

    arXiv 2025

  4. [4]

    Alexander Brinkmann, Roee Shraga, and Christian Bizer. 2025. Extractgpt: Exploring the potential of large language models for product attribute value extraction. In Information Integration and Web Intelligence, pages 38--52, Cham. Springer Nature Switzerland

    work page 2025

  5. [5]

    Rosen, Gerbrand Ceder, Kristin A

    John Dagdelen, Alexander Dunn, Sanghoon Lee, Nicholas Walker, Andrew S. Rosen, Gerbrand Ceder, Kristin A. Persson, and Anubhav Jain. 2024. https://doi.org/10.1038/s41467-024-45563-x Structured information extraction from scientific text with large language models . Nature Communications, 15(1):1418

    doi:10.1038/s41467-024-45563-x 2024

  6. [6]

    Ashraf Elnashar, Jules White, and Douglas C Schmidt. 2025. Enhancing structured data generation with gpt-4o evaluating prompt efficiency across prompt styles. Frontiers in Artificial Intelligence, 8:1558938

    work page 2025

  7. [7]

    Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S. Weld. 2008. https://doi.org/10.1145/1409360.1409378 Open information extraction from the web . Commun. ACM, 51(12):68–74

    work page arXiv 2008

  8. [8]

    Arunim Garg and Vijay Mago. 2021. Role of machine learning in medical research: A survey. Computer science review, 40:100370

    work page 2021

Show all 24 references
  1. [9]

    Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. 2000. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiolo...

    2000

  2. [10]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  3. [11]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. https://arxiv.org/abs/1904.09751 The curious case of neural text degeneration . In International Conference on Learning Representations (ICLR)

    2020 arXiv

  4. [12]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  5. [13]

    Johnson, Tom J

    Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016. https://doi.org/10.1038/sdata.2016.35 Mimic-iii, a freely accessible critical care database . Scienti...

    2016 doi

  6. [14]

    Alex Z Kadhim, Zachary Green, Iman Nazari, Jonathan Baker, Michael George, Ashley Heinson, Matt Stammers, Christopher M Kipps, R Mark Beattie, James J Ashton, and 1 others. 2025. Application of generative artificial intelligence to utilise unstructured clinical data for accele...

    2025

  7. [15]

    Yeonsu Kwon, Jiho Kim, Gyubok Lee, Seongsu Bae, Daeun Kyung, Wonchul Cha, Tom Pollard, Alistair Johnson, and Edward Choi. 2025. https://physionet.org/content/ehrcon-consistency-of-notes/1.0.0/ Ehrcon: Dataset for checking consistency between unstructured notes and structured t...

    2025

  8. [16]

    Yanzeng Li, Bingcong Xue, Ruoyu Zhang, and Lei Zou. 2023. Attgen: Attribute tree generation for real-world attribute joint extraction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2139--2152

    2023

  9. [17]

    Nikita Neveditsin, Pawan Lingras, and Vijay Mago. 2025. Clinical insights: A comprehensive review of language models in medicine. PLOS Digital Health, 4(5):e0000800

    2025

  10. [18]

    Alibaba Cloud Qwen Team. 2024. Qwen3 language model. https://huggingface.co/Qwen. Accessed 2024-05-12

    2024

  11. [19]

    Timo Schick and Hinrich Sch \"u tze. 2021. https://doi.org/10.18653/v1/2021.naacl-main.185 It`s not just size that matters: Small language models are also few-shot learners . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computation...

    2021 doi

  12. [20]

    Mara Schilling-Wilhelmi, Marti \ n o R \' os-Garc \' a, Sherjeel Shabih, Mar \' a Victoria Gil, Santiago Miret, Christoph T Koch, Jos \'e A M \'a rquez, and Kevin Maik Jablonka. 2024. From text to insight: large language models for materials science data extraction. arXiv prep...

    2024 arXiv

  13. [21]

    Sunghwan Sohn, Kavishwar B Wagholikar, Dingcheng Li, Siddhartha R Jonnalagadda, Cui Tao, Ravikumar Komandur Elayavilli, and Hongfang Liu. 2013. Comprehensive temporal information detection from clinical text: medical events, time, and tlink identification. Journal of the Ameri...

    2013

  14. [22]

    Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. 2024. https://doi.org/10.18653/v1/2024.emnlp-industry.91 Let me speak freely? a study on the impact of format restrictions on large language model performance. In Proceedings of the 2024 C...

    2024 doi

  15. [23]

    Yanshan Wang, Liwei Wang, Majid Rastegar-Mojarad, Sungrim Moon, Feichen Shen, Naveed Afzal, Sijia Liu, Yuqun Zeng, Saeed Mehrabi, Sunghwan Sohn, and 1 others. 2018. Clinical information extraction applications: a literature review. Journal of biomedical informatics, 77:34--49

    2018

  16. [24]

    Guineng Zheng, Subhabrata Mukherjee, Xin Luna Dong, and Feifei Li. 2018. Opentag: Open attribute value extraction from product profiles. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1049--1058

    2018

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.