Pith. sign in

REVIEW 3 major objections 4 minor 21 references

ArgHiTZ at ArchEHR-QA 2025: A Two-Step Divide and Conquer Approach to Patient Question Answering for Top Factuality

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Splitting patient-EHR question answering into reranker-based sentence selection followed by small-model drafting yields the most factually grounded answers in the ArchEHR-QA 2025 shared task.

desk verdict A credible shared-task system description whose headline factuality ranking is real but brittle—treat it as a leaderboard result, not a proven claim. read the letter →

arxiv 2506.12886 v1 pith:RYLPB3O5 submitted 2025-06-15 cs.CL

Adrián Cuadrón , Aimar Sagasti , Maitane Urruela , Iker De la Iglesia , Ane G Domingo-Aldama , Aitziber Atutxa , Josu Goikoetxea , Ander Barrena This is my paper · ORCID
classification cs.CL
keywords patientquestionansweringelectronichealthrecordstwo-steppipelinesentencererankingfactualityclinicalNLPsharedtaskprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that patient messages can be answered directly from electronic health records, with source citations, using no external knowledge, no fine-tuning, and no retrieval augmentation, provided the task is split into two steps handled by different tools. The winning recipe is to first score every EHR sentence for relevance to the patient's question with a cross-encoder reranker, keep the sentences above a threshold chosen on the development set, and then prompt the 8B Aloe model to draft a short answer from only those sentences, attaching the original sentence IDs by similarity matching. In the ArchEHR-QA 2025 shared task this pipeline reached an overall score of 0.44, ranking 8th of 30 systems, and the top overall factuality score, with test factuality of 0.605 versus 0.408 for the end-to-end baseline. The paper's broader claim is that how a subtask is solved matters more than whether the task is decomposed, and that a small model can outperform much larger ones on this combined clinical and argumentative writing task.

What carries the argument

The load-bearing mechanism is a two-step pipeline whose first step is a score-thresholded cross-encoder reranker. The reranker takes the concatenation of the patient narrative and the reformulated clinician question as the query and outputs a relevance score for each EHR sentence; the decision boundary is the score that maximizes the Youden index $J = \mathrm{sens} + \mathrm{spec} - 1$ on the ROC curve of the development set. Sentences above the threshold are treated as essential and passed to the second step, where Aloe 8B is prompted to draft a paragraph of at most 75 words using only those sentences; citation IDs are then added by matching each generated sentence to its most similar essential sentence, with a cap per sentence and with title-like sentences excluded. Because the second step keeps and often improves the first step's F-score, the reranker threshold effectively carries the system's factuality.

What would settle it

Compute the reranker threshold that maximizes the Youden index on the test set itself, using the released essential labels. If the test-optimal threshold differs markedly from the development-chosen one and yields a different F1, then the reported top factuality depends on threshold transfer rather than on the two-step architecture; the paper itself notes this risk in Section 6.2.1.2.

Watch

Extended reading notes

Core claim

The central claim is that a divide-and-conquer decomposition—sentence selection by similarity reranking followed by answer generation by a smaller instruction-tuned model—is the most factually reliable way to answer patient questions from EHRs in this task. The paper's best system does not use external knowledge: the patient narrative, the clinician's reformulated question, and the sentence-split EHR are the only inputs. The reranker assigns each sentence a relevance score; a threshold maximizing the Youden index on the 20-document development set separates essential from non-essential sentences; then Aloe 8B writes a coherent answer of at most 75 words using only the selected sentences, and a similarity post-processor attaches the original sentence IDs as citations, which preserves or improves the factuality F-score of the first step. On the test set this two-step reranker system achieves 0.605 factuality and 0.276 relevance, an overall 0.44, the best factuality among all 30 runs, while both prompting-based alternatives and the end-to-end baseline score lower. The paper interprets this as evidence that subtask decomposition helps only when each subtask uses a method matched to it.

Load-bearing premise

The single reranker threshold chosen to maximize the Youden index on the 20 development documents carries over to the test set, so if the test notes are scored differently by the reranker, the reported factuality lead could shrink or vanish; the paper itself flags this as a potential limitation.

Editorial extensions

If this is right

  • For this task, the reranker-based two-step system raises test factuality to 0.605 from 0.408 for end-to-end prompting, at the cost of a lower relevance score of 0.276.
  • All three proposed systems beat the organizers' zero-shot Llama 3.3 70B baseline, whose test overall score is 0.307, even though the best system uses the 8B Aloe model.
  • Prompt-based sentence extraction, whether by generating a list of IDs or by binary per-sentence classification, underperforms the similarity reranker, showing that the choice of method for the extraction subtask is the decisive factor.
  • The citation-by-similarity post-processing means factuality on the test leaderboard can be improved without changing the drafting model, as long as the selected essential sentences are accurate.
  • Smaller instruction-tuned models suffice for drafting; the two 70B models gave no overall advantage and were set aside for cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to make the threshold adaptive: instead of one global cutoff from 20 development documents, estimate a cutoff per query or calibrate the reranker scores on a small labeled sample, which would test whether the reported factuality holds under distribution shift.
  • Because relevance trails factuality across all systems, the bottleneck is drafting, not selection; a text-to-text model fine-tuned on clinical summarization, or a generation step that sees the original sentences with their relevance labels, could raise overall scores without changing the reranker.
  • The two-step recipe is a cheap, reproducible baseline for grounded clinical question answering generally: a cross-encoder selector plus a small generative model can be assembled from public checkpoints and needs no training data beyond the task set.
  • The paper's claim that scale is not decisive is conditional on the task mix of medical expertise, summarization, and rewriting; tasks dominated by reasoning or retrieval could still favor larger models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper is a system description for the ArchEHR-QA 2025 shared task on grounded patient question answering from electronic health records. It compares an end-to-end LLM-prompt baseline with two two-step pipelines: the first step extracts essential sentences either by LLM prompting or by a similarity reranker with a score threshold, and the second step generates a cited patient-facing answer from those sentences. On the development set (20 documents), the reranker-based pipeline using Jina reranking and Aloe 8B achieves the best factuality; on the official test set, the same pipeline reports an overall score of 0.44, rank 8 out of 30, and the highest factuality score of 0.605. The code is publicly released and all experiments use only the provided shared-task data plus off-the-shelf pretrained models.

Significance. If taken at face value, the test-set result is a useful engineering contribution: a training-free, two-step architecture with a reranker for evidence selection achieves top factuality on the official leaderboard while outperforming the organizers' zero-shot Llama-3.3-70B baseline. The paper is transparent, the code is public, and the comparison of three pipeline variants is valuable for the shared-task community. However, the headline claim of 'securing the top position in overall factuality' rests on a single test-set point estimate, with all design choices (reranker, threshold, prompts, post-processing) tuned on only 20 development documents and no uncertainty quantification. The paper's value is therefore primarily as a reproducible system description and an empirical observation that reranker-based sentence selection outperforms prompt-based selection in this setup, rather than as a demonstration of statistically established superiority.

major comments (3)
  1. [Abstract, Section 7, Table 5] The claim of 'securing the top position in overall factuality' is supported only by a single test-set Factuality score of 0.605, with no confidence interval, bootstrap estimate, document-level variance, or margin over the runner-up. Because the reranker model, the threshold in Section 6.2.1.2, and the prompt configurations were all selected on 20 development documents, the reported rank could be a small-sample artifact. I request either uncertainty estimates or at least the runner-up's factuality score, and a rewording that distinguishes 'ranked first on the official leaderboard' from an established statistical superiority.
  2. [Section 6.2.1.2, Table 4] The reranker threshold is fitted to the development labels by maximizing the Youden index, so the development Factuality results in Table 4 are in-sample scores for that fitted threshold. Comparing them directly with prompt-based extractors, which involve no analogous fitted threshold, is not an apples-to-apples comparison. The test-set result is externally scored and therefore still supports the main claim, but the development analysis should be labeled as threshold-optimized and, ideally, supplemented with a threshold-free comparison or a sensitivity analysis.
  3. [Section 3, Tables 1-4] All model and prompt selection decisions are made on a development set of only 20 documents, and no per-document variance is reported. Differences such as Aloe 8B versus Mistral 7B in Table 2 (0.50 versus 0.48 F1) and BAAI versus Alibaba in Table 4 (0.507 versus 0.521) are likely within sampling noise. This does not undermine the official test result, but it does undermine the secondary conclusions that rerankers 'are more effective' than prompting and that larger models do not help; please add a caveat or per-document statistics.
minor comments (4)
  1. [Abstract and Section 3] There are several typos and formatting artifacts, including 'extractessentialsen-' in the abstract, 'for thedevset' in Section 3, and 'a separated file' later in the same section.
  2. [Appendix A.3, A.4] The prompt listings contain mojibake characters such as 'patientâĂŹs' instead of 'patient's'; these should be fixed for readability.
  3. [Table 5] The caption says 'Best results in bold', but in the plain-text rendering no bold is visible; also, the organizers' zero-shot baseline mentioned in Section 7 (0.359 dev, 0.307 test) would be more directly comparable if included in Table 5.
  4. [Section 6.2.1.2] The text notes that Jina obtains the highest F1-score in Table 4, but does not mention that Jina's high recall (0.717) comes with substantially lower precision (0.427); a brief comment on this trade-off would help the reader interpret the threshold choice.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central factuality claim rests on an external leaderboard, and the dev-set threshold tuning is a standard generalization risk rather than a self-fulfilling prediction.

full rationale

The paper's main result (overall 0.44, top factuality on the leaderboard) is measured by the shared task's external evaluation on the test set and is not computed from the paper's own fitted parameters. The reranker threshold is chosen on the 20-document dev set via the Youden index, and the authors explicitly acknowledge in Section 6.2.1.2 that the threshold may not generalize to the test set if the data distribution differs. Reporting dev F1 after tuning on the same dev set is an optimistic evaluation, but the paper does not present that dev number as an independent prediction; the test-set scores in Table 5 are the ones used for the headline claim. The post-processing step that assigns citations by matching generated sentences to first-step essential sentences does mean the final factuality score is largely inherited from the first-step extraction, but the paper states this transparently ('this process preserves—and often improves—the F-Score of the first step'), which is a pipeline property, not a circular derivation of a conclusion from its own premise. There are no load-bearing self-citations; the dataset and task references are to the shared-task organizers, and the reranker and generation models are external. Remaining concerns about the small dev set, missing confidence intervals, and absent runner-up margins are statistical robustness issues, not circularity under the criteria used here.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The system relies on the official ArchEHR-QA evaluation metrics, the representativeness of the 20-document dev set, and the correctness of the hosted models and rerankers. The only fitted numeric parameter is the reranker threshold, tuned on dev. No new scientific entities are postulated.

free parameters (1)
  • reranker score threshold = not reported
    Section 6.2.1.2: sentences with reranker score above the threshold are labeled essential; threshold chosen via Youden index on the 20-document dev set. Exact value not given in the paper.
assumptions (3)
  • domain assumption The official shared task metrics (strict and lenient factuality F1 and relevance metrics BLEU, ROUGE, SARI, BERTScore, AlignScore, Medcon) are valid proxies for answer quality.
    Section 5 lists these metrics as the evaluation; the paper's claim of 'top factuality' is defined by them.
  • domain assumption The 20-document development set is representative enough to select the reranker model and threshold.
    Section 6.2.1.2 chooses the reranker and threshold based on dev performance and then relies on transfer to the test set; the authors flag this as a limitation.
  • domain assumption The third-party models and rerankers (Aloe 8B, Jina reranker) used are the correct, operative versions and their outputs are comparable.
    Section 4 lists the models; no version pinning or API or hardware details are given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ArgHiTZ at ArchEHR-QA 2025: A Two-Step Divide and Conquer Approach to Patient Question Answering for Top Factuality." pith.science (2026). https://pith.science/paper/RYLPB3O5

@misc{pith2026250612886,
  author       = {Pith},
  title        = {Pith review of: ArgHiTZ at ArchEHR-QA 2025: A Two-Step Divide and Conquer Approach to Patient Question Answering for Top Factuality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RYLPB3O5}},
  note         = {Machine review of arXiv:2506.12886}
}
read the original abstract

This work presents three different approaches to address the ArchEHR-QA 2025 Shared Task on automated patient question answering. We introduce an end-to-end prompt-based baseline and two two-step methods to divide the task, without utilizing any external knowledge. Both two step approaches first extract essential sentences from the clinical text, by prompt or similarity ranking, and then generate the final answer from these notes. Results indicate that the re-ranker based two-step system performs best, highlighting the importance of selecting the right approach for each subtask. Our best run achieved an overall score of 0.44, ranking 8th out of 30 on the leaderboard, securing the top position in overall factuality.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 9 canonical work pages

  1. [1]

    Asma Ben Abacha, Yassine Mrabet, Yuhao Zhang, Chaitanya Shivade, Curtis Langlotz, and Dina Demner-Fushman. 2021. https://doi.org/10.18653/v1/2021.bionlp-1.8 Overview of the MEDIQA 2021 shared task on summarization in the medical domain . In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 74--85, Online. Association for Computatio...

    work page doi:10.18653/v1/2021.bionlp-1.8 2021

  2. [2]

    Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. 2019. https://doi.org/10.18653/v1/W19-5039 Overview of the MEDIQA 2019 shared task on textual inference, question entailment and question answering . In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 370--379, Florence, Italy. Association for Computational Linguistics

    work page doi:10.18653/v1/w19-5039 2019

  3. [3]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. https://arxiv.org/abs/2402.03216 Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation . Preprint, arXiv:2402.03216

    arXiv 2024

  4. [4]

    Ashwin Kumar Gururajan, Enrique Lopez-Cuena, Jordi Bayarri-Planas, Adrian Tormos, Daniel Hinjos, Pablo Bernabeu-Perez, Anna Arias-Duart, Pablo Agustin Martin-Torres, Lucia Urcelay-Ganzabal, Marta Gonzalez-Mallo, Sergio Alvarez-Napagao, Eduard Ayguadé-Parra, and Ulises Cortés Dario Garcia-Gasulla. 2024. https://arxiv.org/abs/2405.01886 Aloe: A family of fi...

    arXiv 2024

  5. [5]

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. https://doi.org/10.18653/v1/D19-1259 P ub M ed QA : A dataset for biomedical research question answering . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-I...

    doi:10.18653/v1/d19-1259 2019

  6. [6]

    JinaAI. 2024. jina-reranker-v2-base-multilingual. https://hf.co/jinaai/jina-reranker-v2-base-multilingual

    work page 2024

  7. [7]

    Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. 2023. Mimic-iv-note: Deidentified free-text clinical notes (version 2.2). physionet

    work page 2023

  8. [8]

    Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1--9

    2016

Show all 21 references
  1. [9]

    Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. Bioasq-qa: A manually curated corpus for biomedical question answering. Scientific Data, 10(1):170

    2023

  2. [10]

    Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. 2023. https://arxiv.org/abs/2312.15503 Making large language models a better foundation for dense retrieval . Preprint, arXiv:2312.15503

    2023

  3. [11]

    MetaAI. 2024. Llama-3.3-70b-instruct. https://hf.co/meta-llama/Llama-3.3-70B-Instruct

    2024

  4. [12]

    MistralAI. 2025. Mistral-7b-instruct-v0.3. https://hf.co/mistralai/Mistral-7B-Instruct-v0.3

    2025

  5. [13]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248--260. PMLR

    2022

  6. [14]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of Machine Lea...

    2020

  7. [15]

    Sarvesh Soni and Dina Demner-Fushman. 2025 a . A dataset for addressing patient's information needs related to clinical course of hospitalization. arXiv preprint

    2025

  8. [16]

    Sarvesh Soni and Dina Demner-Fushman. 2025 b . Overview of the archehr-qa 2025 shared task on grounded question answering from electronic health records. In The 24th Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, Vienna, Austria. Association for Co...

    2025

  9. [17]

    Gemma Team. 2024. https://doi.org/10.34740/KAGGLE/M/3301 Gemma

    2024 doi

  10. [18]

    Lawrence KQ Yan, Qian Niu, Ming Li, Yichao Zhang, Caitlyn Heqi Yin, Cheng Fei, Benji Peng, Ziqian Bi, Pohsun Feng, Keyu Chen, and 1 others. 2024. Large language model benchmarks in medical tasks. arXiv preprint arXiv:2410.21348

    2024

  11. [19]

    Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, and 1 others. 2024. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conf...

    2024

  12. [20]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  13. [21]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.