pith. sign in

arxiv: 2605.30107 · v1 · pith:5GZAW4MSnew · submitted 2026-05-28 · 💻 cs.CL

Dial HEALTHDIAL for Advice: A Multilingual and Multi-Parallel Spoken Dialogue Dataset for Knowledge-Grounded Information Seeking

Songbo Hu , Yinhong Liu , Ej Zhou , Evgeniia Razumovskaia , Xiaobin Wang , Alexander Fraser , Ivan Vuli\'c , Anna Korhonen This is my paper

Pith reviewed 2026-06-29 07:53 UTC · model grok-4.3

classification 💻 cs.CL
keywords spoken dialogue datasetmultilingual dialogueknowledge-grounded dialoguehealth information seekingretrieval-augmented generationmultilingual benchmarksWHO content
0
0 comments X

The pith

HEALTHDIAL supplies 6000 spoken dialogues across Arabic, Chinese, English and Spanish grounded in World Health Organization content.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces HEALTHDIAL to address the scarcity of large-scale multilingual spoken dialogue resources suitable for knowledge-grounded systems. It contains 1500 dialogues per language for a total of 6000, each tied to trusted WHO health materials, plus 163 hours of native-speaker audio from diverse dialects and full demographic and sociolinguistic annotations on every speaker. Benchmark evaluations on standard dialogue tasks expose consistent performance gaps across the four languages, including between high-resource ones. The authors release the full dataset, a prototype system, and collection and evaluation tools. This setup allows direct measurement of how well retrieval-augmented generation systems handle spoken health queries in multiple languages.

Core claim

The authors establish that HEALTHDIAL comprises 6000 information-seeking dialogues (1500 per language) grounded in World Health Organization content across Arabic, Chinese, English, and Spanish, together with 163 hours of recorded user speech from native speakers of varied dialects and detailed speaker annotations, and that benchmarks on key dialogue tasks using this resource reveal consistent performance disparities across languages.

What carries the argument

The HEALTHDIAL dataset, a multilingual multi-parallel collection of spoken dialogues each grounded in the same WHO source material, serves as the central object that enables cross-language comparison and evaluation of RAG-based spoken dialogue systems.

If this is right

  • Benchmark results indicate that even high-resource languages exhibit measurable performance gaps on the same health-grounded tasks.
  • Speaker annotations permit analysis of how demographic and regional factors correlate with dialogue quality and system accuracy.
  • The multi-parallel alignment across languages supports direct testing of cross-lingual consistency in retrieval and response generation.
  • Release of the accompanying toolkit allows other researchers to replicate the collection process for additional languages or domains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The dataset could be used to test whether language-specific fine-tuning or adapter modules can close the observed performance gaps.
  • Speech recordings open the possibility of studying how dialectal pronunciation affects retrieval accuracy in spoken health queries.
  • Similar collection methods might be applied to other factual domains where grounding in authoritative sources is required.
  • The parallel structure may support experiments on whether joint training across languages improves factual consistency in generated advice.

Load-bearing premise

The collected dialogues and speech recordings accurately represent natural spoken information-seeking behavior and remain faithfully grounded in the WHO source material without collection-induced artifacts.

What would settle it

A controlled comparison showing that the recorded dialogues contain significantly more scripted or unnatural phrasing than spontaneous health consultations collected in the field would undermine the dataset's claim to represent real spoken behavior.

Figures

Figures reproduced from arXiv: 2605.30107 by Alexander Fraser, Anna Korhonen, Ej Zhou, Evgeniia Razumovskaia, Ivan Vuli\'c, Songbo Hu, Xiaobin Wang, Yinhong Liu.

Figure 1
Figure 1. Figure 1: Overview of the data collection pipeline. The process consists of four main steps: (i) [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of dialogues across the top four [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Average human ratings across key constructs, [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Knowledge filtering accuracy measured by [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: An illustration of dialogue systems based [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: A screenshot of a WHO webpage. The fig￾ure is annotated to show how each component of the webpage corresponds to the attributes of a knowledge snippet. ing snippet in English. We define the alignment as a set of functions that map each non-English snippet to a corresponding English snippet: f LAN : KLAN → KENG for LAN ∈ {ARA, ZHO, SPA}, where f LAN(k) returns the English snippet in KENG that is semanticall… view at source ↗
Figure 7
Figure 7. Figure 7: Transition probabilities in our Markov model. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Screenshot of the annotation interface with the guidelines shown to annotators during English data [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Example set of parallel dialogues in four languages, English, Arabic, Chinese, and Spanish, extracted from [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: Distribution of dialogues by annotator age [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Screenshot of the human evaluation interface with guidelines shown to annotators. The screenshot also [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗
read the original abstract

Creating spoken dialogue datasets is methodologically challenging, and these challenges are amplified when the goal is to build multilingual, multi-parallel datasets at scale. This work introduces HEALTHDIAL, a large-scale, multilingual, and multi-parallel dataset for developing and evaluating retrieval-augmented generation (RAG)-based spoken dialogue systems. The dataset comprises 6,000 information-seeking dialogues (1,500 per language) grounded in trusted content from the World Health Organization (WHO) and 163 hours of user speech recorded from native speakers of diverse dialects across four official WHO languages: Arabic, Chinese, English, and Spanish. Each speaker is annotated with demographic (e.g., gender, age) and sociolinguistic (e.g., primary language, region of origin) variables. We report benchmark results across key dialogue tasks, which reveal consistent performance disparities across languages, even among high-resource ones. To support future research, we release the dataset, a prototype system, and a toolkit for data collection and system evaluation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces HEALTHDIAL, a multilingual and multi-parallel spoken dialogue dataset for knowledge-grounded information seeking. It consists of 6,000 dialogues (1,500 per language) in Arabic, Chinese, English, and Spanish, grounded in World Health Organization (WHO) content, accompanied by 163 hours of recorded user speech from native speakers of diverse dialects. Each dialogue includes annotations for speaker demographics and sociolinguistic variables. The authors provide benchmark results on key dialogue tasks that demonstrate consistent performance disparities across languages and release the dataset, a prototype system, and a toolkit for collection and evaluation.

Significance. If the dialogues are accurately grounded and representative of natural spoken behavior, this dataset would fill an important gap in resources for developing and evaluating multilingual RAG-based spoken dialogue systems, especially in the health domain. The multi-parallel structure across four languages with dialectal variation and demographic annotations enables detailed cross-lingual and sociolinguistic analyses. The public release of the dataset, prototype, and toolkit supports reproducibility and future work. This is particularly valuable given the challenges of creating such datasets at scale.

major comments (2)
  1. [Dataset Construction section] Dataset Construction section: No quantitative verification metrics are reported for grounding accuracy (e.g., percentage of dialogues manually audited against WHO sources) or naturalness (e.g., ratings against spontaneous speech baselines). This is load-bearing for the central claim because the reported benchmark disparities across languages could reflect collection artifacts from prompting or translation mediation rather than genuine cross-lingual differences.
  2. [Benchmark Results section] Benchmark Results section: The exact computation of the reported performance metrics for dialogue tasks (e.g., retrieval precision or generation quality) is not defined with formulas or implementation details, preventing independent verification of the disparity findings.
minor comments (1)
  1. [Abstract] The abstract would benefit from a one-sentence summary of the collection protocol to allow standalone assessment of the dataset's claimed properties.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback on our manuscript. We appreciate the emphasis on ensuring the robustness of our claims regarding dataset quality and metric transparency. Below, we provide point-by-point responses to the major comments and indicate the revisions we plan to make.

read point-by-point responses

  1. Referee: [Dataset Construction section] Dataset Construction section: No quantitative verification metrics are reported for grounding accuracy (e.g., percentage of dialogues manually audited against WHO sources) or naturalness (e.g., ratings against spontaneous speech baselines). This is load-bearing for the central claim because the reported benchmark disparities across languages could reflect collection artifacts from prompting or translation mediation rather than genuine cross-lingual differences.

    Authors: We agree that providing quantitative verification metrics is crucial to support the dataset's quality and to address potential concerns about collection artifacts. The current manuscript does not include these metrics. In the revised version, we will add a dedicated subsection in the Dataset Construction section reporting the results of manual audits for grounding accuracy (including the percentage of dialogues verified against WHO sources) and human evaluation ratings for naturalness relative to spontaneous speech baselines. This will help substantiate that the observed performance disparities reflect genuine cross-lingual differences. revision: yes

  2. Referee: [Benchmark Results section] Benchmark Results section: The exact computation of the reported performance metrics for dialogue tasks (e.g., retrieval precision or generation quality) is not defined with formulas or implementation details, preventing independent verification of the disparity findings.

    Authors: We acknowledge the need for precise definitions of the metrics to allow for reproducibility and independent verification. The manuscript currently lacks explicit formulas and implementation details. We will revise the Benchmark Results section to include detailed formulas for all metrics (e.g., retrieval precision@K, generation quality measures such as BLEU or ROUGE with exact computation methods) along with implementation specifics, such as the libraries and parameters used. revision: yes

Circularity Check

0 steps flagged

No circularity: dataset release with empirical benchmarks only

full rationale

The paper is a data-release contribution that describes collection of 6000 dialogues, records speech, annotates demographics, and reports benchmark results on standard dialogue tasks. No equations, fitted parameters, predictions, or uniqueness theorems are claimed. The central claims (dataset size, language coverage, observed performance disparities) are direct descriptions of the released artifact and its measured properties rather than quantities derived from internal definitions or self-citations. No load-bearing step reduces to a fit or to a prior result by the same authors. This is the normal non-circular outcome for a resource paper.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the validity of the data collection process and the assumption that the recorded dialogues are representative and properly grounded; no free parameters, new entities or non-standard mathematical axioms are introduced.

axioms (1)
  • domain assumption Standard practices in spoken dialogue data collection produce dialogues that remain faithful to the source material and representative of natural user behavior.
    Invoked implicitly when claiming the dialogues are grounded in WHO content and suitable for benchmarking.

pith-pipeline@v0.9.1-grok · 5739 in / 1067 out tokens · 34100 ms · 2026-06-29T07:53:17.242294+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 4 canonical work pages

  1. [1]

    arXiv preprint arXiv:1912.06670 , year=

    SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing. InPro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 5723–5738, Dublin, Ireland. Associa- tion for Computational Linguistics. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohle...

    work page arXiv 2019

  2. [2]

    Exploring and controlling diversity in llm-agent conversation.arXiv preprint arXiv:2412.21102. Jan Clusmann, Fiona R Kolbinger, Hannah Sophie Muti, Zunamys I Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia Löf- fler, Sophie-Caroline Schwarzkopf, Michaela Unger, Gregory P Veldhuizen, and 1 others. 2023. The fu- ture landscape of la...

    work page arXiv 2023

  3. [3]

    how robust ru?

    XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. InProceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411–4421. PMLR. Songbo Hu, Abigail Oppong, Ebele Mogo, Charlotte Collins, Giulia Occhini, Anna Barford, and Anna Korhone...

    work page arXiv 2025

  4. [4]

    InInter- speech, pages 47–51

    Jee haan, i’d like both, por favor: Elicita- tion of a code-switched corpus of hindi-english and spanish-english human-machine dialog. InInter- speech, pages 47–51. Evgeniia Razumovskaia, Goran Glavaš, Olga Majew- ska, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vuli´c. 2022. Crossing the conversational chasm: A primer on natural language processing for ...

    2022

  5. [5]

    url " :

    Differentially private speaker anonymization. Proceedings on Privacy Enhancing Technologies, 2023:98–114. Shuichiro Shimizu, Chenhui Chu, Sheng Li, and Sadao Kurohashi. 2023. Towards speech dialogue transla- tion mediating speakers of different languages. In Findings of the Association for Computational Lin- guistics: ACL 2023, pages 1122–1134, Toronto, C...

    work page arXiv 2023

  6. [6]

    Example (System):Hello, I’m your virtual health assistant

    Opening:Thesysteminitiates the conversa- tion with a greeting and an introduction to its role or the service provided. Example (System):Hello, I’m your virtual health assistant. How can I help you today?

  7. [7]

    Example (User):Hey, I burned my hand cook- ing last week

    Health Concern Presentation:Theuser states their primary health concern, symptom, or question. Example (User):Hey, I burned my hand cook- ing last week. It’s really painful, red, and swollen

  8. [8]

    Example (System):Were you vaccinated for yellow fever before your trip?

    Information Gathering:Thesystemasks clarification questions to gather more context about the user’s symptoms or medical history. Example (System):Were you vaccinated for yellow fever before your trip?

  9. [9]

    Example (System):If the burn is larger than 3 inches or on your face, hands, or joints, you should definitely see a doctor

    Explanation / Medical Education:The systemprovides in-depth information or edu- cates the user about their condition, treatment options, and preventive measures. Example (System):If the burn is larger than 3 inches or on your face, hands, or joints, you should definitely see a doctor

  10. [10]

    Example (System):Until you see a doctor, keep the burn clean and covered with a sterile, non- stick bandage

    Care Planning and Guidance:Thesystem offers specific advice on managing the health issue, including treatment options, preventive measures, lifestyle modifications, and self-care techniques. Example (System):Until you see a doctor, keep the burn clean and covered with a sterile, non- stick bandage

  11. [11]

    Example (System):It’s important to consider your options and what feels right for you

    Decision Support:Theuser or systemmay discuss different options, relevant risks and ben- efits, and explore user preferences. Example (System):It’s important to consider your options and what feels right for you. You can also seek support from a trusted friend, family member, or a professional counsellor

  12. [12]

    Example (System):You can find a local urgent care centre or call your primary care doctor to schedule an appointment

    Healthcare System Navigation:Theuser or systemmay discuss guidance on navigat- ing the healthcare system, including finding a provider, making an appointment, and under- standing insurance coverage and costs. Example (System):You can find a local urgent care centre or call your primary care doctor to schedule an appointment

  13. [13]

    Example (System):In the UK, your medical records are confidential and protected by law

    Legal and Ethical Considerations:The user or systemmay discuss legal and ethical considerations, including informed consent and patient rights. Example (System):In the UK, your medical records are confidential and protected by law

  14. [14]

    Example (System):Your information is safe with us

    Privacy and Confidentiality:Theuser or systemmay inquire about, or proactively assure, the privacy and confidentiality of the user’s information. Example (System):Your information is safe with us. We take your privacy very seriously

  15. [15]

    It’s completely normal to feel scared and overwhelmed

    Emotional Support:Thesystemoffers emotional support, empathy, and reassurance to the user.Example (System):I’m sorry to hear that you’re going through this. It’s completely normal to feel scared and overwhelmed

  16. [16]

    Goodbye! A.3 Dialogue Schemata Creation Figure 7shows the transition probabilities in our hierarchical Markov model

    Closing:Thesystemends the conversation with a summary, an offer of further assistance, or a farewell.Example (System):You’re wel- come! Take care, and I hope you feel better soon. Goodbye! A.3 Dialogue Schemata Creation Figure 7shows the transition probabilities in our hierarchical Markov model. Let au i and as i de- note the discourse acts associated wit...

    2019