Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs

Bolaji Yusuf; Jan \v{C}ernock\'y; J\'an \v{S}vec; Old\v{r}ich Plchot; Pradyoth Hegde; Santosh Kesiraju; \v{S}imon Sedl\'a\v{c}ek

REVIEW 2 major objections 8 minor 35 references

Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs

T0 review · 2 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Aligned speech encoder and open LLM reach 42.17% accuracy on SpokenWOZ

desk verdict Solid empirical study with a genuinely useful ablation suite, but the headline SOTA claim rests on a modified test set and baselines that were not re-evaluated on it — worth refereeing, but the comparison needs fixing. read the letter →

arxiv 2506.08633 v1 pith:VBR74DB3 submitted 2025-06-10 eess.AS cs.CL

Šimon Sedláček , Bolaji Yusuf , Ján Švec , Pradyoth Hegde , Santosh Kesiraju , Oldřich Plchot , Jan Černocký This is my paper · ORCID

classification eess.AScs.CL

keywords dialoguestatetrackingspokenunderstandingspeechencoderLLMalignmentconnectormoduleLoRAfine-tuningWOZjointgoalaccuracyopen-source

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that spoken dialogue state tracking — keeping an updated record of what a user wants across multi-turn spoken conversation — can be done end to end by aligning a pre-trained speech encoder with a large language model through a small trainable connector. The authors build the entire pipeline from openly released components, WavLM as the speech encoder and OLMo-1B as the language model, and report 34.66% joint goal accuracy on the SpokenWOZ test set, which they say surpasses published cascaded baselines. They report a further jump to 42.17% JGA when the connector is attached to the larger Gemma-2-9B-Instruct model. The result matters because it suggests that a fully open, end-to-end trained speech-to-DST system can beat cascade pipelines that transcribe first and then track state from text, without relying on proprietary training data.

What carries the argument

The load-bearing mechanism is a small trainable connector module — a two-layer transformer with a subsampling front end that stacks six neighbouring WavLM embeddings and projects them into the LLM's text embedding space, where they act as soft prompts. The connector is trained in two steps: a frozen-LLM ASR stage that aligns speech encoder and language model on thousands of hours of transcribed speech, and a joint ASR-DST stage in which the encoder stays frozen while low-rank adapters (LoRA) in the LLM are fine-tuned with the connector to autoregressively generate a JSON string containing the user transcription, domains, and slot values. This two-step scheme, not any single component, is what the paper credits for making end-to-end spoken DST work with fully open models.

What would settle it

Take the published SpokenWOZ test set before any filtering, run the two reference systems (the Gemma-2-9B cascade from [27] and the SPACE+WavLM system from [2]) on exactly the nine-conversation-filtered split used in this paper, and compare their JGA to 34.66% and 42.17%; if the baselines rise to the same level once the same deletions are applied, the claimed SOTA is not due to the alignment method.

Watch

Extended reading notes

Core claim

The central claim is that bridging WavLM-large and an LLM with a two-layer transformer connector, trained first for ASR and then for joint ASR-plus-DST with low-rank adapters, yields a spoken dialogue state tracker that exceeds prior published results on SpokenWOZ. The best fully open configuration (WavLM + connector + OLMo-1B) reaches 34.66% JGA after fuzzy-matching post-processing, and replacing the LM with Gemma-2-9B-Instruct raises this to 42.17%. In the paper's own account, the two-stage training is essential: ASR pre-training aligns the modalities on large speech corpora, and joint fine-tuning preserves the model's ability to transcribe the user side at inference time while still emitting the JSON dialogue state. The authors also find that including agent turns in the dialogue history and post-processing slot values with fuzzy matching both give consistent gains.

Load-bearing premise

The comparison to prior work assumes the SpokenWOZ test set the authors evaluate on is the same as the one used to compute the published 25.40% and 25.65% results; the authors remove nine 'corrupted' conversations from the test set, and if those conversations were valid or were included in the prior scores, the reported state-of-the-art gap is partly an artifact of the altered evaluation set.

Editorial extensions

If this is right

An end-to-end aligned speech-LLM system is a viable alternative to ASR-then-DST cascades for spoken task-oriented dialogue, at least on SpokenWOZ.
Fully open models (WavLM-large and OLMo-1B) are sufficient to set a new published JGA mark, which the authors credit to reduced test-contamination risk and the two-stage alignment training.
Fuzzy matching of predicted slot values against the database adds roughly three points of JGA absolute, indicating that much of the remaining error is in exact spelling of named entities rather than in slot selection.
Larger instruction-tuned LLMs (Gemma-2-9B) amplify the same connector-based alignment, improving JGA from 34.66% to 42.17% after the same post-processing.
Training on the TTS-based Speech-Aware MultiWOZ data alongside SpokenWOZ helps the DST head generalize, but the model still transfers poorly to human-verbatim MultiWOZ test data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: the magnitude of the reported gain over prior cascades suggests that connector-based alignment, not just the LLM, is doing the work; a direct test would be to keep the same two-stage training but replace the connector with a fixed-rate subsampler.
Editorial extension: because the nine removed SpokenWOZ conversations are not re-scored for the prior baselines, a clean re-evaluation of those baselines on the filtered test set is needed before the SOTA claim can be taken at face value; this is a testable step the paper does not perform.
Editorial extension: the same connector-plus-two-stage recipe could transfer to other spoken language understanding tasks such as slot filling or spoken question answering, where the JSON output schema would simply be replaced.
Editorial extension: the reported sensitivity to the ASR context (Whisper context raises JGA from 31.91 to 32.89) implies that further gains may come from a speech encoder better matched to the human-read MultiWOZ domain, rather than from scaling the LLM alone.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a clean, honest empirical study of a fully open speech-encoder-to-LLM pipeline for spoken dialogue state tracking. The two-stage ASR-then-DST training is a sensible recipe, the ablations (agent turns, LoRA rank, ASR initialization, fuzzy post-processing, training data augmentation) are exactly what practitioners need, and the choice of WavLM/OLMo to address contamination concerns is thoughtful. The best open pipeline hits 34.66% JGA on SpokenWOZ, and the Gemma-2-9B variant reaches 42.17%. These are real measurements on public data with a standard metric, and the internal consistency across tables is good. I believe the core contribution — that a fully open aligned system can beat a Whisper-to-OLMo cascade — is solid and probably true.

The soft spot is the one the abstract leans on: state-of-the-art. The authors remove nine 'corrupted' test conversations (Section 3.1, footnote 3) but compare against published numbers (25.40, 25.65) that were computed on the untouched official test set. They never show the baselines re-evaluated on the reduced set, and nine conversations can move JGA by several points on a dataset this size. The A10 number (31.91% without fuzzy matching) still clears the cited baselines, so the method is likely ahead regardless, but the headline 42.17% and the phrase 'state of the art' overstate what is actually demonstrated. This is not a fatal flaw, and it is not hidden — the removal is disclosed — but the conclusion should not ignore it.

Minor issues: no error bars or multiple seeds (the ablations are largely single-run), no code release, and the Gemma-2-9B comparison is peripheral to the paper's open-model thesis. The fuzzy matching post-processing is standard in the DST community, so that is not a problem per se, though the 3% absolute gain should be understood as part of the system rather than as model capability.

Who is this for? Someone building open spoken DST systems, or anyone wanting a clear picture of how far a WavLM+connector+small-LLM pipeline can go. It deserves a serious referee. The right outcome is probably acceptance after the authors add a re-evaluated-baseline comparison on the reduced test set, or at minimum a sensitivity analysis showing the effect of removing the nine conversations. I would take it to the reading group as a useful example of how to structure spoken-DST ablations.

Referee Report

2 major / 8 minor

Summary. This paper proposes an end-to-end spoken dialogue state tracking (DST) system that connects a WavLM-large speech encoder to an LLM through a small transformer connector. The training pipeline first pre-trains the connector and encoder for ASR, then fine-tunes the connector and LoRA adapters on a joint ASR-DST objective. Experiments on SpokenWOZ and Speech-Aware MultiWOZ ablate dataset combinations, user/agent dialogue history, LoRA versus full fine-tuning, and fuzzy matching post-processing. The central claims are state-of-the-art results on the SpokenWOZ test set: 34.66% joint goal accuracy (JGA) with the fully open OLMo-1B model and 42.17% JGA with Gemma-2-9B-Instruct, compared to 25.40% and 25.65% for two cited baselines.

Significance. If the results hold, this is a useful empirical contribution: a relatively simple end-to-end alignment of a public speech encoder and an LLM outperforms prior cascaded and aligned systems on SpokenWOZ, and the ablations clarify which design choices matter (ASR pre-training, LoRA, user+agent history, fuzzy matching). The use of openly documented models (WavLM, OLMo), public datasets, and the standard MultiWOZ evaluation script are strengths, as is the transparent reporting of training configurations. The main caveat is that the SOTA comparison rests on a modified test set, so the headline numbers need to be re-anchored to a common evaluation set before the claim is credible.

major comments (2)

[Section 3.1, Table 1, Section 5] The headline state-of-the-art claim is based on a modified SpokenWOZ test set. Section 3.1 states that nine "corrupted" conversations are removed from the SpokenWOZ test set (footnote 3), and Table 1 reports 34.66% and 42.17% JGA on this reduced set. The two prior baselines in Table 1 (25.40% from [27] and 25.65% from [2]) are published numbers for the official test set; the paper does not re-evaluate these baselines on the reduced set, does not report results on the untouched official test set, and does not quantify how many turns the removed dialogues contain. Since JGA is a turn-level metric, the reported SOTA margin could be partly an artifact of the evaluation-set change. Please either re-run the baselines on the reduced set, report results on the full official test set, or explicitly restrict the SOTA claim to the reduced set and justify that restriction.
[Tables 3 and 4; Section 4.2] The paper reports a single run for every configuration and gives no error bars or multiple seeds. This makes it hard to judge the reliability of the main comparisons, e.g., the fuzzy-matching gain (A10 vs A11: 31.91 vs 34.66) and the effect of user+agent history (A8 vs A9: 27.27 vs 31.04). In addition, the fuzzy matching post-processing is applied to the proposed systems but not to the cited baselines; if the baselines would also benefit from this post-processing, the comparison in Table 1 is not controlled. Please provide variance estimates (or at least multiple runs) for the key numbers and, where feasible, report the baselines with the same post-processing.

minor comments (8)

[Section 3.2 vs 4.2] Section 3.2 says the optional final fine-tuning uses batch size 256, but Section 4.2 describes the A11 run with batch size 192; please reconcile the two numbers.
[Section 4.3] Section 4.3 states that Gemma-2-9B-Instruct is trained with "the same ... training hyperparameters as for our other models," yet the LoRA rank is r=8 rather than the r=16 used in Section 3.2; please clarify.
[Section 5] Section 5 contains a typo: "fuzy matching" should be "fuzzy matching."
[Section 4.2] Section 4.2 contains "apart from theprofiledomain" with a missing space; it should read "apart from the profile domain."
[Abstract and Section 3.1] The abstract and Section 3.1 emphasize fully open components, but Gemma-2-9B-Instruct is an open-weights model, not an open-data model; please distinguish this when describing the 42.17% result.
[Section 2.2] The fuzzy matching post-processing is described only by a package name; for reproducibility please specify the matching score threshold and how candidate database values are selected.
[Table 2] Table 2 includes the DSTC-11 reference systems (25.2 and 43.1 JGA on MWOZ dev) but the text does not discuss why the proposed systems are not compared against them; a sentence of context would help.
[General] The paper does not state whether code or model checkpoints will be released, despite the emphasis on openness; please add an availability statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all headline JGA figures are directly measured on held-out SpokenWOZ test data; the modified-test-set concern is a benchmark-fairness issue, not a circular derivation.

full rationale

This is an empirical systems paper with no claimed derivation from first principles. The headline results (34.66% JGA for WavLM+connector+OLMo-1B and 42.17% JGA for Gemma-2-9B-Instruct) are directly measured on held-out SpokenWOZ test data. The two-stage ASR-pretraining and joint ASR-DST fine-tuning procedure uses standard held-out dev-based early stopping, and the fuzzy matching post-processing is applied uniformly to reported variants. No fitted parameter is renamed as a prediction, and no target quantity reduces to an input by construction. The only notable concern is that nine 'corrupted' conversations were removed from the official SpokenWOZ test set (Section 3.1, footnote 3) before comparing against published baselines from [27] and [2]. If those published numbers were computed on the untouched test set, the SOTA comparison may not be strictly apples-to-apples; however, this is a benchmark-validity and fairness concern, not circularity. The cited prior systems are external works, and the paper's central claim does not rest on a self-citation chain. The authors' own prior work appears only as related methodology, not as load-bearing justification. Therefore no circular step is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper's central claim is an empirical benchmark result. It does not introduce new theoretical entities. The listed free parameters are training hyperparameters chosen by hand, and the axioms are the dataset-compatibility, post-processing, and evaluation-set assumptions that the SOTA claim depends on.

free parameters (2)

LoRA rank = 16 (OLMo-1B), 8 (Gemma-2-9B-Instruct)
Chosen by hand in Section 3.2; affects adaptation capacity and final JGA. No systematic sweep is reported, so the impact on the SOTA claim is not quantified.
Training batch sizes and learning rates = ASR: batch 64, lr 1e-4; DST: batch 128, lr 5e-5; FT: batch 256 (OLMo) or 192 (reported for FT-sw), lr not restated
Stated in Sections 3.2 and 4.2; these hyperparameters were chosen by the authors and could affect the reported JGA, but they are not fitted to the test set in a systematic way.

assumptions (4)

domain assumption SpokenWOZ and Speech-Aware MultiWOZ share the same dialogue schema as MultiWOZ 2.1, so joint training or augmentation transfers.
Section 3.1 and 4.2: the authors rely on this to combine datasets; if schemas differ more than stated, the augmentation could hurt performance.
domain assumption Fuzzy matching predicted slot values to the nearest database value is a valid post-processing correction that improves JGA; the algorithm and its threshold are assumed to be appropriate.
Section 2.2 and Table 3 (A5/A11): fuzzy matching yields ~3% absolute JGA gains; there is no manual evaluation of whether these corrections are faithful to the dialogue.
ad hoc to paper Excluding nine corrupted SpokenWOZ test conversations is legitimate and does not bias the SOTA comparison.
Footnote 3 cites a GitHub issue; this modifies the official benchmark and is a post-hoc choice that materially affects comparability with prior published numbers.
domain assumption The two-stage training (ASR pre-training then DST fine-tuning with LoRA) prevents catastrophic forgetting of the LLM while enabling speech-conditioned DST.
Section 2.1: the method assumes that the connector and LoRA can map speech representations into the LLM's text embedding space without full fine-tuning; the ablation A1 vs A3 suggests ASR init helps, but the assumption is not separately validated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs." pith.science (2026). https://pith.science/paper/VBR74DB3

@misc{pith2026250608633,
  author       = {Pith},
  title        = {Pith review of: Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VBR74DB3}},
  note         = {Machine review of arXiv:2506.08633}
}

read the original abstract

In this work, we approach spoken Dialogue State Tracking (DST) by bridging the representation spaces of speech encoders and LLMs via a small connector module, with a focus on fully open-sourced and open-data components (WavLM-large, OLMo). We focus on ablating different aspects of such systems including full/LoRA adapter fine-tuning, the effect of agent turns in the dialogue history, as well as fuzzy matching-based output post-processing, which greatly improves performance of our systems on named entities in the dialogue slot values. We conduct our experiments on the SpokenWOZ dataset, and additionally utilize the Speech-Aware MultiWOZ dataset to augment our training data. Ultimately, our best-performing WavLM + connector + OLMo-1B aligned models achieve state of the art on the SpokenWOZ test set (34.66% JGA), and our system with Gemma-2-9B-instruct further surpasses this result, reaching 42.17% JGA on SpokenWOZ test.

Figures

Figures reproduced from arXiv: 2506.08633 by the authors.

**Figure 1.** (Left) Stage-1 pre-training for ASR. (Right) Stage2 training for joint ASR and dialogue state tracking (DST). Shaded modules are frozen during training. Partly shaded implies that the module is frozen, but additional trainable parameters are added via LoRA. 2.1.1. ASR pre-training Since the speech encoder and LM are pretrained independently of each other, we first bridge their representation spaces before fine-tu… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 32 canonical work pages

[27]

MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Base- lines,

M. Ericet al., “MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Base- lines,” inProceedings of the Twelfth Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association, May 2020, pp. 422–428

work page 2020
[2]

transcription

Method overview Our method treats DST as the problem of mapping from user speech and user-agent dialogue history to a dictionary contain- ing the correct transcription of the user’s speech, and the in- ferred dialogue states (domains, slots). Therefore, we adopt a model which we train to take speech as input and directly output a JSON string representing ...

arXiv 2025
[1]

book-restaurant

Introduction Task-oriented dialogues (ToD) are multiturn conversations be- tween a user and an agent, where the former has a specific goal (e.g., booking a restaurant for 5 people on Friday night) that is achieved with the help of the agent. A key component of auto- mated ToD systems is dialogue state tracking (DST), the task of tracking the user’s intent...

work page
[3]

Experiments 3.1. Datasets We conduct our experiments using two speech-grounded task- oriented dialogue datasets: SpokenWoZ [2] (SWOZ) and Speech-Aware MultiWoZ [1] (MWOZ), built on top of the original text-based MultiWoZ 2.1 [18, 19]. We remove nine originally corrupted conversations from the SpokenWoZ test set3. For SpokenWOZ, we also generate ASR transc...

work page 2000
[4]

dialogue history

from prior works. Model SWOZ test JGA SER (1) Gemma-2-9B-Instruct (cascaded) [27] 25.40 - (2) SPACE+WavLM align [2] 25.65 - Whisper→OLMo-1B, full FT, SW+MW (cascade) 30.74 31.11 WavLM + conn. + OLMo-1B (A11 from Table 3) 34.66 26.80 WavLM + conn. + Gemma-2-9B-Instruct (Table 4)42.17 20.41 We also compare our results to the best system from [2] and the rel...

work page
[5]

Results 4.1. Cascaded system analysis Our experiments focus on ablating three main training setup fac- tors: the training data used, the type of LLM fine-tuning, and the usage of both user and agent turns in the dialogue history. a full overview of our cascaded systems is provided in Table

work page
[6]

First, we observe that fine-tuning on the original Spoken- WoZ transcripts yields worse results than when using Whis- per transcripts, which are of higher quality

We additionally include two text-only baseline systems from DSTC-11 [1] as reference baselines for speech-aware MWOZ. First, we observe that fine-tuning on the original Spoken- WoZ transcripts yields worse results than when using Whis- per transcripts, which are of higher quality. Second, we find it beneficial to combine both SpokenWoZ and SA-MultiWOZ for...

work page
[7]

Conclusions In this work, we propose an end-to-end dialogue state tracking system based on bridging the representation spaces of a pre- trained speech encoder with an LLM via a small transformer connector with a two-step ASR-DST fine-tuning scheme. We use open source models for both the encoder (WavLM-large) and the LLM (OLMo-1B) to mitigate the risk of t...

work page

Show all 35 references

[8]

SEP-210943216 ”ELOQUENCE”, Euro- pean Defence Fund project ARCHER, Czech Ministry of In- terior project No

Acknowledgements The work was supported by European Union’s Horizon Eu- rope project No. SEP-210943216 ”ELOQUENCE”, Euro- pean Defence Fund project ARCHER, Czech Ministry of In- terior project No. VK01020132 ”112” and by Czech Min- istry of Education, Youth and Sports (MoE) th...
[9]

DSTC-11: Speech Aware Task-Oriented Dialog Modeling Track,

H. Soltauet al., “DSTC-11: Speech Aware Task-Oriented Dialog Modeling Track,” inProceedings of The Eleventh Dialog System Technology Challenge. Prague, Czech Republic: Association for Computational Linguistics, Sep. 2023, pp. 226–234

2023
[10]

SpokenWOZ: a large-scale speech-text benchmark for spoken task-oriented dialogue agents,

S. Siet al., “SpokenWOZ: a large-scale speech-text benchmark for spoken task-oriented dialogue agents,” inProceedings of the 37th International Conference on Neural Information Processing Systems, ser. NIPS ’23. Red Hook, NY , USA: Curran Associates Inc., 2023

2023
[11]

Speech-Aware Multi-Domain Dialogue State Generation with ASR Error Correction Modules,

R. Jianget al., “Speech-Aware Multi-Domain Dialogue State Generation with ASR Error Correction Modules,” inProceedings of The Eleventh Dialog System Technology Challenge. Prague, Czech Republic: Association for Computational Linguistics, Sep. 2023, pp. 105–112

2023
[12]

Retrieval Augmented End-to-End Spoken Dialog Models,

M. Wanget al., “Retrieval Augmented End-to-End Spoken Dialog Models,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 12 056–12 060

2024
[13]

MAESTRO: Matched Speech Text Represen- tations through Modality Matching,

Z. Chenet al., “MAESTRO: Matched Speech Text Represen- tations through Modality Matching,” inInterspeech, 2022, pp. 4093–4097

2022
[14]

Connecting Speech Encoder and Large Language Model for ASR,

W. Yuet al., “Connecting Speech Encoder and Large Language Model for ASR,” inIEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), 2024, pp. 12 637– 12 641

2024
[15]

Aligning Pre-trained Models for Spoken Lan- guage Translation,

S. Sedl ´aˇceket al., “Aligning Pre-trained Models for Spoken Lan- guage Translation,” 2024, arXiv:2411.18294

2024 arXiv
[16]

TTS4pretrain 2.0: Advancing the use of Text and Speech in ASR Pretraining with Consistency and Contrastive Losses,

Z. Chenet al., “TTS4pretrain 2.0: Advancing the use of Text and Speech in ASR Pretraining with Consistency and Contrastive Losses,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 7677–7681

2022
[17]

Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognition,

S. Linget al., “Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognition,” inIEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 11 046–11 050

2024
[18]

SLM: Bridge the Thin Gap Between Speech and Text Foundation Models,

M. Wanget al., “SLM: Bridge the Thin Gap Between Speech and Text Foundation Models,” inIEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023, pp. 1–8

2023
[19]

SALMONN: Towards Generic Hearing Abilities for Large Language Models,

C. Tanget al., “SALMONN: Towards Generic Hearing Abilities for Large Language Models,” inThe Twelfth International Con- ference on Learning Representations, 2024

2024
[20]

Integrating Speech Self-Supervised Learning Models and Large Language Models for ASR,

L. Donget al., “Integrating Speech Self-Supervised Learning Models and Large Language Models for ASR,” inInterspeech, 2024, pp. 3954–3958

2024
[21]

An Embarrassingly Simple Approach for LLM with Strong ASR Capacity,

Z. Maet al., “An Embarrassingly Simple Approach for LLM with Strong ASR Capacity,” 2024, arXiv:2402.08846

2024 arXiv
[22]

Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward,

S. Kumaret al., “Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward,” 2025, arXiv:2411.03866

2025 arXiv
[23]

WavLM: Large-Scale Self-Supervised Pre- Training for Full Stack Speech Processing,

S. Chenet al., “WavLM: Large-Scale Self-Supervised Pre- Training for Full Stack Speech Processing,”IEEE Journal of Se- lected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

2022
[24]

OLMo: Accelerating the Science of Lan- guage Models,

D. Groeneveldet al., “OLMo: Accelerating the Science of Lan- guage Models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 15 789–15 809

2024
[25]

LoRA: Low-Rank Adaptation of Large Language Models,

E. J. Huet al., “LoRA: Low-Rank Adaptation of Large Language Models,” inInternational Conference on Learning Representa- tions, 2022

2022
[26]

MultiWOZ - A Large-Scale Multi- Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Mod- elling,

P. Budzianowskiet al., “MultiWOZ - A Large-Scale Multi- Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Mod- elling,” inProceedings of the Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: Association for Computational Linguistics, Oct.-...

2018
[28]

Robust Speech Recognition via Large-Scale Weak Supervision,

A. Radfordet al., “Robust Speech Recognition via Large-Scale Weak Supervision,” inProceedings of the 40th International Con- ference on Machine Learning. PMLR, Jul. 2023

2023
[29]

Shades of BLEU, Flavours of Suc- cess: The Case of MultiWOZ,

T. Nekvinda and O. Du ˇsek, “Shades of BLEU, Flavours of Suc- cess: The Case of MultiWOZ,” inProceedings of the 1st Work- shop on Natural Language Generation, Evaluation, and Metrics (GEM 2021). Online: Association for Computational Linguis- tics, Aug. 2021, pp. 34–46

2021
[30]

SWITCHBOARD: telephone speech corpus for research and development,

J. Godfreyet al., “SWITCHBOARD: telephone speech corpus for research and development,” inProceedings IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 1, 1992

1992
[31]

Librispeech: An ASR corpus based on pub- lic domain audio books,

V . Panayotovet al., “Librispeech: An ASR corpus based on pub- lic domain audio books,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Apr. 2015

2015
[32]

How2: A Large-scale Dataset For Multimodal Language Understanding,

R. Sanabriaet al., “How2: A Large-scale Dataset For Multimodal Language Understanding,” inProceedings of the Workshop on Vi- sually Grounded Interaction and Language (ViGIL). NeurIPS, 2018

2018
[33]

T. M. Gemma Teamet al., “Gemma,” 2024. [Online]. Available: https://www.kaggle.com/m/3301

2024
[34]

Gemma 2: Improving open language models at a practical size,

G. Teamet al., “Gemma 2: Improving open language models at a practical size,” 2024. [Online]. Available: https: //arxiv.org/abs/2408.00118

2024 arXiv
[35]

Schema Augmentation for Zero- Shot Domain Adaptation in Dialogue State Tracking,

C. Richardsonet al., “Schema Augmentation for Zero- Shot Domain Adaptation in Dialogue State Tracking,” 2024, arXiv:2411.00150

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [27]

MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Base- lines,

M. Ericet al., “MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Base- lines,” inProceedings of the Twelfth Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association, May 2020, pp. 422–428

work page 2020

[2] [2]

transcription

Method overview Our method treats DST as the problem of mapping from user speech and user-agent dialogue history to a dictionary contain- ing the correct transcription of the user’s speech, and the in- ferred dialogue states (domains, slots). Therefore, we adopt a model which we train to take speech as input and directly output a JSON string representing ...

arXiv 2025

[3] [1]

book-restaurant

Introduction Task-oriented dialogues (ToD) are multiturn conversations be- tween a user and an agent, where the former has a specific goal (e.g., booking a restaurant for 5 people on Friday night) that is achieved with the help of the agent. A key component of auto- mated ToD systems is dialogue state tracking (DST), the task of tracking the user’s intent...

work page

[4] [3]

Experiments 3.1. Datasets We conduct our experiments using two speech-grounded task- oriented dialogue datasets: SpokenWoZ [2] (SWOZ) and Speech-Aware MultiWoZ [1] (MWOZ), built on top of the original text-based MultiWoZ 2.1 [18, 19]. We remove nine originally corrupted conversations from the SpokenWoZ test set3. For SpokenWOZ, we also generate ASR transc...

work page 2000

[5] [4]

dialogue history

from prior works. Model SWOZ test JGA SER (1) Gemma-2-9B-Instruct (cascaded) [27] 25.40 - (2) SPACE+WavLM align [2] 25.65 - Whisper→OLMo-1B, full FT, SW+MW (cascade) 30.74 31.11 WavLM + conn. + OLMo-1B (A11 from Table 3) 34.66 26.80 WavLM + conn. + Gemma-2-9B-Instruct (Table 4)42.17 20.41 We also compare our results to the best system from [2] and the rel...

work page

[6] [5]

Results 4.1. Cascaded system analysis Our experiments focus on ablating three main training setup fac- tors: the training data used, the type of LLM fine-tuning, and the usage of both user and agent turns in the dialogue history. a full overview of our cascaded systems is provided in Table

work page

[7] [6]

First, we observe that fine-tuning on the original Spoken- WoZ transcripts yields worse results than when using Whis- per transcripts, which are of higher quality

We additionally include two text-only baseline systems from DSTC-11 [1] as reference baselines for speech-aware MWOZ. First, we observe that fine-tuning on the original Spoken- WoZ transcripts yields worse results than when using Whis- per transcripts, which are of higher quality. Second, we find it beneficial to combine both SpokenWoZ and SA-MultiWOZ for...

work page

[8] [7]

Conclusions In this work, we propose an end-to-end dialogue state tracking system based on bridging the representation spaces of a pre- trained speech encoder with an LLM via a small transformer connector with a two-step ASR-DST fine-tuning scheme. We use open source models for both the encoder (WavLM-large) and the LLM (OLMo-1B) to mitigate the risk of t...

work page