{"id":"f81ae9db-3412-4a79-b0d3-c3a2565afde9","arxiv_id":"2506.08633","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An open-source WavLM-plus-connector-plus-LLM pipeline achieves state-of-the-art spoken dialogue state tracking on SpokenWOZ test (34.66% JGA with OLMo-1B, 42.17% with Gemma-2-9B), with detailed ablations.","lead":"This paper reports an end-to-end spoken dialogue state tracking system that links a WavLM speech encoder to a large language model (OLMo-1B or Gemma-2-9B) via a small connector, reaching 34.66% joint goal accuracy with OLMo-1B and 42.17% with Gemma-2-9B on the SpokenWOZ test set. The results matter because they show that fully open-source models can beat closed cascade baselines on this benchmark, and the paper systematically ablates the key design choices.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim rests on a modified SpokenWOZ test set: authors remove nine conversations (Section 3.1) but compare against published baseline numbers that were likely computed on the untouched official test set.","rationale":"The reader's weakest_assumption correctly identifies the modified test set as the most load-bearing fragility in the paper's central claim. I agree that the comparison against 25.40% and 25.65% JGA is only valid if all systems are evaluated on the same set of test dialogues, and the paper does not establish this. The concern is concrete and falsifiable: JGA is computed turn-by-turn, so dropping nine conversations changes the denominator and the set of possibly correct states; no error bars or per-dialogue breakdown are provided to show the removal is immaterial. The paper does have genuine strengths: a transparent two-stage ASR-then-DST training recipe, ablations of agent turns and fuzzy matching, and a fully open OLMo-1B variant that beats the cited baselines even without fuzzy post-processing (A10 at 31.91% vs. 25.40% and 25.65%). Those strengths make the contribution plausible but do not remove the need for a fair SOTA comparison. I also note that the fuzzy post-processing step is applied to the authors' systems but not shown to be applied to the baselines, which further complicates the 34.66% and 42.17% numbers, although the A10 result shows the core method is not solely dependent on fuzzy matching. Since the reader already issued a CONDITIONAL verdict with high confidence, my stress-test does not change that verdict; it reinforces it by pinpointing the specific missing evidence: an identical-set evaluation with all systems treated alike.","tokens_in":8144,"tokens_out":3471,"duration_ms":42532,"concrete_test":"Release or compute results on the official untouched SpokenWOZ test set: identify the exact IDs of the nine removed conversations, then run the A11 OLMo-1B and Gemma-2-9B checkpoints on both the full official test set and the reduced nine-removed set, and re-run the two cited baseline systems on the same full official test set, using the same MultiWOZ evaluation script and applying the same fuzzy post-processing to all systems. If the OLMo result does not exceed both baselines on the identical unmodified test set, or if the Gemma result drops by more than about 1 point JGA, the SOTA claim should be revised to a claim about a cleaned subset until an identical-set comparison is provided.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of state-of-the-art JGA is load-bearing on an apples-to-apples comparison with prior SpokenWOZ results. In Section 3.1 the authors state that they remove nine 'corrupted' conversations from the SpokenWOZ test set, citing a GitHub issue in footnote 3, and then Table 1 reports 34.66% JGA for the OLMo-1B system and 42.17% JGA for the Gemma-2-9B system on this modified set. The two cited prior systems, [27] at 25.40% and [2] at 25.65%, are published numbers for the official test set; the paper does not show that these baselines were re-evaluated on the reduced set, nor does it quantify how many of the removed nine conversations are predicted correctly by either side. Because JGA is per-turn and nine dialogues contain many turns, removing them can shift the metric by several points in a dataset of this size, so the stated SOTA margin could be partly an artifact of the data alteration. The paper's own A10 result (31.91% JGA without fuzzy post-processing) still clears the cited baselines, so the concern is most acute for the headline 42.17% number and for the claim that 34.66% after fuzzy matching is SOTA on the same test set used by prior work. The conditions of this comparison are not acknowledged in the conclusions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes an end-to-end spoken dialogue state tracking (DST) system that connects a WavLM-large speech encoder to an LLM through a small transformer connector. The training pipeline first pre-trains the connector and encoder for ASR, then fine-tunes the connector and LoRA adapters on a joint ASR-DST objective. Experiments on SpokenWOZ and Speech-Aware MultiWOZ ablate dataset combinations, user/agent dialogue history, LoRA versus full fine-tuning, and fuzzy matching post-processing. The central claims are state-of-the-art results on the SpokenWOZ test set: 34.66% joint goal accuracy (JGA) with the fully open OLMo-1B model and 42.17% JGA with Gemma-2-9B-Instruct, compared to 25.40% and 25.65% for two cited baselines.","tokens_in":8352,"tokens_out":10874,"duration_ms":126233,"significance":"If the results hold, this is a useful empirical contribution: a relatively simple end-to-end alignment of a public speech encoder and an LLM outperforms prior cascaded and aligned systems on SpokenWOZ, and the ablations clarify which design choices matter (ASR pre-training, LoRA, user+agent history, fuzzy matching). The use of openly documented models (WavLM, OLMo), public datasets, and the standard MultiWOZ evaluation script are strengths, as is the transparent reporting of training configurations. The main caveat is that the SOTA comparison rests on a modified test set, so the headline numbers need to be re-anchored to a common evaluation set before the claim is credible.","major_comments":[{"comment":"The headline state-of-the-art claim is based on a modified SpokenWOZ test set. Section 3.1 states that nine \"corrupted\" conversations are removed from the SpokenWOZ test set (footnote 3), and Table 1 reports 34.66% and 42.17% JGA on this reduced set. The two prior baselines in Table 1 (25.40% from [27] and 25.65% from [2]) are published numbers for the official test set; the paper does not re-evaluate these baselines on the reduced set, does not report results on the untouched official test set, and does not quantify how many turns the removed dialogues contain. Since JGA is a turn-level metric, the reported SOTA margin could be partly an artifact of the evaluation-set change. Please either re-run the baselines on the reduced set, report results on the full official test set, or explicitly restrict the SOTA claim to the reduced set and justify that restriction.","section":"Section 3.1, Table 1, Section 5"},{"comment":"The paper reports a single run for every configuration and gives no error bars or multiple seeds. This makes it hard to judge the reliability of the main comparisons, e.g., the fuzzy-matching gain (A10 vs A11: 31.91 vs 34.66) and the effect of user+agent history (A8 vs A9: 27.27 vs 31.04). In addition, the fuzzy matching post-processing is applied to the proposed systems but not to the cited baselines; if the baselines would also benefit from this post-processing, the comparison in Table 1 is not controlled. Please provide variance estimates (or at least multiple runs) for the key numbers and, where feasible, report the baselines with the same post-processing.","section":"Tables 3 and 4; Section 4.2"}],"minor_comments":[{"comment":"Section 3.2 says the optional final fine-tuning uses batch size 256, but Section 4.2 describes the A11 run with batch size 192; please reconcile the two numbers.","section":"Section 3.2 vs 4.2"},{"comment":"Section 4.3 states that Gemma-2-9B-Instruct is trained with \"the same ... training hyperparameters as for our other models,\" yet the LoRA rank is r=8 rather than the r=16 used in Section 3.2; please clarify.","section":"Section 4.3"},{"comment":"Section 5 contains a typo: \"fuzy matching\" should be \"fuzzy matching.\"","section":"Section 5"},{"comment":"Section 4.2 contains \"apart from theprofiledomain\" with a missing space; it should read \"apart from the profile domain.\"","section":"Section 4.2"},{"comment":"The abstract and Section 3.1 emphasize fully open components, but Gemma-2-9B-Instruct is an open-weights model, not an open-data model; please distinguish this when describing the 42.17% result.","section":"Abstract and Section 3.1"},{"comment":"The fuzzy matching post-processing is described only by a package name; for reproducibility please specify the matching score threshold and how candidate database values are selected.","section":"Section 2.2"},{"comment":"Table 2 includes the DSTC-11 reference systems (25.2 and 43.1 JGA on MWOZ dev) but the text does not discuss why the proposed systems are not compared against them; a sentence of context would help.","section":"Table 2"},{"comment":"The paper does not state whether code or model checkpoints will be released, despite the emphasis on openness; please add an availability statement.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution for a speech-language processing venue. The main obstacle is the SOTA comparison on a modified test set; this is fixable by re-running baselines or reporting full-set numbers. If the authors can also address the single-run/no-error-bars issue, I would be comfortable with acceptance after major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a clean, honest empirical study of a fully open speech-encoder-to-LLM pipeline for spoken dialogue state tracking. The two-stage ASR-then-DST training is a sensible recipe, the ablations (agent turns, LoRA rank, ASR initialization, fuzzy post-processing, training data augmentation) are exactly what practitioners need, and the choice of WavLM/OLMo to address contamination concerns is thoughtful. The best open pipeline hits 34.66% JGA on SpokenWOZ, and the Gemma-2-9B variant reaches 42.17%. These are real measurements on public data with a standard metric, and the internal consistency across tables is good. I believe the core contribution — that a fully open aligned system can beat a Whisper-to-OLMo cascade — is solid and probably true.\n\nThe soft spot is the one the abstract leans on: state-of-the-art. The authors remove nine 'corrupted' test conversations (Section 3.1, footnote 3) but compare against published numbers (25.40, 25.65) that were computed on the untouched official test set. They never show the baselines re-evaluated on the reduced set, and nine conversations can move JGA by several points on a dataset this size. The A10 number (31.91% without fuzzy matching) still clears the cited baselines, so the method is likely ahead regardless, but the headline 42.17% and the phrase 'state of the art' overstate what is actually demonstrated. This is not a fatal flaw, and it is not hidden — the removal is disclosed — but the conclusion should not ignore it.\n\nMinor issues: no error bars or multiple seeds (the ablations are largely single-run), no code release, and the Gemma-2-9B comparison is peripheral to the paper's open-model thesis. The fuzzy matching post-processing is standard in the DST community, so that is not a problem per se, though the 3% absolute gain should be understood as part of the system rather than as model capability.\n\nWho is this for? Someone building open spoken DST systems, or anyone wanting a clear picture of how far a WavLM+connector+small-LLM pipeline can go. It deserves a serious referee. The right outcome is probably acceptance after the authors add a re-evaluated-baseline comparison on the reduced test set, or at minimum a sensitivity analysis showing the effect of removing the nine conversations. I would take it to the reading group as a useful example of how to structure spoken-DST ablations.","headline":"Solid empirical study with a genuinely useful ablation suite, but the headline SOTA claim rests on a modified test set and baselines that were not re-evaluated on it — worth refereeing, but the comparison needs fixing.","tokens_in":9031,"tokens_out":892,"would_cite":true,"duration_ms":12733,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Aligned speech encoder and open LLM reach 42.17% accuracy on SpokenWOZ","keywords":["dialogue state tracking","spoken dialogue understanding","speech encoder LLM alignment","connector module","LoRA fine-tuning","SpokenWOZ","joint goal accuracy","open-source speech LLM"],"falsifier":"Take the published SpokenWOZ test set before any filtering, run the two reference systems (the Gemma-2-9B cascade from [27] and the SPACE+WavLM system from [2]) on exactly the nine-conversation-filtered split used in this paper, and compare their JGA to 34.66% and 42.17%; if the baselines rise to the same level once the same deletions are applied, the claimed SOTA is not due to the alignment method.","tokens_in":7893,"feed_emoji":"🎙️","tokens_out":6625,"duration_ms":65064,"temperature":0.7,"pith_summary":"This paper tries to show that spoken dialogue state tracking — keeping an updated record of what a user wants across multi-turn spoken conversation — can be done end to end by aligning a pre-trained speech encoder with a large language model through a small trainable connector. The authors build the entire pipeline from openly released components, WavLM as the speech encoder and OLMo-1B as the language model, and report 34.66% joint goal accuracy on the SpokenWOZ test set, which they say surpasses published cascaded baselines. They report a further jump to 42.17% JGA when the connector is attached to the larger Gemma-2-9B-Instruct model. The result matters because it suggests that a fully open, end-to-end trained speech-to-DST system can beat cascade pipelines that transcribe first and then track state from text, without relying on proprietary training data.","feed_headline":"Open speech-LLM pipeline reaches 42.17% on SpokenWOZ","feed_subtitle":"A fully open WavLM + connector + OLMo system beats prior cascades; swapping in Gemma-2-9B adds another 7 points.","key_machinery":"The load-bearing mechanism is a small trainable connector module — a two-layer transformer with a subsampling front end that stacks six neighbouring WavLM embeddings and projects them into the LLM's text embedding space, where they act as soft prompts. The connector is trained in two steps: a frozen-LLM ASR stage that aligns speech encoder and language model on thousands of hours of transcribed speech, and a joint ASR-DST stage in which the encoder stays frozen while low-rank adapters (LoRA) in the LLM are fine-tuned with the connector to autoregressively generate a JSON string containing the user transcription, domains, and slot values. This two-step scheme, not any single component, is what the paper credits for making end-to-end spoken DST work with fully open models.","core_discovery":"The central claim is that bridging WavLM-large and an LLM with a two-layer transformer connector, trained first for ASR and then for joint ASR-plus-DST with low-rank adapters, yields a spoken dialogue state tracker that exceeds prior published results on SpokenWOZ. The best fully open configuration (WavLM + connector + OLMo-1B) reaches 34.66% JGA after fuzzy-matching post-processing, and replacing the LM with Gemma-2-9B-Instruct raises this to 42.17%. In the paper's own account, the two-stage training is essential: ASR pre-training aligns the modalities on large speech corpora, and joint fine-tuning preserves the model's ability to transcribe the user side at inference time while still emitting the JSON dialogue state. The authors also find that including agent turns in the dialogue history and post-processing slot values with fuzzy matching both give consistent gains.","pith_inferences":["Editorial extension: the magnitude of the reported gain over prior cascades suggests that connector-based alignment, not just the LLM, is doing the work; a direct test would be to keep the same two-stage training but replace the connector with a fixed-rate subsampler.","Editorial extension: because the nine removed SpokenWOZ conversations are not re-scored for the prior baselines, a clean re-evaluation of those baselines on the filtered test set is needed before the SOTA claim can be taken at face value; this is a testable step the paper does not perform.","Editorial extension: the same connector-plus-two-stage recipe could transfer to other spoken language understanding tasks such as slot filling or spoken question answering, where the JSON output schema would simply be replaced.","Editorial extension: the reported sensitivity to the ASR context (Whisper context raises JGA from 31.91 to 32.89) implies that further gains may come from a speech encoder better matched to the human-read MultiWOZ domain, rather than from scaling the LLM alone."],"forward_implications":["An end-to-end aligned speech-LLM system is a viable alternative to ASR-then-DST cascades for spoken task-oriented dialogue, at least on SpokenWOZ.","Fully open models (WavLM-large and OLMo-1B) are sufficient to set a new published JGA mark, which the authors credit to reduced test-contamination risk and the two-stage alignment training.","Fuzzy matching of predicted slot values against the database adds roughly three points of JGA absolute, indicating that much of the remaining error is in exact spelling of named entities rather than in slot selection.","Larger instruction-tuned LLMs (Gemma-2-9B) amplify the same connector-based alignment, improving JGA from 34.66% to 42.17% after the same post-processing.","Training on the TTS-based Speech-Aware MultiWOZ data alongside SpokenWOZ helps the DST head generalize, but the model still transfers poorly to human-verbatim MultiWOZ test data."],"supporting_citations":[{"why":"Supplies the WavLM-large speech encoder whose representations are aligned with the LLM.","marker":"[15]"},{"why":"Supplies the fully open OLMo-1B language model used in the main aligned system.","marker":"[16]"},{"why":"Defines the SpokenWOZ dataset and the SPACE+WavLM baseline the paper aims to beat.","marker":"[2]"},{"why":"Defines the DSTC-11 Speech-Aware MultiWOZ dataset used for augmentation and provides reference baselines.","marker":"[1]"},{"why":"Provides the LoRA low-rank adaptation method used to fine-tune the LLM in the joint ASR-DST stage.","marker":"[17]"},{"why":"Supplies Whisper-large-v3 transcripts used for cascaded baselines and as higher-quality reference text for training.","marker":"[20]"},{"why":"Provides the MultiWOZ evaluation script used to compute JGA and slot error rate.","marker":"[21]"},{"why":"Provides the Gemma-2-9B-Instruct cascaded baseline at 25.40% JGA that the aligned systems surpass.","marker":"[27]"},{"why":"Motivates the fuzzy-matching post-processing scheme applied to predicted slot values.","marker":"[3]"}],"fun_headline_variants":["Open speech-LLM pipeline tops SpokenWOZ at 42.17% JGA","Fuzzy matching lifts open DST to 42.17% on SpokenWOZ","WavLM + OLMo + connector: SOTA 42.17% on SpokenWOZ","Two-stage training yields SOTA 42.17% on SpokenWOZ","Gemma-2-9B boosts speech DST to 42.17% JGA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison to prior work assumes the SpokenWOZ test set the authors evaluate on is the same as the one used to compute the published 25.40% and 25.65% results; the authors remove nine 'corrupted' conversations from the test set, and if those conversations were valid or were included in the prior scores, the reported state-of-the-art gap is partly an artifact of the altered evaluation set.","fun_headline_variants_meta":{"raw":{"variants":["Open speech-LLM pipeline tops SpokenWOZ at 42.17% JGA","Fuzzy matching lifts open DST to 42.17% on SpokenWOZ","WavLM + OLMo + connector: SOTA 42.17% on SpokenWOZ","Two-stage training yields SOTA 42.17% on SpokenWOZ","Gemma-2-9B boosts speech DST to 42.17% JGA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001275,"raw_usage":{"total_tokens":5206,"prompt_tokens":927,"completion_tokens":4279,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":4160}},"tokens_in":543,"tokens_out":4279,"duration_ms":31794,"temperature":1.0,"reasoning_tokens":4160,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:05:52.602944+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the published SpokenWOZ test set before any filtering, run the two reference systems (the Gemma-2-9B cascade from [27] and the SPACE+WavLM system from [2]) on exactly the nine-conversation-filtered split used in this paper, and compare their JGA to 34.66% and 42.17%; if the baselines rise to the same level once the same deletions are applied, the claimed SOTA is not due to the alignment method.","supporting_citations":[{"cited_title":"Aligning Pre-trained Models for Spoken Language Translation","cited_arxiv_id":"2411.18294","evidence_quote":"Supplies the WavLM-large speech encoder whose representations are aligned with the LLM."},{"cited_title":"TTS4pretrain 2.0: Advancing the use of Text and Speech in ASR Pretraining with Consistency and Contrastive Losses,","cited_arxiv_id":null,"evidence_quote":"Supplies the fully open OLMo-1B language model used in the main aligned system."},{"cited_title":"book-restaurant","cited_arxiv_id":null,"evidence_quote":"Defines the DSTC-11 Speech-Aware MultiWOZ dataset used for augmentation and provides reference baselines."},{"cited_title":"Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the LoRA low-rank adaptation method used to fine-tune the LLM in the joint ASR-DST stage."},{"cited_title":"Integrating Speech Self-Supervised Learning Models and Large Language Models for ASR,","cited_arxiv_id":null,"evidence_quote":"Supplies Whisper-large-v3 transcripts used for cascaded baselines and as higher-quality reference text for training."},{"cited_title":"MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Base- lines,","cited_arxiv_id":null,"evidence_quote":"Provides the Gemma-2-9B-Instruct cascaded baseline at 25.40% JGA that the aligned systems surpass."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the fuzzy-matching post-processing scheme applied to predicted slot values."}],"review_version":1}