{"id":"2a546726-6ae1-47f0-b35f-98e4ede896f0","arxiv_id":"2607.17230","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuned Voxtral speech models with DAPO reinforcement learning achieved the top Open Medical Concept F1 in both BeTraC tracks while posting the lowest LLM-judged hallucination rates.","lead":"TalTech's systems for the Beyond Transcription Challenge ranked first in both tracks at turning long doctor-patient audio directly into SOAP notes, using fine-tuning and reinforcement learning. The paper suggests that training against a medical-concept metric can improve notes without increasing hallucinations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated LLM-as-judge: the hallucination-rate evidence for RL safety has no human agreement check","rationale":"The paper's official winning results are externally credible and not the focus of concern. The robustness claims on the realistic subset are explicitly anecdotal, and the no-generalization-gap observation is expected given that validation and test come from the same synthetic pipeline. The load-bearing claim is the safety conclusion: that RL against a concept-matching metric does not compromise factual reliability. This claim is supported only by the LLM-as-judge hallucination rates in Table V. The judge is a Gemma 4 31B model that verifies atomic claims against gold transcripts; no human validation, error analysis, or confidence intervals are provided. Without an independent check that this judge reliably identifies hallucinations, the strikingly low hallucination rates—0.08% for the heavyweight system—could reflect judge leniency rather than system behavior. The reported correlations between Concept F1 and judge scores do not resolve this, as both are LLM-derived and may share systematic biases. I also considered whether the missing SFT-vs-RL hallucination comparison is the primary issue; however, the abstract's phrasing 'need not compromise' is an existential claim, so the final low rate is sufficient if the judge is trustworthy. The stronger statement in the conclusion, 'without inflating ... hallucinations,' would require a direct SFT baseline, but this is secondary. The reader's weakest assumption correctly identifies the judge reliability as the decisive uncertainty. A human-annotation study on a sample of atomic claims would settle whether the judge's hallucination flags align with clinical judgment. Given the otherwise strong official results and the paper's transparency, the appropriate verdict remains CONDITIONAL, pending this validation.","tokens_in":6506,"tokens_out":4684,"duration_ms":46674,"concrete_test":"Have two clinician annotators independently judge a random sample of 200 atomic claims extracted from the heavyweight system's EE outputs, marking each as supported, contradicted, or unverifiable against the gold transcript. Compute judge–human agreement on hallucination flags (e.g., Cohen's kappa, precision/recall of the Gemma judge). If κ < 0.6 or judge recall of human-flagged hallucinations is below 80%, the Table V hallucination rates cannot support the safety claim. As a secondary check, run the same judge on the SFT-only lightweight model's EE outputs and compare hallucination rates to the SFT+RL system.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RL against Concept F1 'need not compromise factual reliability' rests on the LLM-as-judge numbers in Table V, specifically the 0.08% hallucination rate for the heavyweight system. Section V-C describes a Gemma 4 31B judge decomposing each note into atomic claims and verifying them against the gold transcript, but the paper provides no human agreement study, no error analysis, and no confidence intervals for this judge. If the judge is systematically lenient—for example, if it fails to flag plausible-sounding but ungrounded statements, or accepts paraphrases that alter clinical meaning—then the reported rates are not evidence of factual reliability, and the central safety conclusion is unsupported. The correlation with Concept F1 (ρ=0.91 and -0.93) does not mitigate this, since both signals derive from LLMs and could share the same blind spots. In addition, Section VI states that RL added C-F1 'without inflating note length or hallucinations,' but Table V only reports final systems; no SFT-only hallucination rates are shown, so even a reliable judge would not directly support the 'without inflating' claim—only the weaker 'need not compromise' claim. The judge-reliability gap is thus the single most load-bearing concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes TalTech's submissions to the Beyond Transcription Challenge (BeTraC), where systems must generate SOAP notes directly from long doctor-patient audio without an intermediate transcription component. The authors screen open-weight speech LLMs for long-audio robustness, then adapt Voxtral Mini (lightweight track) and Voxtral Small (heavyweight track) using LoRA supervised fine-tuning followed by DAPO reinforcement learning whose reward is the challenge's Open Medical Concept F1 (C-F1). Their primary systems ranked first in both tracks according to the official blind evaluation, and an LLM-as-judge evaluation reports low hallucination rates for these systems. The paper further reports that transcript-based fine-tuning transfers to audio input and that the systems remain competitive on acted and, to a limited extent, realistic test subsets.","tokens_in":6796,"tokens_out":4616,"duration_ms":47189,"significance":"If the results hold, the paper provides a practical recipe for direct audio-to-SOAP generation with open-weight models, and it offers evidence that optimizing a concept-matching metric with reinforcement learning need not produce factually unreliable notes. The main strengths are the objective official leaderboard results (first in both tracks), the strong validation-to-test consistency (e.g., 0.571 vs. 0.563 C-F1 for the heavyweight system), and the cross-modal transfer finding. The central safety claim, however, rests on an LLM-as-judge evaluation that is not validated against human judgments, so the factual-reliability conclusion is currently weaker than the abstract suggests. The official ranking itself is not circular, since it is an external blind evaluation against gold references, but the absence of statistical uncertainty and the tiny realistic subset limit some of the secondary claims.","major_comments":[{"comment":"The safety-relevant claim—that RL against Concept F1 'need not compromise factual reliability'—is supported only by the Gemma 4 31B judge's hallucination and contradiction rates. No human agreement study, no judge error analysis, no confidence intervals, and no prompt/decoding details are provided for this judge. If the judge is systematically lenient toward plausible-sounding but ungrounded statements, the reported 0.08% and 0.25% rates are not evidence of factual reliability. The system-level Spearman correlations (ρ=0.91, -0.93) do not mitigate this, because both signals come from LLMs and may share blind spots. This is the single most load-bearing gap and needs to be addressed directly.","section":"§V-C, Table V"},{"comment":"The conclusion states that RL added 0.04–0.05 C-F1 'without inflating note length or hallucinations.' Table V reports only the final RL-trained systems; no SFT-only hallucination/contradiction rates or note lengths are shown. Table III reports validation C-F1/R-2/R-3 but not hallucination rates or lengths. Therefore the data support at most the weaker claim that the final RL-trained systems have low judged hallucination rates, not that RL did not increase hallucination relative to SFT. Please provide SFT-only judge scores and lengths, or phrase the conclusion more narrowly.","section":"§VI, Table V"},{"comment":"The robustness-on-real-recordings claim in the abstract and §V-B relies on the Real. column of Table IV, which contains only three dialogues. The text acknowledges this is 'anecdotal,' but the abstract still asserts that the recipe 'appears to improve robustness on out-of-domain real recordings.' With n=3 and no error bars or case-level analysis, that claim is unsupported beyond anecdote. The mock subset (272 dialogues) provides somewhat stronger but still acted evidence. Please either remove the realistic-subset claim from the abstract or add appropriate uncertainty quantification and explicit hedging.","section":"§V-B, Table IV"}],"minor_comments":[{"comment":"The LoRA target modules are listed as 'all linear'; please specify the exact modules (e.g., attention and MLP projections) for reproducibility.","section":"§IV-A, Table IIa"},{"comment":"The ASR/WER evaluation protocol is not described: which subset of the validation split, what normalization, and what alignment/tool were used? This would help interpret the WER values used for model screening.","section":"Table I"},{"comment":"The KL β is listed as 0. Please clarify whether DAPO was run with no KL penalty and whether any divergence or reward-hacking behavior was observed during RL training.","section":"§IV-B, Table IIb"},{"comment":"For the facts-and-SOAP contrastive model, it is unclear whether the fact table is emitted at inference time as part of the final output or suppressed. If it is emitted, how is it reconciled with the requirement to produce plain-text SOAP notes? This affects interpretation of the 306-word mean length in Table IV.","section":"§IV-C"},{"comment":"There are typographical and formatting issues, including broken 'V oxtral' spacing and 'aboverecall' in §V-C. The paper would benefit from a careful proofreading pass.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The official leaderboard result is solid and the validation/test consistency is a genuine strength. The main risk is the unvalidated LLM-as-judge underlying the hallucination claim; this is fixable within the scope of a revision by adding a human agreement study or at least a small human-annotated sample, and by reporting SFT-only judge scores. The realistic-subset claim should be softened. If the authors provide these, the paper would be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read if you work on medical summarization or speech LLMs. The paper does three things well. It shows a concrete recipe — zero-shot screening for long-audio robustness, LoRA SFT, then DAPO RL with the official metric as reward — that took Voxtral models to first place in both BeTraC tracks, more than doubling baseline Concept F1. It reports a clean validation-to-test transfer and a useful observation: fine-tuning on transcripts transfers to audio almost as well as audio fine-tuning, and may help under domain shift. The authors are transparent about using the validation split for model and reward selection and about the realistic subset being only three dialogues.\n\nThe soft spots are real but not disqualifying. The biggest is the hallucination-rate evidence. The low rates come from the organizers' LLM-as-judge pipeline, not from the authors' own evaluation, and there is no human agreement study, no error analysis, and no confidence intervals. The authors themselves note that the metric and the judge both derive from LLMs, which is honest but doesn't fix the concern. The conclusion that RL added C-F1 'without inflating ... hallucinations' is not directly supported: Table V shows only final systems, and no SFT-only hallucination rates are reported, so that comparison is missing. You can accept the weaker claim that RL against Concept F1 need not produce obviously high hallucination rates; you cannot yet accept the stronger causal claim.\n\nAlso minor: all comparisons are point estimates without repeated runs or significance tests. The 'no generalization gap' claim is consistent with test and validation coming from the same synthetic pipeline; the mock and realistic subsets are the real test, and the realistic subset is anecdotal.\n\nOn balance, this is a solid systems paper for a challenge venue. The central ranking claims are externally evaluated and credible. The safety conclusion needs more support, but that's a fixable weakness, not a fatal one. I'd send it to review with a request for human validation or error analysis of the judge, SFT-only hallucination numbers, and ideally code or checkpoints.","headline":"Winning BeTraC systems with a practical RL recipe, but the core safety claim leans on an external LLM judge nobody has validated.","tokens_in":7273,"tokens_out":2319,"would_cite":true,"duration_ms":22350,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement learning against a medical-concept metric can improve direct audio-to-SOAP summarization without increasing hallucination, the top result in both tracks of the challenge.","keywords":["direct speech summarization","SOAP note generation","Open Medical Concept F1","reinforcement learning","hallucination evaluation","doctor-patient conversation","speech LLM fine-tuning","domain robustness"],"falsifier":"Take the systems' generated notes, inject known hallucinated symptoms or diagnoses into a sample, and have independent clinician annotators mark factual errors; if the human error rate is materially higher than the LLM judge's 0.08–0.25% hallucination rates, the conclusion that the RL recipe preserves factual reliability fails.","tokens_in":6388,"feed_emoji":"🩺","tokens_out":6448,"duration_ms":57264,"temperature":0.7,"pith_summary":"The paper tries to establish that a speech LLM can be trained to produce clinical SOAP notes directly from long doctor-patient audio, skipping any transcription step, and that the training recipe does not trade factual safety for metric score. The authors fine-tune open-weight speech models with supervised learning on reference notes and then reinforcement learning whose reward is the challenge's Open Medical Concept F1. Their systems placed first in both lightweight and heavyweight tracks, and an independent judge-based evaluation found the lowest hallucination rates in both tracks. The conclusion is that optimizing a concept-matching metric through reinforcement learning need not cause the expected degeneracy of listing many plausible medical concepts, because the precision term in F1 and length discipline keep the notes concise.","feed_headline":"Speech-to-SOAP systems win both tracks, hallucinate least","feed_subtitle":"Reward optimization against a medical-concept metric hits top Concept F1 with the lowest judged hallucination rates.","key_machinery":"Open Medical Concept F1—a metric that scores generated notes by MeSH keyword matching and medical named-entity recognition against reference notes—is the load-bearing object: it serves both as training reward and ranking metric. Around it, the recipe couples low-rank supervised fine-tuning (to teach note format) with an on-policy reinforcement learning update that maximizes the metric on rollouts while keeping notes at roughly 295 words. The paper also relies on a zero-shot screening stage that used word-error rate on long audio as a diagnostic for whether a model can follow the full conversation, and on an LLM-based judge that decomposes each note into atomic claims to measure faithfulness","core_discovery":"Starting from zero-shot screening that identified models able to process roughly nine-minute conversations, the authors adapted a small and a large open-weight speech LLM using low-rank supervised fine-tuning followed by reinforcement learning with Open Medical Concept F1 as the reward. On the blind test set the adapted systems achieved Concept F1 of 0.563 (heavyweight) and 0.543 (lightweight), both more than double the baseline, and an LLM-as-judge evaluation measured hallucination rates of 0.08% and 0.25% on the main test split—the lowest in each track. The paper takes this as evidence that RL against such a metric can improve clinical summarization without inflating note length or encoura","pith_inferences":["The safety claim rests on a single LLM judge with no human-agreement study; a stricter human review of the supposedly low hallucination rates would tell whether the RL recipe is genuinely safe for clinical use.","The transcript-to-speech transfer result suggests a cheap general recipe for long-form audio summarization: do the expensive fine-tuning on text, then plug in a frozen speech encoder at inference; this may generalize to other domains beyond medicine.","A natural extension is to replace the F1 reward with a faithfulness-aware reward that uses the judge signal itself, which could push hallucination rates even lower while keeping concept recall.","The strong system-level metric-judge correlation does not guarantee per-note agreement; a per-dialogue correlation study on the realistic subset would test whether the metric is safe for deployment, not just for leaderboard ranking."],"forward_implications":["Reinforcement learning against Open Medical Concept F1 adds 0.04–0.05 Concept F1 over supervised fine-tuning alone, without lengthening notes or increasing judged hallucinations.","Fine-tuning on text transcripts transfers to speech input with only a small drop, so audio-input fine-tuning can be skipped for large models when computing budget is tight.","Models fine-tuned on normalized transcripts generalize better from synthetic to acted and real audio, losing only 0.008 Concept F1 versus 0.038 for the audio-fine-tuned lightweight system.","At system level, Concept F1 correlates strongly with judge-measured faithfulness (Spearman 0.91) and inversely with hallucination rate (−0.93), supporting the metric as a proxy for note quality.","Model scale is secondary: the 3B lightweight system essentially matches the second-best heavyweight submission, and the gain from scaling is smaller than the gain from reinforcement learning."],"fun_headline_variants":["Speech-to-SOAP models win both BeTraC tracks, lowest hallucination","RL vs medical concept F1 yields top SOAP, fewest hallucinations","Speech-to-SOAP via RL wins both tracks, least hallucination","Voxtral with RL wins both BeTraC tracks, lowest hallucination"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's central safety claim—that reinforcement learning does not increase hallucinations—rests on an LLM judge that breaks each note into atomic claims and checks them against the transcript, with no human agreement study, error analysis, or confidence intervals reported.","fun_headline_variants_meta":{"raw":{"variants":["Speech-to-SOAP models win both BeTraC tracks, lowest hallucination","RL vs medical concept F1 yields top SOAP, fewest hallucinations","Speech-to-SOAP via RL wins both tracks, least hallucination","Voxtral with RL wins both BeTraC tracks, lowest hallucination"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001745,"raw_usage":{"total_tokens":6696,"prompt_tokens":678,"completion_tokens":6018,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":422,"completion_tokens_details":{"reasoning_tokens":5936}},"tokens_in":422,"tokens_out":6018,"duration_ms":35896,"temperature":1.0,"reasoning_tokens":5936,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T18:37:47.403386+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the systems' generated notes, inject known hallucinated symptoms or diagnoses into a sample, and have independent clinician annotators mark factual errors; if the human error rate is materially higher than the LLM judge's 0.08–0.25% hallucination rates, the conclusion that the RL recipe preserves factual reliability fails.","supporting_citations":[],"review_version":1}