{"id":"7aade01f-04b7-44c5-8ea6-85069a90249c","arxiv_id":"2606.25552","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"SFL-MTSC improves slot F1 and overall accuracy in zero-shot multi-intent SLU on MAC-SLU by decomposing predictions into intent-specific frames, applying domain-intent grouping and slot clustering, and retaining reliable frames via path support scoring.","lead":"The paper proposes SFL-MTSC, a framework that aggregates LLM predictions at the semantic frame level for multi-intent spoken language understanding to reduce inconsistencies from decoding stochasticity. This approach could help make voice assistants more reliable when users express multiple intents in a single utterance.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Experiments compare SFL-MTSC only to single-path inference, omitting direct comparison to output-level majority voting referenced in the method.","rationale":"The reader's weakest assumption correctly flags the unverified reliability of the decomposition and scoring steps. My concern is narrower and experimental: the claim of improvement is only benchmarked against the weakest baseline (single-path), not against the output-level voting the paper itself contrasts with. This is a direct test of whether the proposed machinery is load-bearing. Agreement is partial because both target the same gap in validating the method's added value.","tokens_in":1643,"tokens_out":351,"duration_ms":14566,"concrete_test":"Generate the same N paths per example as used in the SFL-MTSC experiments, apply output-level majority voting on the full predicted structures, and recompute slot F1, intent accuracy, and overall accuracy on MAC-SLU; if the margin over single-path is comparable to SFL-MTSC, the frame-level components add little beyond basic consistency.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim is that frame-level decomposition, domain-intent grouping, slot clustering, and path support scoring yield higher slot F1 and accuracy than single-path decoding on MAC-SLU. The abstract explicitly positions the approach as an alternative to output-level majority voting, yet reports no results against that baseline (or any other consistency method) on the same generated paths. This leaves open whether the added machinery improves upon simpler aggregation or merely benefits from using multiple samples. If standard majority voting already captures most gains, the load-bearing assumption that the structured frame-level operations are required for reliable multi-intent structures is untested.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes Semantic Frame-Level Multi-Task Self-Consistency (SFL-MTSC), a structured aggregation framework for prompt-based multi-intent spoken language understanding with LLMs. It decomposes LLM predictions into intent-specific semantic frames, applies domain-intent grouping and slot-level clustering, evaluates reliability via path support scoring, and retains reliable frames for re-integration into the final output. This is positioned as an alternative to output-level majority voting. Zero-shot experiments on the MAC-SLU benchmark report improved slot F1 and overall accuracy over single-path inference, with intent accuracy largely stable.","tokens_in":1776,"tokens_out":377,"duration_ms":12998,"significance":"If the empirical gains hold after proper baseline controls, the work could offer a more structured approach to mitigating decoding inconsistencies in multi-intent SLU scenarios. The method explicitly builds on multiple sampled paths but its added machinery (frame decomposition, clustering, and scoring) would need to demonstrate value beyond simpler aggregation to establish broader impact.","major_comments":[{"comment":"Abstract: the positioning of SFL-MTSC as an alternative to output-level majority voting is not supported by any reported comparison to that baseline (or other consistency methods) on the same generated paths; experiments compare only to single-path inference, leaving open whether the frame-level operations are required or if majority voting already captures most gains.","section":"Abstract"},{"comment":"Experiments (and abstract): no details are provided on the implementation of slot-level clustering, path support scoring thresholds, statistical significance testing, error bars, or controls for multiple comparisons, which undermines verification of the reported slot F1 and accuracy improvements on MAC-SLU.","section":"Abstract and Experiments"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and will revise the manuscript accordingly to strengthen the empirical support for SFL-MTSC.","responses":[{"response":"We agree that the abstract's positioning would be better supported by a direct comparison to output-level majority voting (and related consistency methods) performed on the exact same set of sampled paths. The current experiments emphasize gains relative to single-path inference, but adding the requested baseline will clarify whether the frame-level decomposition, grouping, clustering, and scoring provide benefits beyond simpler aggregation. We will include this comparison in the revised version.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the positioning of SFL-MTSC as an alternative to output-level majority voting is not supported by any reported comparison to that baseline (or other consistency methods) on the same generated paths; experiments compare only to single-path inference, leaving open whether the frame-level operations are required or if majority voting already captures most gains."},{"response":"We acknowledge that the manuscript lacks these reproducibility and statistical details. In the revision we will add: complete specifications of the slot-level clustering procedure and its hyperparameters; the exact definition, formula, and threshold(s) for path support scoring; results with error bars across multiple independent sampling runs; appropriate statistical significance tests with p-values; and any adjustments applied for multiple comparisons. These additions will enable verification of the reported improvements.","revision_made":"yes","referee_comment":"[Abstract and Experiments] Experiments (and abstract): no details are provided on the implementation of slot-level clustering, path support scoring thresholds, statistical significance testing, error bars, or controls for multiple comparisons, which undermines verification of the reported slot F1 and accuracy improvements on MAC-SLU."}],"tokens_in":1308,"tokens_out":392,"duration_ms":24546,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper introduces semantic frame-level multi-task self-consistency that decomposes LLM outputs into intent-specific frames, groups them by domain and intent, clusters slots, and scores paths for reliability before re-integrating. This is a concrete step beyond whole-output voting for handling multi-intent structures.\n\nIt does a decent job of targeting the inconsistency problem that comes with stochastic decoding in these tasks. The zero-shot MAC-SLU results show clear lifts in slot F1 and overall accuracy over single-path inference, with intent accuracy mostly unchanged. That suggests the decomposition can preserve useful structure.\n\nThe soft spot is the missing baseline. The abstract positions the method as an alternative to output-level majority voting, yet the experiments skip any direct comparison to it on the same set of paths. Without that, it's impossible to know if the frame grouping, clustering, and path scoring add anything over just taking the most common full output. The gains could come mostly from using multiple samples rather than the extra machinery.\n\nImplementation details on clustering and scoring thresholds are also light, and the abstract gives no error bars or significance tests. Those are fixable but matter for judging how reliable the reported improvements are.\n\nThis is for SLU researchers who already use LLMs and want to stabilize multi-intent predictions. A reader looking for incremental prompting tweaks might get something out of trying the frame-level steps.\n\nIt deserves peer review once the majority-voting comparison is added, because the underlying idea is practical and the current evidence is at least directionally positive.","headline":"SFL-MTSC gives a structured frame-level twist to self-consistency for multi-intent SLU but only beats single-path decoding, not the majority-voting baseline it claims to improve on.","tokens_in":2255,"tokens_out":395,"would_cite":false,"duration_ms":16855,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Semantic frame-level self-consistency improves slot accuracy in multi-intent spoken language understanding.","keywords":["spoken language understanding","multi-intent detection","self-consistency","semantic frames","slot filling","large language models","zero-shot evaluation"],"falsifier":"Running SFL-MTSC on the MAC-SLU benchmark and finding no gain in slot F1 or overall accuracy relative to single-path inference, or finding that valid multi-intent structures are dropped.","tokens_in":2536,"feed_emoji":"","tokens_out":593,"duration_ms":20473,"temperature":0.7,"pith_summary":"The paper proposes a structured aggregation method that breaks LLM outputs into intent-specific frames rather than voting on complete predictions. It groups frames by domain and intent, clusters slots, and scores how well different generation paths support each cluster before keeping the reliable ones. Experiments on the MAC-SLU benchmark show this raises slot F1 and overall accuracy in zero-shot use while intent accuracy stays mostly unchanged.","feed_headline":"Frame-level consistency raises multi-intent SLU slot accuracy","feed_subtitle":"Decomposing outputs into intent frames, clustering slots, and scoring path support improves F1 over single inference on MAC-SLU.","key_machinery":"Semantic Frame-Level Multi-Task Self-Consistency (SFL-MTSC), a framework that aggregates at the semantic frame level through decomposition, grouping, clustering, and path support scoring instead of output-level majority voting.","core_discovery":"SFL-MTSC decomposes predictions into intent-specific frames, performs domain-intent grouping and slot-level clustering, evaluates reliability with path support scoring, and retains reliable frames for re-integration, yielding higher slot F1 and overall accuracy than single-path inference on the MAC-SLU benchmark.","pith_inferences":["Similar frame-level decomposition and support scoring could be tested on other structured prediction tasks where whole-output voting loses internal consistency.","The method implies that component-level consistency checks may be more effective than output-level voting when semantic structures contain multiple independent elements.","If the grouping and clustering steps depend on domain knowledge, performance could vary when that knowledge is incomplete or noisy."],"forward_implications":["Slot F1 and overall accuracy rise over single-path inference in zero-shot multi-intent settings.","Intent accuracy remains largely stable across most experimental conditions.","Reliable frames identified by path support scoring can be retained and re-integrated into a final structured prediction.","The approach targets decoding stochasticity specifically in multi-intent scenarios."],"fun_headline_variants":["Frame consistency raises multi-intent SLU slot F1","Semantic frames improve slot accuracy in multi-intent SLU","Multi-task consistency improves multi-intent SLU slot F1","Intent frame clustering raises slot F1 on MAC-SLU"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Decomposing predictions into intent-specific frames and then grouping, clustering, and scoring them will reliably identify and keep correct structures without introducing selection bias or missing valid combinations.","fun_headline_variants_meta":{"raw":{"variants":["Frame consistency raises multi-intent SLU slot F1","Semantic frames improve slot accuracy in multi-intent SLU","Multi-task consistency improves multi-intent SLU slot F1","Intent frame clustering raises slot F1 on MAC-SLU"]},"model":"grok-4.3","cost_usd":0.009056,"raw_usage":{"total_tokens":4019,"prompt_tokens":578,"num_sources_used":0,"completion_tokens":66,"cost_in_usd_ticks":90562000,"prompt_tokens_details":{"text_tokens":578,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3375,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":578,"tokens_out":66,"duration_ms":27889,"temperature":1.0,"reasoning_tokens":3375,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-25T21:04:36.982826+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running SFL-MTSC on the MAC-SLU benchmark and finding no gain in slot F1 or overall accuracy relative to single-path inference, or finding that valid multi-intent structures are dropped.","supporting_citations":[],"review_version":1}