{"id":"15ac0060-e479-4a49-87be-4ef4b3fa6d73","arxiv_id":"2411.17437","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On a private dataset of 555 real conversations, a zero-shot LLM prompt detects user frustration with higher macro-F1 than sentiment, emotion, dialog-breakdown, and keyword baselines.","lead":"This paper tests whether off-the-shelf models can spot frustrated users in a real customer-service dialog system, and finds that large language models called with a description of the task do best. The result suggests that simple keyword and sentiment tools miss most frustration, which matters for companies automating customer support.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed LLM advantage is not yet supported by the evaluation: the private 555-dialogue benchmark has moderate label agreement, no uncertainty quantification, and the reported gains mix full-conversation LLMs with last-utterance baselines.","rationale":"The reader's weakest assumption was label accuracy, and I agree that moderate inter-annotator agreement threatens the benchmark's validity. However, the most load-bearing issue is broader: even if the labels are accepted, the paper provides no statistical evidence that the observed macro-F1 gaps are real rather than sampling noise. With 555 dialogues and binary outcomes, the difference between 0.86 and 0.71 could easily be within bootstrap variability, especially since the LLM decisions are probably correlated with each other and with the baselines. The absence of any uncertainty quantification is not a cosmetic omission; it is the difference between demonstrating a production-ready detector and reporting a single run of a non-deterministic API. The abstract/body inconsistency reinforces this: if the quantitative headline can change from 16% to 22-33% across versions, the reported effect size is not stable. The matched-input issue is also substantive because it means the '+'33% figure is not a like-for-like comparison of detectors operating on the same input, although the full-conversation setting is the realistic deployment condition and the LLM still wins there. For these reasons, I do not think the paper should be rejected; the claim is plausible and the deployed-data perspective is valuable. But the central claim should remain conditional on a transparent, uncertainty-aware evaluation and on reconciliation of the reported improvement figures. The reader's conditional verdict is therefore the right one, and my stress-test does not move it.","tokens_in":10441,"tokens_out":8461,"duration_ms":83027,"concrete_test":"Obtain per-dialogue predictions of every method and run a paired bootstrap (10,000 resamples) on the macro-F1 differences between GPT-4o-zero-shot and each baseline, plus a McNemar test on binary decisions. Then re-rank all methods on each annotator's individual labels (before adjudication) and on the subset of dialogues where all three annotators agree. If the LLM-versus-DBD gap is not significant at p<0.05, or if the ranking does not hold on high-agreement labels, the headline 'outperforms all other approaches' should be weakened. As a secondary check, recompute relative improvements using matched input (last-utterance LLM versus last-utterance sentiment/emotion) and reconcile the abstract's 16% with Section 4's 22-33%.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the internal benchmark being both accurately labeled and adequately powered. Manual labels reach only Fleiss kappa 0.48 (Section 3.4), and no confidence intervals, bootstrap errors, or significance tests are reported for any macro-F1 difference in Table 1. With moderate inter-annotator agreement, the adjudicated gold labels may contain systematic bias. Section 5.1 explicitly treats requests for human assistance as potential frustration cues, and Figure 3 injects closely related cues directly into the LLM prompt; if annotators and the prompt align on the same surface cues, the LLM F1 may reflect prompt-label agreement rather than robust detection, and the ordering could change under a different annotation standard. The reporting inconsistency between the abstract (16% relative improvement) and Section 4 (+33% over sentiment/emotion, +22% over DBD) further shows the headline effect is not pinned down. Additionally, Table 2 shows that the zero-shot LLM advantage is largely a full-conversation effect: on the last user utterance, GPT-4o-zero-shot scores 0.67 macro-F1, barely above RoBERTa-Sent-LU (0.65) and below DBD (0.71). The +33% headline compares the best full-conversation LLM against the best last-utterance non-LLM baseline, so the claimed magnitude is not a matched-input comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies automatic detection of user frustration in a deployed task-oriented dialog (TOD) system. It introduces a private benchmark of 555 real conversations from two dialog phases (booking negotiations and receptionist transfers), manually annotated by three experts, and compares a deployed keyword matcher, open-source sentiment/emotion models, a feature-based dialog breakdown detection (DBD) baseline, and prompted LLMs (GPT-4o and Llama-3.1-405B) under zero-shot and two-shot in-context learning. The reported results show LLM-based detection achieving the highest macro-F1 scores (0.86–0.87 on full conversations), and the authors provide qualitative analysis of differences between real-world and academic data, and of failure modes across method families.","tokens_in":10716,"tokens_out":5228,"duration_ms":48597,"significance":"If the central finding holds, the paper has clear practical value: zero-shot prompting of a strong LLM would provide a deployable frustration detector without fine-tuning, and the comparison against open-source sentiment, emotion, and dialog-breakdown methods would be a useful industry-oriented result. The paper's strengths include the use of an actual deployed TOD system, the inclusion of several baseline families, and a transparent ICL prompt. However, the evaluation currently has unresolved statistical and comparability issues that affect the magnitude and robustness of the headline claim: the abstract reports a 16% relative improvement while the body reports +33% and +22%; the headline comparison mixes full-conversation LLM results with last-utterance baselines; the DBD+LogReg baseline lacks a described train/test split; and the gold labels rest on only moderate annotator agreement (Fleiss' kappa = 0.48). The direction of the main effect is plausible, but the strength of the claim is not yet commensurate with the evidence.","major_comments":[{"comment":"The abstract reports a 16% relative improvement in F1 score, while Section 4 reports '+33% relative improvement over sentiment and emotion detection methods and +22% relative improvement over the DBD method.' These numbers are not reconciled, and the 16% figure is not tied to a specific model comparison or table row. The authors should state exactly which models, input formats, and baseline variants are being compared, and use one consistent comparison protocol in the abstract and body.","section":"Abstract and Section 4"},{"comment":"The headline comparison is not matched on input length. The best LLM results are full-conversation scores (GPT-4o-zero-shot-FC 0.86, Llama-3.1-405B-two-shot-FC 0.87), while the best sentiment/emotion value used for the +33% claim appears to be RoBERTa-Sent-LU (0.65), a last-utterance variant. On the last user utterance alone, GPT-4o-zero-shot achieves 0.67 macro-F1, barely above RoBERTa-Sent-LU (0.65) and below DBD+LogReg (0.71). The comparisons should be reported within the same input condition (full-conversation vs. last utterance), or the claim 'ICL with LLMs outperforms all other approaches' should be explicitly qualified as holding for the full-conversation setting.","section":"Section 4 and Table 2"},{"comment":"The DBD+LogReg baseline is fit on hand-crafted features but no train/test split, cross-validation procedure, or training details are described. Because the reported DBD result (0.71 macro-F1) is used as the strongest non-LLM baseline, applying the method to the same 555 conversations used for evaluation may give an optimistic estimate. The authors should specify the split or cross-validation protocol, and report bootstrap confidence intervals or paired significance tests for the macro-F1 differences in Table 1 and Table 2.","section":"Section 3.3 and Appendix B"},{"comment":"The gold labels are the foundation of every comparison, yet Fleiss' kappa is only 0.48 (moderate agreement) and the adjudication step is described in one sentence. With N=555, the reported macro-F1 differences could be sensitive to label noise or systematic annotator bias. The authors should report the label distribution, per-annotator agreement on positive/negative cases, and ideally an analysis of whether disagreements correlate with the surface cues listed in the ICL prompt (repetition, negation, requests for human assistance). This would help rule out the possibility that the LLM's advantage partly reflects prompt-label alignment rather than robust frustration detection.","section":"Section 3.4"}],"minor_comments":[{"comment":"The keyword set K is described only as 'mainly composed of profanity, words explicitly indicating negative emotion, and insults'; reporting its size and a few representative examples would improve reproducibility.","section":"Section 3.1"},{"comment":"The figure captions refer to a 'yellow exclamation sign' marker, but the figure is not rendered in the manuscript text; please ensure the markers are visible and explained in the published version.","section":"Section 5.2 and Figure 2"},{"comment":"The reference 'Landis, 1977' should be 'Landis and Koch, 1977' to match the standard citation for Fleiss' kappa interpretation.","section":"References"},{"comment":"The sentence 'Moreover, a drop in performance when full conversations were used' should specify that this refers to the sentiment/emotion baselines, since the LLM and DBD methods do not show the same pattern.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is suitable for an applied NLP/industry venue, and the central engineering claim is plausible. However, the evaluation currently lacks the statistical grounding needed for an archival claim: no uncertainty quantification, an unreported train/test detail for the DBD baseline, and inconsistent headline effect sizes between the abstract and the body. The necessary additions are feasible and would not require new data collection. I see no evidence of circularity in the target definition; the main risk is that label noise and input-format mismatches affect the magnitude of the LLM advantage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the private real-world TOD benchmark: 555 production dialogues with manual frustration labels and a qualitative comparison against EmoWoZ. That alone is worth something, because most frustration work uses lab data or proxies. The paper also does a fair job of showing that keyword matching is nearly useless (1% recall) and that sentiment/emotion models trained on academic data don't transfer well. The ICL prompt design is sensible, and the fact that two different LLMs land in the same ballpark suggests the prompt is doing real work.\n\nThe soft spots are exactly where the stress-test points. First, the headline claim is built on a mismatched comparison. Table 1 pits full-conversation LLMs against last-utterance sentiment/emotion baselines and a last-utterance DBD baseline. When you look at Table 2 with matched inputs, GPT-4o-zero-shot on the last user utterance gets 0.67 macro-F1, below DBD's 0.71. So the \"+33% over sentiment and +22% over DBD\" is really \"LLM with full context beats baselines with one utterance.\" That's still an interesting result, but it's a different claim, and the abstract's 16% versus the body's 22–33% tells you the headline number isn't pinned down.\n\nSecond, the annotation quality is a real concern. Fleiss kappa of 0.48 is moderate at best, and with no error bars or significance tests on a dataset of 555, the ranking in Table 1 could shift under a different annotation standard. The prompt explicitly lists cues like repetition and requests for human assistance, and Section 5.1 treats those as frustration signals, so there is a risk of label-prompt alignment inflating the LLM numbers. I wouldn't call it circular—the labels were made by humans, not by the LLM—but the prompt and the annotation guidelines may share surface cues.\n\nThird, the DBD+LogReg baseline is fitted without any described train/test split. If the logistic regression was trained on the same 555 examples it is evaluated on, that gives it an unfair advantage, not the LLM, so the direction of bias is conservative. Still, it should be disclosed.\n\nWho is this for? Industry practitioners who want a quick read on whether off-the-shelf LLM prompting can replace keyword filters in production TOD systems. It is not a rigorous academic benchmark yet. I would send it to peer review, but with major revision requests: add confidence intervals, report matched-input comparisons, clarify the DBD fitting procedure, and reconcile the abstract with the body. The core finding—that full-conversation LLM ICL beats last-utterance classical methods on real user frustration—is plausible and worth publishing after those fixes.","headline":"Useful industry benchmark and a plausible LLM advantage, but the evaluation is not yet tight enough to trust the headline magnitude.","tokens_in":11288,"tokens_out":1515,"would_cite":true,"duration_ms":17531,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that in-context learning with LLMs detects user frustration in deployed task-oriented dialog systems better than keyword matching, sentiment and emotion models, and dialog-breakdown detectors, with a zero-shot GPT-4o…","keywords":["user frustration detection","task-oriented dialog","in-context learning","large language models","sentiment analysis","emotion detection","dialog breakdown detection","real-world deployment"],"falsifier":"Re-annotate a random subset of the 555 conversations with a separate, larger annotator pool and a more prescriptive frustration rubric; if the LLM in-context learning method no longer beats the dialog-breakdown baseline by the reported margin, or if the new labels substantially change which conversations count as frustrated, the main comparison collapses.","tokens_in":10247,"feed_emoji":"😤","tokens_out":7099,"duration_ms":58936,"temperature":0.7,"pith_summary":"The paper sets out to show that user frustration in a real, deployed task-oriented dialog system is detected best by prompting a large language model in context, rather than by keyword rules or off-the-shelf sentiment and emotion models. It compares several approaches on 555 real conversations from a production booking and receptionist system, manually labeled for frustration. Its central result is that a zero-shot LLM prompt reaches a macro-F1 of 0.86, a 33% relative improvement over sentiment and emotion methods and a 22% improvement over dialog-breakdown feature classifiers. The authors argue that real-world frustration is often expressed through repetition, negation, and unresolved tasks rather than profanity, which explains why keyword matching catches only about 1% of frustrated conversations. A practical payoff is that a deployed system could adopt a domain-described prompt with no fine-tuning as a frustration detector.","feed_headline":"Prompted LLMs spot user frustration better than existing detectors","feed_subtitle":"Out-of-the-box LLM prompts beat keyword, sentiment, emotion, and breakdown detectors on a deployed system.","key_machinery":"The central object is the in-context learning prompt P(T, D, H), built from a human-written task description that lists explicit and subtle frustration cues, a domain description (receptionist routing or booking negotiation), the conversation history formatted with USER and SYSTEM prefixes, and output instructions asking for a single 0 or 1 label. This prompt carries the argument because it lets a frozen LLM apply general knowledge of frustration and dialogue without any gradient updates, and it is the component varied across zero-shot and two-shot settings and across GPT-4o and Llama-3.1-405B to demonstrate generality. The scoring machinery is an internal benchmark of 555 real production conversations labeled by three experts, with disagreements resolved manually.","core_discovery":"The paper's central claim is that in-context learning with LLMs, rather than fine-tuning or task-specific training, gives the best frustration detection among the methods tested, and that the advantage comes from LLMs reading both semantics and dialogue structure. On its internal benchmark, a zero-shot GPT-4o prompt reaches macro-F1 0.86 on full conversations and Llama-3.1-405B with two shots reaches 0.87, while the best non-LLM baseline, logistic regression on dialog-breakdown features, reaches 0.71. The paper also finds that sentiment and emotion models degrade when given full conversation history instead of the last user utterance, indicating they capture isolated emotion rather than contextual frustration. It concludes that academic emotion benchmarks such as EmoWoZ differ from real-world production data in urgency, frustration triggers, repetition, and requests for human agents, and that these differences explain why open-source models transfer poorly.","pith_inferences":["Editorial inference: the same prompt could be run on a sliding window of recent turns rather than the whole conversation, giving real-time signals; the paper only labels conversations as a whole.","Editorial inference: because the benchmark covers one week of production and only 555 conversations, the margin over dialog-breakdown detection may shrink on longer or noisier production streams; a rollout with continuous monitoring would test this.","Editorial inference: combining dialog-breakdown features as a cheap first filter with an LLM as a second-stage verifier could cut LLM cost while keeping recall, a hybrid the paper does not evaluate.","Editorial inference: the qualitative finding that repetition and negation are strong cues suggests a testable prompt extension that explicitly asks the model to count repeated user requests or rejections before labeling frustration."],"forward_implications":["A deployed task-oriented dialog system can adopt frustration detection by writing a domain description into an in-context learning prompt, with no fine-tuning or in-domain training data.","The zero-shot GPT-4o configuration reaches recall 0.96 for frustrated conversations, so most frustrated users would be flagged for dialog repair or transfer to a human agent.","The prompt transfers across model families with similar macro-F1, so the method is not tied to a single LLM vendor.","Keyword matching, at 100% precision but about 1% recall, and off-the-shelf sentiment and emotion models are not adequate production detectors for this task.","Dialog-breakdown features such as repetition and negation remain a competitive, lower-cost alternative where LLM inference is impractical."],"supporting_citations":[{"why":"Establishes the in-context learning paradigm that the LLM-based detection method is built on.","marker":"Brown et al., 2020"},{"why":"Supplies the RoBERTa-Sent sentiment analysis baseline used for comparison.","marker":"Hartmann et al., 2023"},{"why":"Provides EmoWoZ, the academic emotion-in-TOD dataset used both as a comparison corpus and as training data for the DistilBERT-EmoWoZ baseline.","marker":"Feng et al., 2022"},{"why":"Supplies the DistilBERT-EmoWoZ emotion detection baseline.","marker":"Huang, 2024"},{"why":"Supplies the DistilRoBERTa-Emo emotion detection baseline.","marker":"Hartmann, 2022"},{"why":"Supplies the dialog-breakdown detection features used in the DBD+LogReg baseline.","marker":"Bodigutla et al., 2020"},{"why":"Supports the claim that in-context learning prompts can be adapted to new domains by describing the domain.","marker":"Feng et al., 2024"}],"fun_headline_variants":["LLM prompts beat older detectors for user frustration","Prompted LLMs best frustration detection in real dialogs","In-context LLMs outdo sentiment and breakdown tools","Zero-shot LLM prompts detect frustration best"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings in the paper assume the manual frustration labels are accurate enough to compare methods, even though three annotators only reached moderate agreement (Fleiss' kappa = 0.48).","fun_headline_variants_meta":{"raw":{"variants":["LLM prompts beat older detectors for user frustration","Prompted LLMs best frustration detection in real dialogs","In-context LLMs outdo sentiment and breakdown tools","Zero-shot LLM prompts detect frustration best"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000283,"raw_usage":{"total_tokens":1654,"prompt_tokens":908,"completion_tokens":746,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":684}},"tokens_in":524,"tokens_out":746,"duration_ms":6055,"temperature":1.0,"reasoning_tokens":684,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:06:19.631593+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random subset of the 555 conversations with a separate, larger annotator pool and a more prescriptive frustration rubric; if the LLM in-context learning method no longer beats the dialog-breakdown baseline by the reported margin, or if the new labels substantially change which conversations count as frustrated, the main comparison collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides EmoWoZ, the academic emotion-in-TOD dataset used both as a comparison corpus and as training data for the DistilBERT-EmoWoZ baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DistilBERT-EmoWoZ emotion detection baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DistilRoBERTa-Emo emotion detection baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the dialog-breakdown detection features used in the DBD+LogReg baseline."}],"review_version":1}