{"id":"5df6db39-1925-44ec-9edf-a2ec31a1c008","arxiv_id":"2608.10810","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CUE-Bench provides 51,823 Chinese discourse instances annotated with a nine-way Affective Stance defined by explicit-implicit polarity, plus pragmatic intent and fine-grained emotion labels.","lead":"CUE-Bench is a new Chinese-language dataset for testing whether AI can read the emotion that is implied but not stated, such as sarcasm, polite complaints, or indirect refusal. It gives each conversation snippet four linked labels, from surface wording to hidden feeling, plus a structured map that connects them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold-label reliability for the dominant Veiled Negative class is the load-bearing weak point; the paper's own audit flags 62% of audited Veiled Negative instances as problematic, yet the 20k model-agreement instances were never human-reviewed.","rationale":"I agree with the reader that gold-label reliability for Veiled Negative is the weakest load-bearing assumption. The label-quality problem is more fundamental than the abstract-to-Table 3 aggregation discrepancy: the latter is a reporting fix, while the former means there may be no trustworthy target to learn from or evaluate against. I considered whether the matrix-guided CoT gains could be explained by prompt surface effects, but that is secondary: if the labels are unreliable, the measured gains are uninterpretable regardless of prompt design. The manuscript itself flags residual annotation noise in Limitations §7, so this concern is not manufactured. Because the decisive experiment cannot be run from the paper alone and no data or code is released, I keep the reader's CONDITIONAL verdict rather than moving to REJECT: the construction is described in enough detail that a clean-subset analysis can settle the question.","tokens_in":17443,"tokens_out":7508,"duration_ms":79079,"concrete_test":"Release the dataset and run a stratified blind expert re-annotation of 500 instances from the final 51,823, oversampling Veiled Negative and covering all three pipeline segments (model-agreement, human-verified, LLM-adjudicated), using Appendix C guidelines and majority adjudication. Then recompute Table 3 for every model on the subset where original gold labels agree with the expert majority. If Veiled Negative agreement is below 0.6, or if the pragmatic-intent and fine-grained-emotion gains materially shrink or disappear on this clean subset, the benchmark's gold labels and headline improvements are not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two load-bearing pillars: CUE-Bench measures unsaid affect, and matrix-guided prompting improves intent and emotion prediction. Both presuppose that the gold labels are correct. The paper's own evidence undercuts this for the most frequent stance class. Veiled Negative is 22.3% of the data, but in the RQ2 audit 458 audited Veiled Negative instances are marked problematic, covering 62% of audited instances of that class (§5.4, Figure 4). This is not a rare tail class; it is the category the benchmark is specifically designed to stress-test. Moreover, the audit covers only the model-disagreement pool. The 20,000 'high-confidence' model-agreement instances (Appendix B.3) receive no human verification at all, and the 89% adjudicator-accuracy validation and the 3.1% contamination estimate are computed from the disagreement pool, so errors in the agreement pool are unmeasured. The inter-annotator statistics do not rescue the labels: stance α=0.52, intent α=0.34, emotion α=0.31 (Table 2), and the improved conditional kappas are computed only after excluding instances with inconsistent stance, which selects the easiest subset and cannot validate the full benchmark. If Veiled Negative gold labels are systematically noisy, both benchmark validity and the reported 7.8/3.5-point gains are measured against LLM-adjudicator bias rather than human affective judgment. This is especially dangerous because the evaluation prompts use the same matrix vocabulary that generated the labels. Even if some of the 62% are order-inconsistency cases already discarded by filtering, the paper does not report the post-filter error rate for Veiled Negative, so the retained labels in that class remain unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CUE-Bench, a Chinese benchmark for 'unsaid' emotion understanding built around an Explicit–Implicit Stance Matrix. For 51,823 context–target instances drawn from five Chinese discourse sources, it annotates explicit affective signal, implicit affective tendency, a derived nine-way Affective Stance, eight pragmatic intents, and twenty-five fine-grained emotions. The authors propose Matrix-Guided Chain-of-Thought prompting, in which an LLM first predicts explicit/implicit orientations, maps them through a fixed matrix to stance, then predicts intent and emotion. Experiments on five LLMs claim that this method improves fine-grained emotion recognition by 3.5 percentage points and pragmatic intent detection by 7.8 percentage points over strong baselines, and oracle-conditioning ablations are used to argue that stance is a useful intermediate representation.","tokens_in":17783,"tokens_out":6697,"duration_ms":59454,"significance":"The paper proposes a genuinely multi-layer affective annotation scheme that links surface expression to inferred affect, and the Explicit–Implicit Stance Matrix is simple and interpretable. If the gold labels are reliable, CUE-Bench would be a useful Chinese resource for evaluating pragmatic and affective inference, and the matrix-guided prompting recipe is practical and easy to reproduce. The authors also provide detailed annotation guidelines, a dataset card, conditional agreement analysis, and an explicit statement of limitations. However, the paper's own reliability evidence raises serious questions about the most frequent stance class, and two of the headline claims rest on aggregation and oracle-conditioning choices that overstate what is demonstrated.","major_comments":[{"comment":"The headline gains of 3.5 pp for fine-grained emotion and 7.8 pp for pragmatic intent are obtained by averaging the deltas over all five models and all three metrics (Acc/F1/W-F1) for each task, but this aggregation is never stated and the per-metric picture is weaker: macro-F1 for fine-grained emotion decreases on GPT-4o-mini (−0.006) and LLaMA-3.1-8B (−0.015), and several stance macro-F1 deltas are negative (Table 3). Please report per-metric aggregated deltas with sign and state the aggregation rule explicitly, or revise the abstract claims accordingly.","section":"Abstract; §5.2, Table 3"},{"comment":"The most load-bearing weakness is gold-label reliability for Veiled Negative, the most frequent stance (22.3%). The RQ2 audit of 1,500 model-disagreement instances finds 458 Veiled Negative instances marked problematic, i.e., 62% of audited instances of that class, yet the 20,000 model-agreement instances in Appendix B.3 receive no human review, and the 89% adjudicator-accuracy and 3.1% contamination estimates are computed from the disagreement pool. Moreover, stance α=0.52, intent α=0.34, and emotion α=0.31 (Table 2) are modest, and the conditional κ values are computed after excluding instances with inconsistent stance, which selects the easier subset. Since both benchmark validity and the prompting gains presuppose that gold labels are correct, please provide per-class reliability evidence for Veiled Negative, audit a sample of the agreement pool, and quantify how label noise affects the reported deltas.","section":"§5.4 RQ2; Figure 4; Appendix B.3; Table 2"},{"comment":"The oracle-conditioning ablations insert gold Affective Stance and gold Pragmatic Intent into the prompt and measure downstream intent/emotion prediction. This is an upper-bound exercise: it shows that gold intermediate labels are informative, not that the model's own predicted stance or intent helps. The conclusion (§6) that these ablations 'confirm the value of Affective Stance as an intermediate representation' is therefore too strong. Please either rephrase the claim or add an evaluation that conditions on the model's predicted stance.","section":"§5.3; Table 4; §6"},{"comment":"Because Affective Stance is defined as the deterministic map ϕ(e_i,h_i) and annotators are instructed to revisit the base signals rather than override the mapped stance (Appendix C.3), stance labels inherit any noise in the explicit/implicit sign labels and cannot serve as independent validation of the matrix. The paper should state this explicitly and treat stance reliability as a consequence of sign-label reliability, not as an independent guarantee.","section":"§4.2; Appendix C.3"}],"minor_comments":[{"comment":"The benchmark name is written as 'CUE Bench' in the abstract but as 'CUE-Bench' elsewhere; please use a consistent spelling.","section":"Abstract"},{"comment":"The model name 'Qwen 3-8B' is inconsistent with 'Qwen-3-8B' used in §5.1; please unify.","section":"Table 3"},{"comment":"The red line showing the problem-case rate is not labeled in the legend; please add a legend entry.","section":"Figure 4"},{"comment":"The phrase 'strongest baseline' is not identified per model; from Table 3 the best baseline differs across models (e.g., CoT for DeepSeek-V4-Flash, Few-shot for LLaMA-3.1-8B). Please specify the baseline used for each reported delta.","section":"§5.2"},{"comment":"The definition of 'problematic' conflates LLM adjudication inconsistency with human–LLM disagreement; please clarify whether the human review is treated as ground truth in this audit.","section":"§5.4 RQ2"},{"comment":"The English stance names in Table 7 are inconsistent with the Chinese matrix labels shown in Figure 2 (e.g., 'POSITIVE' vs '一致性正面'); please unify the English terminology.","section":"Table 7"}],"recommendation":"major_revision","confidential_remarks":"The Veiled Negative reliability issue is the key risk and is reflected in major comment 2; the revision should be judged primarily on how it addresses that issue. The paper's own audit data could be leveraged to produce per-class reliability estimates. I also note that no public release URL or code is provided; if the benchmark is a central contribution, the paper should state how the data will be accessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nCUE-Bench is a real resource: 51,823 Chinese discourse instances with explicit and implicit affect, a nine-way stance, eight intents, and 25 emotions. The Explicit-Implicit Stance Matrix is a simple, interpretable way to tie surface wording to latent affect, and the multi-layer annotation is more than most benchmarks offer. The pipeline is described honestly: dual-model agreement, human verification of 10k disagreement cases, LLM adjudication with order reversal, and a limitation section that names residual noise. The conditional kappas (0.79 intent, 0.67 emotion given consistent stance) suggest the stance helps align annotators.\n\nThe soft spots are real, though. The dominant class, Veiled Negative (22.3%), is exactly the category the benchmark is designed to stress-test, and the paper's own audit finds 62% of audited instances in that class problematic (458 cases). That audit only covers the model-disagreement pool; the 20k instances where two models agreed never get human review. The 89% adjudicator accuracy and 3.1% contamination estimate come from the disagreement pool, so errors in the agreement pool are unmeasured. If Veiled Negative labels are systematically noisy, both the benchmark's validity and the reported prompting gains are on shaky ground. The raw IAA is also low: α=0.52 stance, 0.34 intent, 0.31 emotion. These are defensible for subjective tasks, but the paper leans on conditional agreement computed after dropping inconsistent-stance cases, which is the easy subset.\n\nThe headline gains (3.5 pp emotion, 7.8 pp intent) are also presented without saying they come from averaging over five models and three metrics. On macro-F1, some models show negative deltas. The intent gains are consistent and real, but the abstract overstates the emotion result.\n\nNo data or code is released, which makes the resource hard to evaluate. And the oracle ablations show what stance can do when the label is given, not that models can produce it; that is fine as a diagnostic but should not be read as a capability result.\n\nI would engage with this paper. The schema and the dataset construction are worth a serious referee. But the authors need to sample-verify the agreement pool, report Veiled Negative reliability separately, state the aggregation in the abstract, and release at least a sample before this is citable as a benchmark. Give it conditional acceptance at best.","headline":"A substantial Chinese affective benchmark with a clever stance matrix, but the dominant Veiled Negative class is under-verified and the headline gains rest on an unstated aggregation.","tokens_in":18338,"tokens_out":3542,"would_cite":false,"duration_ms":31852,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that structuring affect as a nine-cell explicit–implicit stance matrix lets LLMs infer pragmatic intent and fine-grained emotion more accurately, with gains of 7.8 and 3.5 percentage points over strong prompting baselines.","keywords":["affective stance","implicit emotion","pragmatic intent","fine-grained emotion","Chinese discourse","benchmark construction","chain-of-thought prompting","explicit-implicit stance matrix"],"falsifier":"Take a random sample of 300 instances from the Veiled Negative class, remove the original gold label, have three fresh expert annotators independently select from the nine stances, and measure majority agreement with the gold label; if it falls below 60 percent, the most frequent class is too noisy to support the reported benchmark gains.","tokens_in":17253,"feed_emoji":"😶","tokens_out":11139,"duration_ms":102381,"temperature":0.7,"pith_summary":"This paper tries to establish that affective meaning in discourse is best understood as a relation between what is said and what is left unsaid. It introduces CUE-Bench, a Chinese benchmark of 51,823 context–utterance pairs annotated with an explicit affective signal, an implicit affective tendency, nine affective stances, eight pragmatic intents, and twenty-five fine-grained emotions. The core claim is that the Explicit-Implicit Stance Matrix, which composes the explicit signal and implicit tendency into nine named stances, is a usable intermediate layer for affective inference. The paper shows that prompting LLMs through that matrix in a fixed order improves pragmatic intent detection by 7.8 percentage points and fine-grained emotion recognition by 3.5 percentage points over strong prompting baselines. If this is right, affective NLP gains both a multi-layer Chinese resource and a training-free prompting method that pushes models from surface polarity toward intended meaning.","feed_headline":"A nine-stance matrix lifts intent detection by 7.8 points","feed_subtitle":"Chinese benchmark with 51,823 utterances scores the gap between surface sentiment and hidden meaning.","key_machinery":"The Explicit-Implicit Stance Matrix is a $3\\times3$ table: rows are the explicit affective signal $e_i\\in\\{+,0,-\\}$, columns are the implicit affective tendency $h_i\\in\\{+,0,-\\}$, and each cell is the Affective Stance $s_i=\\phi(e_i,h_i)$. It is the load-bearing intermediate representation of the paper because the stance is not an independent judgment but a deterministic composition of the two affect layers, and the same ordered pair drives the annotation protocol, the conditional inter-annotator agreement analysis, and the Matrix-Guided Chain-of-Thought prompting protocol.","core_discovery":"On the paper's own terms, the discovery is that an utterance's affective stance can be defined as the composition of two low-cost signals: the explicit affective signal $e_i$ (the polarity anchored in surface wording) and the implicit affective tendency $h_i$ (the affect recoverable from context and pragmatic force). The map $\\phi(e_i,h_i)$ fills a $3\\times3$ matrix whose nine cells are named stances: Positive, Formulaic Positive, Sarcastic Negative, Understated Positive, Neutral, Veiled Negative, Affiliative Positive, Reportive Negative, and Negative. CUE-Bench annotates these layers on 51,823 Chinese instances and evaluates three connected tasks; its central experimental claim is that a Matrix-Guided Chain-of-Thought prompt, which forces the model to state explicit signal, implicit tendency, stance, intent, and then emotion, improves pragmatic intent detection by 7.8 percentage points and fine-grained emotion recognition by 3.5 percentage points over the strongest tested prompting baselines. The paper interprets these gains as evidence that the stance matrix is a structured intermediate representation, not just a label set.","pith_inferences":["Because the most frequent stance, Veiled Negative, also has the highest problematic-case rate in the audit, aggregate accuracy is partly a score on the hardest cell of the matrix; a per-class headline would likely show the benchmark's difficulty is concentrated there.","The 7.8-point and 3.5-point figures are accuracy- and weighted-F1-based; under macro-F1 the emotion gains are near zero or negative for several models, so the improvement should be read as distribution-weighted rather than class-balanced.","A prompt-only control with an equally structured but semantically arbitrary intermediate schema (for example, surface polarity, topic, then final label) would separate the stance matrix's contribution from the general benefit of enforced structured reasoning."],"forward_implications":["Affective NLP evaluation can test the reasoning path rather than only the final label, so a model that guesses the right emotion for the wrong reason is no longer indistinguishable from one that recovers the stance.","The same 51,823 instances support three connected tasks, letting developers pinpoint whether a dialogue system fails at surface reading, stance resolution, or final emotion discrimination.","Matrix-guided prompting is a training-free intervention that yields consistent pragmatic-intent gains across all five tested LLM families, making it directly usable in deployed Chinese dialogue and customer-service systems.","The smaller and less stable gains on fine-grained emotion identify category-level emotion discrimination, not stance recovery, as the remaining bottleneck."],"supporting_citations":[{"why":"Supplies the stance-triangle concept that grounds the paper's definition of Affective Stance.","marker":"Du Bois, 2007"},{"why":"Supplies the politeness theory that motivates many of the explicit-implicit mismatches in Chinese discourse.","marker":"Brown and Levinson, 1987"},{"why":"Supplies the conversational-implicature logic behind the what-is-said versus what-is-meant distinction.","marker":"Grice, 1975"},{"why":"IEST is the implicit-emotion benchmark whose single-layer design CUE-Bench extends.","marker":"Klinger et al., 2018"},{"why":"GoEmotions is the fine-grained emotion benchmark used as a design and comparison reference, including for agreement expectations.","marker":"Demszky et al., 2020"},{"why":"Provides evidence that LLMs can perform annotation tasks, motivating the constrained LLM adjudication step in the pipeline.","marker":"Gilardi et al., 2023"},{"why":"Grounds the paper's decision not to reject low inter-annotator agreement scores with fixed thresholds.","marker":"Wong et al., 2021"},{"why":"Supplies the emotion-wheel reference for the benchmark's twenty-five fine-grained emotion categories.","marker":"Plutchik, 2001"}],"fun_headline_variants":["Nine-stance matrix lifts Chinese intent detection by 7.8 points","Hidden affect decoded: stance matrix boosts intent by 7.8%","CUE-Bench's nine stances add 7.8 points to intent in Chinese","Stance matrix: +7.8% intent, +3.5% emotion on 51k Chinese","Surfacing unsaid affect: matrix gains on Chinese discourse"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result depends on the gold labels for the benchmark's most common category—neutral-sounding speech with a hidden negative attitude—being trustworthy, and the paper's own audit found problems in 62% of the sampled cases in that category.","fun_headline_variants_meta":{"raw":{"variants":["Nine-stance matrix lifts Chinese intent detection by 7.8 points","Hidden affect decoded: stance matrix boosts intent by 7.8%","CUE-Bench's nine stances add 7.8 points to intent in Chinese","Stance matrix: +7.8% intent, +3.5% emotion on 51k Chinese","Surfacing unsaid affect: matrix gains on Chinese discourse"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1486,"prompt_tokens":946,"completion_tokens":540,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":434}},"tokens_in":562,"tokens_out":540,"duration_ms":5663,"temperature":1.0,"reasoning_tokens":434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:52:30.960830+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 300 instances from the Veiled Negative class, remove the original gold label, have three fresh expert annotators independently select from the nine stances, and measure majority agreement with the gold label; if it falls below 60 percent, the most frequent class is too noisy to support the reported benchmark gains.","supporting_citations":[],"review_version":1}