{"id":"59255960-415a-4d31-be77-81216a85ae36","arxiv_id":"2508.07625","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A confidence-based multimodal emotion recognition system built from an existing trusted classification method reports high trusted F1 scores using a new evaluation metric whose threshold is fitted to the test set.","lead":"The paper applies an existing evidential deep learning framework to multimodal emotion recognition, adding confidence based fusion and new 'trusted' evaluation metrics. It claims state-of-the-art trusted performance on IEMOCAP and Music-video, but the evaluation methodology, including test-set threshold selection and limited baselines, undercuts the claim.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Trusted metrics are computed at thresholds selected on the test set, so the reported trusted-SOTA gap may be an artifact of test-label leakage.","rationale":"The reader's weakest assumption identifies exactly this test-set threshold selection, and it is the most load-bearing issue for the central claim. The paper's key quantitative evidence is trusted F1 / trusted accuracy comparing TER to baselines; all of these numbers are computed at per-model thresholds chosen on test labels. This is not a minor methodological nit: it means the evaluation protocol is not a fixed decision rule and can inflate any model's trusted metrics. The claim that TER's trusted F1 exceeds its own ordinary F1 is especially fragile because the threshold can be adjusted to make precision and recall artificially balanced. A validation-set threshold is the minimal fix; without it, no trusted comparison is interpretable. I therefore keep the reader's REJECT verdict: the concern is severe and the central claim is unsupported as presented. If the re-evaluation with validation-set thresholds still shows a large trusted gap, a conditional acceptance could be considered, but that remains to be demonstrated.","tokens_in":18106,"tokens_out":4137,"duration_ms":56756,"concrete_test":"Recompute Tables 5-7 with per-model trusted thresholds selected on a validation split (e.g., 20% of the training partition) and then frozen when computing trusted metrics on the test split. If TER's trusted F1 no longer exceeds all baselines, or if trusted F1 no longer exceeds its own F1, the headline trusted-SOTA result is leakage-driven. Report the fraction of accepted samples at the chosen threshold as well.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The trusted-performance claim depends on thresholds chosen per model as the intersection of the trusted P-R curve with the y=x line (Section 3.3.2, Fig. 4, Table 7). The paper never restricts this choice to a validation set; the same test labels used to compute trusted precision/recall are used to select the threshold. Since every model is free to choose its own test-fitted threshold, the reported trusted F1 values (e.g., TER 0.7511 vs bc-LSTM 0.5642 on IEMOCAP) compare models at different operating points chosen with access to ground truth. This also explains the paper's unusual result that trusted F1 exceeds ordinary F1 for TER: a threshold tuned on test labels can always be moved to a favorable precision/recall trade-off, and the trusted F1 is then not a fixed-procedure metric. Without a held-out threshold selection step, the central claim that TER is the most reliable model is not supported by the tables.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TER, an end-to-end audio-visual emotion recognition model in which Video Swin Transformer and Multi-VGGish branches produce evidential outputs; a confidence module converts softplus evidence into belief masses and an uncertainty mass, and a combining-beliefs module fuses modalities via Dempster-Shafer combination. Training uses a 'trusted cross-entropy' loss that adds an uncertainty term. The main novelty is a trusted evaluation criterion: trusted precision/recall, trusted accuracy, and trusted F1 are computed from high-confidence predictions, with the confidence threshold chosen as the intersection of the trusted P-R curve with y=x. Experiments on IEMOCAP and Music-video report that TER reaches state-of-the-art trusted F1 (0.7511 and 0.9035) and high accuracy on Music-video (82.40%), and that the trusted loss improves SOTA models' trusted performance.","tokens_in":18375,"tokens_out":7833,"duration_ms":84161,"significance":"Confidence-based late fusion and reliability evaluation are timely for affective computing. The DS fusion module is a reasonable adaptation of prior work [12], and the paper explicitly aims to provide a new evaluation criterion. If the evaluation protocol were sound, the trusted F1 metric would be a useful complement to accuracy in high-stakes applications. The paper is clearly structured and reports ablations of modality, fusion, and loss components. However, the reported trusted metrics are invalidated by test-set threshold selection, and the classification SOTA comparison is too narrow; the current experimental evidence does not support the central claims. The proposed trusted loss and fusion ideas may still be valuable after a rigorous re-evaluation.","major_comments":[{"comment":"The trusted threshold is chosen per model as the intersection of the trusted P-R curve with the y=x line, but the P-R curve is built from test predictions and test labels. No validation split or fixed threshold is used. Consequently every trusted metric in Tables 2-7 is evaluated at a test-fitted operating point. This is label leakage: the threshold itself is a fitted parameter. It explains the surprising result that TER's trusted F1 (0.7511) exceeds its ordinary F1 (0.6007) and that the gaps over bc-LSTM (0.5642) are large. Please select thresholds on a held-out validation set (or fix a priori), report all trusted metrics at that threshold, and show error bars over seeds.","section":"Sec. 3.3.2, Fig. 4, Tables 5-7"},{"comment":"Eq. (10) defines Trusted Accuracy as HT/(HT+HF), which is identical to Trusted Precision in Eq. (7). Thus 'Trusted Acc.' in Tables 2-7 is not a new metric and carries no information beyond trusted precision. The paper should either acknowledge this equivalence or define a genuinely different trusted accuracy, e.g., the fraction of all test samples that are both correct and high-confidence.","section":"Sec. 3.3.3 and 3.3.4, Eqs. (7)-(10)"},{"comment":"The Music-video SOTA comparison includes only CIM, COGMEN, and MMGCN, three conversational models that were developed for text-based emotion recognition in conversation. The authors state that they disable the text feature input for these models. Those models are therefore evaluated outside their design setting, and no audio-visual or AVCA baseline is included. This does not support the claim that TER is state-of-the-art on Music-video (82.40%). The comparison should include recent audio-visual methods (e.g., [50]-[52] cited in Section 2) trained on the same visual+audio features and protocol.","section":"Sec. 4.3, Table 6"},{"comment":"The trusted cross-entropy loss includes a term -u_i log uhat_i for the ground-truth uncertainty u_i, but the paper never specifies how u_i is defined for labeled examples. With the usual one-hot labels u_i=0, this term vanishes and the loss reduces to CE on the belief masses; with u_i>0 it changes the target distribution. This underspecification affects the central training contribution in Table 4 and Fig. 5. Please define how u_i is constructed (label smoothing? estimated uncertainty?) and ablate the choice.","section":"Sec. 3.2, Eq. (5)"},{"comment":"All reported numbers are single runs with no standard deviations or significance tests. Differences such as TER Acc. 0.6014 vs DAG-ERC 0.6794 on IEMOCAP are large, but the claimed trusted-performance advantages need repeated runs, especially because the threshold selection step adds another source of variance. Report mean±std over at least 3-5 seeds and a paired significance test for the main claims.","section":"Sec. 4, Tables 2-7"}],"minor_comments":[{"comment":"Typos: 'Accuarcy' in the Section 3.3.4 heading and 'Performence' in the Section 4.4 heading.","section":"Sec. 3.3.4, Sec. 4.4"},{"comment":"Caption says 'branchs'; should be 'branches'.","section":"Fig. 4 caption"},{"comment":"The phrase 'dissolution research' should be 'ablation study'.","section":"Sec. 4.2.1"},{"comment":"The caption mentions 'six evaluation metrics,' but the table lists five performance metrics plus the threshold; the wording should be clarified.","section":"Table 7 caption"},{"comment":"The DS combination rule is presented in simplified form; please provide a direct citation or a short derivation to [12].","section":"Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The test-label threshold issue is the primary reason for the major revision. If a validation-set threshold yields smaller or reversed gaps, the paper's central claim would collapse; the authors should be required to rerun all trusted-metric experiments before acceptance. The Music-video comparison also needs appropriate audio-visual baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for passing along the TER paper. I read it through, and I can see why you flagged it. The headline claim—TER is the most reliable model on IEMOCAP and Music-video—doesn't hold up under the evaluation protocol. The trusted thresholds are selected on the test set, so the trusted metrics are effectively optimized for each model against the labels they're meant to predict. That alone makes the trusted-SOTA gap untrustworthy.\n\nWhat's actually new and worth credit: the paper applies Han et al.'s evidential confidence to audio-video emotion recognition with a Video Swin Transformer + Multi-VGGish front end. That specific combination isn't in the prior literature, and the authors do run ablations. In their setup, the combining-beliefs module beats early and fixed late fusion, and the motivating idea—flagging low-confidence predictions rather than forcing a label—is sound. They are also transparent about the threshold selection, which lets a reader see the problem instead of hiding it.\n\nThe soft spots, in order. The threshold is the big one. Section 3.3.2 describes choosing the threshold as the intersection of the trusted P-R curve with y=x, built from test predictions; Table 7 lists per-model thresholds. Nothing confines this to a validation set. It's a fitted parameter, so the trusted F1 values compare models at different, test-chosen operating points. That explains the odd result where trusted F1 exceeds ordinary F1 for TER—a threshold tuned on ground truth can be moved to a favorable spot.\n\nThere are smaller issues too. Eq. 10 defines Trusted Accuracy identically to Eq. 7 (Trusted Precision), so that extra metric is redundant. Eq. 5's Trusted CE adds -u_i log u_i, but ground-truth u_i is 0 for one-hot labels, so the term vanishes and the loss becomes ordinary cross-entropy. The Music-video SOTA comparison disables text inputs for the baselines, handicapping them, and the IEMOCAP baselines are all text-centric conversational models, not audio-video models. No error bars or significance tests. The paper isn't dishonest; it's just an evaluation loaded with choices that don't survive scrutiny.\n\nWho is this for? Affective computing researchers working on uncertainty-aware fusion might take the architectural idea and rerun the evaluation properly. I wouldn't cite the headline numbers. But the paper deserves a real referee—the problem is meaningful, and the flaws are the kind a careful reviewer can catch and the authors can fix. Send it to review, but expect a major-revision decision.","headline":"The trusted-SOTA claim is unsupported because the thresholds are fitted on test labels; the architecture is a reasonable application of evidential fusion, but the evaluation needs an overhaul.","tokens_in":18816,"tokens_out":2720,"would_cite":false,"duration_ms":31475,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multimodal emotion model hits 0.90 trusted F1 by fusing on uncertainty","keywords":["multimodal emotion recognition","evidential deep learning","confidence estimation","trusted evaluation","Dempster-Shafer fusion","uncertainty","trusted cross-entropy loss","IEMOCAP"],"falsifier":"Recompute the trusted metrics in Tables 5-7 using a single global threshold (e.g., $u=0.5$) or a threshold selected on a held-out validation split; if TER no longer leads in trusted F1 or its trusted F1 no longer exceeds its standard F1, the claimed trusted superiority is an artifact of per-model test-set threshold selection.","tokens_in":18031,"feed_emoji":"🎭","tokens_out":6613,"duration_ms":72166,"temperature":0.7,"pith_summary":"The paper proposes trusted emotion recognition (TER), a model that trains each audio and video branch to output an evidential confidence value alongside its class prediction, fuses the branches with a Dempster-Shafer combination rule, and trains with a trusted cross-entropy loss that rewards correct high-confidence predictions. It also introduces trusted precision and trusted recall, a trusted threshold chosen where the trusted precision-recall curve crosses the line $\\mathrm{TP}=\\mathrm{TR}$, and trusted accuracy and trusted F1 as evaluation metrics. On the IEMOCAP and Music-video benchmarks, TER is reported to match or beat state-of-the-art accuracy while achieving the best trusted F1 scores: 0.7511 and 0.9035, respectively.","feed_headline":"Multimodal emotion model hits 0.90 trusted F1 by fusing on uncertainty","feed_subtitle":"A confidence-aware loss and Dempster-Shafer fusion make high-confidence emotion predictions correct on two benchmarks.","key_machinery":"The evidential confidence module (based on the Dirichlet/evidence framework in [12]) computes belief mass $b_c$ and uncertainty $u$ from softplus-transformed logits, turning each branch's output into a subjective opinion. The Combining Beliefs module applies the Dempster-Shafer combination rule (Eq. 4) to fuse the two modalities. The trusted cross-entropy loss (Eq. 5) and the trusted threshold selection (intersection of the trusted P-R curve with $\\mathrm{TP}=\\mathrm{TR}$) carry the argument: they convert uncertainty estimates into a training signal and an evaluation protocol. Together they let the model prefer 'I don't know' over guessing, and let the evaluator measure predictions only wher","core_discovery":"At the center of TER is the replacement of the softmax classifier with an evidential output: a softplus layer produces Dirichlet evidence per class, from which a belief mass $b_c$ and uncertainty $u$ are computed. Each modality therefore reports not just a label but how unsure it is. A combining-beliefs module fuses the two modalities with the Dempster-Shafer combination rule (Eq. 4), which discounts conflicting evidence and lets the more confident modality dominate. The trusted cross-entropy loss (Eq. 5) includes the uncertainty term so the model is penalized for high-uncertainty wrong answers as well as ordinary classification error. For evaluation, the trusted threshold for each model is","pith_inferences":["The reported trusted metrics depend on fitting each model's threshold to test labels; a validation-set-selected threshold or a fixed global threshold would likely reduce the gap between TER and the baselines, so part of the claimed advantage may be a property of the evaluation protocol rather than the model alone.","Evidential confidence values from different architectures retrofitted with the same module may not be directly comparable, since no calibration step aligns their scales; the per-model thresholds in Table 7 mask this.","The same trust-aware loss and trusted-F1 objective could transfer to other multimodal tasks where silent wrong answers are costly, such as medical imaging or robot perception.","A cleaner head-to-head test would report trusted F1 at a predefined threshold (e.g., $u=0.5$) to decouple model quality from threshold optimization."],"forward_implications":["High-confidence predictions from TER are claimed to be correct more often than the model's overall accuracy, making the model usable selectively in risk-sensitive settings such as screening or triage.","The trusted-loss retrofit is nearly drop-in: swapping softmax for softplus and adding the uncertainty term to the loss improved trusted F1 of existing state-of-the-art models on IEMOCAP by an average of about 20% (Table 7).","The trusted evaluation criterion shifts model selection from raw accuracy to reliability: a slightly less accurate model with a higher trusted F1 becomes preferable for deployment.","Because the method is end-to-end and uses no manual feature engineering, it can be applied directly to raw video and audio, and the modular confidence scheme extends naturally to additional modalities."],"supporting_citations":[{"why":"Supplies the evidential confidence module (Dirichlet-based belief mass and uncertainty) and the Dempster-Shafer combination rule that TER extends to emotion recognition.","marker":"[12]"},{"why":"Video Swin Transformer is the visual feature extractor for the video branch.","marker":"[29]"},{"why":"VGGish is the audio feature extractor used inside the Multi-VGGish module.","marker":"[30]"},{"why":"Precision-recall analysis on binary classification is adapted to define trusted precision, trusted recall, and the trusted threshold.","marker":"[31]"},{"why":"IEMOCAP is one of the two benchmark datasets used for training and trusted-performance evaluation.","marker":"[32]"},{"why":"Music-video is the other benchmark dataset used for training and trusted-performance evaluation.","marker":"[33]"},{"why":"DAG-ERC is a state-of-the-art baseline whose trusted performance TER surpasses and whose trusted retrofit is measured in Table 7.","marker":"[38]"},{"why":"MMGCN is a state-of-the-art baseline used for comparison on both datasets and for trusted-loss generalizability testing.","marker":"[41]"}],"fun_headline_variants":["Uncertainty-driven fusion lifts emotion recognition to 0.90 F1","Evidential fusion with uncertainty beats baselines in emotion recognition","Dempster-Shafer fusion makes emotion model reliable at 0.90 F1","Confidence-aware multimodal fusion hits 0.90 trusted F1"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The trusted threshold is fit per model to the test-set trusted precision-recall curve, so the trusted metrics assume it is fair to choose each model's confidence cut-off with the test answer key in hand; if the cut-off were fixed or chosen on a validation split, the reported trusted-F1 gaps could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty-driven fusion lifts emotion recognition to 0.90 F1","Evidential fusion with uncertainty beats baselines in emotion recognition","Dempster-Shafer fusion makes emotion model reliable at 0.90 F1","Confidence-aware multimodal fusion hits 0.90 trusted F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000828,"raw_usage":{"total_tokens":3458,"prompt_tokens":753,"completion_tokens":2705,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":2625}},"tokens_in":497,"tokens_out":2705,"duration_ms":18500,"temperature":1.0,"reasoning_tokens":2625,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:57:59.691816+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the trusted metrics in Tables 5-7 using a single global threshold (e.g., $u=0.5$) or a threshold selected on a held-out validation split; if TER no longer leads in trusted F1 or its trusted F1 no longer exceeds its standard F1, the claimed trusted superiority is an artifact of per-model test-set threshold selection.","supporting_citations":[{"cited_title":"Trusted multi-view classification with dynamic evidential fusion,","cited_arxiv_id":null,"evidence_quote":"Supplies the evidential confidence module (Dirichlet-based belief mass and uncertainty) and the Dempster-Shafer combination rule that TER extends to emotion recognition."},{"cited_title":"Video swin transformer,","cited_arxiv_id":null,"evidence_quote":"Video Swin Transformer is the visual feature extractor for the video branch."},{"cited_title":"CNN architectures for large-scale audio classification,","cited_arxiv_id":null,"evidence_quote":"VGGish is the audio feature extractor used inside the Multi-VGGish module."},{"cited_title":"The relationship between precision-recall and roc curves,","cited_arxiv_id":null,"evidence_quote":"Precision-recall analysis on binary classification is adapted to define trusted precision, trusted recall, and the trusted threshold."},{"cited_title":"IEMOCAP: Interactive emotional dyadic motion capture database,","cited_arxiv_id":null,"evidence_quote":"IEMOCAP is one of the two benchmark datasets used for training and trusted-performance evaluation."},{"cited_title":"Deep learning-based late fusion of multimodal information for emotion classification of music video,","cited_arxiv_id":null,"evidence_quote":"Music-video is the other benchmark dataset used for training and trusted-performance evaluation."},{"cited_title":"Directed acyclic graph network for conversational emotion recognition,","cited_arxiv_id":null,"evidence_quote":"DAG-ERC is a state-of-the-art baseline whose trusted performance TER surpasses and whose trusted retrofit is measured in Table 7."},{"cited_title":"MMGCN: Multimodal fusion via deep graph convolution network for emotion recognition in conversation,","cited_arxiv_id":null,"evidence_quote":"MMGCN is a state-of-the-art baseline used for comparison on both datasets and for trusted-loss generalizability testing."}],"review_version":1}