{"id":"ac1a2524-f093-42b1-9a89-6c0b3e228b45","arxiv_id":"2505.18168","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"SEKE uses human emotion labels to prompt GPT-4o, filters its answers by resampling uncertain ones, and builds FEID, a multi-grained facial emotion instruction set that improves VLLM emotion perception.","lead":"Researchers used GPT-4o plus human emotion labels to automatically generate a training set with three levels of facial emotion descriptions, basic emotion, valence-arousal, and action units, then fine-tuned an open vision-language model on it. The resulting model beats several larger commercial and open models on a new facial emotion benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains may partly reflect LLM-based answer extraction bias: SEKE outputs are template-aligned by construction, while baseline free-form answers are parsed by an unvalidated LLM, which could depress baseline scores.","rationale":"The reader's conditional verdict is appropriate, and my concern is one of the two threats the reader named under 'weakest_assumption'. I focus on the LLM-based extraction step because it is the most direct threat to the central empirical claim: it affects every number in Table 2, it is asymmetric between SEKE and the baselines, and the paper supplies no validation of it. The same-source subject split is also a real concern, but it bears on generalization beyond FEAB, whereas extraction bias bears on whether FEAB itself was scored fairly. The metric definition in Section 4.2 also calls RMSE 'MAE', which further signals that the evaluation pipeline needs careful auditing before the headline comparisons are accepted. My recommendation is unchanged: conditional acceptance pending an independent check of the parsing/scoring pipeline and, ideally, release of the appendices, data, and code. I do not see grounds for rejection, because the central effect is internally consistent and the manual FEAB labels provide external grounding; the issue is verification, not demonstrated fraud or error.","tokens_in":14711,"tokens_out":8510,"duration_ms":92028,"concrete_test":"Select a random subset of 200 FEAB test items per model. Have two annotators, blind to model identity, read the raw model responses and independently record the stated discrete expression, valence/arousal values, and AU activation list using a pre-specified rule for absent or ambiguous AUs. Compute expression accuracy, AU F1, and valence/arousal MAE from the human-extracted values and compare them with the reported LLM-extracted values for the same items. If the SEKE-versus-GPT-4o gap changes by more than a few points on any metric after human extraction, the extraction step is a significant confound and the reported advantage is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing point is Section 4.2's evaluation pipeline: after inference, an LLM converts each model's free-form answer into a universal template before metrics are computed, with details deferred to the missing Appendix C. This conversion is not shown to be neutral. The SEKE model was fine-tuned on FEID, whose answers were generated using the same 11 rewrite templates and output format, so SEKE's responses should already match the extractor's expected schema. Baselines such as GPT-4o, LLaVA-OneVision, and Qwen2.5-VL were not trained on that format; the extractor must infer their numeric expression labels, valence/arousal values, and 17 AU flags from heterogeneous text, with no manual validation or inter-annotator check reported. If the extractor omits, misparses, or normalizes away baseline values, the headline gaps (61.1 to 69.4 accuracy, 29.0 to 54.9 AU F1, 0.342 to 0.295 valence MAE) would be inflated by format alignment rather than by emotion knowledge. The internal ablations in Table 4 share the same extraction pipeline, so they cannot rule out this bias. If this concern lands, it directly undermines the 'significantly outperforms' claim even on FEAB, independent of the subject-split question.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SEKE, a pipeline that uses GPT-4o to generate facial emotion instruction data by combining manual annotations from existing datasets (expression, valence/arousal, action units) with a knowledge-guided prompt and an uncertainty-aware Monte Carlo sampling scheme (SV-UAMC). The generated dataset, FEID, is used to fine-tune LLaVA-OneVision, yielding the SEKE model. The authors also create a benchmark, FEAB, with subject-independent splits from the same six source datasets, and report that SEKE outperforms closed- and open-source VLLMs on expression accuracy, average AU F1, and valence/arousal MAE on FEAB. Ablations compare FEID with prior instruction datasets and test the contributions of prior knowledge and SV-UAMC.","tokens_in":15015,"tokens_out":2990,"duration_ms":29717,"significance":"If the central claim holds, the paper makes a useful practical contribution: it provides a relatively low-cost method for generating multi-grained emotion instruction data and releases FEID and FEAB, which could support future work in affective VLLMs. The paper also ships a systematic ablation design (Tables 3-4 and Figure 4) that isolates the contribution of the generated data, and the idea of using epistemic-uncertainty-aware self-verification for data generation is interesting. However, the empirical support for the headline claim currently depends on an unvalidated LLM-based answer-extraction step whose details are deferred to a missing appendix, and on a single-run comparison without error bars or significance tests. The significance of the resource is real, but the reported performance gaps need stronger evaluation evidence.","major_comments":[{"comment":"The paper defines MAE as \"taking the square root of the average squared differences between predicted and actual values,\" which is the definition of RMSE, not MAE. Since all valence/arousal results (Tables 2-5 and Figure 5) are labeled MAE, the numbers may actually be RMSE values. This must be corrected, and the experiments should be rerun or re-reported with the true metric, or the text should explicitly state that RMSE is used.","section":"§4.2, Evaluation Metrics"},{"comment":"The evaluation pipeline uses an LLM to convert free-form model outputs into a universal template before computing metrics, and details are deferred to Appendix C, which is missing from the submission. This step is load-bearing: SEKE was fine-tuned on FEID answers generated with the same 11 rewrite templates and output format, so its outputs are expected to match the extractor's schema, while baselines such as GPT-4o, Qwen2.5-VL, and LLaVA-OneVision were not. If the extractor normalizes, omits, or misparses baseline answers, the reported gaps (e.g., 61.1 to 69.4 expression accuracy, 29.0 to 54.9 AU F1, 0.342 to 0.295 valence MAE) would be inflated by format alignment. The authors need to validate the extractor with manual inspection, report inter-annotator agreement or an error analysis, and release the extraction code so the comparison is neutral.","section":"§4.2 and Appendix C"},{"comment":"All results in Table 2 appear to come from a single evaluation run, yet the abstract and Section 4.3 state that SEKE \"significantly outperforms\" state-of-the-art methods. Without standard deviations over multiple runs or a significance test, the word \"significantly\" is not supported. At minimum, the authors should report variance across seeds for the SEKE model and for the strongest baselines.","section":"Table 2 and §4.3"},{"comment":"FEAB is constructed by holding out 1/10 of subjects from the same six datasets whose remaining images form FEID. Because FEID training images come from the same datasets, the model may learn dataset-specific statistics (lighting, camera, label distribution) rather than general emotion perception. The paper should provide cross-dataset evidence or an analysis controlling for this. In addition, the reliability experiment in §4.5.2 uses Aff-Wild2 samples with manual labels for all three descriptions; the text does not state whether these samples are disjoint from FEID training, and if they overlap, the reported accuracy of generated missing descriptions would be circular.","section":"§4.1 and §4.5.2"},{"comment":"The UAMC sampling procedure depends on several hand-set parameters: N=5, an initial sampling count of 2, and the acceptance probability p_acc = 1/2 + (1/2) U_bar. These choices are not justified by a sensitivity analysis or an ablation. Since the efficiency and reliability claims of SV-UAMC rest on these parameters, the authors should vary N and the acceptance rule to show that the reported gains are not tied to a particular setting.","section":"§3.2, Algorithm 1 and Eq. (4)"}],"minor_comments":[{"comment":"There is a typo: \"for metics computing\" should read \"for metrics computing.\"","section":"Section 4.2"},{"comment":"Several entries in Table 2 appear to be missing spaces or contain OCR artifacts (e.g., \"31.716.632.4\", \"69.969.38.2\", \"41 33\"). The table should be reformatted so each numeric column is unambiguous.","section":"Table 2"},{"comment":"References [24] and [25] cite the same paper (LLaVA-NeXT-Interleave) twice; one duplicate should be removed.","section":"References [24] and [25]"},{"comment":"The text references Appendices A-D (templates, sample distributions, extraction details, visualization examples), but none of these appendices are present in the submitted version. All referenced material must be included for the claims to be checkable.","section":"Appendices"},{"comment":"The phrase \"significantly outperforms\" in the abstract is not backed by any statistical test in the body; consider softening the wording until such tests are added.","section":"Abstract and Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The missing Appendix C and the unvalidated LLM-based answer extraction are the main blockers for me. If the extraction step is shown to be unbiased and the metric definition is corrected, the paper could become a solid contribution. I would also encourage the editor to ask the authors to clarify the overlap between the Aff-Wild2 reliability experiment and the FEID training set, since that is a potential circularity. The resource contribution (FEID/FEAB) is valuable, but the evaluation needs to be made trustworthy before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on arXiv:2505.18168. The core contribution is a data-generation pipeline: feed GPT-4o partial manual labels as prior knowledge, have it infer the missing emotion description (e.g., AUs from expression+VA), then run an uncertainty-aware Monte Carlo sampling scheme to decide when to re-sample. They release FEID, an instruction dataset covering all three annotation types, and FEAB, a benchmark with held-out subjects from the same six datasets. That is genuinely useful. The idea of using epistemic uncertainty to adaptively sample is reasonable and the ablations show both modules individually help. The dataset and benchmark are potentially reusable for affective computing.\n\nBut the evaluation has a load-bearing weak point. In Section 4.2 they compute metrics after running an LLM extractor that converts each model's free-form output into a universal template. They do not show that the extractor is neutral. SEKE is fine-tuned on FEID, which was written using the same 11 rewrite templates, so its outputs are likely already in the expected schema. Baseline models like GPT-4o or Qwen2.5-VL were not trained on that format; the extractor has to parse their heterogeneous free-form text. No manual validation or inter-annotator agreement is reported. If the extractor drops or misparses baseline outputs, the headline gaps (61.1→69.4 accuracy, 29.0→54.9 AU F1) are inflated by format alignment, not emotion knowledge. The internal ablations share the same pipeline, so they don't rule this out. Appendix C, where the extraction details would presumably live, is missing from this version. That is not just a missing appendix; it is the key to trusting the main claim.\n\nOther soft spots: the MAE is described as the square root of the average squared differences, which is RMSE, not MAE. No error bars or significance tests anywhere. The Aff-Wild2 reliability check masks one label type and generates it from the other two—reasonable, but it is the same correlation prior the method injects, so it tells you less than the paper implies. The subject split for FEAB is fine; holding out subjects from the same datasets is standard, though it does not test cross-dataset transfer.\n\nWho should read this: someone working on affective instruction data or VLLM evaluation for emotion tasks. The recipe is worth trying even if the numbers need re-validation. I'd send it to review, but with a strong request to release the data, the code, and a validated extractor, and to fix the metric definition.","headline":"A useful recipe for generating multi-grained emotion instruction data, but the main empirical claim is undermined by an unvalidated LLM extraction step that likely favors the fine-tuned model.","tokens_in":15551,"tokens_out":3406,"would_cite":false,"duration_ms":29841,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a vision LLM on self-verified three-level emotion instruction data beats GPT-4o on all three facial emotion tasks.","keywords":["facial emotion analysis","vision large language model","instruction data generation","action unit detection","valence-arousal estimation","epistemic uncertainty","Monte Carlo sampling","self-verification"],"falsifier":"Re-run the SEKE model on a facial emotion test set drawn from sources with no subject or image overlap with the six datasets used to build FEID; if the margin over GPT-4o collapses, the gains are dataset-specific. Separately, parse the same model outputs with two independent extraction methods, a deterministic regex parser and a different LLM, and compare the metrics; material discrepancies would show that the evaluation pipeline, not the model, produced part of the reported advantage.","tokens_in":14525,"feed_emoji":"😊","tokens_out":6703,"duration_ms":59125,"temperature":0.7,"pith_summary":"The paper tries to establish that a vision large language model can acquire fine-grained facial emotion perception without expensive expert annotation. Its pipeline, SEKE, uses a closed-source VLLM to fill in missing emotion labels, guided by existing human annotations and by the correlations among three description levels: discrete expression, valence-arousal, and action units. An uncertainty-aware Monte Carlo sampling step filters the most reliable predictions and feeds them back into a summary prompt, producing the FEID instruction dataset. Fine-tuning LLaVA-OneVision on FEID is reported to surpass GPT-4o and six open-source models on the FEAB benchmark across expression classification, action-unit detection, and valence/arousal estimation. If these results hold, they indicate that data completeness and correlation reasoning, rather than model scale, are the deciding factors for VLLM emotion understanding.","feed_headline":"Fine-tuned VLLM beats GPT-4o on facial emotion analysis","feed_subtitle":"A self-verification loop turns sparse human labels into three-level emotion instructions that lift accuracy by 8.3 points.","key_machinery":"The load-bearing mechanism is the inherent correlation among the three emotion description levels, used as a generative constraint. Given partial manual labels, SEKE prompts GPT-4o to predict the missing labels while reasoning through these correlations, and each FEID sample records all three descriptions plus correlation reasoning. Reliability is enforced by SV-UAMC: the model samples repeatedly, starting at two and capped at five, estimates epistemic uncertainty per task from sample variance, and probabilistically continues sampling when the maximum normalized uncertainty is high; the collected predictions and uncertainty statistics are then summarized into a final instruction. Fine-tuning LLaVA-OneVision on the 26,238 generated samples is the training mechanism that turns this data into a perception model.","core_discovery":"The central claim is that comprehensive, correlated instruction data is what unlocks emotion perception in VLLMs. On the FEAB benchmark, the SEKE fine-tuned model reaches 69.4% expression accuracy, 54.9% average action-unit F1, and valence and arousal MAE of 0.295 and 0.231, against GPT-4o's 61.1%, 29.0%, 0.342, and 0.291. Ablations show that dropping either the human prior-knowledge guidance or the uncertainty-aware self-verification step lowers performance, and that training on any single description type degrades all three tasks. The paper reads this as evidence that a model learns to triangulate coarse and fine cues when every training sample carries all three descriptions plus a reasoning sentence connecting them.","pith_inferences":["The same fill-in-the-missing-labels loop should transfer to other domains where multiple correlated annotation levels exist, such as structured medical image findings, provided one level is cheap to obtain.","Because GPT-4o both generated the data and is beaten by the fine-tuned model, the pipeline suggests that diversity and correlation structure in synthetic instruction data can elicit capabilities the generator cannot reliably exhibit zero-shot; that is a testable generalization.","The subject-holdout split still shares image provenance with training data, so evaluating the SEKE model on an entirely unrelated emotion dataset would clarify whether the gains are genuine perception or dataset statistics.","The uncertainty-aware acceptance rule is a general-purpose quality filter: using a different closed-source generator or a larger sampling cap would show whether the filter's benefit scales."],"forward_implications":["On FEAB, the SEKE-trained LLaVA-OneVision beats GPT-4o by 8.3 points in expression accuracy, 25.9 points in average action-unit F1, and lowers valence/arousal MAE by 0.047/0.060.","Ablation against alternative instruction datasets (EmoVIT, MAFW, MERR-Fine) shows FEID produces the strongest downstream model, implying existing emotion instruction data lacks enough fine-grained cues.","Removing either prior-knowledge guidance or SV-UAMC from the generation pipeline measurably hurts the fine-tuned model, so both components carry the reported gain.","Training on only one of the three description types declines on all tasks, implying the correlation knowledge, not any single label type, drives the improvement.","On Aff-Wild2, SEKE-generated missing labels align with manual labels better than direct GPT queries, with a 16.9% action-unit F1 improvement."],"supporting_citations":[{"why":"provides the LLaVA-OneVision base model that SEKE fine-tunes on FEID.","marker":"[23]"},{"why":"is the closed-source VLLM used to generate FEID and the strongest comparison baseline.","marker":"[41]"},{"why":"supplies the multi-task expression, valence-arousal, and action-unit annotations used for FEID, FEAB, and the masking reliability test.","marker":"[22]"},{"why":"supplies the peak-expression frames used for FEID and FEAB.","marker":"[36]"},{"why":"supplies in-the-wild expression images used for FEID and FEAB.","marker":"[27]"},{"why":"supplies expression and valence-arousal annotations used for FEID and FEAB.","marker":"[38]"},{"why":"supplies action-unit intensity annotations used for FEID and FEAB.","marker":"[37]"},{"why":"supplies dense action-unit annotations used for FEID and FEAB.","marker":"[57]"},{"why":"Emotion-LLaMA is the closest emotion-instruction competitor and defines a baseline that FEID must beat.","marker":"[9]"},{"why":"EmoVIT is the key instruction-dataset baseline in the ablation comparison.","marker":"[52]"}],"fun_headline_variants":["Self-verification boosts VLLM facial emotion perception","SEKE outdoes GPT-4o on emotion benchmark","Emotion instruction data lifts VLLM accuracy by 8.3 points","Three-level emotion labels improve vision LLM analysis","Uncertainty-aware sampling refines VLLM emotion labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation's fairness rests on the assumption that holding out one tenth of subjects from the same six datasets that supplied FEID training images removes dataset-specific transfer, and that the LLM used to extract numeric answers from free-form model text is unbiased; if either fails, the reported superiority could be an artifact of the test setup rather than of the generated instructions.","fun_headline_variants_meta":{"raw":{"variants":["Self-verification boosts VLLM facial emotion perception","SEKE outdoes GPT-4o on emotion benchmark","Emotion instruction data lifts VLLM accuracy by 8.3 points","Three-level emotion labels improve vision LLM analysis","Uncertainty-aware sampling refines VLLM emotion labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000957,"raw_usage":{"total_tokens":4084,"prompt_tokens":957,"completion_tokens":3127,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":3043}},"tokens_in":573,"tokens_out":3127,"duration_ms":22603,"temperature":1.0,"reasoning_tokens":3043,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:40:01.212322+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the SEKE model on a facial emotion test set drawn from sources with no subject or image overlap with the six datasets used to build FEID; if the margin over GPT-4o collapses, the gains are dataset-specific. Separately, parse the same model outputs with two independent extraction methods, a deterministic regex parser and a different LLM, and compare the metrics; material discrepancies would show that the evaluation pipeline, not the model, produced part of the reported advantage.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the peak-expression frames used for FEID and FEAB."},{"cited_title":"Affectnet: A database for facial expression, valence, and arousal computing in the wild.IEEE Transactions on Affective Computing, 10(1):18–31, 2017","cited_arxiv_id":null,"evidence_quote":"supplies expression and valence-arousal annotations used for FEID and FEAB."},{"cited_title":"Disfa: A spontaneous facial action intensity database.IEEE Transactions on Affective Computing, 4(2):151–160, 2013","cited_arxiv_id":null,"evidence_quote":"supplies action-unit intensity annotations used for FEID and FEAB."},{"cited_title":"A high-resolution spontaneous 3d dynamic facial expression database","cited_arxiv_id":null,"evidence_quote":"supplies dense action-unit annotations used for FEID and FEAB."}],"review_version":1}