{"id":"17dc4341-f116-4771-a9a4-e56e9028a0c1","arxiv_id":"2505.13419","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FEALLM is a multimodal LLM fine-tuned on a new, aligned facial expression and action unit reasoning dataset, reporting improved facial emotion analysis on its benchmark and zero-shot gains on RAF-DB, AffectNet, BP4D, and DISFA.","lead":"The authors built a new instruction dataset and benchmark for teaching multimodal AI models to read facial expressions and the small facial muscle movements behind them, and designed a model called FEALLM that extracts local face details. The model reports better accuracy on emotion recognition and action unit detection than several large vision-language models, including in zero-shot tests on standard datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPT-4o instruction text is generated from ground-truth FE/AU labels; if not visually grounded, claimed emotional reasoning is label-derived, and FEABench AU gains may reflect label co-occurrence shortcuts rather than true AU perception.","rationale":"I read the paper as claiming two things: (1) FEALLM sets the best performance on FEABench and shows impressive zero-shot generalization, and (2) the mechanism is emotional synergy and causal reasoning between AU and FE. The first claim is supported by controlled comparisons against the LLaVA-1.5-7B LoRA baseline, but the second rests entirely on the GPT-4o text generation. The paper's own Section 3.1 establishes that GPT-generated labels can be unreliable, yet the free-text descriptions receive no equivalent scrutiny. The reader's weakest_assumption identifies exactly this issue, and I agree. The concrete test of regenerating instruction data without the image would directly show whether the image contributes to the text; if the label-only model performs equally, the reasoning is label-derived and the mechanism claim fails, even if some performance numbers survive. Because the reader's CONDITIONAL verdict already accounts for this uncertainty and the paper could address it with additional experiments, I recommend UNCHANGED rather than moving the verdict. This is a scientific concern about the data-generation protocol, not a judgment about the authors.","tokens_in":16270,"tokens_out":14156,"duration_ms":135148,"concrete_test":"Regenerate the FEA instruction data using the identical GPT-4o prompt template but omit the <Image> input, so GPT-4o receives only the FE and AU labels and must generate the facial movement and emotion reasoning descriptions without seeing the face. Use this label-only version to fine-tune the same FEALLM architecture with the same hyperparameters and data splits as in the paper. Compare against the original image-conditioned model on FEABench FER accuracy, AU average F1, and the zero-shot BP4D/DISFA AU F1 scores. If the label-only model performs within a small margin (e.g., 1.5% AU F1) of the image-conditioned model, the generated text is not visually grounded and the reasoning/synergy claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is the FEA instruction data pipeline in Section 3.2. GPT-4o is prompted with both the image and the ground-truth FE and AU labels, and the resulting 'facial movement description' and 'emotion reasoning description' are used as supervision. The only validation step is checking consistency with the original annotations, not visual grounding of the generated text. Section 3.1 itself warns that GPT-generated annotations are unreliable (EmoLA's FE accuracy about 91%, AU F1 about 76%, AU10 58.1%), but no similar manual review is applied to GPT-4o's free-text descriptions. Consequently, the text may describe movements that are plausible given the labels but not actually present in the image. Fine-tuning on such text can teach the model to produce fluent reasoning that is label-derived rather than image-grounded. Because FEABench's evaluation set is sampled from the same Aff-Wild2 source, FE-AU co-occurrence statistics are shared between training and evaluation, so AU F1 gains can be inflated by learning those statistics from the label-conditioned text rather than by genuinely perceiving AUs in novel images. This directly threatens the central claims of emotional synergy/reasoning and of robust zero-shot generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FEALLM, a multimodal large language model for joint facial expression recognition (FER) and action unit detection (AUD). The authors construct an FEA instruction dataset from Aff-Wild2 (14,892 training images, 1,335 test images from disjoint subjects), using GPT-4o prompted with both the face image and ground-truth FE/AU labels to generate three types of instruction text: emotion summaries, facial movement descriptions, and emotion reasoning descriptions. They also introduce FEABench, a benchmark that evaluates FER and AUD jointly on the held-out portion of this data. FEALLM augments LLaVA-1.5-7B with a Local Clue Aggregator (LCA) that processes 16 cropped facial regions and a Multi-perspective Projector (MPP) that fuses shallow visual-encoder features with local features via cross-attention. Fine-tuning only the LCA, MPP, and LoRA modules, the model achieves 67.36% FER accuracy and 59.60% average AU F1 on FEABench, and is evaluated zero-shot on RAF-DB, AffectNet, BP4D, and DISFA. Ablations show positive contributions from the instruction types and from the LCA and MPP modules.","tokens_in":16569,"tokens_out":5033,"duration_ms":46241,"significance":"If the results are validated under a fairer comparison protocol, the paper would make a useful contribution by providing a joint FER/AUD instruction dataset and benchmark, and by demonstrating that MLLMs can be adapted to facial detail perception through local crops and shallow-feature fusion. The dataset and code release are positive; the claim to be the first to establish causal reasoning between AUs and FEs in instruction data, if properly supported, would be of interest to the affective computing and MLLM communities. However, the central claims of genuine emotional reasoning and impressive zero-shot generalization are currently not fully supported by the experimental design, and several load-bearing evaluation choices need to be addressed before the paper's main conclusions can be accepted.","major_comments":[{"comment":"The FEA instruction data generation pipeline uses GPT-4o with both the image and the ground-truth FE/AU labels as input, and the only validation step is checking consistency between the generated text and the labels. Section 3.1 itself warns that GPT-generated annotations are unreliable (e.g., EmoLA's AU F1 of 76% and AU10 F1 of 58.1%), yet no manual review or visual-grounding check is applied to GPT-4o's free-text facial-movement and reasoning descriptions. If the generated descriptions contain plausible but image-inconsistent statements (e.g., 'raised cheeks' when AU6 is annotated but not visibly present), fine-tuning on this text can teach the model to produce label-derived rationales rather than perceptually grounded reasoning. Because the FEABench evaluation set is sampled from the same Aff-Wild2 source as the training set, shared FE-AU co-occurrence statistics could inflate AUD gains without genuine AU perception. This directly threatens the paper's central claims of emotional synergy and causal reasoning, and the authors should provide evidence that the generated text is visually grounded, for example by human evaluation on a sampled subset or by an image-only generation setting without label prompting.","section":"Section 3.2, Figure 3"},{"comment":"The FEABench comparison is mostly against zero-shot MLLMs, which is not an apples-to-apples baseline for a model fine-tuned on the proposed FEA instruction dataset. The only fine-tuned baseline is LLaVA-1.5+LoRA. To support the claim that FEALLM is the best method on FEABench and that the LCA and MPP modules are responsible for the gains, the authors should fine-tune other MLLMs (e.g., Qwen-VL-Chat, InternVL2, MiniCPM-o) on the same FEA instruction data under the same training protocol. Without such comparisons, the margins in Table 1 may largely reflect the presence or absence of instruction tuning rather than the proposed architecture.","section":"Section 5.2, Table 1"},{"comment":"No error bars, multiple seeds, or significance tests are reported for any experiment. This is particularly important because training runs for only one epoch, the fine-tuned baseline margins are small (e.g., FEABench average AU F1 59.60 vs. 55.79; BP4D average 45.98 vs. 45.51), and the zero-shot AUD results show the fine-tuned LLaVA+LoRA baseline outperforming FEALLM on several individual AUs (e.g., BP4D AU6 74.83 vs. 72.90, AU12 85.58 vs. 84.88; DISFA AU12 60.20 vs. 54.50). The authors should report mean and standard deviation over at least three independent fine-tuning runs, or provide a clear justification for why a single run is sufficient.","section":"Section 5.2, Tables 1, 3, 4, 5"},{"comment":"The zero-shot AUD evaluation on BP4D and DISFA uses 2% uniform frame sampling and only evaluates the AUs shared with the FEA instruction dataset. This protocol deviates from standard practice in the AUD literature, which typically reports frame-level F1 on the full test set, and it makes the numbers in Table 5 not directly comparable with established AUD baselines. The authors should either adopt the standard evaluation protocol or clearly state the deviation and its implications in the main text. In addition, given that the fine-tuned LLaVA+LoRA baseline beats FEALLM on several AUs, the paper's phrase 'impressive generalization capability' is not sufficiently supported","section":"Section 5.3, Table 5"},{"comment":"The claim that adding emotion reasoning instructions yields 'substantial progress' in both FER and AUD is overstated: the gain is +3.71 accuracy points for FER (63.65 to 67.36) but only +0.81 F1 points for AUD (58.79 to 59.60). Moreover, the ablation adds reasoning text as an additional training signal, so it does not isolate the effect of reasoning supervision from the effect of simply having more instruction examples. A control condition that adds an equal amount of non-reasoning instruction text (e.g., more emotion-summary or facial-movement templates) is needed to support the claim that reasoning instructions, not just data quantity, drive the improvement.","section":"Section 5.4, Table 3"}],"minor_comments":[{"comment":"The paper states that 'more than ten question templates' are used per instruction type and refers to the Appendix for details, but the submitted version does not include an appendix with these templates or the cropping illustration, which limits reproducibility of the dataset construction.","section":"Section 3.2"},{"comment":"The pre-training dataset is referred to as 'LLaVA-LCS-558K'; this appears to be a typo for 'LLaVA-558K' or 'LLaVA-LCS-558K' depending on the actual split, and should be corrected for clarity.","section":"Section 5.1"},{"comment":"The evaluation metric for AUD is described as F1, but the paper does not specify how per-AU F1 scores are aggregated into the reported average (macro-average over AUs is implied but should be stated explicitly), nor how the regular-expression extraction handles AUs that are mentioned in negation or absent.","section":"Section 3.3"},{"comment":"There is a grammatical error in the caption and example text: 'a open mouth' should be 'an open mouth'.","section":"Figure 7"},{"comment":"The learnable scaling factors gamma_1 and gamma_2 are introduced but their initialization is not specified; please state the initial values and whether they are scalar or per-channel parameters.","section":"Section 4.3, Eq. (6)-(7)"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a useful dataset/benchmark and a reasonable architecture, but the evaluation falls short of supporting the headline claims. The most important issue is the absence of a visual-grounding check for the GPT-4o-generated instruction text, which bears directly on the 'causal reasoning' contribution; the second issue is the weak baseline comparison. I would encourage the editor to ask for the missing appendix material as well, since the submission references it but does not include it. The novelty claim of being 'the first' to introduce FE-AU causal reasoning should also be carefully checked against Emotion-LLaMA and EmoLLM, which already use reasoning/chain-of-thought for FER, even if not jointly with AUD."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on FEALLM. It is a reasonable empirical contribution. The authors build a new FE+AU instruction dataset from Aff-Wild2, create a joint benchmark (FEABench), and propose a LLaVA-based architecture with a local-clue aggregator (LCA) and a multi-perspective projector (MPP). The ablations are clean: both modules help, and adding the reasoning instructions improves over summary+movement alone. The zero-shot results on RAF-DB, AffectNet, BP4D, and DISFA are external and give independent support to the generalization claim. Gains over the fine-tuned LLaVA-1.5+LoRA baseline are modest (67.36 vs 59.97 FER accuracy; 59.60 vs 55.79 AU F1) but consistent across tasks.\n\nThe soft spots are real but not fatal. No error bars or multiple seeds are reported, and the fine-tuned baseline is only one architecture. The zero-shot AUD protocol uses 2% frame sampling and only shared AUs; this is disclosed and not unreasonable, but it limits comparability. The larger issue is the instruction-data pipeline: GPT-4o is prompted with the image plus the ground-truth FE/AU labels, and the generated emotion reasoning text is validated only for consistency with those labels, not for visual grounding. So the \"causal reasoning\" claim is not supported by the evidence. The text is label-derived, and the model likely learns to produce fluent post-hoc rationales rather than true causal chains. That is a limitation of the reasoning claim, not a fatal flaw: at inference the model sees only the image, so the FER/AUD perception results remain meaningful. The FEABench AU gains could partly reflect label co-occurrence statistics shared with the training set, but the external zero-shot benchmarks mitigate that worry.\n\nThe paper overstates its reasoning contribution. It should be asked to temper that language and add variance reporting. The dataset and benchmark, if released, will be useful to the affective computing community. I would send it to peer review rather than desk reject it; with revisions on the claims and evaluation rigor, it could be a solid contribution.\n\nRecommendation: engage with it, but push for the release of data/code and for more careful wording on causal reasoning.","headline":"Useful FE+AU instruction dataset, joint benchmark, and architecture with modest but consistent gains; the causal reasoning claim outruns the evidence.","tokens_in":17133,"tokens_out":3043,"would_cite":true,"duration_ms":30079,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FEALLM claims that a multimodal LLM trained on aligned FE and AU instruction data with explicit reasoning, plus local-detail and shallow-feature fusion, outperforms existing MLLMs on facial emotion analysis and generalizes zero-shot.","keywords":["facial emotion analysis","multimodal large language model","action unit detection","facial expression recognition","instruction tuning","emotion reasoning","zero-shot generalization","FEABench"],"falsifier":"Take a fixed set of face images and feed the text generator deliberately wrong AU labels while keeping the image unchanged; if the generated reasoning text follows the wrong labels fluently and plausibly, then the reasoning is label-conditioned rather than image-grounded, and a model trained on it will inherit that spurious link. A second check is to have human raters mark whether each AU mentioned in the reasoning text is actually visible in the image; systematic mismatches would falsify the grounding claim.","tokens_in":16087,"feed_emoji":"😊","tokens_out":6680,"duration_ms":60576,"temperature":0.7,"pith_summary":"The paper sets out to show that a multimodal large language model can perform facial emotion analysis, recognizing both coarse expressions and fine-grained action units, when trained on instruction data that explicitly connects the two. It builds a facial emotion analysis instruction dataset from expert-annotated images, adds generated reasoning text that derives an emotion from its action units, and introduces FEABench to evaluate both tasks together. It then proposes a model architecture that feeds local face crops and shallow visual features into the language model, arguing that these details let the model link muscle movements to emotions. A sympathetic reader would care because the claim, if true, moves MLLMs from black-box expression classification toward interpretable, causal emotion reasoning that transfers to new datasets.","feed_headline":"Facial emotion model ties AUs to expressions, beating prior MLLMs","feed_subtitle":"Reasoning text plus local facial features lift both expression accuracy and AU detection, even zero-shot.","key_machinery":"The central machinery is the pairing of an instruction dataset with a detail-capturing model. The Local Clue Aggregator (LCA) crops each face into sixteen local regions (eight directions at half and three-quarters of the side length), runs each through a convolutional block, applies self-attention over the region tokens, and projects them into the LLM embedding space. The Multi-perspective Projector (MPP) takes feature maps from five layers of the visual encoder, uses the deepest as query and the shallower ones as key and value in a cross-attention step, fuses those with the LCA's local features through a second cross-attention with a learnable residual, and aligns the result with a two-layer MLP. The claim carried by these modules is that low-level texture and local muscle regions, which global visual features suppress, are precisely the evidence needed to reason from action units to emotions.","core_discovery":"On the proposed FEABench, the model reaches 67.36% accuracy on facial expression recognition and 59.60% average F1 across twelve action units, outperforming the same base model fine-tuned on the same instruction data without the new modules (59.97% and 55.79%). In zero-shot settings it reaches 69.95% on RAF-DB and 41.91% on AffectNet, and average AU F1 of 45.98% on BP4D and 42.61% on DISFA. The paper attributes these gains to two mechanisms: instruction data that includes emotion-reasoning descriptions bridging AUs to emotions, and an architecture that adds local facial details and low-level encoder features to the global visual tokens. It also reports that the reasoning instruction component alone lifts FER accuracy from 63.65% to 67.36% and AU F1 from 58.79% to 59.60%, and that removing any cropping direction or shallow/local feature in the modules hurts both tasks.","pith_inferences":["A natural stress test is to swap the AU labels supplied to the text generator while keeping the image fixed; if generated reasoning follows the wrong labels fluently, the model is learning label-conditioned scripts rather than image-grounded causes.","The reasoning text may also be replaceable by deterministic templates that map each AU set to canned emotion chains; matching FEALLM's gains with templates would show the content, not the generation model, is what matters.","The LCA/MPP recipe suggests a general design principle for MLLMs on fine-grained recognition: explicitly crop salient regions and re-inject shallow features instead of relying on global attention to discover small visual clues.","FEABench could be extended to score reasoning consistency, checking that the emotion the model states is logically entailed by the AUs it reports, so that emotional reasoning is measured rather than just fluency."],"forward_implications":["Joint instruction tuning with aligned FE and AU descriptions lets one model handle both expression recognition and action-unit detection, so separate task-specific heads are not required.","Emotion-reasoning text contributes beyond the underlying labels alone: adding it improves both FER and AUD, suggesting that explicit muscle-to-emotion chains are a useful training signal.","Local crops and shallow visual features each add independent gains, so detail injection is a viable way to improve MLLMs on fine-grained visual tasks.","Zero-shot results on four external datasets indicate that this style of instruction tuning transfers to new subjects and new annotation schemes without task-specific fine-tuning."],"supporting_citations":[{"why":"Supplies the Aff-Wild2 images and expert-verified FE and AU labels from which the training and evaluation splits are sampled.","marker":"[18]"},{"why":"Provides the generator model used to produce emotion summaries, facial movement descriptions, and reasoning text from image plus labels.","marker":"[1]"},{"why":"Defines the base LLaVA-1.5-7B architecture and pretraining alignment data that FEALLM initializes from and that the LoRA baseline uses.","marker":"[30]"},{"why":"Provides the CLIP ViT-L/14 visual encoder whose shallow and deep feature maps feed the Multi-perspective Projector.","marker":"[39]"},{"why":"Provides the Vicuna-7B language model that is frozen and adapted with LoRA during instruction fine-tuning.","marker":"[10]"},{"why":"Supplies the low-rank adaptation method used to fine-tune the frozen LLM efficiently.","marker":"[14]"},{"why":"Defines the Facial Action Coding System whose AU labels and muscle-movement semantics underpin the dataset and reasoning text.","marker":"[13]"},{"why":"Contrasts prior instruction tuning with misaligned FE and AU labels, motivating the aligned, joint dataset in this paper.","marker":"[26]"}],"fun_headline_variants":["Facial emotion AI links expressions to action units, boosts zero-shot","New model ties facial expressions to AUs, excels on FEABench","Emotion reasoning in MLLMs lifts AU detection and expression accuracy","FEALLM: causal emotional reasoning improves facial emotion analysis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the text generated by GPT-4o, given only the image and its correct labels, is genuinely grounded in what is visible in the face and causally faithful to how AUs produce emotions; if the reasoning is fluent but merely restates the labels, the claimed emotional synergy and reasoning would not be real.","fun_headline_variants_meta":{"raw":{"variants":["Facial emotion AI links expressions to action units, boosts zero-shot","New model ties facial expressions to AUs, excels on FEABench","Emotion reasoning in MLLMs lifts AU detection and expression accuracy","FEALLM: causal emotional reasoning improves facial emotion analysis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1633,"prompt_tokens":1009,"completion_tokens":624,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":550}},"tokens_in":625,"tokens_out":624,"duration_ms":6311,"temperature":1.0,"reasoning_tokens":550,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:13:35.656071+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed set of face images and feed the text generator deliberately wrong AU labels while keeping the image unchanged; if the generated reasoning text follows the wrong labels fluently and plausibly, then the reasoning is label-conditioned rather than image-grounded, and a model trained on it will inherit that spurious link. A second check is to have human raters mark whether each AU mentioned in the reasoning text is actually visible in the image; systematic mismatches would falsify the grounding claim.","supporting_citations":[],"review_version":1}