Pith. sign in

REVIEW 4 major objections 5 minor 29 references

AU-LLM: Micro-Expression Action Unit Detection via Enhanced LLM-Based Feature Fusion

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read The paper claims that a large language model, fed a single fused visual token that combines mid-level texture and high-level semantic features, can outperform prior methods on micro-expression Action Unit detection, reaching 81.4% average…

desk verdict A useful empirical first for micro-expression AU detection with an LLM, but the paper never isolates the LLM's contribution, so the SOTA claim rests on a missing ablation. read the letter →

arxiv 2507.21778 v1 pith:3QYLDQKW submitted 2025-07-29 cs.CV

classification cs.CV
keywords Micro-expressionActionUnitdetectionLargeLanguageModelFeatureFusionLoRACASMEIISAMMAffectiveComputing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a large language model can serve as a reasoning engine for micro-expression Action Unit (AU) detection, a task where subtle, fleeting facial muscle movements make ordinary classifiers struggle. To do this it introduces AU-LLM, which fuses mid-level local-texture features and high-level semantic features from a 3D-CNN backbone into a single visual token and feeds that token to a LoRA-tuned LLM. The authors report that this design beats previous state-of-the-art methods on two standard benchmarks, reaching 81.4% average F1 on CASME II (prior best 78.4%) and 61.9% on SAMM (prior best 57.3%), and that it also generalizes under cross-domain protocols. If true, this would establish LLM-based reasoning as a viable route for fine-grained, low-intensity facial analysis rather than a technique limited to high-level visual understanding.

What carries the argument

The load-bearing mechanism is the Enhanced Fusion Projector (EFP), an MLP that maps the concatenation of flattened mid-level and high-level visual features to a single information-dense token in the LLM's embedding space. The EFP is what bridges the vision-language semantic gap: a linear projection would discard low-intensity local texture cues, while the MLP's non-linear transformation preserves both local textural detail (from Fmid) and global semantic context (from Fhigh) in one vector. Around it sit the LED temporal-filtering module that amplifies subtle motion, a 3D-CNN backbone with a Squeeze-and-Excitation channel recalibration layer, LoRA fine-tuning of the LLM's attention projections, and the Asymmetric Loss that down-weights negative samples. The final classification is read off the hidden state of the last token, so the entire pipeline is trained end-to-end for multi-label AU prediction.

What would settle it

Run the same LOSO protocol on CASME II with the backbone and EFP intact but swap the LoRA-tuned LLM for a multi-layer perceptron or linear classifier trained on the identical fused token; if average F1 matches or exceeds 81.4%, the paper's central claim that LLM reasoning drives the improvement collapses.

Watch

Extended reading notes

Core claim

The central discovery is that micro-expression AU detection can be reframed as a vision-language reasoning problem, provided the visual representation is distilled correctly. The paper's AU-LLM pipeline extracts temporally filtered spatio-temporal features with a 3D-CNN whose LED module amplifies onset-apex-offset motion patterns, then forms a compact token by concatenating mid-level features (local texture) with high-level features (global semantics) and passing the concatenation through an MLP. This fused token is prepended to a text prompt such as 'Analyze the facial features to classify action units' and processed by a 1.5B-parameter LLM fine-tuned with LoRA; the last token's hidden state feeds a linear AU classifier. With DeepSeek-R1-Distill-Qwen-1.5B as the LLM, the method reports 81.4% average F1 on CASME II and 61.9% on SAMM, and in bidirectional cross-domain tests it outperforms LED-based baselines by 13.4 and 6.7 percentage points respectively. The paper attributes these gains to the EFP's non-linear multi-level fusion, the LLM's reasoning, and the asymmetric loss that counters AU class imbalance.

Load-bearing premise

The claim that the LLM's reasoning is what lifts performance rests on an untested assumption: no experiment in the paper replaces the LLM with an ordinary classifier fed the same fused token, so the fused representation alone might explain the gains.

Editorial extensions

If this is right

  • LLM-based reasoning is a workable paradigm for low-intensity, fine-grained facial analysis, not just macro-expression recognition, opening micro-expression AU detection to the tools of multimodal LLMs.
  • Fusing mid-level texture with high-level semantic features beats using either alone in this framework, so future AU detectors should preserve local detail when adapting visual features for language models.
  • A non-linear fusion projector gives a clear gain over a linear projector (the EFP− variant), implying that the vision-to-language mapping is a major source of accuracy, not a trivial embedding lookup.
  • Cross-domain results (49.9% on CASME II→SAMM and 52.8% on SAMM→CASME II) predict that LLM-based detectors can transfer between datasets with different subjects, ethnicities, and distributions better than the tested baselines.
  • The authors' stated future direction—interactive, context-aware micro-expression question answering—is a direct corollary if AU detection with this architecture is robust.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never replaces the LLM with a plain classifier on the same fused token, so the strongest possible reading—that LLM reasoning itself drives the gain—remains unproven; the EFP's fused representation could plausibly be doing most of the work.
  • A natural extension is to test whether the EFP token alone, without the LLM, already beats the LED-based baselines; that experiment would isolate the contribution of the fusion projector from the language model.
  • The single-token design may cap the LLM's spatial reasoning; feeding multiple regional tokens (per facial region) might improve AU localization even further, at the cost of more LLM computation.
  • Because the method's gains appear consistently across three 1.5B LLMs, the result may transfer to other parameter-efficient fine-tuning settings, but whether it scales with larger LLMs is untested and could go either way.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AU-LLM, a framework for micro-expression Action Unit (AU) detection that combines a 3D-CNN backbone (with LED temporal filtering and a Squeeze-and-Excitation layer) with a Large Language Model (LLM). Mid-level and high-level visual features are fused by an MLP-based Enhanced Fusion Projector (EFP) into a single visual token, which is prepended to a short text prompt and passed to a LoRA-tuned 1.5B LLM. The last token hidden state is fed to a linear classifier for multi-label AU prediction. Experiments on CASME II and SAMM report average F1 of 81.4% and 61.9% for the best configuration (DeepSeek-R1), outperforming the prior state of the art (78.4% and 57.3%). The paper also includes ablations of the EFP components and bidirectional cross-domain evaluations between CASME II and SAMM.

Significance. If the result holds, AU-LLM would be the first successful application of an LLM to micro-expression AU detection, and the EFP's compact single-token fusion is an interesting design for bridging the vision-language gap in fine-grained tasks. Strengths include the use of standard benchmarks and protocols (LOSO and cross-domain), the reported comparisons against several strong baselines, and the public code release. However, the central attribution of the improvement to the LLM's reasoning is currently unsupported because no experiment replaces the LLM with a non-LLM classifier on the same token. The absence of error bars and the suspicious duplicate values in the cross-domain table further weaken the quantitative claims. With the requested ablations and verification, the contribution could be a solid advance for the micro-expression analysis community.

major comments (4)
  1. [Section 4.4, Tables 1–3] The experiments do not include a baseline that replaces the LoRA-tuned LLM with a standard classifier (e.g., an MLP or a linear head) on the same fused token Tv. Every ablated variant reported in Table 3 still ends with the LLM, so the 3.0-point gain over SSSNet LED on CASME II and the 4.6-point gain on SAMM cannot be attributed to the LLM's reasoning rather than to the 3D-CNN+EFP representation. Because the title, abstract, and conclusion all credit the LLM reasoning for the improvement, this missing control is load-bearing and should be added.
  2. [Table 4] In the CASME II→SAMM block, ResNet-18 and LED SSSNet report identical F1 values for AU2 (47.0), AU7 (41.5), and AU12 (45.1), which is implausible for two different models and strongly suggests a copying or reporting error. This table is the basis for the cross-domain generalization claim (13.4 percentage-point improvement over LED SSSNet), so the numbers must be verified and corrected before the claim can be evaluated.
  3. [Section 4.3] No standard deviations, confidence intervals, or significance tests are reported for any F1 result, although the LOSO protocol produces multiple folds and the training procedure involves stochastic optimization. The reported margin of 3.0 points over SSSNet LED on CASME II may be within run-to-run or fold variation; the state-of-the-art claim requires variance estimates or statistical testing.
  4. [Section 4.3, Tables 1–2] The paper evaluates three LLM backbones (DeepSeek-R1, Qwen2, Qwen2.5) and then presents the best-performing one on the test set as the proposed AU-LLM result, without a pre-specified choice, a shared validation set, or a multiple-comparison correction. This post-hoc selection inflates the reported improvement and should be justified, for example by reporting all variants as the method's performance range or by selecting the backbone based on validation performance alone.
minor comments (5)
  1. [Equation (1)] The formula for the LED filter is typeset incorrectly: the exponents and subscripts in the piecewise expression are garbled, making the equation unreadable. Please use the original formulation from [20] or fix the LaTeX.
  2. [Section 4.1, Table 2] The text says evaluation on SAMM is 'cross-dataset validation,' but Table 2 does not state whether the models were trained on CASME II and tested on SAMM or trained on SAMM directly with LOSO. Please clarify the protocol and ensure that all compared methods use the identical setting.
  3. [Abstract, Section 1] The claim that this is the 'first' use of an LLM for AU detection should be supported by a more thorough search of recent literature, including concurrent or arXiv-only works; consider softening the novelty claim or adding the missing references.
  4. [Section 3.3] The exact text prompt (e.g., 'Please analyze the features and classify AU') is only described in prose; include the full prompt template and tokenization details to improve reproducibility.
  5. [Table 3, Figure 2] The caption of Figure 2 does not explain what is plotted (e.g., per-AU F1 curves or bar charts), and the figure is only referenced in passing; please describe the visualization content in the text and caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivations and evaluations are self-contained, with no fitted parameter renamed as a prediction and no load-bearing self-citation chain.

full rationale

The paper's central pipeline extracts mid-level and high-level features from a 3D-CNN backbone, fuses them through the EFP MLP, and classifies via a LoRA-tuned LLM; success is measured by macro F1 on held-out subjects under LOSO and cross-domain protocols. None of the reported results is obtained by construction: the EFP fusion function in Eqs. (2)-(3) is a learned projection from concatenated visual features, not a definitional restatement of the final F1, and the LLM's logits come from a trained classifier on the hidden state rather than from the input features themselves. The LED temporal filter is imported from an external prior work [20] with explicit equations and learnable parameters, and the baselines, datasets, and loss are standard external components. Self-citations to related works by overlapping authors, such as [22], [23], and [25], are used only as background or for conventional design choices, not as load-bearing evidence for the claimed SOTA improvement. The absence of an ablation that replaces the LLM with a non-LLM head on the identical fused token is a legitimate concern about whether the LLM specifically causes the gain, but that is an empirical attribution gap rather than circularity: no equation or fitting procedure forces the LLM result to equal the visual-feature result. The test-set selection of the best among three LLM variants and the suspicious duplicated values in Table 4 are evaluation-quality issues, not definitional circularity. Accordingly, the paper does not exhibit any step where a prediction reduces by construction to its inputs.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claims rest on standard supervised-learning assumptions: benchmark annotations are correct, the pretrained LLMs do not leak test-subject information, macro F1 is the right metric, and the hand-chosen hyperparameters do not determine the qualitative result. No new physical or conceptual entities are introduced; the EFP is a neural network module, not a separate entity.

free parameters (4)
  • ASL focusing factors = gamma+ = 0, gamma- = 4
    Chosen by hand in Section 3.4; controls loss weighting for positive and negative AU examples and affects training dynamics and final F1.
  • LoRA rank and alpha = r = 16, alpha = 32
    Chosen in Section 4.2; determines the capacity of the LLM adaptation and is not derived from data or theory.
  • Optimizer hyperparameters = lr = 3e-5, weight decay = 0.005, batch size = 256
    Chosen in Section 4.2; standard Adam settings with no sensitivity analysis reported.
  • LED filter initial parameters = alpha, r1, r2 initialized per prior work
    Equation (1) uses learnable alpha, r1, r2 initialized from prior work; the choice of initial values influences temporal filtering but is not justified in this paper.
assumptions (3)
  • domain assumption CASME II and SAMM AU annotations are correct ground truth for the evaluated labels.
    Section 4.1 defines the AU label sets; the entire evaluation rests on annotation quality and label consistency across subjects.
  • domain assumption The pretrained LLMs (Qwen2, Qwen2.5, DeepSeek-R1-Distill) do not leak information about the test subjects or the AU labels.
    Section 4.2 loads 1.5B LLMs from Hugging Face; if these models were trained on images or annotations from CASME II or SAMM, the LOSO and cross-domain results would be inflated.
  • domain assumption Macro F1 over the chosen AUs is the correct primary metric for comparing methods on imbalanced multi-label AU detection.
    Section 4.1 justifies macro F1; the ranking of methods could change under a different metric such as micro F1 or per-class AUC.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AU-LLM: Micro-Expression Action Unit Detection via Enhanced LLM-Based Feature Fusion." pith.science (2026). https://pith.science/paper/3QYLDQKW

@misc{pith2026250721778,
  author       = {Pith},
  title        = {Pith review of: AU-LLM: Micro-Expression Action Unit Detection via Enhanced LLM-Based Feature Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3QYLDQKW}},
  note         = {Machine review of arXiv:2507.21778}
}
read the original abstract

The detection of micro-expression Action Units (AUs) is a formidable challenge in affective computing, pivotal for decoding subtle, involuntary human emotions. While Large Language Models (LLMs) demonstrate profound reasoning abilities, their application to the fine-grained, low-intensity domain of micro-expression AU detection remains unexplored. This paper pioneers this direction by introducing \textbf{AU-LLM}, a novel framework that for the first time uses LLM to detect AUs in micro-expression datasets with subtle intensities and the scarcity of data. We specifically address the critical vision-language semantic gap, the \textbf{Enhanced Fusion Projector (EFP)}. The EFP employs a Multi-Layer Perceptron (MLP) to intelligently fuse mid-level (local texture) and high-level (global semantics) visual features from a specialized 3D-CNN backbone into a single, information-dense token. This compact representation effectively empowers the LLM to perform nuanced reasoning over subtle facial muscle movements.Through extensive evaluations on the benchmark CASME II and SAMM datasets, including stringent Leave-One-Subject-Out (LOSO) and cross-domain protocols, AU-LLM establishes a new state-of-the-art, validating the significant potential and robustness of LLM-based reasoning for micro-expression analysis. The codes are available at https://github.com/ZS-liu-JLU/AU-LLMs.

Figures

Figures reproduced from arXiv: 2507.21778 by the authors.

Figure 1
Figure 1. The overall framework of our proposed AU-LLM. A video sequence is first [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Visualization of ablation studies on both datasets, showing the perfor [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Visualization of model attention. (a) Original micro-expression samples. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 21 canonical work pages

  1. [1]

    NeurIPS (2022)

    Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. NeurIPS (2022)

  2. [2]

    NeurIPS (2020)

    Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. NeurIPS (2020)

  3. [3]

    In: IEEE TAFFC (2018)

    Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: Samm: A spontaneous micro-facial movement dataset. In: IEEE TAFFC (2018)

  4. [4]

    DeepSeek-AI: Deepseek-r1: A 671b moe model with fine-grained sparsity (2024)

  5. [5]

    Consulting Psychologists Press (1978)

    Ekman, P., Friesen, W.V.: Facial action coding system. Consulting Psychologists Press (1978)

  6. [6]

    In: CVPR (2016)

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)

  7. [7]

    In: ICLR (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. In: ICLR (2022)

  8. [8]

    In: CVPR (2018)

    Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: CVPR (2018)

Show all 29 references
  1. [9]

    In: SCIA (2025)

    Khor, H.Q., Li, Y., Jiang, X., Zhao, G.: Infused suppression of magnification arte- facts for micro-au detection. In: SCIA (2025)

  2. [10]

    In: AAAI (2019)

    Li, G., Zhu, X., Zeng, Y., Wang, Q., Lin, L.: Semantic relationships guided repre- sentation learning for facial action unit recognition. In: AAAI (2019)

  3. [11]

    Neurocomputing (2021)

    Li, Y., Huang, X., Zhao, G.: Micro-expression action unit detection with spatial and channel attention. Neurocomputing (2021)

  4. [12]

    In: FG (2021)

    Li, Y., Peng, W., Zhao, G.: Micro-expression action unit detection with dual-view attentive similarity-preserving knowledge distillation. In: FG (2021)

  5. [13]

    In: ICMI (2021)

    Li, Y., Zhao, G.: Intra-and inter-contrastive learning for micro-expression action unit detection. In: ICMI (2021)

  6. [14]

    In: CVPR (2021)

    Miriam Jacob, G., Stenger, B.: Facial action unit detection with transformers. In: CVPR (2021)

  7. [15]

    In: ICCV (2021)

    Ridnik, T., Ben-Baruch, E., Zamir, N., Noy, A., Friedman, I., Protter, M., Zelnik- Manor, L.: Asymmetric loss for multi-label classification. In: ICCV (2021)

  8. [16]

    arXiv preprint arXiv:1412.6550 (2014)

    Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gkioxari, G., Bengio, Y.: Fit- nets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014)

  9. [17]

    Team, Q.: Qwen2 technical report (2024)

  10. [18]

    In: ICCV (2017)

    Tung, F., Mori, G.: Similarity-preserving knowledge distillation. In: ICCV (2017)

  11. [19]

    IEEE TAFFC (2023)

    Varanka, T., Li, Y., Peng, W., Zhao, G.: Data leakage and evaluation issues in micro-expression analysis. IEEE TAFFC (2023)

  12. [20]

    In: Image Analysis (2024)

    Varanka, T., Peng, W., Zhao, G.: Learnable eulerian dynamics for micro-expression action unit detection. In: Image Analysis (2024)

  13. [21]

    In: ACM MM (2020)

    Xie, H.X., Lo, L., Shuai, H.H., Cheng, W.H.: Au-assisted graph attention convo- lutional network for micro-expression recognition. In: ACM MM (2020)

  14. [22]

    arXiv preprint arXiv:2505.03621 (2025)

    Xie, Y., Zhao, B., Dai, M., Zhou, J.P., Sun, Y., Tan, T., Xie, W., Shen, L., Yu, Z.: Physllm: Harnessing large language models for cross-modal remote physiological sensing. arXiv preprint arXiv:2505.03621 (2025)

  15. [23]

    arXiv preprint arXiv:2503.23450 (2025)

    Xing, B., Yuan, K., Yu, Z., Liu, X., K¨ alvi¨ ainen, H.: Au-ttt: Vision test-time training model for facial action unit detection. arXiv preprint arXiv:2503.23450 (2025)

  16. [24]

    In: PLOS ONE (2014) AU-LLM: Enhancing Micro-Expression AU Detection with LLMs 11

    Yan, W.J., Li, X., Wang, S.J., Zhao, G., Liu, Y.J., Chen, Y.H., Fu, X.: CASME II: An improved spontaneous micro-expression database and the baseline evaluation. In: PLOS ONE (2014) AU-LLM: Enhancing Micro-Expression AU Detection with LLMs 11

  17. [25]

    In: ECCV (2024)

    Yuan, K., Yu, Z., Liu, X., Xie, W., Yue, H., Yang, J.: Auformer: Vision transformers are parameter-efficient facial action unit detectors. In: ECCV (2024)

  18. [26]

    arXiv preprint arXiv:1612.03928 (2016)

    Zagoruyko, S., Komodakis, N.: Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928 (2016)

  19. [27]

    IEEE TPAMI (2007)

    Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE TPAMI (2007)

  20. [28]

    In: CVPR (2015)

    Zhao, K., Chu, W.S., De la Torre, F., Cohn, J.F., Zhang, H.: Joint patch and multi-label learning for facial action unit detection. In: CVPR (2015)

  21. [29]

    IJCV (2022)

    Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. IJCV (2022)

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.