{"id":"cf2b4588-2aee-47b6-b362-7c7529b6f864","arxiv_id":"2506.02891","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"OpenFace 3.0 shows a single lightweight multi-task model can handle four facial behavior tasks at speeds competitive with specialized toolkits, though the 'rivals SOTA' claim is not equally supported across all four tasks.","lead":"OpenFace 3.0 is an open-source toolkit that detects facial landmarks, action units, gaze, and emotions with one lightweight multi-task model that runs in real time on a CPU. It claims to match or beat specialized state-of-the-art systems on several benchmarks while using a single model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table III's SOTA baselines are not protocol-matched; the MPIIGaze 2.56° result is suspiciously better than published SOTA. The headline claim rests on these numbers, so reproducing one anomalous row under the standard protocol would settle the concern.","rationale":"The paper is honest engineering work: the ablations show multitask learning helps AU and gaze, the user study is small but reasonable, and the code is promised open-source. The central claim of 'rivals SOTA' rests entirely on Table III, and the baselines are not protocol-matched. The MPIIGaze result is so far outside the published distribution that it is likely explained by a different evaluation setup. If reproduced as reported, the claim survives; if not, the performance claim must be scoped down to 'competitive with toolkits', which is still useful but not SOTA-rivaling. The reader's conditional verdict is the right one: the toolkit is probably real and useful, but the headline performance comparison needs verification before it can be accepted at face value.","tokens_in":17175,"tokens_out":4493,"duration_ms":47373,"concrete_test":"Reproduce the MPIIGaze row: run the released OpenFace3.0 checkpoint on the standard MPIIGaze within-dataset subject-exclusive protocol using L2CS-Net's official evaluation code. If the angular error is substantially above 2.56° (e.g., >3.5°), the Table III comparison is protocol-incomparable and the headline claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table III is the evidential core of the 'exceeds or rivals SOTA' claim, but the SOTA rows are not reproducible as stated. No train/test split, evaluation protocol, or model instance is given for any baseline. The single most damning example is MPIIGaze: OpenFace3.0 (MTL w/ unc.) reports 2.56° angular error, below the cited SOTA 3.14° and far below typical subject-exclusive MPIIGaze results (~3.5–4.5°) from L2CS-Net and other published models. A protocol mismatch (e.g., person-dependent evaluation or evaluating on the training set) would explain this. Similarly, WFLW is reported with 'NME inter-ocular' although the standard WFLW metric is NME normalized by face size; if OpenFace3.0 uses inter-ocular while the 4.00/4.14 baselines use face-size normalization, the comparison is invalid. The table itself also shows AU results (59/59 for MTL w/ unc.) below SOTA small (61/62), so the caption overstates. Because every headline number flows from unverified baselines, the central claim is conditional on protocol comparability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents OpenFace 3.0, an open-source toolkit for facial landmark detection, action unit detection, gaze estimation, and emotion recognition using a single lightweight multi-task model. The architecture combines a stacked-hourglass landmark module with an EfficientNet backbone, followed by task-specific heads (a dynamic graph AU module, per-eye FC gaze heads, and an emotion classifier). Training uses a three-stage procedure with homoscedastic uncertainty-based loss weighting. Experiments benchmark the system against SOTA, SOTA-small, and prior toolkits on 300W, WFLW, DISFA, BP4D, MPIIGaze, Gaze360, and AffectNet, and include efficiency comparisons and a small user study.","tokens_in":17427,"tokens_out":5368,"duration_ms":50078,"significance":"If the reported performance is reliable, this is a valuable contribution: a single 29.4M-parameter model that runs four facial analysis tasks in real time on a CPU, with open-source code and an intuitive Python API. The paper also demonstrates the benefit of a three-stage multitask training scheme and uncertainty weighting in an ablation. The system has the potential to be a widely used research tool if the evaluation claims are made rigorous. However, the headline claim of matching or exceeding state-of-the-art specialized models is not currently supported by the evidence presented, and the evaluation protocol is underspecified.","major_comments":[{"comment":"The caption of Table III states that 'Across all tasks, our approach either exceeds or performs comparably to SOTA models,' but the reported numbers contradict this. For DISFA and BP4D, the best OpenFace 3.0 F1 scores are 59/60 and 59/62, respectively, which are below the listed SOTA (66/66) and below or at the SOTA-small values (61/62). AffectNet accuracy of 0.60 is below the listed SOTA of 0.65. Only the landmark results (300W 2.87, WFLW 4.02) and MPIIGaze (2.56) beat the cited SOTA-small values. The text in Section IV-B must be revised to describe the actual results rather than a blanket claim, and the caption should not overstate the findings.","section":"Section IV-B, Table III"},{"comment":"The SOTA and SOTA-small baseline numbers are not protocol-matched to OpenFace 3.0. The paper does not specify which exact model variant or checkpoint produced each baseline value, which train/test splits were used for 300W, WFLW, MPIIGaze, Gaze360, DISFA, BP4D, or AffectNet, or whether the evaluation protocols (e.g., subject-exclusive vs. random split, metric normalization) coincide. The most striking case is MPIIGaze: OpenFace 3.0 MTL w/ unc. reports 2.56° angular error, below the cited SOTA of 3.14° and well below typical subject-exclusive results from published gaze-estimation models. This strongly suggests a protocol mismatch, such as evaluating in a person-dependent manner or on training data. The authors should report their evaluation protocol in full and rerun the baselines under the same protocol, or substantially temper the 'rivals SOTA' claim.","section":"Section IV-A, Table III baselines"},{"comment":"The paper states that landmark accuracy is measured by NME normalized by inter-ocular distance for both 300W and WFLW. However, the standard WFLW benchmark measures NME normalized by face size (inter-pupil distance or bounding box size), and the cited SOTA baselines on WFLW (4.00 for SPIGA, 4.14 for SLPT) were reported in their original papers with face-size normalization. Comparing inter-ocular NME against face-size-normalized NME is not a valid comparison. The authors should cite the exact normalization used for each baseline and report results in the metric that matches the published literature, or clearly state that a non-standard normalization is used and provide the standard metric as well.","section":"Section IV-A.1, metric definition"},{"comment":"All results in Table III and the ablation study are point estimates from what appears to be a single training run. No confidence intervals, standard deviations, or number of random seeds are reported. Consequently, differences such as the MPIIGaze improvement from 4.25° to 2.56° with uncertainty weighting, or the DISFA F1 change from 60 to 59, cannot be assessed for statistical significance. The authors should provide at least three independent runs (or resampling-based uncertainty estimates) and report mean ± standard deviation, so that the claimed benefits of multitasking and uncertainty weighting can be evaluated.","section":"Section IV-B and IV-C, Table III"}],"minor_comments":[{"comment":"The text says 'Training details for our landmark detection module are in Sections III-D and III-D' and later references 'Section' without a number; these cross-references are broken and should be fixed to point to the correct subsections.","section":"Section III-D"},{"comment":"In the gaze estimation paragraph, 'trained specifically for this tak' contains a typo; it should read 'task'.","section":"Section IV-B"},{"comment":"The reference to 'Appendix VII-.0.b' is malformed and does not match the actual appendix structure; the user study participant details should be referenced correctly.","section":"Section V"},{"comment":"The ethics statement contains the duplicated phrase 'for for' in 'training models for for facial analysis tasks.'","section":"Section VII"},{"comment":"The face-orientation bins (Easy: 0°–15°, Medium: 15°–45°, Hard: >45°) are not defined in terms of which rotation axis (yaw, pitch, or roll) is used, nor is it explained how the orientation is computed from the model outputs. Clarify this for reproducibility.","section":"Table V"},{"comment":"The pilot user study has only 8 participants, and the UMUX-LITE and NASA-TLX scores are reported as averages without variance or individual-level data. Since the study is described as a pilot, this limitation should be stated explicitly, and the scores should be presented with ranges or distributions.","section":"Section V and Appendix"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong systems/toolkit contribution with open-source code and a thoughtful multitask architecture. The main risk is that the central 'rivals SOTA' claim rests on protocol-incomparable baseline numbers, and the reported results do not consistently support the Table III caption. The MPIIGaze result in particular warrants careful reproduction. I would recommend major revision with the requirement that the authors either provide protocol-matched baselines or clearly scope the claims to 'competitive with lightweight models' rather than SOTA."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWhat you should know: this is a systems/toolkit paper, not a research breakthrough. OpenFace 3.0 assembles known components—RetinaFace, stacked Hourglass, EfficientNet, ME-GraphAU-style dynamic graph, L2CS-Net-style gaze heads, Kendall uncertainty weighting—into a single 29.4M-parameter model that runs four facial analysis tasks in real time on a CPU. That is genuinely useful. The field has been wanting an open, maintained successor to OpenFace 2.0, and the authors ship code, a pip package, and a reasonable interface.\n\nWhat it does well: the three-stage training schedule is thoughtful (landmark module first, freeze, then classifiers, then joint fine-tune), and the Table V angled-face analysis is a nice illustration that multi-task sharing helps non-frontal emotion recognition. The usability pilot is small but appropriate for a tool publication.\n\nThe soft spots are real and not minor. The headline 'exceeds or rivals SOTA' is contradicted by the paper's own Table III: AU F1 and AffectNet accuracy are below the listed SOTA and SOTA small numbers. The body text is more careful than the abstract and caption, but the framing still overreaches. More importantly, the baseline column is not protocol-matched. The MPIIGaze 2.56° figure is suspicious; published subject-exclusive results are typically above 3°, and the paper gives no split or evaluation details. That number anchors the strongest claim. The WFLW metric label 'NME int ocul' may not match the normalization used by the cited baselines. Without model instances, train/test splits, and metric conventions for every 'SOTA' row, the comparison is unverifiable.\n\nThis paper deserves a serious referee, because an open, efficient, multi-task facial analysis toolkit is valuable and the artifact is real. But the claims need re-scoping, the baselines need per-model protocol citations, and the release needs a commit hash. I would not accept it as-is, but I would send it to review with the expectation of major revision. For a reading group, it is worth discussing as a case study in how evaluation protocol can make or break a SOTA claim.","headline":"A genuinely useful lightweight multi-task OpenFace successor, but the SOTA claims are overplayed and the key MPIIGaze number needs protocol verification.","tokens_in":17965,"tokens_out":3126,"would_cite":true,"duration_ms":32657,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenFace 3.0 shows that one compact multitask model can perform four facial behavior analysis tasks in real time on a CPU while matching specialized models.","keywords":["facial behavior analysis","multi-task learning","facial landmark detection","action unit detection","eye gaze estimation","emotion recognition","lightweight model","real-time toolkit"],"falsifier":"Run OpenFace 3.0 and each named baseline (SPIGA, SLPT, ME-GraphAU, L2CS-Net, MCGaze, DDAMFN, EfficientFace) on the same held-out splits of 300W, WFLW, MPIIGaze, Gaze360, DISFA, BP4D, and AffectNet, with identical normalization and scoring; if OpenFace 3.0 no longer matches or beats the published numbers, the central claim is false.","tokens_in":16967,"feed_emoji":"😀","tokens_out":9255,"duration_ms":87379,"temperature":0.7,"pith_summary":"OpenFace 3.0 sets out to show that one compact multitask model can do the work of several specialized facial-analysis systems: detect facial landmarks, recognize facial action units, estimate eye gaze, and classify eight emotions. The paper claims this 29.4M-parameter model runs in real time on a CPU, uses less memory than earlier toolkits, and matches or beats specialized state-of-the-art models on landmarks and gaze while remaining competitive on action units and emotion. If the claim holds, researchers get a single open-source tool that replaces multiple heavyweight pipelines and works in-the-wild, including on angled faces. The paper also reports that multitask training with uncertainty-based loss weighting improves gaze and action-unit performance and keeps emotion recognition stable on non-frontal faces.","feed_headline":"One lightweight model runs four facial analysis tasks in real time","feed_subtitle":"OpenFace 3.0 matches specialized state-of-the-art models on landmarks, gaze, AUs, and emotion.","key_machinery":"The load-bearing mechanism is the unified facial representation: landmark heatmap coordinates from four stacked Hourglass networks are concatenated with EfficientNet's final-layer contextual features, pretrained on VGGFace2, to make one shared input. Three heads consume it: an action-unit detector builds a per-face cosine-similarity graph over AU features and updates it with a graph convolutional layer, a gaze head predicts yaw and pitch for each eye through separate fully connected layers, and an emotion head classifies eight classes. Training proceeds in three stages (landmark-only, then classifiers with a frozen backbone, then full fine-tuning), with a homoscedastic-uncertainty loss weighting that rebalances tasks of different units and dataset sizes. This mechanism is what lets 29.4M parameters cover all four tasks and keeps inference at 38 ms per frame on a CPU.","core_discovery":"On the paper's own terms, the discovery is that sharing one facial representation across four tasks is not a compromise: it is a performance lever. OpenFace 3.0 concatenates landmark coordinates from stacked Hourglass networks with contextual features from an EfficientNet backbone pretrained on VGGFace2, then feeds this unified representation to three lightweight heads. Across its single-task, multitask, and uncertainty-weighted multitask variants, the system reports NME of 2.87 on 300W and 4.02 on WFLW for landmarks, angular errors of 2.56 on MPIIGaze and 10.6 on Gaze360, F1 scores of 60 on DISFA and 62 on BP4D for action units, and 0.60 accuracy on AffectNet's eight emotion classes. Table III states that across all tasks the approach either exceeds or performs comparably to specialized state-of-the-art models, state-of-the-art small models, and other toolkits, while Table IV shows 29.4M parameters and 38 ms per frame on CPU. The paper further finds that multitask training with uncertainty weighting improves gaze substantially and stabilizes emotion recognition on faces tilted beyond 45 degrees.","pith_inferences":["Inference: the angle-based evaluation suggests gaze data acts as a regularizer for emotion recognition on non-frontal faces; a direct test would be to train the emotion head with and without the gaze task while holding all other components fixed.","Inference: because the shared representation is task-agnostic, adding additional output heads such as valence/arousal, head pose, or identity should cost only a few million parameters, making OpenFace 3.0 a natural seed for a broader face-behavior foundation model.","Inference: for downstream users, the fair comparison is end-to-end behavior on their own video, including face detection, alignment, and task head, since detection failures propagate to all four outputs; per-task benchmark numbers alone may overstate or understate real-world performance."],"forward_implications":["A single 29.4M-parameter model can replace separate toolkits for landmarks, action units, gaze, and emotion, cutting deployment cost and memory footprint.","Real-time CPU inference makes facial behavior analysis practical on edge devices, robotics, and webcam-based interfaces without GPU hardware.","Shared multitask features improve robustness on non-frontal faces, so in-the-wild videos with head movement should see smaller accuracy drops than specialized emotion models.","Uncertainty-weighted multitask training improves gaze estimation substantially, suggesting task weighting is a key tuning lever for future toolkits.","Open-source release of training code lets the community add tasks or datasets and retrain the same architecture."],"supporting_citations":[{"why":"Supplies the prior OpenFace 2.0 toolkit compared against on landmarks, action units, and gaze.","marker":"[6]"},{"why":"Source of the dynamic graph approach for action unit detection that OpenFace 3.0 adapts.","marker":"[47]"},{"why":"Provides the homoscedastic uncertainty weighting used to balance losses across tasks.","marker":"[36]"},{"why":"Supplies RetinaFace face detection and alignment before landmark extraction.","marker":"[18]"},{"why":"Provides VGGFace2 pretraining for the EfficientNet backbone.","marker":"[11]"},{"why":"Defines the STAR loss used to train the landmark detection module.","marker":"[86]"},{"why":"Inspires the separate per-angle gaze losses used in the gaze estimation head.","marker":"[1]"},{"why":"Provides the AffectNet dataset with eight emotion labels for training and evaluation.","marker":"[50]"},{"why":"Provides the 300W landmark dataset used for evaluation.","marker":"[59]"}],"fun_headline_variants":["OpenFace 3.0: one model, four facial tasks, real-time CPU","Shared representation boosts all four facial analysis tasks","Lightweight multitask model rivals specialized facial analyzers","OpenFace 3.0 matches SOTA on landmarks, gaze, AUs, emotion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison rests on treating previously published state-of-the-art numbers from other papers as directly comparable to OpenFace 3.0, even though the training data, model sizes, validation splits, and metric conventions behind those numbers may differ.","fun_headline_variants_meta":{"raw":{"variants":["OpenFace 3.0: one model, four facial tasks, real-time CPU","Shared representation boosts all four facial analysis tasks","Lightweight multitask model rivals specialized facial analyzers","OpenFace 3.0 matches SOTA on landmarks, gaze, AUs, emotion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1453,"prompt_tokens":984,"completion_tokens":469,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":393}},"tokens_in":600,"tokens_out":469,"duration_ms":4297,"temperature":1.0,"reasoning_tokens":393,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:13:04.667276+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run OpenFace 3.0 and each named baseline (SPIGA, SLPT, ME-GraphAU, L2CS-Net, MCGaze, DDAMFN, EfficientFace) on the same held-out splits of 300W, WFLW, MPIIGaze, Gaze360, DISFA, BP4D, and AffectNet, with identical normalization and scoring; if OpenFace 3.0 no longer matches or beats the published numbers, the central claim is false.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the dynamic graph approach for action unit detection that OpenFace 3.0 adapts."},{"cited_title":"Kendall, Y","cited_arxiv_id":null,"evidence_quote":"Provides the homoscedastic uncertainty weighting used to balance losses across tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies RetinaFace face detection and alignment before landmark extraction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides VGGFace2 pretraining for the EfficientNet backbone."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the STAR loss used to train the landmark detection module."},{"cited_title":"Mollahosseini, B","cited_arxiv_id":null,"evidence_quote":"Provides the AffectNet dataset with eight emotion labels for training and evaluation."},{"cited_title":"Sagonas, E","cited_arxiv_id":null,"evidence_quote":"Provides the 300W landmark dataset used for evaluation."}],"review_version":1}