{"id":"985af96b-7665-4968-a95c-d2caeb9c89d8","arxiv_id":"2504.15624","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FaceInsight, an MLLM with segmentation inputs, correlation priors, and logic rules, reports higher face-attribute, age/gender/race, and expression accuracy than nine general MLLMs across six benchmarks.","lead":"This paper introduces FaceInsight, a multimodal large language model specialized for face perception that adds face segmentation maps, graph-based attribute correlations, and logic-rule losses to a LLaVA-style architecture. The authors report consistent accuracy gains over nine general-purpose vision-language models on attribute, demographic, and expression benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training/test contamination in the six benchmark datasets is inseparable from the headline performance claim, and the paper provides no code, splits, or error bars to rule it out.","rationale":"The reader's weakest assumption is exactly the concern I identify: the evaluation protocol uses the same six datasets for training and testing, with no split description, and this creates a direct leakage path for the headline claim. I agree with the CONDITIONAL verdict because the architecture and ablation are plausible, but the evaluation protocol is not yet sufficient to support the claim of consistent state-of-the-art face perception. The strongest independent evidence in the paper is the ablation study (Fig. 8), which shows modest, internally consistent gains from the added components on CelebA; that suggests the method itself has some real effect, but the ablation does not extend to the six-dataset headline claim. The paper also compares against only three fine-tuned MLLMs, and all comparisons are single-run without error bars. The paper does include some useful details (e.g., the threshold for the correlation prior, the reweighting in Eq. 7, and the logical rules), but none of those details resolve the train/test contamination question. My concrete test would settle the concern: a clear, released split protocol or a held-out evaluation, plus variance reporting. If those are provided and the margins hold, the claim is credible; if they are not, the paper should be read as an architecture proposal with unverified benchmark gains.","tokens_in":19136,"tokens_out":1818,"duration_ms":15227,"concrete_test":"Ask the authors to release the exact train/test split protocols for all six datasets, or to rerun the evaluation on a held-out subset of each benchmark that was not used for training or for computing the correlation prior map in Eq. (5). If the splits are already disjoint, then report model performance on the held-out test portion only, and separately recompute the correlation prior from the training portion only; if the reported numbers change materially or drop below the compared fine-tuned baselines, the headline claim of consistent superiority is not supported. Additionally, report the standard deviation or 95% confidence interval across at least three random seeds for the main comparison tables.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that FaceInsight 'consistently outperforms nine compared MLLMs across all datasets' under both training-free and fine-tuned settings. This claim rests on evaluating a model trained on exactly the six datasets used for testing (Section 4.2: 'We trained our framework simultaneously on six training datasets'; the six datasets listed in Section 4.1 are MAAD, CelebA, FairFace, UTKFace, ExpW, RAF-DB). No train/test split protocol is described, no held-out partitions are identified, and no code or released models are provided. The correlation prior map in Eq. (5)-(6) is explicitly derived from co-occurrence frequencies 'in the training set', so if the training set is the same benchmark image pool as the test set (or even a disjoint-image but same-annotation-population sample), the CCM encodes label-distribution information that is exactly the information needed to score well on the benchmark's attribute space. The segmentation maps are also generated by a third-party face parser (Pyfacer/RetinaFace), which may have been trained on face data that overlaps with these benchmarks; this is a weaker but still relevant source of leakage. The comparison protocol compounds the problem: the nine general MLLMs are mostly evaluated zero-shot and must be parsed into the binary attribute space, while FaceInsight is directly trained with BCE loss on the same task and datasets. Yet the paper claims superiority even in the 'training-free' setting for FaceInsight, which is never clearly defined for a model that has been trained on these datasets. Finally, Tables 1-3 report no variance or significance testing, so it is impossible to tell whether the margins (e.g., 93.55 vs. 87.77 on MAAD accuracy) are robust. The load-bearing concern is therefore that the reported margins measure benchmark memorization or label-prior leakage rather than face perception ability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FaceInsight, a multimodal large language model for face perception that combines three components: (1) face segmentation maps as an auxiliary perceptual modality, (2) a correlation constraint module (CCM) that injects pairwise attribute-dependency priors into the text embedding via a graph convolutional network, and (3) a logical constraint module that adds a loss derived from 22 hand-authored first-order logic rules enforcing mutual exclusions and group constraints among facial attributes. The model is built on the LLaVA-v1.5 architecture with a CLIP ViT image encoder pre-trained on FLIP. Experiments cover face attribute recognition (MAAD, CelebA), age/gender/race estimation (FairFace, UTKFace), and facial expression prediction (ExpW, RAF-DB). The authors compare FaceInsight against nine MLLMs, claim consistent superior performance under both 'training-free' and fine-tuned settings, and provide ablations on the contribution of each module.","tokens_in":19442,"tokens_out":3380,"duration_ms":32770,"significance":"If the central performance claim were established under a clean evaluation protocol, FaceInsight would be a meaningful contribution: it addresses a real deficiency of general MLLMs on fine-grained face perception, and the combination of segmentation maps, correlation priors, and logic constraints is a plausible and well-motivated design. The paper also includes ablation studies that isolate the effect of each component, which is a strength. However, the reported evidence does not currently support the headline claim of 'consistently outperforms nine compared MLLMs across all datasets' because the evaluation protocol has a potentially load-bearing train/test overlap, the comparison is not apples-to-apples (task-trained FaceInsight versus mostly zero-shot MLLMs), no error bars or statistical tests are provided, and no code or model weights are released. These issues are fixable in principle but require substantial additional experimentation and reporting.","major_comments":[{"comment":"The central evaluation claim is not supported because the training and test sets are the same six datasets. Section 4.2 states 'We trained our framework simultaneously on six training datasets,' and the six datasets listed in Section 4.1 are exactly MAAD, CelebA, FairFace, UTKFace, ExpW, and RAF-DB, which are also the test sets. The paper does not describe any train/test split protocol, does not identify held-out partitions, and provides no code or released checkpoints. Without disjoint train and test images, and without evidence that the annotation population used for training is independent of the test population, the reported margins in Tables 1–5 cannot be attributed to face perception ability rather than to memorization or label-distribution leakage. Please specify the exact split protocol (including identity-exclusion rules), release the splits or code, and additionally evaluate on at least one benchmark not used for training.","section":"§4.1–4.2, Tables 1–5"},{"comment":"The correlation prior maps that drive the CCM are computed directly from co-occurrence frequencies in the training set, and this same training set is drawn from the benchmark pool used for evaluation. This means the CCM encodes the label-distribution statistics of the evaluation domains, including attribute base rates and pairwise associations, and can inflate performance even if the visual features are weak. The authors acknowledge overfitting concerns and apply a threshold, but they do not address the more direct circularity: the 'facial knowledge' injected is distribution-specific rather than an independent source of knowledge. Please either derive the priors from an external/unrelated corpus, or evaluate on held-out datasets with different label distributions, and ablate the CCM against a randomized or shuffled prior matrix to quantify how much of the gain comes from prior leakage.","section":"§3.2, Eqs. (5)–(6)"},{"comment":"The comparison protocol is not balanced. FaceInsight is trained with a binary cross-entropy loss on the exact task labels of the six evaluation datasets, while the nine comparison MLLMs are evaluated zero-shot (only three are fine-tuned, on 'task-specific datasets' with unspecified splits). The paper still reports that FaceInsight outperforms the MLLMs in both settings, but this conflates task-specific training with general face perception ability. The label 'training-free' for FaceInsight is misleading because the model has already been trained on the six datasets. Please add strong specialist baselines (e.g., FaceXFormer, FairFace, MIVolo, or other dedicated face-perception models) trained and evaluated on the same splits, report the fine-tuning protocol for comparison MLLMs in detail, and include error bars or confidence intervals across multiple runs so that the 'consistently outperforms' claim is statistically grounded.","section":"§4.2, §4.3, Figs. 3–6"},{"comment":"The ablation study reports only single-point mAP numbers on CelebA with no error bars, no number of runs, and no statistical test. The text is also internally unclear about how the component contributions add up: it states an average mAP improvement of 2.3%, then reports gains of 0.82%, 1.48%, and 0.95% for the sequential additions, which do not obviously sum to 2.3%. Please clarify whether the reported numbers are cumulative or incremental, and provide means and standard deviations over multiple seeds. In addition, Fig. 8b sweeps only ω on CelebA; the choice ω=0.8 should be justified on a validation split that is disjoint from the reported test set.","section":"§4.5, Fig. 8"}],"minor_comments":[{"comment":"The dataset name appears as 'TRAF-DB' in the text but as 'RAF-DB' everywhere else, including Table 5; please correct the typo.","section":"§4.1"},{"comment":"The phrase 'training-free settings' is applied to FaceInsight, but the model is trained on the six datasets before evaluation. Please rephrase to distinguish zero-shot evaluation of comparison MLLMs from the fully trained FaceInsight, or introduce a genuinely zero-shot variant of FaceInsight.","section":"Abstract and §4.2"},{"comment":"The figure contains a passage of garbled path-like tokens ('/uni00000018/uni00000018/...') that appears to be a text corruption artifact; this should be removed or replaced with the intended caption or annotation text.","section":"Fig. 1"},{"comment":"The text mentions FaRL as a baseline, but Table 4 lists only FairFace, FaceXFormer, and FaceInsight. Either add FaRL to the table or remove the reference to it in this comparison.","section":"§4.3.2, Table 4"},{"comment":"The text refers to 'Pyfacer [50]' and cites RetinaFace [50]; please clarify whether Pyfacer is a library wrapping RetinaFace and provide a proper citation or URL for the library used.","section":"§3.1"},{"comment":"The logical loss multiplies V_i(rule_j) by P(y_ij), but it is not clear how P(y_ij) is selected for rules with multiple antecedents or how V_i is computed for rules involving negation. A short derivation or pseudocode would improve reproducibility.","section":"§3.2, Eq. (9)"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong motivating problem and a sensible architectural proposal, but the evaluation protocol is the main obstacle. The same six datasets are used for training and testing with no split description, and the comparison is mostly against zero-shot general MLLMs. I would not recommend rejection because the issues are addressable: the authors can supply disjoint splits, add specialist baselines, report error bars, and evaluate on a held-out benchmark. However, without those changes the headline claim is not established. I also note the unusual garbled text in Fig. 1 and the discrepancy between 'Pyfacer' and the cited RetinaFace reference; these should be fixed during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does one genuinely useful thing: it combines face segmentation maps, a GCN over attribute co-occurrence statistics, and logic-rule constraints into an MLLM for face perception. That specific combination is new relative to Face-MLLM, FaVChat, and EMO-LLaMA, and the ablation study shows each component contributes something. The motivation is also sound—general MLLMs do hallucinate on face queries, and the qualitative examples make the case vividly. Credit where due: the logic rules are externally authored, so that part is not circular, and the architecture is clearly described.\n\nThe soft spot is the evaluation, and it is a load-bearing one. The model is trained on the same six datasets it is tested on, and the paper never states the train/test split protocol. That alone makes the reported margins (e.g., 93.55 vs. 87.77 on MAAD) impossible to interpret. The 'training-free' comparison is also unfair: FaceInsight's adapter and GCN are trained on those benchmarks, while the nine general MLLMs are mostly evaluated zero-shot. Calling FaceInsight 'training-free' in that context is misleading. There are no error bars or significance tests, so we cannot tell whether even the fine-tuned gains are robust. No code or released models are provided, which would have cleared up most of these doubts.\n\nI am not saying the central idea is wrong. The correlation prior derived from training labels is a legitimate way to encode attribute dependencies—the problem is that the paper does not establish that the test split is disjoint from the training distribution in a way that makes the benchmark results meaningful. The Fix is not hard: report exact splits, evaluate with the same protocol for all models, add variance estimates, and release the code. Without that, the headline claim 'consistently outperforms nine MLLMs' is not supported by the evidence as presented, even though the architecture plausibly could deliver such gains.\n\nWho is this for? Researchers working on face-centric MLLMs will find the architectural ideas worth trying, and the ablation gives a decent hint about what helps. But the paper should not be accepted as-is. Send it to peer review with a clear request for the missing evaluation details. If the authors provide the splits and code, the community can judge the method fairly. If they cannot, the claim will not survive scrutiny.","headline":"Plausible architecture and a useful problem, but the evaluation as reported cannot support the headline performance claim.","tokens_in":20035,"tokens_out":1518,"would_cite":false,"duration_ms":16969,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FaceInsight claims that adding segmentation maps, co-occurrence priors, and logic rules to an MLLM beats nine general models on face attribute, demographic, and expression tasks.","keywords":["face perception","multimodal large language model","visual-textual alignment","face attribute recognition","age gender and race estimation","facial expression recognition","face segmentation","logical constraints"],"falsifier":"Compute the correlation priors only from the training split, fine-tune the general MLLMs on the same instruction-following splits, and evaluate both on held-out identities; if FaceInsight's recall and F1 advantage over the fine-tuned general models collapses, the gains come from label leakage rather than the proposed modules.","tokens_in":18954,"feed_emoji":"👤","tokens_out":9308,"duration_ms":78016,"temperature":0.7,"pith_summary":"FaceInsight argues that general multimodal large language models fail at face perception because they lean too heavily on language-driven reasoning and do not capture the dense, overlapping structure of facial attributes. The paper's central claim is that feeding an MLLM three face-specific signals—segmentation maps, an attribute co-occurrence correlation map refined by a graph convolutional network, and first-order logic rules penalizing contradictory outputs—makes it outperform nine general MLLMs on face attribute recognition, age/gender/race estimation, and expression prediction. If true, this would give a practical recipe for making vision-language models reliable in face-critical applications such as authentication, human-computer interaction, and healthcare, where hallucinated or contradictory facial descriptions are costly. The paper reports the largest gains not in accuracy but in recall, precision, and F1, which is where the comparison models produce many incorrect or conflicting attributes.","feed_headline":"Face-reading MLLM beats nine rivals with three extra cues","feed_subtitle":"Segmentation maps, attribute correlations, and logic rules push face perception accuracy above general MLLMs.","key_machinery":"The load-bearing mechanism is the correlation constraint module, built on an asymmetric conditional-probability matrix $P=(p_{ij})$ with $p_{ij}=m_{ij}/n_i$, where $m_{ij}$ is the co-occurrence count of facial attributes $i$ and $j$ and $n_i$ is the frequency of $i$ in the training set. Thresholding $P$ at $\\tau$ gives a sparse adjacency map $\\hat{C}$, reweighted so each attribute keeps weight $1-\\omega$ on itself and distributes $\\omega$ over its co-occurring neighbors; $L$ graph-convolutional layers refine prompt features by $T_p^l = \\rho(\\hat{C} T_p^{l-1} W^{l-1})$ with a residual connection, producing the final text embedding $\\hat{T}_p = T_p^0 + T_p^L$. A parallel logical constraint module turns 22 first-order rules into a violation probability and adds a logical loss $\\mathcal{L}_c$ to the binary cross-entropy loss, so the full objective is $\\mathcal{L} = \\mathcal{L}_{bce} + \\mathcal{L}_c$. Face segmentation maps generated by Pyfacer are encoded by a second pretrained ViT and injected as extra tokens, giving the LLM both the global image and localized structural cues.","core_discovery":"FaceInsight claims that a multimodal large language model can be made to perceive faces accurately by aligning facial knowledge through three targeted additions: feeding face segmentation maps as an auxiliary perceptual modality; refining text-prompt embeddings with a graph convolutional network driven by an asymmetric conditional-probability correlation map among facial attributes; and adding a logical-loss term that penalizes violations of 22 hand-coded rules such as 'bald implies not bangs and not wavy hair.' Trained with binary cross-entropy plus the logical loss, the model reports the best accuracy, recall, precision, and F1 across face attribute recognition on MAAD and CelebA, age/gender/race estimation on FairFace and UTKFace, and expression prediction on ExpW and RAF-DB, beating nine general MLLMs in training-free comparisons and three fine-tuned MLLMs in the fine-tuned setting. Ablations on CelebA attribute the average 2.3% mAP gain to the three components, with each step adding a positive increment.","pith_inferences":["Beyond the paper's claims, the training-set-dependent correlation priors make the evaluation protocol the main open question: comparing a model trained on the six datasets against zero-shot general MLLMs conflates task-specific supervision with architectural benefit.","A cleaner test of the mechanism would compute the correlation matrix only from a training split, hold out entire identities, and fine-tune the general MLLMs on the same split; the paper does not describe such a protocol.","The hand-coded 22 rules suggest a natural scaling path: mine logical rules from a larger face-description corpus or learn them as soft constraints, which would let the logical module cover attribute vocabularies beyond the training sets.","The asymmetry of the correlation map is a general idea: directional attribute dependencies like 'bald implies no bangs' could improve consistency in other dense attribute-recognition domains, such as medical image description."],"forward_implications":["Adding the three modules to a LLaVA-style MLLM does not require retraining the LLM or the image encoder; only the adapter and graph convolutional network are updated.","The logical constraint loss can be reused in any vision-language task with mutually exclusive or co-occurring labels to suppress contradictory outputs.","The recipe is claimed to transfer to other fine-grained visual perception tasks, as the paper states in its conclusion.","Because the biggest gains are in recall, precision, and F1, the method most helps applications where false or contradictory facial descriptions matter more than raw accuracy."],"supporting_citations":[{"why":"It documents that general MLLMs fail at face understanding, motivating the comparison the paper aims to beat.","marker":"[1]"},{"why":"It defines the LLaVA-v1.5 architecture and instruction-tuning recipe that FaceInsight extends and that the fine-tuned baselines use.","marker":"[18]"},{"why":"It provides FaceXFormer, a unified face-perception model compared against FaceInsight in Table 4.","marker":"[42]"},{"why":"It supplies the face segmentation maps used as the auxiliary perceptual modality.","marker":"[50]"},{"why":"It supplies the MAAD dataset used for face attribute recognition evaluation and training.","marker":"[51]"},{"why":"It supplies the CelebA dataset used for attribute evaluation and for the ablation study.","marker":"[52]"},{"why":"It supplies the FairFace demographic dataset and also serves as a baseline model in Table 4.","marker":"[53]"},{"why":"It supplies the UTKFace dataset used for age, gender, and race evaluation.","marker":"[54]"},{"why":"It supplies the ExpW dataset used for facial expression evaluation.","marker":"[55]"},{"why":"It supplies the RAF-DB dataset used for facial expression evaluation.","marker":"[56]"}],"fun_headline_variants":["FaceInsight adds three cues to beat nine face-reading rivals","Three extra cues give FaceInsight edge over general MLLMs","Segmentation, correlations, and rules: FaceInsight wins","Face perception MLLM: three components, nine rivals beaten"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported margins assume that the labels used to train FaceInsight are strictly separated from the images used to test it, and that measuring it against general models that were not trained on face-task labels is a fair comparison.","fun_headline_variants_meta":{"raw":{"variants":["FaceInsight adds three cues to beat nine face-reading rivals","Three extra cues give FaceInsight edge over general MLLMs","Segmentation, correlations, and rules: FaceInsight wins","Face perception MLLM: three components, nine rivals beaten"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000662,"raw_usage":{"total_tokens":2993,"prompt_tokens":880,"completion_tokens":2113,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":2041}},"tokens_in":496,"tokens_out":2113,"duration_ms":15161,"temperature":1.0,"reasoning_tokens":2041,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:21:14.549476+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the correlation priors only from the training split, fine-tune the general MLLMs on the same instruction-following splits, and evaluate both on held-out identities; if FaceInsight's recall and F1 advantage over the fine-tuned general models collapses, the gains come from label leakage rather than the proposed modules.","supporting_citations":[{"cited_title":"Retinaface: Single-shot multi-level face localisation in the wild","cited_arxiv_id":null,"evidence_quote":"It supplies the face segmentation maps used as the auxiliary perceptual modality."},{"cited_title":"Maad-face: A massively annotated attribute dataset for face images","cited_arxiv_id":null,"evidence_quote":"It supplies the MAAD dataset used for face attribute recognition evaluation and training."},{"cited_title":"Fairface: Face attribute dataset for bal- anced race, gender, and age for bias measurement and mitigation","cited_arxiv_id":null,"evidence_quote":"It supplies the FairFace demographic dataset and also serves as a baseline model in Table 4."},{"cited_title":"Age progression/regression by con- ditional adversarial autoencoder","cited_arxiv_id":null,"evidence_quote":"It supplies the UTKFace dataset used for age, gender, and race evaluation."},{"cited_title":"From facial expression recognition to interpersonal relation prediction","cited_arxiv_id":null,"evidence_quote":"It supplies the ExpW dataset used for facial expression evaluation."}],"review_version":1}