{"id":"43e6e9c5-6376-4ca3-9595-ee2414b29ac2","arxiv_id":"2507.15470","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A federated learning system that fuses facial and physiological signals for in-vehicle emotion recognition reports 87% accuracy, but the custom dataset and code are not released.","lead":"This paper builds a car emotion-recognition system that keeps camera and body-sensor data on local devices, sending only model updates to a server. The authors' prototype, tested on three Raspberry Pis, reports 87% accuracy when combining face and heart-rate signals.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 87% fusion accuracy cannot follow from Eq. 16 as written: a two-classifier majority vote is undefined on disagreement, and no documented tie-break yields 87%.","rationale":"The reader's conditional verdict is reasonable but understates the problem: the tie-break issue is not merely an unspecified implementation detail; it makes the central headline number (87% fusion) mathematically incompatible with the published fusion equation under any standard interpretation. The per-modality accuracies (77% and 74%) combined with a two-way hard vote cannot produce 87% without an undocumented, label-informed tie-break, so the central claim fails as written unless Eq. 16 is a typographical error. I credit the paper for the working Flower-based prototype, the Raspberry Pi deployment, the use of FER2013/FERPlus for the visual modality, and the honest prototype framing, and I am not accusing anyone of fabricating results: the experiments may be real and the 87% may be obtainable with soft voting. But the paper currently provides no code, no dataset, and no reproducible fusion rule, and the abstract/conclusion state 18 rounds while Table I lists 10 rounds with 430 s/round and the results section says 20 rounds and 120 s/round. These inconsistencies compound the fusion ambiguity. The correct fix is straightforward: specify and implement a well-defined fusion rule, release the code and the custom dataset (or its statistics), and reconcile the round counts and timings. Until then the current manuscript should not be accepted; hence I move the reader's CONDITIONAL to REJECT (reject the present version), while noting that a revised version may well resolve the concern. Agreement with the reader is partial because the reader flagged the undefined tie-break as part of a broader dataset concern, whereas I regard it as the single most load-bearing flaw in the central claim.","tokens_in":10406,"tokens_out":10290,"duration_ms":120199,"concrete_test":"Ask the authors to release the exact fusion code and the test-set predictions for both modalities, then recompute fused accuracy under Eq. 16 with each candidate tie-break (always CNN, always RF, random, class-prior, and any other rule claimed). Additionally recompute with soft voting, i.e., argmax_c [p_visual(c)+p_physio(c)]. If no well-defined tie-break and no soft-voting variant reproduces 87%, the reported fusion result is unsupported; if soft voting does reproduce it, Eq. 16 must be corrected. A quick independent check requires no data: derive the expected random-tie accuracy, 75.5%, directly from the reported per-modality accuracies, which already shows the stated hard-voting rule cannot produce the claimed 87%.","verdict_should_be":"REJECT","load_bearing_attack":"The central fusion claim is not reproducible from the stated method. Equation 16 defines the fused prediction as argmax_c [1(hat_y_visual=c) + 1(hat_y_physio=c)]. With two classifiers and seven classes, whenever the CNN and RF predictions differ, two classes tie at one vote each and the argmax is undefined; the paper never defines a tie-breaking rule. If ties are always resolved to the visual prediction, the fused accuracy is exactly the CNN accuracy (77%); if always to the physiological prediction, it is the RF accuracy (74%). If ties are broken uniformly at random between the two candidate labels, the expected fused accuracy is the simple average of the two individual accuracies, (0.77 + 0.74)/2 = 75.5%, not 87%. Achieving 87% under hard voting would require a tie-break that selects the correct modality on a large fraction of disagreements, with no stated basis for doing so (the required joint correct-rate is P(both correct)=0.64). It is possible the implementation actually used soft voting by averaging the two models' probability vectors, but then Eq. 16 is misreported and must be corrected; it is also possible that an unstated tie-break was tuned on the unreleased physiological test set, which is described without subject counts, per-class counts, or label-validation details (Section IV-B). Either way, the headline 87% does not follow from the method as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FedMultiEmo, a privacy-preserving multimodal federated learning framework for real-time in-vehicle emotion recognition. Visual features from facial images are classified by a CNN, physiological signals (heart rate, electrodermal activity, skin temperature) are classified by a Random Forest, and the two modality predictions are combined by decision-level majority voting. Training uses Federated Averaging across three Raspberry Pi clients orchestrated by a Flower server, with client updates weighted by local dataset size. The paper reports 77% accuracy for the federated CNN, 74% for the Random Forest, and 87% for the fused system on FER2013 plus a self-collected physiological dataset, along with convergence in 18 rounds, an average round time of 120 seconds, and a per-client memory footprint below 200 MB. The claimed contribution is a practical, privacy-preserving multimodal emotion recognition system that matches centralized baselines while keeping raw data local.","tokens_in":10707,"tokens_out":4161,"duration_ms":46980,"significance":"If the reported results are correct, FedMultiEmo would be a useful systems demonstration: it combines two complementary modalities in a federated setting, provides an end-to-end edge-cloud deployment on Raspberry Pi clients, uses the public FER2013/FERPlus benchmark for the visual modality, and reports confusion matrices for all three classification configurations. The explicit comparison of individual, centralized, and federated training costs is also informative. However, the central numerical claims rest on three load-bearing points that are not yet established: the fusion rule as written cannot produce the reported 87% accuracy, the self-collected physiological dataset is not characterized enough for reproducibility, and several reported experimental constants are internally inconsistent. The paper does not provide code, data, or machine-checked proofs, so the credibility of the headline numbers depends entirely on the completeness and correctness of the experimental description. These issues are fixable in a revision, but they currently prevent the reader from verifying the central claim.","major_comments":[{"comment":"The stated majority-vote fusion rule is undefined whenever the CNN and Random Forest predictions disagree, and no tie-breaking rule is given. With two classifiers and seven classes, any disagreement produces two classes tied at one vote, so the argmax in Eq. (16) is not well defined. If ties were always resolved to the visual prediction, the fused accuracy would equal the CNN accuracy (77% or 78%); if always to the physiological prediction, it would equal the Random Forest accuracy (74%); if broken uniformly at random, the expected fused accuracy would be roughly the average of the two, about 75.5%. The reported 87% fused accuracy therefore cannot follow from Eq. (16) as written. The paper must either state the actual tie-breaking rule and show how it yields 87%, or correct Eq. (16) to describe the actually implemented fusion (e.g., soft voting over probability vectors) and report the corresponding accuracy.","section":"Section III-F, Eq. (16)"},{"comment":"The bespoke physiological dataset is the sole basis for the 74% Random Forest accuracy and the 87% fusion accuracy, but it is not characterized sufficiently for reproducibility. The paper does not state the number of subjects, the number of sessions, the per-class sample counts, how the video-elicited ground-truth labels were validated (e.g., self-report, multiple annotators, or external measures), or how train/test splits were constructed. It is also unclear how the CNN trained on FER2013 and the Random Forest trained on this custom set are evaluated jointly at test time, since no paired test set or data synchronization procedure is described. Without this information, the fusion result cannot be interpreted or reproduced.","section":"Section IV-B"},{"comment":"The number of federated rounds and the per-round time are reported inconsistently: the abstract and conclusion state 18 rounds, Section IV-D states 20 global rounds, and Table I lists 10 rounds; the abstract gives an average round time of 120 seconds while Table I gives 430 seconds per round. These contradictions directly affect the convergence and real-time claims and must be reconciled with a single consistent measurement protocol.","section":"Section IV-D and Table I"},{"comment":"The federated aggregation of the Random Forest is not specified. Equation (17) is a FedAvg update for weight vectors, but a 200-tree Random Forest does not have a natural weight-average representation, and the text does not describe how the server aggregates tree ensembles across the three clients. Without this detail, the claim that the physiological model is trained in a federated manner is incomplete.","section":"Section III-G and Section IV-D"},{"comment":"The reported CNN accuracy is 77% in the abstract but 78% in Section IV-E.1, and the precision/recall values for the Random Forest (67%/73%) and CNN (77%/77%) are not reconciled with the accuracy figures for the same models. Since these numbers appear in the paper's central performance summary, they need to be made consistent and traceable to specific evaluation runs.","section":"Section IV-E.1"}],"minor_comments":[{"comment":"Equation (4) states that a 4th-order filter has T = 4 coefficients, but a 4th-order Butterworth filter has more than four coefficients; the notation conflates filter order with filter length and should be corrected.","section":"Section III-B, Eq. (4)"},{"comment":"The moving average in Eq. (5) is written with s[t - i] rather than the filtered signal from Eq. (4); the notation should be made consistent so that the smoothing step is clear.","section":"Section III-B, Eq. (5)"},{"comment":"Equation (8) defines HRV as the mean absolute successive difference of heart rate samples, which is not the standard definition of heart rate variability; additionally, with 1 Hz sampling and a 5-second window, the feature is computed over very few samples and its statistical reliability should be discussed.","section":"Section III-C, Eq. (8)"},{"comment":"References [11] and [26] both appear to describe the same Fed-PhyERS system but with different authors and venues; these should be unified or clearly distinguished.","section":"References [11] and [26]"},{"comment":"The phrase 'personalized Federated Averaging' is used, but Eq. (17) is standard FedAvg with client weighting by dataset size; the paper should clarify what personalization mechanism beyond standard weighting is being claimed.","section":"Section III-G"},{"comment":"The confusion matrices in Fig. 2 are difficult to read at print resolution; the class labels, color scales, and percentage values should be enlarged or a higher-resolution version provided.","section":"Figure 2"},{"comment":"The number of clients N in Eqs. (1) and (17) is never stated explicitly in the problem formulation; Section IV-A later says three clients, but the notation would benefit from an explicit statement of N in the setup.","section":"Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a systems-oriented demonstration, and the practical prototype is a strength. However, the headline fusion result is not reproducible from the stated method, and the experimental description contains several internal contradictions. Since the authors can potentially correct the fusion description, supply the dataset characterization, and reconcile the reported constants, I recommend major revision rather than rejection. If the authors cannot provide the missing tie-breaking or soft-voting details and the physiological dataset statistics, the 87% claim should be withdrawn or substantially qualified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a believable prototype paper—FL plus vision and physiological emotion recognition on Raspberry Pi clients—but the headline 87% fusion number does not follow from the method as written, and several reported numbers do not line up. Worth engaging as a fixable workshop-level paper, not as a clean result.\n\nWhat is genuinely there: the authors built an end-to-end thing. Three Raspberry Pi 4 clients, a Flower server, USB cameras, a Fitbit Versa 3, and an I2C EDA sensor. That is real engineering. Using FER2013 with FERPlus relabeling is sensible. The framing—two modalities, decision-level fusion, everything federated—is a natural composition, and the qualitative claim that fusion helps over either unimodal model is plausible. I also do not see a citation problem: FedAvg is cited to McMahan, and the self-citations are used as background context, not to inflate novelty.\n\nThe soft spots are real and they sit in the quantitative core. Eq. 16 defines fusion as argmax_c [1(yhat_visual=c)+1(yhat_physio=c)]. With two classifiers and seven classes, any disagreement produces a two-way tie, and the paper never defines a tie-break. If ties go to the CNN you just get 77%; to the RF, 74%; uniformly random, 75.5%. The claimed 87% therefore cannot be reproduced from the stated rule. Possibly they used soft voting by averaging probability vectors, but then Eq. 16 is misreported. Either way the central accuracy claim is not supported by the written method. This is not a minor typo; it is the paper's headline result.\n\nThere are also smaller but telling inconsistencies: CNN 77% in the abstract vs 78% in Section IV.E.1; rounds 18 vs 20 vs 10 in Table I; per-round time 120 s vs 430 s. And the physiological dataset—which carries half the fusion weight—is described without subject count, per-class counts, or label-validation details. The RF results and the fusion gain rest on that unreleased dataset.\n\nProportionately: the prototype and the architectural composition are real contributions, and all of these issues are fixable with a corrected equation, a consistent results table, and a proper dataset appendix. But as it stands, the 87% should not be quoted. If you cite this for the prototype, cite it for the prototype, not for the numbers.\n\nRecommendation: send it to peer review, but expect heavy revision. The core idea is sound and the engineering is present; the presentation of the quantitative evidence is not yet trustworthy.","headline":"The prototype is real and the integration story is plausible, but the headline 87% fusion accuracy doesn't follow from the stated majority-vote rule and the reported numbers are inconsistent.","tokens_in":11274,"tokens_out":2042,"would_cite":false,"duration_ms":21849,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedMultiEmo claims that decision-level fusion of a federated facial-expression CNN and a federated physiological random forest reaches 87% emotion-recognition accuracy—matching the centralized baseline while keeping all raw data local.","keywords":["federated learning","emotion recognition","multimodal fusion","driver monitoring","edge AI","privacy","random forest","convolutional neural network"],"falsifier":"Count the number of test samples on which the two modalities disagree and re-run the fusion evaluation with a fixed tie-break (for example, always choose the visual prediction). If ties are frequent and the accuracy moves by more than a few points when the tie-break changes, the 87% figure is not robust; likewise, subject-level cross-validation of the physiological dataset would settle whether the 74% and the fusion gain generalize.","tokens_in":1333,"feed_emoji":"🚗","tokens_out":2096,"duration_ms":115529,"temperature":0.7,"pith_summary":"The paper sets out to show that emotion recognition in a car can be federated—trained across devices without raw data leaving them—without giving up accuracy. It pairs a convolutional neural network that reads facial expressions with a random forest that reads heart rate, electrodermal activity, and skin temperature, and fuses the two at the decision level by majority vote. On a public facial-expression dataset plus a lab-collected physiological dataset, the fused model reaches 87% accuracy, while the vision-only and physiology-only models reach 77% and 74%, respectively, matching the centralized baseline. If the result holds, adaptive driver-assistance and comfort features can be trained on sensitive occupant data while that data stays inside the vehicle.","feed_headline":"Face plus pulse hits 87% in privacy-safe emotion recognition","feed_subtitle":"Federated system fuses face and heart signals in the car, matching centralized accuracy with raw data kept local.","key_machinery":"The load-bearing mechanism is decision-level majority-vote fusion of two heterogeneous classifiers trained under federated averaging. Each client trains a small CNN on its own facial images and a 200-tree random forest on hand-crafted physiological features (heart-rate variability, maximum electrodermal activity, and temperature fluctuation over five-second windows); only the model updates are sent to the server, and the server aggregates them by a FedAvg rule weighted by each client's local data volume. At inference, each modality produces a vote over the seven emotion classes, and Eq. (16) selects the class with the highest summed indicator votes. Because fusion happens after prediction, models of completely different types—a gradient-trained deep network and a non-gradient ensemble—can be combined without sharing gradients, features, or raw samples.","core_discovery":"FedMultiEmo claims that multimodal, decision-level fusion lets a federated system match or exceed centralized accuracy for real-time emotion recognition. The central discovery is concrete: a federated CNN on facial images reaches about 77% accuracy, a federated random forest on physiological features reaches 74%, and their majority-vote fusion reaches 87%—a 10-point gain over the vision-only model—while only model updates, not raw images or signals, are exchanged. The fusion works because the two modalities fail in complementary ways: vision degrades under poor lighting or occlusion, while physiological signals are unaffected by the camera but vary across individuals. The prototype also shows that the approach is deployable: it converges in about 18 rounds with an average round time of 120 seconds and a per-client memory footprint under 200 MB on low-cost edge hardware.","pith_inferences":["Subject-level rather than sample-level evaluation of the self-collected physiological dataset would test whether the 74% random-forest figure and the fusion gain persist across drivers, since physiological responses vary strongly between individuals.","A confidence-weighted fusion rule, which the paper lists as future work, could push accuracy beyond 87% or reveal which emotion classes carry the fusion gain; the confusion matrices suggest disgust benefits most from adding the physiological modality.","The same decision-level design could be adapted to other vehicle-sensing pairs—for example, voice plus facial expression—whenever the modalities are heterogeneous enough that a shared neural architecture would be wasteful."],"forward_implications":["Adaptive in-vehicle responses such as climate, seat, and stress-relief actions could be triggered by emotion state without transmitting facial images or physiological streams to a central server.","The roughly 10-point accuracy gain from fusion over the visual-only model indicates that the system can keep working when the camera view degrades, because the physiological channel still votes.","Because training converges in about 18 rounds with sub-200 MB per-client memory, the approach fits the compute and communication budgets of low-cost in-car edge devices rather than requiring cloud GPUs.","The same majority-vote design can be reused for other heterogeneous model pairs, since only output class votes are exchanged at fusion time."],"supporting_citations":[{"why":"supplies the public facial-expression dataset used to train the visual CNN.","marker":"[30]"},{"why":"provides the crowd-sourced relabeling the authors use to improve visual label quality.","marker":"[31]"},{"why":"supplies the federated orchestration layer used to coordinate the distributed training deployment.","marker":"[29]"},{"why":"provides the Federated Averaging algorithm whose client-data-volume weighting the paper adopts.","marker":"[25]"},{"why":"is the prior federated physiological emotion-recognition system this work contrasts with and extends to multimodal automotive use.","marker":"[26]"}],"fun_headline_variants":["Federated fusion hits 87% for in-car emotion detection","Privacy-safe emotion AI matches central model at 87%","Multimodal federated learning reaches 87% in-car accuracy","Federated face+physiology: 87% emotion accuracy, privacy kept"],"cache_read_input_tokens":13312,"weakest_assumption_plain":"The 87% fusion result rests on a self-collected physiological dataset whose number of subjects, per-class sample counts, and label-validation procedure are not reported, and on an unspecified tie-breaking rule for the majority vote.","fun_headline_variants_meta":{"raw":{"variants":["Federated fusion hits 87% for in-car emotion detection","Privacy-safe emotion AI matches central model at 87%","Multimodal federated learning reaches 87% in-car accuracy","Federated face+physiology: 87% emotion accuracy, privacy kept"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000749,"raw_usage":{"total_tokens":3361,"prompt_tokens":995,"completion_tokens":2366,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":2290}},"tokens_in":611,"tokens_out":2366,"duration_ms":17260,"temperature":1.0,"reasoning_tokens":2290,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:30:44.874419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count the number of test samples on which the two modalities disagree and re-run the fusion evaluation with a fixed tie-break (for example, always choose the visual prediction). If ties are frequent and the accuracy moves by more than a few points when the tie-break changes, the 87% figure is not robust; likewise, subject-level cross-validation of the physiological dataset would settle whether the 74% and the fusion gain generalize.","supporting_citations":[{"cited_title":"Challenges in representation learning: A report on three machine learning contests,","cited_arxiv_id":null,"evidence_quote":"supplies the public facial-expression dataset used to train the visual CNN."},{"cited_title":"Training deep networks for facial expression recognition with crowd-sourced label distribution,","cited_arxiv_id":null,"evidence_quote":"provides the crowd-sourced relabeling the authors use to improve visual label quality."},{"cited_title":"Fed-phyers: Federated learning- based multi-modal emotion recognition using physiological signals,","cited_arxiv_id":null,"evidence_quote":"is the prior federated physiological emotion-recognition system this work contrasts with and extends to multimodal automotive use."}],"review_version":1}