{"id":"a0e3cf24-ae43-4785-afbb-18011f554bf6","arxiv_id":"2504.21184","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AffectEval is a modular open-source framework for affective computing that claims to cut programming effort by up to 90% while replicating prior stress and affect detection results.","lead":"The authors built AffectEval, an open-source Python framework that packages the steps of an affective computing pipeline into reusable, customizable modules. The paper reports that pipelines built with it replicate or beat two prior stress-detection studies while using up to 90% fewer lines of code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 contradicts the paper's central 'same or better across all experiments' claim, so the validation does not support the strongest claim as stated.","rationale":"The reader correctly flagged that the replication may not be controlled and that the paper contains a contradiction between the textual claim and Table 2. My stress-test sharpens this into a narrower, internally checkable failure: the paper's own results include multiple rows where AffectEval is substantially worse than the original, so the central claim 'same or better across all experiments' is false regardless of whether the original methods were faithfully reproduced. This is not a disagreement with outside consensus or a matter of missing evidence; it is an internal inconsistency between the prose and the reported numbers. The framework itself may be useful, and the open-source release and modular design are positive contributions, but the empirical validation as reported does not support the strongest claim. A conditional acceptance would require the authors to rewrite the central claim and provide controlled replication evidence; given that the current claim is directly contradicted by Table 2, the appropriate verdict for the paper in its present form is REJECT, with the expectation that a revised version could be reconsidered.","tokens_in":15651,"tokens_out":4878,"duration_ms":53424,"concrete_test":"Re-run the WESAD SVM experiment using AffectEval but with the exact label-generation and feature-extraction code from Zhou et al.; if accuracy recovers to roughly 86.0% and AUC to roughly 72.0%, then AffectEval's default pipeline is not equivalent to the original, and the 'same or better' claim should be scoped to the specific AffectEval configuration. If accuracy remains near 57.16%, the discrepancy is inherent to the AffectEval pipeline and not explainable by seeding or unspecified parameters, confirming that the universal claim in Section 4.3 is false.","verdict_should_be":"REJECT","load_bearing_attack":"Section 4.3 states: 'Our pipelines achieved the same or better accuracies, AUC scores, and F1-scores across all experiments.' Table 2 directly contradicts this. For WESAD, AffectEval's SVM accuracy is 57.16 ± 1.25 versus the original 86.0, and the ensemble accuracy is 86.29 versus 99.0. For the Schmidt 3-class task, AffectEval's AB accuracy is 77.60 ± 4.10 versus 80.34, and LDA accuracy is 69.83 ± 2.61 versus 79.35. These are not 'same or better' results; at least four rows are worse. The table caption attributes discrepancies to 'preprocessing and feature extraction methods from different libraries' and unspecified model parameters, which is itself an admission that the pipelines are not controlled replications. Because the central claim in the abstract and Section 4.3 is universal, one counterexample row is sufficient to falsify it. The LOC-reduction estimate in Section 4.4 is separately unverifiable because the baselines are not public, but the performance claim is checkable from the paper's own numbers and fails.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AffectEval, a modular, object-oriented framework for building affective computing pipelines from physiological signals. The framework provides six components (signal acquisition, preprocessing, feature extraction, feature selection, label generation, and classification) with default behaviors that users can override, and it establishes a standardized dataset folder format. The authors validate AffectEval by replicating two prior studies: Schmidt et al. (2018) on the WESAD dataset (three-class and binary affect classification) and Zhou et al. (2023) on the APD and WESAD datasets (binary stress detection). They report that their pipelines achieve the same or better accuracy, AUC, and F1-scores across all experiments, and they claim up to a 90% reduction in programming effort measured by raw lines of code. The paper also compares AffectEval with prior frameworks (metaFERA, AffectToolbox) and discusses limitations such as lack of real-time and distributed support.","tokens_in":15898,"tokens_out":2951,"duration_ms":29908,"significance":"AffectEval is a potentially useful open-source contribution: it is the first framework, to the authors' knowledge, that covers all pipeline components including feature selection and label generation, and it explicitly supports multimodal physiological signals across multiple application domains. The reuse of pipeline structure across two different studies demonstrates a concrete engineering benefit, and the line-of-code reduction, if properly verified, would be a meaningful usability improvement. The validation approach is also commendable in principle: reproducing external benchmarks (Schmidt et al.) gives independent grounding. However, the paper's central empirical claim—'same or better across all experiments'—is contradicted by its own Table 2, and the replication is not controlled due to acknowledged differences in preprocessing, feature extraction libraries, and unspecified model parameters. The line-of-code metric is unverifiable because the baseline code is not public. These issues undermine the paper's stated contributions in its current form, although they are addressable through revised claims and additional disclosure.","major_comments":[{"comment":"The claim that 'Our pipelines achieved the same or better accuracies, AUC scores, and F1-scores across all experiments' is false as stated. Table 2 reports several AffectEval results that are substantially worse than the original: WESAD SVM accuracy is 57.16 ± 1.25 vs. 86.0; WESAD ensemble accuracy is 86.29 vs. 99.0; Schmidt et al. three-class AB accuracy is 77.60 ± 4.10 vs. 80.34; LDA accuracy is 69.83 ± 2.61 vs. 79.35. Since the sentence makes a universal claim, even a single counterexample invalidates it. The abstract and conclusion repeat the 'same or higher' phrasing, so this is not a local omission but a load-bearing misstatement that must be corrected and replaced with an accurate summary of the comparison.","section":"Section 4.3, Table 2"},{"comment":"The validation is not a controlled replication. The text first says the authors used 'the same preprocessing methods, physiological features, labels, and classification models outlined by the authors,' but later attributes discrepancies to 'preprocessing and feature extraction methods from different libraries' and 'model parameters that were not specified in previous work.' These two statements are inconsistent: if the pipelines differ in preprocessing, feature extraction, or model configuration, then the accuracy comparison does not isolate the effect of the framework. The authors should either align their implementation exactly with the original protocols (to the extent possible), or explicitly reframe the results as 'an AffectEval-based reimplementation' rather than a replication, and analyze which methodological differences explain the large performance gaps (e.g., WESAD SVM dropping from 86.0 to 57.16).","section":"Section 4.3"},{"comment":"The 90%/89% reduction in lines of code is not verifiable from the information provided. The original code for Schmidt et al. and Zhou et al. is not public, so the comparison baseline is an estimate made by the authors. The metric 'raw lines of code' is also a poor proxy for programming effort unless the counting methodology (including comments, blank lines, library calls, and custom functions) is precisely defined and the original code is made available or described in detail. Without these, the headline 'reduces programming effort by up to 90%' is an unsubstantiated quantitative claim. Please provide the counting rules and, if possible, the original code or a detailed reconstruction of the baseline.","section":"Section 4.4"},{"comment":"The label-generation thresholds are free parameters whose influence on the comparison is not analyzed. For APD, the authors choose a fixed SUDS threshold of 50; for WESAD, they use a per-subject average STAI threshold. If the original studies used different binarization rules, the label distributions would differ and the classification accuracies would not be comparable. The paper does not confirm that these thresholds match those of Schmidt et al. and Zhou et al., nor does it report the class balance produced by these thresholds. Please verify the thresholds against the original papers or state explicitly that these choices are part of the AffectEval implementation, and discuss their effect on the reported results.","section":"Section 4.2"}],"minor_comments":[{"comment":"The sentence 'one-hot encoding is automatically performed is automatically performed for categorical features' contains a duplicated phrase; it should read 'one-hot encoding is automatically performed for categorical features.'","section":"Section 3.5"},{"comment":"In the Zhou et al. section of Table 2, the AffectEval Ensemble row shows identical values for APD and WESAD (86.29 accuracy, 67.52 AUC vs. 86.29 accuracy, 67.53 AUC). This is almost certainly a transcription or copy-paste error and should be corrected, as it currently obscures the WESAD ensemble comparison (original: 99.0 accuracy, 96.9 AUC).","section":"Table 2"},{"comment":"The abstract claims 'reduces programming effort by up to 90%, as measured by the reduction in raw lines of code,' while Section 4.4 reports 90% for Schmidt et al. and 89% for Zhou et al. The phrasing should be made consistent and the 'up to' qualifier clarified to indicate which experiment yields the maximum.","section":"Abstract and Section 4.4"},{"comment":"The phrase 'we replicate a subset of the findings of Schmidt et al. using the physiological modalities from WESAD' is slightly inconsistent with Section 4.3's blanket 'replicating [40, 51]' language; please use consistent terminology about the scope of replication throughout.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"One author (Emily Zhou) is also the first author of Zhou et al. [51], one of the two studies being replicated. This creates a conflict of interest that is not disclosed in the manuscript. While the replication of Schmidt et al. provides some independent grounding, the self-replication aspect increases the risk of hidden tuning or implicit benchmark familiarity. The authors should be asked to disclose this relationship explicitly, and the published version should include a statement about it. The paper's fit with the journal is acceptable as a software/engineering contribution, but the central validation claim must be corrected before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The framework is worth having. AffectEval packages the affective-computing pipeline into six clean components, adds feature selection and label generation to the usual architecture, ships pre-implemented behaviors, and is open source. That is a real contribution for researchers who are tired of wiring biosppy, neurokit, and scikit-learn together from scratch. The literature review and the comparison against metaFERA and AffectToolbox are also useful: the paper correctly identifies that those tools either require too much manual implementation or do not cover physiological modalities.\n\nThe problem is the validation. Section 4.3 says the pipelines \"achieved the same or better accuracies, AUC scores, and F1-scores across all experiments,\" and the abstract repeats the performance message. Table 2 says otherwise. For WESAD, AffectEval's SVM gets 57.16% accuracy versus the original 86.0%, and the ensemble gets 86.29% versus 99.0%. For Schmidt's three-class task, AdaBoost and LDA are also clearly worse. This is not a minor discrepancy; it is a direct contradiction between the paper's headline claim and its own results table. A single counterexample row would have been enough to falsify a universal claim, and there are several.\n\nThe table caption blames the differences on \"preprocessing and feature extraction methods from different libraries\" and unspecified model parameters. That is an admission that the replication is not controlled. If the preprocessing or feature set differs, the comparison is not a fair test of the framework; it is just a different pipeline. The authors should either rerun the original methods exactly, or report the results honestly as \"comparable in some cases, better in others, worse in others,\" with an analysis of why. The line-of-code reduction is also unverifiable: the baselines are not public, and the estimate covers only a subset of components. It is a reasonable illustration, but \"up to 90%\" should be presented as an estimate, not a measured claim.\n\nOne more note: the first author of this paper is also the first author of the Zhou et al. study being replicated. That is not fatal, but it makes the comparison less independent than it could be.\n\nAll that said, the underlying framework is plausible and the paper is written clearly. The core problem is overclaiming, not a broken artifact. I would send this to peer review, with a strong request to fix the performance claims and either tighten or soften the replication narrative. The framework deserves to be in the literature; the current validation does not support the current abstract.","headline":"AffectEval is a genuinely useful modular framework, but the paper's central 'same or better' claim is contradicted by its own Table 2, so the validation needs a major fix.","tokens_in":16414,"tokens_out":2963,"would_cite":false,"duration_ms":32100,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AffectEval claims to cut affective-computing pipeline code by up to 90% while matching or beating prior accuracy.","keywords":["affective computing","software framework","pipeline","physiological signals","emotion recognition","stress detection","reproducibility","modular architecture"],"falsifier":"Re-run both replicated studies from their original released code on the same raw WESAD and APD files, using identical preprocessing, features, labels, train/test splits, and random seeds, then run the AffectEval pipelines alongside; the central claim fails if AffectEval's metrics are not at least comparable or if the line-count reduction disappears when custom label-generation and preprocessing code is included.","tokens_in":15449,"feed_emoji":"💓","tokens_out":7096,"duration_ms":65240,"temperature":0.7,"pith_summary":"AffectEval is a proposed take on a recurring problem in affective computing: building emotion- and stress-recognition pipelines from scratch for each new dataset or application, which wastes effort and makes results hard to compare. The paper claims that a modular framework with six pipeline components—signal acquisition, preprocessing, feature extraction, feature selection, label generation, and classification—can cover the full range of multimodal physiological affect-recognition tasks with pre-implemented defaults that users can override. To support this, the authors reimplemented two published studies on the WESAD and Anxiety Phases datasets and report that their pipelines matched or exceeded the original accuracy, AUC, and F1 scores while cutting raw lines of code by up to 90 percent. If the claim holds, researchers could assemble end-to-end affective computing pipelines in far less time and reuse the same components across domains such as stress, general affect, and clinical applications.","feed_headline":"One framework rebuilds emotion-detection pipelines with 90% less code","feed_subtitle":"Modular components reproduce two published stress and affect studies with equal or better scores, in far fewer lines of code.","key_machinery":"The carrying mechanism is the component-and-pipeline architecture. Six classes—Signal Acquisition, Signal Preprocessor, Feature Extractor, Feature Selector, Label Generator, and Classification—each inherit from an abstract base class, ship with default behaviors built on standard signal-processing and machine-learning libraries, and allow method-level overrides. Users instantiate the components they need, wire them into an ordered list whose input and output types are compatible, and hand the list to a Pipeline that executes them in sequence. This ordering plus the standardized dataset layout is what makes the framework reusable across signals and domains; the same pipeline structure is reused for the two replicated studies with only the component parameters changed.","core_discovery":"The central claim is that a single modular framework can serve as a general substrate for affective computing research. AffectEval organizes the standard pipeline into independent components, each with a default implementation and an interface that lets users substitute custom methods, then chains them in an ordered list executed by a Pipeline object. The paper adds feature selection and label generation as first-class components beyond the usual acquisition-preprocessing-feature-extraction-classification stages, and it introduces a fixed folder-and-CSV dataset format to cut setup work. As evidence, it reproduces the affect-classification experiments of two prior studies on two multimodal datasets and reports performance that is the same or better across all experiments, with effort reductions of 90% and 89% measured in raw lines of code.","pith_inferences":["The 90% figure counts only preprocessing, feature extraction, and classification code; data formatting and label generation are excluded because they are needed regardless. For a new project, those excluded steps may dominate, so the realized effort saving could be smaller than 90%.","A fairer test of the framework's promise would measure the time a new user needs to go from raw data to working pipeline on a dataset the authors did not touch; the paper does not provide this measure.","Because the framework's components are defined by input/output compatibility, the same architecture should extend naturally to audio, text, or video signals; adding default methods for those modalities would be a concrete test of the design rather than a rewrite."],"forward_implications":["A researcher working on a new stress- or emotion-detection dataset can start from AffectEval's defaults and only customize the parts that differ, instead of writing a full pipeline by hand.","Because components are swappable, the same framework provides a controlled setting for comparing preprocessing methods, feature extractors, feature selectors, and classifiers on identical data and labels.","The standard dataset format lowers the barrier to reproducing published experiments, since acquisition and label-generation scaffolding no longer need to be recreated for each paper.","If the replication results hold, the framework offers a single codebase that spans multiple domains—general affect, stress, and potentially depression detection—without sacrificing classification performance."],"supporting_citations":[{"why":"Supplies the WESAD dataset and the affect-classification experiments that AffectEval reproduces; its reported metrics are the baseline for the same-or-better claim.","marker":"[40]"},{"why":"Supplies the stress-detection experiments on APD and WESAD that AffectEval reproduces, including the label-generation strategy and baseline metrics.","marker":"[51]"},{"why":"Defines the canonical affective-computing pipeline stages that AffectEval extends with feature selection and label generation, and serves as the comparison meta-framework.","marker":"[35]"},{"why":"Provides the Anxiety Phases Dataset used as one of the two validation datasets for binary stress detection.","marker":"[41]"},{"why":"Supplies default signal-preprocessing and feature-extraction behaviors that AffectEval wraps; the paper cites library differences as a source of performance discrepancies.","marker":"[31]"},{"why":"The GUI-based comparative framework whose limited modality support motivates AffectEval's multimodal design.","marker":"[34]"},{"why":"Supplies the default classifiers and feature selector behind AffectEval's Classification and Feature Selector components.","marker":"[37]"}],"fun_headline_variants":["AffectEval: cut affect-coding effort by 90%","Modular framework reproduces affect studies with 90% less code","Emotion pipelines: one framework, 90% less code","AffectEval: build multimodal emotion pipelines with 90% less code","Reusable components cut emotion-detection code by 90%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim stands on the premise that AffectEval's reproduction pipelines genuinely match the original studies' preprocessing, feature extraction, labels, and evaluation protocols; if those details drift, the reported accuracy comparison is not controlled and the effort reduction is measured against an unverified baseline.","fun_headline_variants_meta":{"raw":{"variants":["AffectEval: cut affect-coding effort by 90%","Modular framework reproduces affect studies with 90% less code","Emotion pipelines: one framework, 90% less code","AffectEval: build multimodal emotion pipelines with 90% less code","Reusable components cut emotion-detection code by 90%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000277,"raw_usage":{"total_tokens":1596,"prompt_tokens":834,"completion_tokens":762,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":671}},"tokens_in":450,"tokens_out":762,"duration_ms":7010,"temperature":1.0,"reasoning_tokens":671,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:10:24.409558+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run both replicated studies from their original released code on the same raw WESAD and APD files, using identical preprocessing, features, labels, train/test splits, and random seeds, then run the AffectEval pipelines alongside; the central claim fails if AffectEval's metrics are not at least comparable or if the line-count reduction disappears when custom label-generation and preprocessing code is included.","supporting_citations":[{"cited_title":"Alarcão, Teresa Chambel, and Manuel J","cited_arxiv_id":null,"evidence_quote":"Defines the canonical affective-computing pipeline stages that AffectEval extends with feature selection and label generation, and serves as the comparison meta-framework."},{"cited_title":"Lau, Jan C","cited_arxiv_id":null,"evidence_quote":"Supplies default signal-preprocessing and feature-extraction behaviors that AffectEval wraps; the paper cites library differences as a source of performance discrepancies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The GUI-based comparative framework whose limited modality support motivates AffectEval's multimodal design."}],"review_version":1}