Pith. sign in

REVIEW 3 major objections 4 minor 22 references

Go witheFlow: Real-time Emotion Driven Audio Effects Modulation

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Pith's one-line read witheFlow maps a performer's EEG attention, ECG stress, and audio emotion into real-time gain changes on DAW effect chains, keeping the musician in creative control.

desk verdict Honest, clearly-described proof-of-concept for bio-signal-driven audio FX; the load-bearing risk is the unvalidated audio emotion regressor, but the paper itself flags it. read the letter →

arxiv 2510.02171 v3 pith:E3UAUPRQ submitted 2025-10-02 cs.SD cs.AIeess.AS

classification cs.SDcs.AIeess.AS
keywords witheFlowreal-timeaudioeffectsmodulationbiosignal-drivenmusicperformancevalence-arousalBaevskyStressIndexEEGattentionrelaxationrule-basedmixinglogichuman-AIco-creativity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

witheFlow is a proof-of-concept system that lets a live musician's physiology and the emotional character of their dry audio drive the mix of audio effects in real time. It combines EEG-derived attention and relaxation, an ECG-derived Baevsky Stress Index, and valence-arousal estimates from a neural audio model, then applies YAML-defined gain rules to boost or suppress parallel effect channels in a DAW. The paper argues that this kind of AI-human collaboration — where machines handle real-time technical processing and the performer keeps creative agency — is a useful alternative to AI as an autonomous composer. A sympathetic reader would care because it demonstrates a concrete, local, open-source path toward emotionally responsive instruments that performers can customize and override.

What carries the argument

The load-bearing components are the Baevsky Stress Index (a heart-rate-variability metric computed from the mode and range of RR intervals), the attention and relaxation ratios derived from EEG alpha and beta power over four-second windows, and the valence-arousal regressor built by replacing the final layer of a PANNs CNN10 with a two-output linear head trained on DEAM. The mixing logic is a transparent piecewise map: the input domain (stress, attention, valence, arousal) is partitioned into regions, each paired with a Python gain function; user-editable YAML files define the region conditions and functions. This formalization is what makes the system interpretable and prepares it for a fut

What would settle it

Collect solo recordings with human valence-arousal ratings, run the system's regressor on them, and check correlation with the ratings; if predictions are uncorrelated or systematically shifted, the V-A-distance rules in all four rulesets are being driven by unreliable features. A simpler laboratory test is to feed synthetic V-A values into the mixing logic and verify that gain changes match the specified quadrant rules exactly.

Watch

Extended reading notes

Core claim

The central claim is that a laptop-runnable system can close a feedback loop between a performer's measurable internal state and their sonic output without altering musical content. Signals from commercial EEG and ECG sensors are converted into attention/relaxation and stress values; a PANNs CNN10 retrained on DEAM regresses the dry audio's valence and arousal; a rule-based mixing logic, formalized as a piecewise function over a partitioned input domain, translates these values into gain changes on multiple effects channels. The gain logic is organized by stress/attention quadrants: high stress pushes the mix toward V-A positions far from the dry signal, while low stress keeps it close, with

Load-bearing premise

The load-bearing premise is that the audio emotion regressor — PANNs CNN10 trained on DEAM — gives trustworthy valence-arousal estimates for dry solo performance audio in real time; the paper itself notes that emotion datasets typically contain full productions rather than solo performances.

Editorial extensions

If this is right

  • Live performers can shape their effect mix hands-free, with gain decisions driven by measurable internal state rather than manual knob-twiddling.
  • Because the rules are editable YAML/Python files, musicians can inspect exactly why a gain changed and customize or reverse the behavior, supporting trust and agency.
  • Local, laptop-based processing keeps biosignal data private and avoids cloud latency, making the approach practical in real performance settings.
  • The piecewise-rule formalization points directly to a learnable mixing logic, where decision trees could refine user-defined rules while staying interpretable.
  • The system's reliance on solo performance audio makes annotated solo-performance datasets with real-time labels a critical next resource.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the DEAM-trained regressor is systematically biased on solo instrument audio, every biosignal ruleset inherits that bias, since all rules compare the dry signal's V-A position with each effect channel's V-A position; a validation study on solo recordings would settle this.
  • The architecture is feature-agnostic, so the same rule-based loop could be closed with video, motion, or other physiological signals without redesigning the mixing logic.
  • The closed loop between internal state and sound raises a testable safety question: whether the system's own audio changes could feed back into the performer's physiology over long sessions; the foot-pedal override mitigates but does not measure this.
  • A quantitative evaluation framework for emotion-driven mixing systems, which the paper identifies as missing, could be built from the V-A-distance principle underlying the rules.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces witheFlow, a proof-of-concept system for real-time audio effects modulation driven by a combination of biosignal features (EEG-derived attention/relaxation, ECG-derived Baevsky Stress Index) and audio-based valence-arousal estimates. It describes the system architecture, DAW routing, a customizable YAML rule-based mixing logic, artifact handling, and reports anecdotal positive feedback from collaborating musicians. The central contribution is the integration and open-source implementation rather than a formal empirical evaluation; the paper explicitly states that a quantitative evaluation framework is future work.

Significance. If the system's underlying emotion estimates are trustworthy, witheFlow is a relevant contribution to real-time co-creative performance systems: it is lightweight, local, interpretable, and provides performer control via MIDI override. The architecture is coherent, the formulas for attention/relaxation and Baevsky SI are standard, and the decision to use transparent rule-based mixing rather than an opaque end-to-end model is well motivated. The paper's strengths include its open-source commitment and its careful ethical discussion of biosignal privacy and psychological safety. The main risk is that the audio V-A regressor, which is load-bearing for all four rulesets, is not validated on the solo dry instrumental material that the system actually processes.

major comments (3)
  1. [§2, Audio Emotion Regressor; Table 1] The audio V-A regressor is load-bearing for every rule in Table 1, since each rule compares the dry signal's V-A position with each effect channel's V-A position. The manuscript concedes that DEAM contains full music productions rather than solo performances, but reports no validation of the regressor on DEAM's test split or on solo/instrumental material. If the V-A estimates are systematically biased for dry solo input, all four rulesets are corrupted and the 'emotion-driven' behavior reduces to an arbitrary effect switcher. Please add quantitative validation (e.g., DEAM test-set metrics and a small solo-performance evaluation) or explicitly restrict the system claims to the design/proof-of-concept level.
  2. [§3, Discussion; §5, Conclusion] The abstract and conclusion state that the system 'enhances' music performance, but the only supporting evidence is the statement in Section 3 that musicians 'have reported a generally positive experience.' No details are given on the number of musicians, instruments, tasks, instructions, or data collection. Either report a systematic qualitative evaluation (even a small-N study with a clear protocol) or reframe the claims to say that the system is intended to enhance and that only anecdotal feedback has been collected so far. The paper's own Section 3 says formal evaluation is future work, so the current wording overstates the evidence.
  3. [§2, Mixing Logic; Table 1] Table 1 defines behavior in terms of 'High/Low' stress, attention, valence, and arousal, but the threshold values and calibration procedure are not specified. This makes the default behavior non-reproducible from the text alone. Please provide the default threshold values (or point to the exact configuration lines in the GitHub repository) and describe how thresholds are calibrated for individual performers.
minor comments (4)
  1. [Title/Abstract] The system name is typeset inconsistently: 'Go witheFlow', 'witheFlow', and 'witheflow' (in the YouTube link). Please standardize.
  2. [§2, Audio Routing] Typo: 'The performer creates up toneffects chains' should read 'up to n effects chains' (or similar).
  3. [§2, Audio Emotion Regressor] Use a space in '30kHz' and state the original PANNs input sampling rate for comparison.
  4. [§4, Ethical considerations] The institution is named 'National and Technical University of Athens' here but 'National Technical University of Athens' in the affiliation. Please make this consistent.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the system is an openly described proof-of-concept whose components come from external sources and whose rules are disclosed design choices, not derived predictions.

full rationale

The paper does not derive any result from its own output. The two feature extractors are external: the audio emotion regressor is a PANNs CNN10 trained on the DEAM dataset, and the biosignal features (attention/relaxation, Baevsky SI) are standard definitions with external citations. The mixing logic is presented as hand-written, customizable rules, with the paper explicitly stating 'the mixing logic is fully customizable' and that the functions 'can easily be customized by the end user' — these are design choices, not predictions fitted to data. The formalization as a piecewise function over a partitioned domain is a description of the implemented rule set, not a derivation that presupposes its conclusion. The only self-citations ([7], [8]) are used for framing and for a preliminary dataset analysis; neither is load-bearing for the system's claimed functionality. The paper's own concession that 'datasets typically contain full music productions and not solo performances' and its statement that 'a formal framework for quantitative evaluation' remains future work identify a correctness/validation risk, but not circularity: the audio regressor is trained externally and no evaluation on solo material is claimed. Overall, the claim is a working prototype with disclosed components and no hidden fit-by-construction step.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper adds no fitted constants in the physics sense, but it depends on hand-chosen thresholds and windows, on unvalidated domain bridges (EEG→attention, HRV→stress, DEAM-model→solo-audio affect), and on an ad hoc rule table that encodes design preference in place of evidence.

free parameters (3)
  • High/Low thresholds for stress, attention, and V-A rule conditions
    The YAML rulesets use conditions of the form a < x < b; the values of a and b define the partition of the input domain D and determine which gain function fires (Section 2, Mixing Logic; Table 1). No threshold values are disclosed.
  • Feature window sizes (EEG 4 s / 1000 samples, ECG 15 s, audio 5 s) = EEG 1000 samples; ECG 15 s; audio 5 s
    Hand-chosen design parameters trading responsiveness against feature stability; they define the effective temporal resolution of every stream (Section 2, Biosignals and Audio Emotion Regressor).
  • V-A coordinates assigned to each FX channel
    The 'closest'/'furthest' and arousal-direction gain functions require a valence-arousal position per effect channel, but the paper does not specify how these positions are assigned or calibrated (Section 2, Mixing Logic).
assumptions (5)
  • domain assumption Increased beta power indicates heightened attention and increased alpha power indicates greater relaxation (§2, Biosignals).
    Assumed from cited literature [12], [16]; no validation is given for performing musicians under motion and sensor-artifact conditions.
  • domain assumption The Baevsky Stress Index computed from 15-s RR-interval windows quantifies the performer's momentary stress (§2, Biosignals).
    Baevsky SI is a standard HRV metric from [15], but the 15-s sliding-window adaptation for real-time use is unvalidated in the paper.
  • domain assumption Valence-Arousal from Russell's circumplex adequately represents the emotion of dry solo audio, and PANNs CNN10 retrained on DEAM estimates it reliably in the deployment domain (§2, Audio Emotion Regressor).
    The paper itself concedes that DEAM contains 'full music productions and not solo performances', so generalization to the deployment domain is assumed rather than shown.
  • ad hoc to paper The intuitive mapping of Table 1 holds: high stress is 'somehow expressed via their performance' so boosting distant FX is appropriate; low stress calls for proximity; attention can be used as an indicator of intent (§2, Mixing Logic).
    This is the load-bearing design hypothesis connecting biosignals to gain actions, stated verbatim as 'the assumption that the musician's stress is somehow expressed via their performance'. No experiment tests it.
  • standard math The piecewise-function partition notation correctly formalizes the mixing logic (§2, Mixing Logic).
    Unproblematic and near-tautological; included for completeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Go witheFlow: Real-time Emotion Driven Audio Effects Modulation." pith.science (2026). https://pith.science/paper/E3UAUPRQ

@misc{pith2026251002171,
  author       = {Pith},
  title        = {Pith review of: Go witheFlow: Real-time Emotion Driven Audio Effects Modulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E3UAUPRQ}},
  note         = {Machine review of arXiv:2510.02171}
}
read the original abstract

Music performance is a distinctly human activity, intrinsically linked to the performer's ability to convey, evoke, or express emotion. Machines cannot perform music in the human sense; they can produce, reproduce, execute, or synthesize music, but they lack the capacity for affective or emotional experience. As such, music performance is an ideal candidate through which to explore aspects of collaboration between humans and machines. In this paper, we introduce the witheFlow system, designed to enhance real-time music performance by automatically modulating audio effects based on features extracted from both biosignals and the audio itself. The system, currently in a proof-of-concept phase, is designed to be lightweight, able to run locally on a laptop, and is open-source given the availability of a compatible Digital Audio Workstation and sensors.

Figures

Figures reproduced from arXiv: 2510.02171 by the authors.

Figure 1
Figure 1. Overview of how audio Valence-Arousal (VA) interact with audio effects based on biosig [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The witheFlow System. beta power correlates with heightened attention [12], while elevated alpha power indicates greater relaxation [16]. These metrics are defined as: Attention = Beta Power Alpha Power + Beta Power , Relaxation = Alpha Power Alpha Power + Beta Power . The ECG signal is sampled at 1000 Hz. Stress is estimated using a sliding window approach: every 0.5 seconds (500 samples), RR intervals from the pre… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 4 linked inside Pith

  1. [1]

    Developing a benchmark for emotional analysis of music.PLOS ONE, 12(3):e0173392, 2017

    Anna Aljanaki, Yi-Hsuan Yang, and Mohammad Soleymani. Developing a benchmark for emotional analysis of music.PLOS ONE, 12(3):e0173392, 2017

  2. [2]

    A systematic review of artificial intelligence-based music generation: Scope, applications, and future trends

    Miguel Civit, Javier Civit-Masot, Francisco Cuadrado, and Maria J Escalona. A systematic review of artificial intelligence-based music generation: Scope, applications, and future trends. Expert Systems with Applications, 209:118190, 2022

  3. [3]

    Using artificial intelligence for enhancing human creativity

    Mayssa Elfar and Mina Dawood. Using artificial intelligence for enhancing human creativity. Journal of Art, Design and Music, 2:106–120, 07 2023

  4. [4]

    Ddsp: Differentiable digital signal processing.arXiv preprint arXiv:2001.04643, 2020

    Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts. Ddsp: Differentiable digital signal processing.arXiv preprint arXiv:2001.04643, 2020

  5. [5]

    Eeg-neurofeedback for optimising performance

    John H Gruzelier. Eeg-neurofeedback for optimising performance. i: A review of cognitive and affective outcome in healthy participants.Neuroscience & Biobehavioral Reviews, 44:124–141, 2014

  6. [6]

    Plumbley

    Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880–2894, 2020

  7. [7]

    Challenges and perspectives in interpretable music auto-tagging using perceptual features.IEEE Access, 2025

    Vassilis Lyberatos, Spyridon Kantarelis, Edmund Dervakos, and Giorgos Stamou. Challenges and perspectives in interpretable music auto-tagging using perceptual features.IEEE Access, 2025

  8. [8]

    Music interpretation and emotion perception: A computational and neurophysiological investigation, 2025

    Vassilis Lyberatos, Spyridon Kantarelis, Ioanna Zioga, Christina Anagnostopoulou, Giorgos Stamou, and Anastasia Georgaki. Music interpretation and emotion perception: A computational and neurophysiological investigation, 2025

Show all 22 references
  1. [9]

    Bahar Mahmud, Guan Hong, and Bernard Fong. A study of human–ai symbiosis for creative work: Recent developments and future directions in deep learning.ACM Transactions on Multimedia Computing, Communications and Applications, 20(2):1–21, 2023

  2. [10]

    Autonomy, authenticity, authorship and intention in computer generated art

    Jon McCormack, Tom Gifford, and Pat Hutchings. Autonomy, authenticity, authorship and intention in computer generated art. InProceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 15, pages 53–59, 2019

  3. [11]

    Music genre classification: A review of deep-learning and traditional machine-learning approaches

    Ndiatenda Ndou, Ritesh Ajoodha, and Ashwini Jadhav. Music genre classification: A review of deep-learning and traditional machine-learning approaches. In2021 IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS), pages 1–6. IEEE, 2021

  4. [12]

    Increase in beta power reflects attentional top-down modulation after psychosocial stress induction

    Ismael Palacios-García, Jaime Silva, Mario Villena-González, Germán Campos-Arteaga, Clau- dio Artigas-Vergara, Nicolas Luarte, Eugenio Rodríguez, and Conrado A Bosman. Increase in beta power reflects attentional top-down modulation after psychosocial stress induction. Frontier...

  5. [13]

    AR Editions, Inc., 1995

    Joseph Rothstein.MIDI: A comprehensive introduction, volume 7. AR Editions, Inc., 1995

  6. [14]

    James A. Russell. A circumplex model of affect.Journal of Personality and Social Psychology, 39(6):1161–1178, 1980

  7. [15]

    Stress index calculation and analysis based on heart rate variability of ecg signal with arrhythmia

    Tanmay Kumar Sahoo, Ashutos Mahapatra, and Nersisson Ruban. Stress index calculation and analysis based on heart rate variability of ecg signal with arrhythmia. In2019 Innovations in Power and Advanced Computing Technologies (i-PACT), volume 1, pages 1–7. IEEE, 2019

  8. [16]

    Elec- troencephalographic biomarkers of relaxation: A systematic review and meta-analysis.bioRxiv, March 2024

    Kairi Sugimoto, Hideaki Kurashiki, Yuting Xu, Mitsuaki Takemi, and Kaoru Amano. Elec- troencephalographic biomarkers of relaxation: A systematic review and meta-analysis.bioRxiv, March 2024

  9. [17]

    Towards a human-centric design framework for ai assisted music production

    Augoustinos Tsiros and Alessandro Palladini. Towards a human-centric design framework for ai assisted music production. InProceedings of the International Conference on New Interfaces for Musical Expression, pages 399–404, 2020. 7

  10. [18]

    Ai see, you see: Human-ai musical collaboration in augmented reality

    Yichen Wang and Charles Patrick Martin. Ai see, you see: Human-ai musical collaboration in augmented reality. InProceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA ’25, New York, NY , USA, 2025. Association for Computing Machinery

  11. [19]

    Midi-ddsp: Detailed control of musical performance via hierarchical modeling.arXiv preprint arXiv:2112.09312, 2021

    Yusong Wu, Ethan Manilow, Yi Deng, Rigel Swavely, Kyle Kastner, Tim Cooijmans, Aaron Courville, Cheng-Zhi Anna Huang, and Jesse Engel. Midi-ddsp: Detailed control of musical performance via hierarchical modeling.arXiv preprint arXiv:2112.09312, 2021

  12. [20]

    A clustering-based approach to automatic harmonic analysis: An exploratory study of harmony and form in mozart’s piano sonatas

    Jason Yust, Jaeseong Lee, and Eugene Pinsky. A clustering-based approach to automatic harmonic analysis: An exploratory study of harmony and form in mozart’s piano sonatas. Transactions of the International Society for Music Information Retrieval, 2022

  13. [21]

    Symbolic music representations for classification tasks: A systematic evalua- tion.arXiv preprint arXiv:2309.02567, 2023

    Huan Zhang, Emmanouil Karystinaios, Simon Dixon, Gerhard Widmer, and Carlos Eduardo Cancino-Chacón. Symbolic music representations for classification tasks: A systematic evalua- tion.arXiv preprint arXiv:2309.02567, 2023

  14. [22]

    A survey of ai music generation tools and models.arXiv preprint arXiv:2308.12982, 2023

    Yueyue Zhu, Jared Baca, Banafsheh Rekabdar, and Reza Rawassizadeh. A survey of ai music generation tools and models.arXiv preprint arXiv:2308.12982, 2023. 8

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.