REVIEW 4 major objections 5 minor 44 references
Feel my Speech: Automatic Speech Emotion Conversion for Tangible, Haptic, or Proxemic Interaction Design
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that converting automatic speech-emotion recognition outputs into physical, felt, and spatial forms makes emotions usable as interaction design material, and it contributes a starter kit for doing so.
desk verdict A clear design-position paper that packages existing SER with simple actuator mappings; the argument is plausible and honestly framed, but the central claim rests entirely on untested assumptions about SER reliability and user perception. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the three-part SEC pipeline. Sensing: a wav2vec 2.0-based transformer model, exposed through an affect-analysis toolbox, predicts continuous valence, arousal, and dominance from raw speech audio. Mapping: a configurable command-line tool chunks the audio stream and applies user-specified rules that convert the emotion values into physical or spatial outputs. Display: an Arduino sketch drives a vibrotactile motor and a Processing sketch visualizes the values, both receiving the SER results over a serial or TCP connection. The work of the pipeline is to turn an abstract ML output into a felt, manipulable design material while keeping each stage replaceable.
What would settle it
A controlled study would settle it: record speech from actors expressing distinct emotions, run the SEC pipeline, have blind participants feel the resulting haptic outputs (or observe the proxemic behaviour), and ask them to classify the intended emotion; if classification is at chance, or if users report the sensations as arbitrary, the central claim that converted speech emotions are meaningful design material fails. A complementary check is to measure the SER model's valence, arousal, and dominance accuracy on noisy, real-world microphone input against its benchmark performance.
Extended reading notes
Core claim
The central claim is that speech emotion conversion (SEC) is a useful new design resource: instead of stopping at digital emotion estimates, an interactive system can translate those estimates into physical displays and proxemic actions, letting people feel or see emotions as bodily happenings. The paper's contribution is the method and starter kit, which chain three replaceable parts: a pre-trained transformer-based SER model that predicts continuous valence, arousal, and dominance from raw audio; configurable mapping rules that turn those values into commands; and maker-friendly output devices (a vibrotactile motor on Arduino, a visualization in Processing) that render the commands. The author argues this opens design opportunities such as haptic collars or blankets that let pets feel their owner's tone of voice, social robots that adjust their approach distance based on detected anger or happiness, and somaesthetic or artistic explorations of emotion as a felt material.
Load-bearing premise
The whole approach assumes the transformer-based model's real-time valence, arousal, and dominance predictions are accurate enough in real, context-sensitive settings that the resulting physical vibrations, warmth, or robot distances will feel emotionally meaningful to users.
Editorial extensions
If this is right
- Designers, artists, and educators can prototype emotion-driven physical interfaces without building or fine-tuning ML models themselves.
- Companion animals, including deaf cats and dogs, could receive emotional tone through haptic or thermal feedback in collars or blankets.
- Social robots could use detected speech emotion to set proxemic parameters, such as keeping a larger distance when a person sounds angry and a smaller one when they sound happy.
- Artistic and somaesthetic researchers can treat speech emotions as raw material and calibrate familiarity or strangeness by adjusting mapping options.
- SEC can be extended to media contexts such as gaming, VR/AR, TV, or radio to make emotional content tangible and more accessible.
Reading between the lines
- Because emotion-to-physical mappings are culturally and personally variable, a natural next step is participatory design sessions that elicit preferred mappings for different user groups; the paper suggests such methods but does not test them.
- The same pipeline could be paired with emerging animal vocal emotion recognition to create a bidirectional emotional communication channel between humans and companion animals, a direction the paper mentions only as emerging.
- A testable extension is using SEC as an accessibility aid that renders emotional tone through touch for deaf or hard-of-hearing users; the paper does not develop this use case explicitly.
- Whether the felt physical sensations are emotionally meaningful remains an open empirical question; a user study measuring recognition of mapped emotions would settle it.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper argues that converting automatic speech emotion recognition (SER) outputs into physical, haptic, tangible, or proxemic manifestations would make emotions more usable as design material in interaction design. It describes a method and a 'starter kit' for speech emotion conversion (SEC), composed of a command-line SER tool extracted from the AffectToolbox and example Arduino/Processing sketches that display emotion values via vibrotactile or visual feedback. The paper then sketches design opportunities for animal-computer interaction, proxemic interaction, and somaesthetic/artistic research, with concrete examples such as a haptic collar for deaf pets and robot approach distance regulated by detected anger level. The contribution is primarily conceptual: the paper argues for a new design space rather than providing empirical evidence of effectiveness.
Significance. If the argument lands, the paper identifies a genuine and underexplored design material—real-time paralinguistic emotion values as input to physical displays—and connects it to several active HCI research areas (mid-air haptics, somaesthetic design, animal-computer interaction, proxemics). The author leverages prior work on the AffectToolbox and Wagner et al.'s transformer-based SER model, and the proposed pipeline is technically plausible. However, the paper makes no empirical claims and presents no evaluation; its significance rests on the openness of the design space and the availability of the starter kit. Those two pillars are currently weak: the kit is only 'planned' to be open-sourced, and no evidence is provided that the converted physical signals are perceived as emotionally meaningful by humans or animals. The paper is a useful design provocation but not yet a demonstrated contribution.
major comments (4)
- [Abstract and Section 2] The abstract states that 'a starter kit for speech emotion conversion is provided,' but Section 2 says 'I plan to make the starter kit available open source' (footnote 1), and no code, binary, instructions, or screenshots of usage beyond a GUI screenshot are included in the manuscript. The central methodological contribution is therefore not actually delivered to the reader. Please either include the kit as a supplement (with installation/run instructions and a minimal demo) or revise the abstract and text to state explicitly that the kit is planned but not yet released, and adjust the contribution claim accordingly.
- [Section 2.2.1 and Section 3.2] The pipeline delegates emotion sensing to the AffectToolbox's black-box SER model [31], but the paper provides no accuracy or reliability assessment in the target conditions: live microphone input, variable audio chunk sizes, background noise, multiple speakers, or context-dependent interactions. Yet Section 3.2's concrete example sets a robot's approach distance based on detecting anger versus happiness; if the SER model misclassifies valence or arousal, the robot could make socially inappropriate or unsafe proxemic decisions. The claim that SEC lets emotions be 'directly felt and socially communicated' is load-bearing and requires at least a sanity check of SER accuracy in the envisioned scenarios, or an explicit reframing of these examples as speculative design illustrations rather than demonstrated capabilities.
- [Section 3.2] The emotion-to-distance mapping is arbitrary: 'person(s) sounding angry could map to a minimum proximity of 2 meters... if the speech sound changes to happy then the distance could shrink to 1 meter' (Section 3.2). No proxemics literature (e.g., Hall's zones) or user data is cited to justify these thresholds, and the paper itself acknowledges in Section 2.2.2 that 'there is no one solution fits all.' If these numbers are intended only as illustrative, state that explicitly; if they are meant as a design recommendation, support them with evidence or participatory design. Without this distinction, the example is at risk of being read as a validated mapping rather than a placeholder.
- [Section 3.1] The animal-interaction scenarios (haptic collar, pet blanket that converts speech emotions to haptic/thermal feedback) assume that companion animals perceive the haptic or thermal output as emotionally meaningful. The paper cites Quaranta et al. for cats' recognition of human emotional valence via visual and auditory signals, and Totakura et al. for animal vocal emotion recognition, but provides no evidence that animals can decode tactile or thermal displays as emotion signals. The phrase 'Haptic feedback is a modality that is already used for pets' refers to training collars, which typically deliver punitive or corrective stimuli, not emotion valence. Please mark these scenarios as open design speculations rather than near-term opportunities, or include relevant evidence about animals' haptic perception.
minor comments (5)
- [Section 1] The phrase 'closing the "valence gab"' should be 'valence gap.' Also, 'continuos' in Section 2.1 and 'soemasthetic' in the Section 3.3 heading are misspelled; 'proexmic' and 'tangilbe' appear in Section 4. A proofread would improve the paper's polish.
- [Section 2.2] The text repeatedly uses 'audio junks' where 'audio chunks' is meant. This is a terminological error that may confuse readers who are not native speakers.
- [Section 1] The paper uses the terms 'emotions' and 'affective states' colloquially and deliberately avoids formal definitions. This is acceptable for a design paper, but it should be acknowledged earlier that the validity of the ser model's continuous dimensions (valence, arousal, dominance) is not the focus; otherwise the central argument may be conflated with a claim about the ground truth of emotion.
- [Section 2.2.2] The description of 'two options' for sending SER results to external boards/apps is slightly ambiguous: one is a serial connection to Arduino, the other a TCP socket to Processing. Clarify which components are wired and which are network-based, as the figure (Figure 1) does not show the connection topology.
- [Section 3.3] The connection between SEC and somaesthetic interaction design would be strengthened by a more specific example of how the mapped physical sensation (e.g., vibration intensity, warmth) supports first-person or collaborative bodily reflection. As written, the link remains programmatic.
Circularity Check
No circularity: the paper makes no derived prediction and contains no fitted-input-as-output step.
full rationale
The paper is a design-and-method proposal, not a derivation. It describes a starter kit that pipes speech emotion recognition (SER) values into tangible/haptic/proxemic displays, and it explicitly leaves the mapping from emotion values to physical output as an open design decision: "The design space to map speech emotions to physical sensations and behaviours is vast and there is no one solution fits all." No equation is derived, no parameter is fitted, and no prediction is claimed that could reduce by construction to its inputs. The SER model itself is externally cited (Wagner et al. [31]) and is used as a black-box building block, not redefined by this paper. The AffectToolbox [18] is also an external tool from a different author set. The self-citations in the paper serve only as examples of prior design explorations (art installations, robot humor, participatory gesture elicitation, heartbeat-somaesthetic designs, taste as playful modality); none of these is load-bearing for the paper's central argument. The central claim that emotions can be experienced as bodily sensations is a design proposition supported by examples and related work, not a result that is equivalent to its own inputs. Therefore no self-definitional, fitted-input, self-citation-load-bearing, uniqueness-imported, ansatz-smuggled, or renaming step is present.
Assumptions & free parameters
assumptions (4)
- domain assumption Continuous emotion dimensions (valence, arousal, dominance) adequately represent the emotional content of speech.
- domain assumption Paralinguistic features alone are sufficient for reliable emotion recognition in the targeted scenarios.
- domain assumption Users will perceive physical mappings (vibration intensity, proximity distance) as meaningful emotional signals.
- domain assumption The audio chunk size used for SER analysis does not critically alter the emotional interpretation in a way that invalidates the design.
Cite this review
Pith. "Pith review of Feel my Speech: Automatic Speech Emotion Conversion for Tangible, Haptic, or Proxemic Interaction Design." pith.science (2026). https://pith.science/paper/73VHVRO4
@misc{pith2026241207722,
author = {Pith},
title = {Pith review of: Feel my Speech: Automatic Speech Emotion Conversion for Tangible, Haptic, or Proxemic Interaction Design},
year = {2026},
howpublished = {\url{https://pith.science/paper/73VHVRO4}},
note = {Machine review of arXiv:2412.07722}
}
read the original abstract
Innovations in interaction design are increasingly driven by progress in machine learning fields. Automatic speech emotion recognition (SER) is such an example field on the rise, creating well performing models, which typically take as input a speech audio sample and provide as output digital labels or values describing the human emotion(s) embedded in the speech audio sample. Such labels and values are only abstract representations of the felt or expressed emotions, making it challenging to analyse them as experiences and work with them as design material for physical interactions, including tangible, haptic, or proxemic interactions. This paper argues that both the analysis of emotions and their use in interaction designs would benefit from alternative physical representations, which can be directly felt and socially communicated as bodily sensations or spatial behaviours. To this end, a method is described and a starter kit for speech emotion conversion is provided. Furthermore, opportunities of speech emotion conversion for new interaction designs are introduced, such as for interacting with animals or robots.
Figures
Reference graph
Works this paper leans on
-
[31]
Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, and Björn W Schuller. 2023. Dawn of the transformer era in speech emotion recognition: closing the valence gap. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 9 (2023), 10745– 10759
work page 2023
-
[1]
Imtiaj Ahmed, Ville Harjunen, Giulio Jacucci, Eve Hoggan, Niklas Ravaja, and Michiel M Spapé. 2016. Reach out and touch me: Effects of four distinct haptic technologies on affective touch in virtual reality. In Proceedings of the 18th ACM International Conference on Multimodal Interaction . 341–348
work page 2016
-
[2]
Miquel Alfaras, Vasiliki Tsaknaki, Pedro Sanches, Charles Windlin, Muhammad Umair, Corina Sas, and Kristina Höök. 2020. From biodata to somadata. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems . 1–14
work page 2020
-
[3]
Shahin Amiriparian, Artem Sokolov, Ilhan Aslan, Lukas Christ, Maurice Gerczuk, Tobias Hübner, Dmitry Lamanov, Manuel Milling, Sandra Ottl, Ilya Podure- mennykh, et al. 2021. On the impact of word error rate on acoustic-linguistic speech emotion recognition: An update for the deep learning era. arXiv preprint arXiv:2104.10121 (2021)
arXiv 2021
-
[4]
Ilhan Aslan, Hadrian Burkhardt, Julian Kraus, and Elisabeth André. 2016. Hold my heart and breathe with me: Tangible somaesthetic designs. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction . 1–6
work page 2016
-
[5]
Ilhan Aslan, Dominik Neu, Daniela Neupert, Stefan Grafberger, Nico Weise, Pascal Pfeil, and Maximilian Kuschewski. 2023. How to Compliment a Human-Designing Affective and Well-being Promoting Conversational Things. Interaction Design and Architecture (s) Journal 58 (2023), 157–184
work page 2023
-
[6]
Ilhan Aslan, Tabea Schmidt, Jens Woehrle, Lukas Vogel, and Elisabeth André
-
[7]
Ilhan Aslan, Andreas Seiderer, Chi Tai Dang, Simon Rädler, and Elisabeth André
Show all 44 references
-
[8]
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems 33 (2020), 12449–12460
2020
-
[9]
Till Ballendat, Nicolai Marquardt, and Saul Greenberg. 2010. Proxemic inter- action: designing for a proximity and orientation-aware environment. In ACM International Conference on Interactive Tabletops and Surfaces . 121–130
2010
-
[10]
Alan M Beck and Aaron H Katcher. 2003. Future directions in human-animal bond research. American behavioral scientist 47, 1 (2003), 79–93
2003
-
[11]
Björn Bittner, Ilhan Aslan, Chi Tai Dang, and Elisabeth André. 2019. Of smarthomes, IoT plants, and implicit interaction design. (2019)
2019
-
[12]
Lukas Christ, Shahin Amiriparian, Manuel Milling, Ilhan Aslan, and Björn W Schuller. 2024. Modeling Emotional Trajectories in Written Stories Utilizing Transformers and Weakly-Supervised Learning. arXiv preprint arXiv:2406.02251 (2024)
2024 arXiv
-
[13]
Jennifer Golbeck and Carman Neustaedter. 2012. Pet video chat: monitoring and interacting with dogs over distance. In CHI’12 Extended Abstracts on Human Factors in Computing Systems . 211–220
2012
-
[14]
Saul Greenberg, Nicolai Marquardt, Till Ballendat, Rob Diaz-Marino, and Miaosen Wang. 2011. Proxemic interactions: the new ubicomp? interactions 18, 1 (2011), 42–50
2011
-
[15]
Molly J Hall, Anthony Ng, Robert J Ursano, Harry Holloway, Carol Fullerton, and Jacob Casper. 2004. Psychological impact of the animal-human bond in disaster preparedness and response. Journal of Psychiatric Practice® 10, 6 (2004), 368–374
2004
-
[16]
Kristina Hook. 2018. Designing with the body: Somaesthetic interaction design
2018
-
[17]
Yulan Ju, Dingding Zheng, Danny Hynds, George Chernyshov, Kai Kunze, and Kouta Minamizawa. 2021. Haptic empathy: Conveying emotional meaning through vibrotactile feedback. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems . 1–7
2021
-
[18]
Silvan Mertes, Dominik Schiller, Michael Dietz, Elisabeth André, and Florian Lingenfelser. 2024. The AffectToolbox: Affect Analysis for Everyone. In2024 12th International conference on affective computing and intelligent interaction (ACII) . IEEE
2024
-
[19]
Martin Murer, Ilhan Aslan, and Manfred Tscheligi. 2013. LOLL io: exploring taste as playful modality. In Proceedings of the 7th international conference on tangible, embedded and embodied interaction . 299–302
2013
-
[20]
Marianna Obrist, Sriram Subramanian, Elia Gatti, Benjamin Long, and Thomas Carter. 2015. Emotions mediated through mid-air haptics. In Proceedings of the 33rd annual ACM conference on human factors in computing systems . 2053–2062
2015
-
[21]
Marianna Obrist, Alexandre N Tuch, and Kasper Hornbaek. 2014. Opportunities for odor: experiences with smell and implications for technology. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . 2843–2852
2014
-
[22]
Ville Paananen, Mohammad Sina Kiarostami, Lee Lik-Hang, Tristan Braud, and Simo Hosio. 2023. From digital media to empathic spaces: A systematic review of empathy research in extended reality environments. Comput. Surveys 56, 5 (2023), 1–40
2023
-
[23]
Angelo Quaranta, Serenella d’Ingeo, Rosaria Amoruso, and Marcello Siniscalchi
-
[24]
Hannes Ritschel, Silvan Mertes, Florian Lingenfelser, Thomas Kiderle, and Elis- abeth André. 2023. The Affective Bar Piano. In Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents . 1–3
2023
-
[25]
Eike Schneiders, Alan Chamberlain, Joel E Fischer, Steve Benford, Simon Castle- Green, Victor Ngo, Ayse Kucukyilmaz, Pepita Barnard, Ju Row Farr, Matt Adams, et al. 2023. TAS for cats: An artist-led exploration of trustworthy autonomous sys- tems for companion animals. In Proc...
2023
-
[26]
Animals 10, 7 (2020), 1107
Emotion recognition in cats. Animals 10, 7 (2020), 1107
2020
-
[27]
Varun Totakura, Mohana Krishna Janmanchi, Durganath Rajesh, and MT Hussan
-
[28]
Andreas Triantafyllopoulos, Anton Batliner, Simon Rampp, Manuel Milling, and Björn Schuller. 2024. INTERSPEECH 2009 Emotion Challenge Revisited: Bench- marking 15 Years of Progress in Speech Emotion Recognition. arXiv preprint arXiv:2406.06401 (2024)
2024 arXiv
-
[29]
Björn Schuller, Adria Mallol-Ragolta, Alejandro Peña Almansa, Iosif Tsangko, Mostafa M Amin, Anastasia Semertzidou, Lukas Christ, and Shahin Amiriparian
-
[30]
Chi Thanh Vi, Damien Ablart, Elia Gatti, Carlos Velasco, and Marianna Obrist
-
[32]
International Journal of Scientific & Technology Research 9, 2 (2020), 6007–6011
Prediction of animal vocal emotions using convolutional neural network. International Journal of Scientific & Technology Research 9, 2 (2020), 6007–6011
2020
-
[33]
Klaus Weber, Hannes Ritschel, Ilhan Aslan, Florian Lingenfelser, and Elisabeth André. 2018. How to shape the humor of a robot-social behavior adaptation based on reinforcement learning. In Proceedings of the 20th ACM international conference on multimodal interaction . 154–162
2018
-
[34]
Christiana Tsiourti, Astrid Weiss, Katarzyna Wac, and Markus Vincze. 2017. Designing emotionally expressive robots: A comparative study on the perception of communication modalities. In Proceedings of the 5th international conference on human agent interaction . 213–222
2017
-
[35]
Graham Wilson, Martin Halvey, Stephen A Brewster, and Stephen A Hughes
-
[36]
Charles Windlin, Anna Ståhl, Pedro Sanches, Vasiliki Tsaknaki, Pavel Karpashe- vich, Madeline Balaam, and Kristina Höök. 2019. Soma Bits-mediating technology to orchestrate bodily experiences. In RTD 2019-Research through Design Confer- ence 2019, the Science Centre, Delft, on...
2019
-
[37]
Mounia Ziat, Katherine Chin, and Roope Raisamo. 2020. Effects of visual locomo- tion and tactile stimuli duration on the emotional dimensions of the cutaneous rabbit illusion. In Proceedings of the 2020 International Conference on Multimodal Interaction. 117–124
2020
-
[38]
Froma Walsh. 2009. Human-animal bonds I: The relational significance of com- panion animals. Family process 48, 4 (2009), 462–480
2009
-
[40]
Danielle Wilde, Anna Vallgårda, and Oscar Tomico. 2017. Embodied design ideation methods: Analysing the power of estrangement. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems . 5158–5170
2017
-
[2011]
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Some like it hot: thermal feedback for mobile devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . 2555–2564
-
[2017]
International Journal of Human-Computer Studies 108 (2017), 1–14
Not just seeing, but also feeling art: Mid-air haptic experiences integrated in a multisensory art exhibition. International Journal of Human-Computer Studies 108 (2017), 1–14
2017
-
[2018]
In Proceedings of the 20th ACM International Conference on Multimodal Interaction
Pen+ mid-air gestures: Eliciting contextual gestures. In Proceedings of the 20th ACM International Conference on Multimodal Interaction . 135–144
-
[2020]
In Proceedings of the 2020 International Conference on Multimodal Interaction
PiHearts: Resonating Experiences of Self and Others Enabled by a Tangi- ble Somaesthetic Design. In Proceedings of the 2020 International Conference on Multimodal Interaction. 433–441
2020
-
[2024]
arXiv preprint arXiv:2409.08907 (2024)
Affective Computing Has Changed: The Foundation Model Disruption. arXiv preprint arXiv:2409.08907 (2024)
2024 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.