Pith. sign in

REVIEW 3 major objections 4 minor 30 references

TONUS: Neuromorphic human pose estimation for artistic sound co-creation

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a human pose estimation network with a spiking encoder can run partly on a neuromorphic chip, using an order of magnitude fewer operations than dense baselines, and applies it to an interactive sound installation.

desk verdict A genuinely hardware-constrained SNN encoder for event-based pose estimation, undermined by an unfair DHP19 comparison and an on-chip claim the paper itself concedes is not yet demonstrated. read the letter →

arxiv 2507.15734 v1 pith:HV3LQSKE submitted 2025-07-21 cs.NE

Jules Lecomte (1) , Konrad Zinner (2) , Michael Neumeier (1) , Axel von Arnim (1) ((1) Fortiss GmbH , Munich , Germany , (2) Hochschule für Musik und Theater Munich , Germany) This is my paper · ORCID
classification cs.NE
keywords neuromorphiccomputingspikingneuralnetworkshumanposeestimationevent-basedcamerassoundinstallationmediaartIntelLoihi2multihead
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to prove that human pose estimation can be done with a spiking neural network encoder designed for the Intel Loihi 2 chip, with an ordinary ANN decoder handling the output heads, and that this hybrid is sparse and cheap enough for a real-time artistic installation. The authors report the first human pose estimation network that runs partly on a neuromorphic chip, with 19.13 GMAC and 0.7 GAC operations per second versus hundreds to thousands of GMAC for dense baselines, and a 26-fold sparsity factor over an equivalent dense ANN. They also built and exhibited TONUS, a sound installation in which an event camera and this network turn a visitor's limb movements into soundscapes and neural-visual feedback. Accuracy on the DHP19 benchmark is lower than the best dense models (MPJPE 12.07 pixels for the multihead SNN), but the paper argues this is enough for interactive artistic use.

What carries the argument

The load-bearing object is the hybrid SNN-encoder / ANN-decoder pose network. The encoder is a spiking CNN using two neuron models: CUBA (a current-based leaky integrate-and-fire neuron matching Loihi 2) and PLIF (a parametric leaky integrate-and-fire neuron from SpikingJelly); events are binned into 10 ms windows with graded or binary spikes, and the encoder's last-layer membrane potentials are fed as floating values into a multihead decoder with heatmap, center, regression, and offset heads. A Kalman filter smooths each joint's 2D position and velocity. The design's purpose is to push all accumulate-only, sparse operations into the chip-portable encoder while keeping the deconvolution-based decoder on a GPU, with the multihead loss adapted from CenterNet.

What would settle it

Measure the trained encoder on a Loihi 2 board (or a cycle-accurate simulator) with real event-stream input and compare end-to-end latency and wall-clock power, including event ingestion, against the dense ANN baseline; if the spiking system is not faster and lower-power in practice, or if the MPJPE degrades sharply under chip timing, the central practical claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a small encoder–decoder network can perform event-based human pose estimation while satisfying the constraints of neuromorphic hardware: the encoder is a spiking CNN whose final membrane potentials are passed as floating values to a CenterNet-style multihead decoder (heatmap, center, regression, offset), and the whole encoder uses only accumulate operations with binary spikes. Against DHP19, the multihead SNN achieves MPJPE 12.07 with 19.13 GMAC and 0.7 GAC operations per second, at least an order of magnitude fewer operations than the state-of-the-art dense networks and roughly 26 times sparser than an equivalent dense ANN. The paper also claims to be the first to demonstrate an artistic human–machine interaction built on neuromorphic sensing, exhibited at Festival der Zukunft in Munich. The authors are explicit that accuracy trails larger dense models and that the real-time port to Loihi 2 still faces hardware limitations, but they present the operation-count and sparsity advantage as the core contribution.

Load-bearing premise

The argument assumes the spiking encoder can actually run on Loihi 2 in real time with the claimed energy and latency benefits, but the paper reports that the port was limited by missing fast I/O and that the decoder cannot run on the chip, so chip execution and energy savings are projections rather than measured on-device results.

Editorial extensions

If this is right

  • If the claims hold, event-based human pose estimation can run with roughly 26x sparsity and accumulate-only operations, putting body tracking on very low-power edge hardware.
  • The architecture shows that hardware constraints (no skip connections, no deconvolutions) can be absorbed into the encoder design without destroying usable accuracy, opening hybrid SNN/ANN designs for other keypoint tasks.
  • The multihead formulation plus Kalman filtering becomes a reusable recipe for stabilizing sparse event-stream predictions in interactive settings.
  • For media art, the work demonstrates a privacy-preserving 'Green AI' interaction model: visitors are represented by movement traces and neural activity, not recognizable images, and the installation can run on modest computing budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the fast I/O limitation is solved, the decoder could likely be replaced by regression and convolution layers already supported on Loihi 2, making a fully on-chip HPE pipeline plausible; the paper hints at this but does not claim it.
  • The operation-count comparison leaves out event ingress, memory traffic, and conversion costs; measuring wall-clock chip power would tell whether the energy savings survive real deployment.
  • The 'partly on a neuromorphic chip' qualifier is important: if any fully spiking HPE network that runs entirely on neuromorphic hardware appears, the novelty claim narrows to the hybrid art-system combination rather than to feasibility.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents TONUS, an artistic sound installation that uses an event camera and a spiking-neural-network encoder for human pose estimation. The encoder is designed for Intel Loihi 2 with binary spikes and quantized weights, while a multihead ANN decoder runs on a GPU. The authors train on a custom 12-subject dataset, evaluate on DHP19, and report MPJPE and operation counts, along with an ablation study and a description of a public exhibition. They claim this is the first human pose estimation network able to run partly on a neuromorphic chip and that it outperforms state-of-the-art networks in floating operations.

Significance. If the hardware port were actually demonstrated, the contribution would be significant for low-power edge human pose estimation and for interactive media art. The paper's strengths include a reasonably detailed architecture, a thorough ablation study (Kalman filter, time bins, neuron model, multihead loss), and a real public installation that validates the artistic concept. The authors also honestly acknowledge several limitations in the conclusion. However, the headline claim of on-chip capability is not supported by any chip measurement, and the accuracy comparison suffers from a training/test dataset mismatch. As it stands, the contribution is a promising design study for a neuromorphic-compatible HPE encoder rather than a demonstrated neuromorphic HPE system.

major comments (3)
  1. [Section I vs. Section VI] The Introduction states 'To the best of our knowledge, this work is the first human pose estimation network able to run partly on a neuromorphic chip.' This claim is contradicted by the paper's own Section VI, which says the real-time port to Loihi 2 'was until now limited by the missing fast I/O interface' and that 'our current decoder is not currently runnable on the chip because of unsupported deconvolution layers.' No on-chip execution, latency, power, or throughput measurement is reported anywhere; the installation itself ran on a GPU (Section IV-B, Table II). The claim must therefore be downgraded to 'designed for' or 'portable in principle' unless an actual on-chip experiment is added.
  2. [Section III-C and Section V-A] The model is trained on a custom 12-subject dataset (Section III-C), while the accuracy comparison in Table III is performed on DHP19 (Section V-A). All cited baselines (Calabrese, Baldwin, Goyal, Aydin) are trained on DHP19. This train/test domain mismatch is not disclosed in the comparison and confounds the MPJPE numbers: a model trained on a different distribution naturally performs worse on DHP19, so the comparison is not apples-to-apples. Please retrain on DHP19, or clearly state that the reported MPJPE is a cross-dataset evaluation and avoid direct comparison without this caveat.
  3. [Section V-B] The energy and sparsity analysis is entirely analytical: Eq. (9) and Eq. (10) count operations from the model's own spike statistics, and the energy savings are derived from generic CMOS per-operation costs (0.38 pJ vs. 1.69 pJ). No measurement on Loihi 2 or any other neuromorphic hardware is provided. The closing statement that 'running models on neuromorphic hardware allows great energy savings' is therefore unsupported by direct evidence. The analysis is fine as an estimate, but the wording should be softened to reflect that this is a projected, not measured, benefit.
minor comments (4)
  1. [Section III-A] The text says '16 bins of 625 ms' but the total window is 10 ms, so each bin should be 0.625 ms; please correct this typo.
  2. [Table III] The table title says 'on DHP19 and our new dataset', but the table only reports MPJPE on DHP19; no accuracy numbers for the custom dataset are shown. Either add those results or amend the title.
  3. [Equation (9)] Equation (9) is typeset ambiguously: the expression 'ΣN i=1Si N ∗ Cout ∗ K 2 s' does not clearly convey the intended averaging of spike counts over the test set. Please rewrite the formula with explicit parentheses or an indicator for the mean.
  4. [Throughout] There are several typographical issues: 'CUrrent BAsed' should be 'current-based', 'magntude' should be 'magnitude', and 'wiht' should be 'with'. In addition, Table III uses 'GFLOP/s' but the values appear to be per-forward-pass operation counts rather than rates; please clarify the units.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the model's accuracy and operation counts are evaluated against external datasets and published baselines, and the on-chip portability claim is an unsupported design assertion, not a self-referential derivation.

full rationale

The paper's derivation chain is not circular. The pose-estimation accuracy is measured on the external DHP19 dataset and a custom recorded dataset, and compared in Table III with previously published ANN and SNN baselines (Calabrese, Baldwin, Goyal, Aydin). The operation counts in Eq. (9)-(10) are computed from the model's own spike statistics, not fitted to match a target, and the sparsity factor and energy comparison are analytical estimates based on standard CMOS energy numbers. The ablation study in Section V-C varies time-bin resolution, neuron model, encoder type, Kalman filtering, and multihead loss, all independently evaluated. The paper's central novelty claim 'this work is the first human pose estimation network able to run partly on a neuromorphic chip' is indeed weakened by the paper's own admission in Section VI that the real-time port to Loihi 2 was limited by the missing fast I/O interface and that the decoder cannot run on the chip because of unsupported deconvolution layers. However, that is a support gap or unverified design assertion, not a circular step: no fitted parameter is renamed as a prediction, and no cited result is equivalent to the claim by construction. The claim is a design intent expressed as a capability, but the derivation of the measured results does not assume the conclusion. No load-bearing self-citation or definitional circularity is present. Accordingly, the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical entities, forces, or conserved quantities are postulated. The neural totem and LED/audio mappings are artistic artifacts, not scientific entities. The central claims rest on the model's learned weights and several hand-chosen hyperparameters, plus the domain assumption that the evaluation protocol (custom training set, DHP19 test set) is informative.

free parameters (5)
  • Membrane threshold and decay constants (CUBA alpha_u, alpha_v; PLIF learned decay)
    Neuron dynamics parameters that affect spike timing and accuracy; they are hyperparameters set by hand or learned during training, not derived from first principles.
  • Number of time bins per sample = 8 or 32 depending on neuron model (CUBA best at 8, PLIF best at 32)
    Selected by ablation on the custom validation set (Table IV); the final configuration for Table III is not explicitly stated.
  • Heatmap mask threshold tau = 0.1
    Chosen by hand for post-processing to select high-confidence heatmap pixels; affects final joint predictions.
  • Loss weights and training hyperparameters (epochs, learning rate, batch size)
    Training details are not fully specified in the paper; these are fitted to the training data and affect final accuracy.
  • Kalman filter process and measurement noise parameters
    The constant-velocity Kalman filter requires noise covariances, which are not reported; these affect the smoothed MPJPE of 15.19.
assumptions (4)
  • standard math Backpropagation Through Time and SLAYER surrogate gradient assumptions
    Training relies on the SLAYER algorithm [2] and standard BPTT for SNNs, whose surrogate gradient approximations are unproved in the paper.
  • domain assumption Evaluating on DHP19 without training on DHP19 is a fair comparison
    The paper tests on DHP19 while training on its own dataset, assuming the metric transfers and the domain gap does not dominate the comparison; this is not justified in the text.
  • domain assumption Operation counting formula (Eq. 9) reflects on-chip energy cost
    The number of AC/MAC operations is computed from spike counts and kernel sizes, assuming each spike triggers an operation and that the 0.38 pJ vs 1.69 pJ energy values apply to the eventual Loihi 2 implementation, which is not measured.
  • domain assumption Event camera and environment in the installation match training distribution
    The authors use the same sensor in the installation to keep the data distribution consistent, but the installation environment (lighting, visitor motion) is not quantitatively validated against the training set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TONUS: Neuromorphic human pose estimation for artistic sound co-creation." pith.science (2026). https://pith.science/paper/HV3LQSKE

@misc{pith2026250715734,
  author       = {Pith},
  title        = {Pith review of: TONUS: Neuromorphic human pose estimation for artistic sound co-creation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HV3LQSKE}},
  note         = {Machine review of arXiv:2507.15734}
}
read the original abstract

Human machine interaction is a huge source of inspiration in today's media art and digital design, as machines and humans merge together more and more. Its place in art reflects its growing applications in industry, such as robotics. However, those interactions often remains too technical and machine-driven for people to really engage into. On the artistic side, new technologies are often not explored in their full potential and lag a bit behind, so that state-of-the-art research does not make its way up to museums and exhibitions. Machines should support people's imagination and poetry in a seamless interface to their body or soul. We propose an artistic sound installation featuring neuromorphic body sensing to support a direct yet non intrusive interaction with the visitor with the purpose of creating sound scapes together with the machine. We design a neuromorphic multihead human pose estimation neural sensor that shapes sound scapes and visual output with fine body movement control. In particular, the feature extractor is a spiking neural network tailored for a dedicated neuromorphic chip. The visitor, immersed in a sound atmosphere and a neurally processed representation of themselves that they control, experience the dialogue with a machine that thinks neurally, similarly to them.

Figures

Figures reproduced from arXiv: 2507.15734 by the authors.

Figure 1
Figure 1. The model architecture with a spiking encoder. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. The inside arrangement as it was in the original plan. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 2
Figure 2. Visualisation of multihead outputs and ground truth [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Left: the totem presenting the neuromorphic hardware [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Technical setup of the installation. submitted to the Festival der Zukunft in Munich, there is a neural totem facing the visitor, as well as a projected neuronal body representation of him. The totem not only carries the museally presented neuromorphic chip Loihi2 and …
Figure 6
Figure 6. Figure 6: Four different HPE results overlayed on input events. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Impact of the Kalman filter (MPJPE over time). [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 27 canonical work pages

  1. [1]

    A 240 × 180 130 db 3 µs latency global shutter spatiotemporal vision sensor,

    C. Brandli, R. Berner, M. Yang, S.-C. Liu, and T. Delbruck, “A 240 × 180 130 db 3 µs latency global shutter spatiotemporal vision sensor,” IEEE Journal of Solid-State Circuits , vol. 49, no. 10, pp. 2333–2341, 2014

    2014

  2. [2]

    Slayer: Spike layer error reassignment in time,

    S. B. Shrestha and G. Orchard, “Slayer: Spike layer error reassignment in time,” 2018

    work page 2018

  3. [3]

    Taking neuromorphic computing to the next level with loihi 2

    Intel Labs, “Taking neuromorphic computing to the next level with loihi 2.” https://www.intel.com/content/www/us/en/research/ neuromorphic-computing-loihi-2-technology-brief.html, 2022

    work page 2022

  4. [4]

    Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time,

    H.-S. Fang, J. Li, H. Tang, C. Xu, H. Zhu, Y . Xiu, Y .-L. Li, and C. Lu, “Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time,” 2022

    work page 2022

  5. [5]

    Openpose: Realtime multi-person 2d pose estimation using part affinity fields,

    Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y . Sheikh, “Openpose: Realtime multi-person 2d pose estimation using part affinity fields,” 2019

    work page 2019

  6. [6]

    Objects as points,

    X. Zhou, D. Wang, and P. Kr ¨ahenb¨uhl, “Objects as points,” 2019

    2019

  7. [7]

    Movenet,

    TensorFlow, “Movenet,” 2021

    work page 2021

  8. [8]

    Dhp19: Dynamic vision sensor 3d human pose dataset,

    E. Calabrese, G. Taverni, C. Awai Easthope, S. Skriabine, F. Corradi, L. Longinotti, K. Eng, and T. Delbruck, “Dhp19: Dynamic vision sensor 3d human pose dataset,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , June 2019

    work page 2019

Show all 30 references
  1. [9]

    Moveenet: Online high-frequency human pose estimation with an event camera,

    G. Goyal, F. Di Pietro, N. Carissimi, A. Glover, and C. Bartolozzi, “Moveenet: Online high-frequency human pose estimation with an event camera,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4023–4032, June 2023

    2023

  2. [10]

    Time-ordered recent event (tore) volumes for event cameras,

    R. W. Baldwin, R. Liu, M. Almatrafi, V . Asari, and K. Hirakawa, “Time-ordered recent event (tore) volumes for event cameras,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 2, pp. 2519–2532, 2023

    2023

  3. [11]

    Neuromorphic high-frequency 3d dancing pose estimation in dynamic environment,

    Z. Zhang, K. Chai, H. Yu, R. Majaj, F. Walsh, E. Wang, U. Mahbub, H. Siegelmann, D. Kim, and T. Rahman, “Neuromorphic high-frequency 3d dancing pose estimation in dynamic environment,” Neurocomputing, vol. 547, p. 126388, 2023

    2023

  4. [12]

    A hybrid ann-snn architecture for low-power and low-latency visual perception,

    A. Aydin, M. Gehrig, D. Gehrig, and D. Scaramuzza, “A hybrid ann-snn architecture for low-power and low-latency visual perception,” 2024

    2024

  5. [13]

    3d human pose estimation in RGBD images for robotic task learning,

    C. Zimmermann, T. Welschehold, C. Dornhege, W. Burgard, and T. Brox, “3d human pose estimation in RGBD images for robotic task learning,” CoRR, vol. abs/1803.02622, 2018

    2018 arXiv

  6. [14]

    Robust human pose tracking for realistic service robot applications,

    M. Vasileiadis, S. Malassiotis, D. Giakoumis, C.-S. Bouganis, and D. Tzovaras, “Robust human pose tracking for realistic service robot applications,” in 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 1363–1372, 2017

    2017

  7. [15]

    Maira product page

    Neura Robotics, “Maira product page.” https://neura-robotics.com/ products/maira

  8. [16]

    Sign language recognition based on computer vision,

    W. Li, H. Pu, and R. Wang, “Sign language recognition based on computer vision,” in 2021 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA) , pp. 919–922, 2021

    2021

  9. [17]

    Eventhands: Real-time neural 3d hand pose estimation from an event stream,

    V . Rudnev, V . Golyanik, J. Wang, H.-P. Seidel, F. Mueller, M. Elgharib, and C. Theobalt, “Eventhands: Real-time neural 3d hand pose estimation from an event stream,” in International Conference on Computer Vision (ICCV), 2021

    2021

  10. [18]

    Kinect adventures

    Microsoft Game Studios, “Kinect adventures.” https://en.wikipedia.org/ wiki/Kinect Adventures!, 2010

    2010

  11. [19]

    Kinect fun lab: Air band

    Relentless Software, Good Science Studio, “Kinect fun lab: Air band.” https://en.wikipedia.org/wiki/Kinect Fun Labs, 2011

    2011

  12. [20]

    Trigger playground experiment

    Kling Klang Klong, “Trigger playground experiment.” https://vimeo. com/129906348, 2015

    2015

  13. [21]

    Momentum

    Kling Klang Klong, “Momentum.” https://www.klingklangklong.com/ works/momentum, 2015

    2015

  14. [22]

    Disco dusche

    Kling Klang Klong, “Disco dusche.” https://www.klingklangklong.com/ works/disco-dusche, 2018

    2018

  15. [23]

    Klangderwisch

    K. Essl, “Klangderwisch.” https://www.youtube.com/watch?v= 39zlwZ837Gc, 2015

    2015

  16. [24]

    Bot & Dolly, “Box.” https://www.youtube.com/watch?v= lX6JcybgDFo&list=PLywSrQu7PoR7bea3DqK1BA VvG3ON25KEt, 2013

    2013

  17. [25]

    Glass sound installation

    M. Koshenkova and R. Deutsch, “Glass sound installation.” https://www. youtube.com/watch?v=dKtB6iyXn80, 2013

    2013

  18. [26]

    Grasping sound

    E. Sharpe, “Grasping sound.” https://www.youtube.com/watch?v= Jj10zQsX3zo, 2016

    2016

  19. [27]

    Mirror of truth

    R. Castagnola, “Mirror of truth.” https://www.youtube.com/watch?v= RQ29I4pn1Rc, 2016

    2016

  20. [28]

    Noise and silence

    Group Gravity, “Noise and silence.” https://www.youtube.com/watch? v=enocRynbT0E, 2008

    2008

  21. [29]

    Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence,

    W. Fang, Y . Chen, J. Ding, Z. Yu, T. Masquelier, D. Chen, L. Huang, H. Zhou, G. Li, and Y . Tian, “Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence,” Science Advances, vol. 9, no. 40, p. eadi1480, 2023

    2023

  22. [30]

    Markerless motion capture and analysis with simi shape 3d

    Simi Reality Motion System GmbH, “Markerless motion capture and analysis with simi shape 3d.” http://www.simi.com/en/products/ movement-analysis/markerless-motion-capture.html?type=rss%27A% 3D0%27A%3D0%27A%3D0, 2023

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.