REVIEW 3 major objections 5 minor 42 references
Calliphony: A Calligraphy-Driven Interface for Real-Time Generative Music Performance
T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Calliphony establishes that brush motion, measured only as rotational speed, can serve as an expressive external control layer for a real-time symbolic music generation model.
desk verdict A credible NIME systems demo with a genuinely new external-trigger control of Notochord, but the key integration with the model's timing state is under-specified and unmeasured. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the distance-integral trigger: a continuous speed value derived from the brush's tri-axial gyroscope rotation is integrated over time in Max, and each time the accumulated movement distance crosses a user-set threshold, the system queries Notochord to predict and output the next melody note. The same speed signal, through threshold-based onset and stop values with hysteresis, switches the chord, bass, and sub-melody channels on and off. This mechanism is what converts calligraphic motion into note-onset density, harmonic layering, and textural change.
What would settle it
Record the same gyroscope speed profile with a brush and with an ordinary stick, and also vary brush pressure while holding rotation constant; if both pairs of trials produce identical music, then rotation speed alone carries the expressivity, and the calligraphy-specific sensor dimensions contribute nothing to the control signal.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that brush motion does not need to enter the neural network as an input feature to control it; the motion can instead control when and under what constraints the model is queried. Calliphony converts tri-axial gyroscope rotation into a scalar speed value, integrates that value over time, and triggers Notochord to predict the next melody note whenever the accumulated movement distance reaches a preset threshold. Pitch and velocity remain the model's choices, but inter-onset interval is replaced by the performer's motion, making note density a direct function of writing speed. A second speed threshold with hysteresis switches chords, bass, and a sub-melody on or off, so acceleration thickens the texture and deceleration strips it back to a monophonic line or silence.
Load-bearing premise
The system assumes that a scalar speed value from the brush's rotation, integrated over time, is expressive enough to shape musical density and texture, even though it deliberately ignores pressure, contact area, ink flow, and brush-tip deformation.
Editorial extensions
If this is right
- A calligrapher can perform a multi-track composition (melody, chords, bass, and sub-melody) using only brush speed, with no keyboard or sequencer input.
- Pauses and slow passages become rests and sparse texture, so the phrasing of handwritten characters is translated directly into musical phrasing.
- Because the control signal is a scalar accumulation of rotation speed, the same interaction layer can be attached to any moving object or body part without retraining the generative model.
- Scale filtering and pitch-repeat suppression keep the generated melody tonally coherent even though the model was not trained for calligraphy-specific control.
- The generative model can be swapped for another real-time MIDI model without redesigning the gesture interface, since the interface operates on query timing rather than model internals.
Reading between the lines
- The authors do not test whether pressure, contact area, or brush-tip deformation would add musical expressivity if sensed; their own discussion identifies these as missing dimensions, so the system's ceiling is set by the speed-only sensor.
- A natural extension is to invert the mapping and use accumulated distance to control pitch contour, scale choice, or timbre rather than onset density, a variation the authors do not explore.
- The same distance-integral control could be applied to other time-based embodied practices, such as conducting, dance, or tai chi, where velocity and pauses carry phrase structure; this follows from the paper's architecture but is not claimed by it.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Calliphony, a calligraphy-driven interface for real-time generative music performance. An M5Stack gyroscope attached to a calligraphy brush streams angular velocity to a Max patch, which derives a scalar speed value. This speed controls two mechanisms in a Python-based generative layer built around the Notochord autoregressive MIDI model: a distance-integral trigger that queries the model for the next melody note when accumulated motion reaches a threshold, and threshold-based activation of chord, bass, and sub-melody tracks. Output is routed to Ableton Live through LoopMIDI. The authors report an informal five-minute live performance at an art event and acknowledge that no formal recording or user study was conducted. The paper's contributions are framed as a performance-oriented prototype and a cross-modal performance scenario.
Significance. If the described integration works as stated, the system is a plausible and potentially useful NIME-style contribution: it uses an external, non-musical gestural signal to control the query timing and structural activation of an existing real-time generative model, rather than treating the model as an autonomous composer. The paper is concrete about the hardware/software pipeline, uses a publicly available model, and is honest about the absence of formal evaluation. The strength is the clear architectural description: Max for sensor processing, Python for model control, and Ableton for rendering, with OSC between components. The main weakness is that the central mechanism—replacing Notochord's internal inter-onset interval with an externally triggered query—is under-specified at the level of the model's hidden state and event feedback, which is load-bearing for the density-control claim. In addition, the 'low-latency' claim is unmeasured. With a precise description of the inference loop and at least a small quantitative or logged demonstration, the paper would substantiate its claims.
major comments (3)
- [§3.2.1] The core mechanism is under-specified: the paper states that 'the inter-onset interval is no longer decided by the model' and that the system triggers Notochord queries when a distance integral reaches a threshold, but it does not explain how Notochord's event representation and GRU hidden state are updated under this scheme. In Notochord, each event is a tuple of instrument, pitch, inter-onset interval, and velocity, and the model samples the IOI as part of the event; in the original auto mode, note endings are represented by zero-velocity events. If Calliphony discards the sampled IOI and query times are set externally, the hidden state must be updated with the actual elapsed time between triggers and with a representation of note-off/velocity-zero, but the paper does not specify which values are fed back. If the sampled IOI is fed back while the actual timing is externally controlled, the model's internal clock diverges from the performance timeline, and future pitch/chord samples come from an out-of-distribution hidden state; if the actual IOI is fed back, that should be stated explicitly. This is load-bearing for the central claim that brush motion controls note-onset density, and it must be resolved in a revised version.
- [Abstract and §3.3] The abstract calls the pipeline 'low-latency,' but no latency measurement is reported anywhere in the paper. For an interactive music performance system, the real-time claim should be supported by at least a round-trip latency measurement from sensor input to MIDI/audio output, or by CPU load and buffer-size data. Without this, the suitability of the system for live performance is asserted rather than demonstrated. The authors should either add such a measurement or explicitly qualify the claim as unquantified.
- [§4] The only reported evaluation is informal audience feedback from a single five-minute performance that was not recorded, and the paper explicitly acknowledges that this does not constitute user-study data. This is acceptable for a demo-oriented proceedings paper, but it means the functional claims—distance-integral density control, threshold-based layer activation, scale-based pitch constraints—are not verified. A short logged demonstration, such as a recorded sensor trace with the resulting MIDI output, or a simple offline analysis of event-density versus brush-speed correlation, would substantially strengthen the central claims.
minor comments (5)
- [§3.1] The method for obtaining a scalar speed value from accumulated rotation angles does not discuss gyroscope drift, offset calibration, or integration error; given that the signal is accumulated over time, a brief note on how drift was handled in practice would improve reproducibility.
- [§3.1 and §2.1] The paper emphasizes brush direction and turns as expressive features in calligraphy, but the control signal is a scalar speed value that discards directional information; please clarify whether direction is intentionally unused or whether future work plans to exploit it.
- [§1] There are several typographical spacing errors, including 'This paper presentsCalliphony' and 'poemShui Diao Ge Touon a water-writing cloth'; these should be corrected in a final pass.
- [§2.2] The sentence 'Early work such as Hé: Calligraphy as a Musical Interface extract computable features' has a subject-verb agreement error ('extract' should be 'extracts'); this is a minor presentational fix.
- [§4 and §7] The paper promises to release code and a video demo upon publication; if possible, the authors should include a link or a supplementary demo file in the revised version, since the unrecorded performance cannot otherwise be inspected.
Circularity Check
No significant circularity: the control mapping is by construction and Notochord is an external model; self-citations are background only.
full rationale
The paper is an engineering and prototype description rather than a derivation or empirical prediction. The claimed control mechanism in Section 3.2.1 is implemented by definition: integrating a speed signal and triggering a Notochord query at threshold crossings directly defines note-onset density; the paper does not present this relationship as a discovered prediction, so there is no fitted input renamed as a prediction. Notochord is an external, publicly released model [30], and the system uses its query and feed interfaces without fitting any parameters to the authors' own data. Thresholds and exclusion probabilities are user-adjustable interface parameters, not fitted constants. Self-citations such as [18,19,21,37,40] appear only as background on calligraphy generation and symbolic music modeling; none is load-bearing for the Calliphony pipeline, and no uniqueness theorem from the authors' prior work is invoked. The only substantive concern raised by the system description is whether overriding Notochord's inter-onset interval modality without specifying how the GRU hidden state accounts for actual elapsed time puts the model out of distribution; that is a correctness or under-specification risk, not circularity, because it does not make any output equal to an input by construction. Therefore no circular steps are identified.
Assumptions & free parameters
free parameters (3)
- distance-integral trigger threshold =
unspecified, user-adjustable
- onset and stop thresholds for accompaniment =
unspecified, user-adjustable
- pitch-repeat exclusion probability =
user-adjustable 0..1
assumptions (2)
- domain assumption Notochord's autoregressive GRU model, with the publicly released checkpoint, provides coherent symbolic generation when queried under constraints.
- domain assumption Integration of tri-axial angular velocity over time yields a scalar that is monotonically related to perceived writing energy.
Cite this review
Pith. "Pith review of Calliphony: A Calligraphy-Driven Interface for Real-Time Generative Music Performance." pith.science (2026). https://pith.science/paper/OFSLE2Z7
@misc{pith2026260803040,
author = {Pith},
title = {Pith review of: Calliphony: A Calligraphy-Driven Interface for Real-Time Generative Music Performance},
year = {2026},
howpublished = {\url{https://pith.science/paper/OFSLE2Z7}},
note = {Machine review of arXiv:2608.03040}
}
read the original abstract
While music generative models have recently gained significant attention, how they can be effectively integrated into live music performances still requires further exploration. This paper presents Calliphony, a calligraphy-driven interface for real-time generative music performance. Specifically, we build a low-latency pipeline that captures brush motion with an attachable sensor and maps it to control signals for real-time symbolic music generation. Using a generative model, the system produces multi-track MIDI in performance settings, while brush-derived control signals constrain event timing and activate additional musical layers. The generated melody is then extended with real-time harmony and additional voices, and finally rendered through a DAW for live staging. Calliphony contributes: (1) a performance-oriented prototype that uses calligraphic motion as an external control layer for a real-time symbolic music generation model, controlling note density, pitch constraints, and accompaniment-layer activation; and (2) a cross-modal performance scenario that extends calligraphy beyond a primarily visual practice into an audiovisual, AI-assisted setting.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. MusicLM: Generating music from text.arXiv preprint arXiv:2301.11325, 2023
arXiv 2023
-
[2]
BachDuet: A human-machine duet improvisation system
Christodoulos Benetatos and Zhiyao Duan. BachDuet: A human-machine duet improvisation system. InLate-Breaking/Demo at the 20th International Society for Music Information Retrieval Conference (ISMIR), Delft, The Netherlands,
-
[3]
KAHEI CHENG, Irina Kruchinina, and Matin Esmaeili. Phantom of utopia. InProceedings of the International Conference on New Interfaces for Musical Expression, pages 142–145, 2024
work page 2024
-
[4]
Die schönheit der vergänglichkeit
Se-Lien Chuang and Andreas Weixler. Die schönheit der vergänglichkeit. InProceedings of the International Conference on New Interfaces for Musical Expression, pages 70–73, 2024
work page 2024
-
[5]
MusicGen: Simple and controllable music generation.arXiv preprint arXiv:2306.05284, 2023
Jade Copet, Felix Kreuk, Gabriel Synnaeve, Yossi Adi, et al. MusicGen: Simple and controllable music generation.arXiv preprint arXiv:2306.05284, 2023
arXiv 2023
-
[6]
Dawn Delbanco. Chinese calligraphy, 2008. The Metropolitan Museum of Art, accessed: 2026-02-03
work page 2008
-
[7]
Rahul Dey and Fathi M. Salem. Gate-variants of gated recurrent unit (GRU) neural networks. In2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), pages 1597–1600. IEEE, 2017
work page 2017
-
[8]
Jukebox: A generative model for music.arXiv preprint arXiv:2005.00341, 2020
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Rad- ford, and Ilya Sutskever. Jukebox: A generative model for music.arXiv preprint arXiv:2005.00341, 2020
arXiv 2005
Show all 42 references
-
[9]
MMM: Exploring conditional multi-track music generation with the transformer, 2020
Jeff Ens and Philippe Pasquier. MMM: Exploring conditional multi-track music generation with the transformer, 2020
2020
-
[10]
The four treasures of the study: Ink, inkstone, brush, and paper, 2019
Howard Hotson. The four treasures of the study: Ink, inkstone, brush, and paper, 2019. University of Oxford, accessed: 2026-02-03
2019
-
[11]
Music transformer.arXiv preprint arXiv:1809.04281, 2018
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew Dai, Matthew Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer.arXiv preprint arXiv:1809.04281, 2018
2018 arXiv
-
[12]
Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions
Yu-Siang Huang and Yi-Hsuan Yang. Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions. InProceedings of the 28th ACM International Conference on Multimedia, MM ’20, pages 1180– 1188, New York, NY, USA, 2020. Association for Computing ...
2020
-
[13]
Self-cultivation through art: Chinese calligraphy and the body
Ruyu Hung. Self-cultivation through art: Chinese calligraphy and the body. Educational Philosophy and Theory, 2021. Published online: 14 Sep 2021
2021
-
[14]
Infilling piano performances
Daphne Ippolito, Cheng-Zhi Anna Huang, Curtis Hawthorne, and Douglas Eck. Infilling piano performances. InNeurIPS Workshop on Machine Learning for Creativity and Design, 2018. Workshop paper / online supplement
2018
-
[15]
Hé: Calligraphy as a musical interface
Laewoo Kang and Hsin-Yi Chien. Hé: Calligraphy as a musical interface. InProceedings of the International Conference on New Interfaces for Musical Expression, pages 352–355, 2010
2010
-
[16]
Collaboration and recursion: Reflections on calligraphy and feedback
Iurii Kuzmin, Omar Al Kanawati, and Raul Masu. Collaboration and recursion: Reflections on calligraphy and feedback. InProceedings of the International Conference on New Interfaces for Musical Expression, pages 175–183, 2025
2025
-
[17]
George E. Lewis. Too many notes: Computers, complexity and culture in Voyager.Leonardo Music Journal, 10:33–39, 2000
2000
-
[18]
CalliffusionV2: Personalized natural calligraphy generation with flexible multi-modal control, 2024
Qisheng Liao, Liang Li, Yulang Fei, and Gus Xia. CalliffusionV2: Personalized natural calligraphy generation with flexible multi-modal control, 2024
2024
-
[19]
Calliffusion: Chinese calligraphy generation and style transfer with diffusion modeling
Qisheng Liao, Gus Xia, and Zhinuo Wang. Calliffusion: Chinese calligraphy generation and style transfer with diffusion modeling. InProceedings of the International Conference on Computational Creativity, 2023
2023
-
[20]
Moyun: A diffusion-based model for style-specific chinese calligraphy generation, 2024
Kaiyuan Liu, Jiahao Mei, Hengyu Zhang, Yihuai Zhang, Xingjiao Wu, Daoguo Dong, and Liang He. Moyun: A diffusion-based model for style-specific chinese calligraphy generation, 2024
2024
-
[21]
Polyffusion: A diffusion model for polyphonic score generation with internal and external controls
Lejun Min, Junyan Jiang, Gus Xia, and Jingwei Zhao. Polyffusion: A diffusion model for polyphonic score generation with internal and external controls. arXiv preprint arXiv:2307.10304, 2023
2023 arXiv
-
[22]
Symbolic music generation with diffusion models
Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. InProceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2021
2021
-
[23]
This time with feeling: Learning expressive musical performance.Neural Computing and Applications, 32:955–967, 2020
Sageev Oore, Ian Simon, Sander Dieleman, Douglas Eck, and Karen Simonyan. This time with feeling: Learning expressive musical performance.Neural Computing and Applications, 32:955–967, 2020
2020
-
[24]
Calligraphy
Oxford Bibliographies. Calligraphy. Oxford Bibliographies: Chinese Studies. Retrieved February 8, 2026
2026
-
[25]
The Continuator: Musical interaction with style.Journal of New Music Research, 32(3):333–341, 2003
François Pachet. The Continuator: Musical interaction with style.Journal of New Music Research, 32(3):333–341, 2003
2003
-
[26]
Colin Raffel and Daniel P. W. Ellis. Extracting ground-truth information from MIDI files: A MIDIfesto. InProceedings of the International Society for Music Information Retrieval Conference (ISMIR), pages 796–802, 2016
2016
-
[27]
PopMAG: Pop music accompaniment generation
Yi Ren, Jinzheng He, Xu Tan, Tao Qin, Zhou Zhao, and Tie-Yan Liu. PopMAG: Pop music accompaniment generation. InProceedings of the 28th ACM Inter- national Conference on Multimedia, MM ’20, pages 1190–1198. Association for Computing Machinery, 2020
2020
-
[28]
A hierarchical latent vector model for learning long-term structure in music
Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck. A hierarchical latent vector model for learning long-term structure in music. InProceedings of the 35th International Conference on Machine Learning (ICML), 2018
2018
-
[29]
MIT Press, 1993
Robert Rowe.Interactive Music Systems: Machine Listening and Composing. MIT Press, 1993
1993
-
[30]
Notochord: A flexible probabilistic model for real-time MIDI performance.arXiv preprint arXiv:2403.12000, 2024
Victor Shepardson, Jack Armitage, and Thor Magnusson. Notochord: A flexible probabilistic model for real-time MIDI performance.arXiv preprint arXiv:2403.12000, 2024
2024 arXiv
-
[31]
Learning a latent space of multitrack measures, 2018
Ian Simon, Adam Roberts, Colin Raffel, Jesse Engel, Curtis Hawthorne, and Douglas Eck. Learning a latent space of multitrack measures, 2018
2018
-
[32]
Four treasures of a scholar’s studio, 2026
Smithsonian National Museum of Asian Art. Four treasures of a scholar’s studio, 2026. Smithsonian Institution, accessed: 2026-02-03
2026
-
[33]
Computer assisted melo-rhythmic generation of traditional chinese music from ink brush calligraphy
Will WW Tang, Stephen Chan, Grace Ngai, and Hong-va Leong. Computer assisted melo-rhythmic generation of traditional chinese music from ink brush calligraphy. InProceedings of the International Conference on New Interfaces for Musical Expression, pages 84–89, 2013
2013
-
[34]
Decision of the intergovernmental committee: 4.COM 13.08
UNESCO. Decision of the intergovernmental committee: 4.COM 13.08. UN- ESCO Intangible Cultural Heritage, 2009. Retrieved February 8, 2026
2009
-
[35]
Chinese calligraphy, 2009
UNESCO Intangible Cultural Heritage. Chinese calligraphy, 2009. Representa- tive List of the Intangible Cultural Heritage of Humanity, accessed: 2026-02-03
2009
-
[36]
SongDriver: Real-time music accompaniment generation without logical la- tency nor exposure bias
Zihao Wang, Qihao Liang, Kejun Zhang, Yuxing Wang, Chen Zhang, Pengfei Yu, Yongsheng Feng, Wenbo Liu, Yikai Wang, Yuntai Bao, and Yiheng Yang. SongDriver: Real-time music accompaniment generation without logical la- tency nor exposure bias. 2022
2022
-
[37]
Whole-song hierarchical genera- tion of symbolic music using cascaded diffusion models.arXiv preprint arXiv:2405.09901, 2024
Ziyu Wang, Lejun Min, and Gus Xia. Whole-song hierarchical genera- tion of symbolic music using cascaded diffusion models.arXiv preprint arXiv:2405.09901, 2024
2024 arXiv
-
[38]
Gesture-driven DDSP synthesis for digitizing the chinese erhu
Wenqi WU and Hanyu QU. Gesture-driven DDSP synthesis for digitizing the chinese erhu. In Doga Cavdir and Florent Berthaut, editors,Proceedings of the International Conference on New Interfaces for Musical Expression, pages 505–510, Canberra, Australia, June 2025
2025
-
[39]
Adaptive accompaniment with ReaLchords
Yusong Wu, Tim Cooijmans, Kyle Kastner, Adam Roberts, Ian Simon, Alexan- der Scarlatos, Chris Donahue, Cassie Tarakajian, Shayegan Omidshafiei, Aaron Courville, Pablo Samuel Castro, Natasha Jaques, and Cheng-Zhi Anna Huang. Adaptive accompaniment with ReaLchords. InProceedings...
2024
-
[40]
Deep music analogy via latent representation disentanglement.arXiv preprint arXiv:1906.03626, 2019
Ruihan Yang, Dingsu Wang, Ziyu Wang, Tianyao Chen, Junyan Jiang, and Gus Xia. Deep music analogy via latent representation disentanglement.arXiv preprint arXiv:1906.03626, 2019
1906 arXiv
-
[41]
Chinese calligraphy
Chiang Yee and The Editors of Encyclopaedia Britannica. Chinese calligraphy. Encyclopaedia Britannica, 2009. Accessed: 2026-02-08
2009
-
[2019]
Extended abstract (Late-Breaking/Demo, unrefereed)
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.