Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Musical Agent Systems: MACAT and MACataRT

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MACAT and MACataRT are real-time musical agents that improvise with humans using small personal audio corpora.

desk verdict A useful system/practice report on two working MAX/MSP musical agents, but the conclusion overreaches: the factor-oracle and real-time synthesis additions are plausible, not demonstrated as better than baseline. read the letter →

arxiv 2502.00023 v1 pith:V57VIKBU submitted 2025-01-19 cs.MA cs.AIcs.HCcs.SDeess.AS

Keon Ju M. Lee , Philippe Pasquier This is my paper · ORCID
classification cs.MAcs.AIcs.HCcs.SDeess.AS
keywords musicalagentsconcatenativesynthesisfactororacleself-organizingmapaudiomosaicingreal-timeimprovisationsmalldataco-creativeAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents two working musical agents, MACAT and MACataRT, built for real-time human–AI co-creation. The central claim is that both systems let a musician bring a small corpus of their own recordings and get an improvising partner that responds live, either leading a solo piece (MACAT) or collaborating through reactive and proactive audio mosaicing (MACataRT). The authors argue that this small-data, artist-in-the-loop approach preserves expressive nuance, keeps training transparent and ethical, and broadens what a performer can do in live improvisation. They offer live performances, including a top-ten finalist entry in the 2024 AI Music Song Contest, as evidence that the systems function in real musical practice.

What carries the argument

The load-bearing mechanism is the pairing of concatenative sound synthesis with the factor oracle. Concatenative synthesis rebuilds sound by selecting and stitching short segments from a user-supplied corpus according to audio descriptors, preserving the expressive character of the original recordings. The factor oracle is a suffix automaton, trained offline on the sequence of segment (or SOM node) indices, that recognizes recurring patterns and generates new sequences by forward or backward jumps; a probability parameter controls how often the oracle moves forward rather than backwards. MACAT adds a self-organizing map—a 2D grid that clusters perceptually similar segments—and a machine-listening loop that feeds the current audio state back into the oracle. MACataRT instead applies the oracle to the indices produced by CataRT-style audio mosaicing, so the same corpus can drive either reactive accompaniment or autonomous continuation.

What would settle it

A controlled listening study comparing MACataRT's proactive mode against the same system with the factor oracle disabled and the segment order randomized would settle whether the temporal model adds musical coherence; if listeners cannot distinguish the oracle version from the scrambled baseline, the central creative claim is unsupported.

Watch

Extended reading notes

Core claim

The paper's central claim is that MACAT and MACataRT are functioning musical agent systems for real-time co-creative improvisation, built on corpus-based concatenative synthesis and trained on small, personalized audio corpora. MACAT is the agent-led system: it clusters corpus segments with a self-organizing map, learns the node sequence with a variable-order Markov model, and uses a factor oracle plus a self-listening loop to choose what to play next, with a congruence parameter that lets the performer choose between repetition and exploration. MACataRT is the collaborative system: it keeps CataRT-style audio mosaicing and descriptor targeting, and adds a factor oracle trained on the sequence of segment indices, giving it two modes, reactive improvisation (respond to the live input) and proactive improvisation (continue from learned patterns). The paper offers live performances, including a top-ten finalist piece in the 2024 AI Music Song Contest, as evidence that the systems work in musical practice.

Load-bearing premise

The conclusion that these systems broaden creative options rests on treating successful live performances as sufficient evidence, because the paper deliberately sets aside quantitative evaluation.

Editorial extensions

If this is right

  • A musician with a small set of personal recordings can train an improvising agent on a laptop CPU, with no GPU or large dataset, and use it in live performance.
  • MACataRT's factor-oracle layer turns a reactive mosaicing instrument into one that can also generate autonomously, so the same corpus supports both following the human and leading the music.
  • MACAT's self-listening feedback allows a single performer to play alongside a machine that hears, clusters, and re-sequences its own output in real time.
  • Because all source material comes from a small curated corpus, a performance's audio provenance remains traceable, which supports the paper's ethical transparency argument.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the reactive/proactive split suggests a general design pattern—the same corpus and synthesizer can serve both as an accompanist and as an autonomous continuation engine, and musicians' preferences between those two roles are a testable question the paper does not isolate.
  • Beyond the paper: since every generated segment is drawn from a known corpus, the architecture could log segment indices and produce an exact audit trail of which source recordings contributed to a performance, making the claimed transparency mechanically verifiable.
  • Beyond the paper: a natural next comparison is to pit these small-data agents against a large-scale generative model on stylistic coherence with the same performer; the paper's design implies the small-data systems would win on personalization and traceability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents two MAX/MSP musical agent systems, MACAT and MACataRT, which extend the authors' prior MASOM system and IRCAM's CataRT with real-time concatenative synthesis and factor-oracle-based temporal modeling. It describes the systems' workflows, interfaces, machine listening components, and an ethical rationale built around small, personalized audio corpora. The paper reports live performances, including a Top-10 finalist entry in the 2024 AI Music Song Contest, and argues that these systems 'significantly broaden the creative options for musicians' in real-time co-creative improvisation.

Significance. The main strength is the detailed, well-illustrated description of two apparently functional open-source systems, with publicly available code and real-world performance documentation. The design philosophy of small data and artist-in-the-loop control is timely and relevant to the ethical AI-in-music discourse. However, the paper's stated value claim—that the systems 'significantly broaden creative options'—is not supported by any user study, listening test, or baseline comparison; as it stands the contribution is a credible systems/design report rather than an empirically validated claim about creative expansion.

major comments (3)
  1. [Section 4 / Abstract] The sentence 'MACAT and MACataRT demonstrate how artist-in-the-loop AI agents can significantly broaden the creative options for musicians' is a load-bearing claim that the evidence does not currently support. The paper presents no user study, listening test, structured rehearsal observation, or analysis of the cited performance recordings. Section 3.1 explicitly declines quantitative evaluation, but the conclusion makes a comparative and magnitude claim ('significantly broaden') that requires some form of evidence. Either temper the claim to a demonstration of functional tools or add an evaluation component such as expert assessment, performance transcripts, or a comparison of MACataRT with CataRT on a defined musical task.
  2. [Section 2.2 / Figure 1(d)] The claimed advantage of MACataRT over CataRT is the addition of a factor oracle temporal model to address CataRT's lack of temporal structure. This is described and shown in workflow diagrams, but no evidence shows that the factor oracle produces improved temporal coherence or creative utility relative to CataRT's KNN-based selection. Add at least a small case study or baseline comparison—for example, generation from the same corpus with and without the factor oracle—to make this load-bearing design claim credible.
  3. [Appendix A] The ethical claims—that small personalized datasets make training data 'straightforward tracking' and 'foster trust and confidence'—are presented as conclusions, but the paper provides no user data or systematic traceability analysis. As a design rationale these statements are acceptable, but as assertions about user experience and transparency they require either evidence or explicit reframing as intended properties.
minor comments (5)
  1. [Section 3.2] The phrase 'Echoes of Synthetic F orest' contains a typo; it should read 'Forest'.
  2. [Reference [7]] The title 'Adaptive Concatenative Sound Synthesis and Its Application to Micromontage Compositior' has a typo; 'Compositior' should be 'Composition'.
  3. [Appendix B.3] The acronym 'MFFC' should be 'MFCC' in the first bullet under extracted audio features.
  4. [Section 2.1 vs. Appendix B.2] Please clarify the relationship between VMM and the factor oracle in generation: Section 2.1 says MACAT learns sequences with VMM and uses FO for pattern recognition, while Appendix B.2 says the current MASOM architecture 'is not equipped with the VOMM ... but with Factor Oracle.' This appears inconsistent and should be reconciled.
  5. [Appendix B.3] The text refers to 'Equation 2a' when discussing spectral flatness statistics, but the equations are numbered (1) and (2); the reference should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; the paper is a system description whose claims rest on implementation and live practice, not on fitted inputs or self-referential reasoning.

full rationale

The paper is a systems/workshop description rather than a derivation chain. It introduces two MAX/MSP musical agents as extensions of MASOM and CataRT, and the load-bearing content is implementation, interface design, and live performance use, not an equation chain that maps inputs to predicted outputs. The Valence/Arousal regression coefficients in Appendix B are imported from prior external affect-recognition work and are not used to support the paper's central claim; they are part of MASOM's machine-listening module and are not fitted to data in this paper. Self-citations to MASOM and CataRT describe the systems being extended rather than invoking an unverified uniqueness theorem or smuggling in an ansatz. No fitted parameter is renamed as a prediction, and no quantity derived here is equivalent to an input by construction. The closest weakness is that the conclusion that the systems 'significantly broaden the creative options for musicians' is supported mainly by anecdotal live performances and an explicit methodological choice to prioritize research-creation over quantitative evaluation; that is an evidence-scope concern, not circularity. The derivation chain, such as it is, is self-contained with respect to the claims actually made.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims are about system design and use, not about a new physical or mathematical entity. The paper introduces no fitted parameters in its own contributions; the affect coefficients in Appendix B are inherited from prior external work and are not used to support the claimed novelty. The three listed axioms are the main unverified premises on which the claimed creative benefits rest.

assumptions (3)
  • domain assumption Small, personalized corpora preserve artistic nuance better than large corpora.
    Section 1.2 asserts this as the motivation for the systems, but no empirical comparison is provided.
  • domain assumption Factor-oracle temporal modeling produces musically coherent sequences appropriate for improvisation.
    Sections 2.2 and Appendix C assume that the factor oracle learns useful musical patterns; no listening tests or analyses support this.
  • domain assumption Research-creation methodology is appropriate because no quantitative measures exist for personalized improvisation systems.
    Section 3.1 explicitly invokes this assumption to justify replacing quantitative evaluation with live performance showcases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Musical Agent Systems: MACAT and MACataRT." pith.science (2026). https://pith.science/paper/V57VIKBU

@misc{pith2026250200023,
  author       = {Pith},
  title        = {Pith review of: Musical Agent Systems: MACAT and MACataRT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V57VIKBU}},
  note         = {Machine review of arXiv:2502.00023}
}
read the original abstract

Our research explores the development and application of musical agents, human-in-the-loop generative AI systems designed to support music performance and improvisation within co-creative spaces. We introduce MACAT and MACataRT, two distinct musical agent systems crafted to enhance interactive music-making between human musicians and AI. MACAT is optimized for agent-led performance, employing real-time synthesis and self-listening to shape its output autonomously, while MACataRT provides a flexible environment for collaborative improvisation through audio mosaicing and sequence-based learning. Both systems emphasize training on personalized, small datasets, fostering ethical and transparent AI engagement that respects artistic integrity. This research highlights how interactive, artist-centred generative AI can expand creative possibilities, empowering musicians to explore new forms of artistic expression in real-time, performance-driven and music improvisation contexts.

Figures

Figures reproduced from arXiv: 2502.00023 by the authors.

Figure 1
Figure 1. The workflow of each musical agent system for the comparison: (a) MASOM, (b) MACAT, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The latest MASOM interface extended by the authors. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Diagram of original MASOM architecture, as presented in the original MASOM publication. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The interface of MACAT system [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: The interface of MACataRT system [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Call and Response: Modelling Reciprocal Coordination in Human-AI Vocal Ensembles

    cs.HC 2026-08 conditional novelty 6.0 of 10

    The paper frames unconducted vocal ensemble singing as coupled dynamic systems without an external reference and proposes an architecture for an AI singer that enters, rather than tracks, the ensemble's collective state.

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    Tatar, K., & Pasquier, P. (2019). Musical agents: A typology and state of the art towards musical metacreation. Journal of New Music Research

    work page 2019

  2. [2]

    Pasquier, P., Eigenfeldt, A., Bown, O., & Dubnov, S. (2017). An introduction to musical metacreation. Computers in Entertainment (CIE)

    work page 2017

  3. [3]

    J., & Norvig, P

    Russell, S. J., & Norvig, P. (2016). Artificial intelligence: a modern approach. Pearson

    work page 2016

  4. [4]

    Wooldridge, M. (2009). An introduction to multiagent systems. John Wiley & sons

    work page 2009

  5. [5]

    Puckette, M. (1991). Combining event and signal processing in the MAX graphical programming environment. Computer music journal

    work page 1991

  6. [6]

    Schwarz, D., Beller, G., Verbrugghe, B., & Britton, S. (2006). Real-time corpus-based concatenative synthesis with CataRT. In the proceedings of the Conference on Digital Audio Effects (DAFx)

    work page 2006

  7. [7]

    Sturm, B. L. (2006). Adaptive Concatenative Sound Synthesis and Its Application to Micromontage Compositior. Computer Music Journal

    work page 2006

  8. [8]

    Hackbarth, B., Schnell, N., Esling, P., & Schwarz, D. (2013). Composing morphology: Concatenative synthesis as an intuitive medium for prescribing sound in time. Contemporary Music Review

    work page 2013

Show all 25 references
  1. [9]

    Einbond, A., Bresson, J., Schwarz, D., & Carpentier, T. (2021). Instrumental Radiation Patterns as Models for Corpus-Based Spatial Sound Synthesis: Cosmologies for Piano and 3D Electronics. In the proceedings of the International Computer Music Conference (ICMC)

    2021

  2. [10]

    Vigliensoni, G., Perry, P., & Fiebrink, R. (2022). A small-data mindset for generative AI creative work. In the proceedings of the Machine Learning for Creativity and Design Workshop at the conference on Neural Information Processing Systems (NIPS)

    2022

  3. [11]

    M., & Pasquier, P

    Abuzuraiq, A. M., & Pasquier, P. (2024). Seizing the Means of Production: Exploring the Landscape of Crafting, Adapting and Navigating Generative AI Models in the Visual Arts. In the proceedings of the Computer-Human Interaction (CHI) workshop on Generative AI and HCI

    2024

  4. [12]

    Thelle, N. J. W., & Wærstad, B. I. G. (2023). Co-Creatives Spaces: The machine as a collaborator. In Proceedings of the Conference on New Interfaces for Musical Expression (NIME)

    2023

  5. [13]

    Rowe, R. (1992). Interactive music systems: machine listening and composing. MIT press

    1992

  6. [14]

    & Pasquier, P

    Tatar, K. & Pasquier, P. (2017). MASOM: A Musical Agent Architecture based on Self-Organizing Maps, Affective Computing, and Variable Markov Models. In the proceedings of the Conference on International Workshop on Musical Metacreation (MuMe)

    2017

  7. [15]

    Kohonen, T. (1990). The self-organizing map. In the proceedings of the conference on the Institute of Electrical and Electronics Engineers (IEEE)

    1990

  8. [16]

    Begleiter, R., El-Yaniv, R., & Yona, G. (2004). On prediction using variable order Markov models. Journal of Artificial Intelligence Research

    2004

  9. [17]

    Assayag, G., & Dubnov, S. (2004). Using factor oracles for machine improvisation. Journal of Soft Computing

    2004

  10. [18]

    Cover, T., & Hart, P. (1967). Nearest neighbor pattern classification. IEEE transactions on information theory

    1967

  11. [19]

    Lazier, A., & Cook, P. (2003). MOSIEVIUS: Feature driven interactive audio mosaicing. In the proceedings of the conference on Digital audio effects (DAFx). [20]Stévance, S., & Lacasse, S. (2017). creation in Music and the Arts: Towards a Collaborative Interdiscipline. Routledge

    2003

  12. [21]

    Top 10 Finalist AI Music Song Contest (2024) URL: https://www.aisongcontest.com/the-2024-finalists (Last accessed: 14th of November, 2024)

    2024

  13. [22]

    KeRa’s Music Performance as a Top 10 Finalist at Zürich, Switzerland (2024) URL: https://www.youtube.com/live/2ojQEGNUXic?si=nEmzbR_yCyhyk1aS&t=4499 (Last accessed: 14th of November, 2024). 5 A Ethical Implication Our musical agent systems present multiple advantages over larg...

    2024

  14. [23]

    So, the musical agent should be pre-trained first (offline training) to use it during live performance

    Prior knowledge: The audio recordings (chosen by users) are used to train by applying AI algorithms, and the audio would be input data for this model. So, the musical agent should be pre-trained first (offline training) to use it during live performance

  15. [24]

    The musical agent learns to generate music based on musical memory (from self-organizing map nodes) and statistical learning (learning through the connectivity of each SOM node)

    Past actions: Automatic music generation in the agent is based on the training data (in audio recordings), and the recordings are converted to multiple audio segments using the multi-granular segmentation based on the Fast Fourier Transform algorithm. The musical agent learns ...

  16. [25]

    SOM is an artificial neural network that represents, visualizes, and clusters high-dimensional input data in a 2-dimensional topology based on each calculated feature vector

    Goal & values: The goal of the agent is automatic music composition and free improvisation for original music content based on the training data’s style of music. SOM is an artificial neural network that represents, visualizes, and clusters high-dimensional input data in a 2-d...

  17. [26]

    Drop a folder here!

    Observations: The agent can efficiently listen to a massive amount of music by extracting and analyzing audio features as part of the machine listening module. The agent partially sees the environment because it can perceive the current status (SOM nodes are visualized in real...

    2000

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.