{"id":"a84afdf9-183c-445d-b9a4-01cb27e36ba0","arxiv_id":"2505.04203","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ELGAR generates whole-body cello performance motion from audio using a diffusion transformer, with new contact losses and a normalized motion-capture dataset.","lead":"This paper presents ELGAR, a diffusion-based system that generates whole-body cello playing motions directly from audio. It introduces hand and bow contact losses and a new dataset, and evaluates with instrument-specific metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never tests whether generated motions are actually controlled by the audio, so the 'directly from audio' claim is not yet evidenced; reported metrics cannot rule out generic, audio-agnostic output.","rationale":"The reader's weakest assumption concerns the reliability of audio-derived contact positions used both as training targets and as evaluation ground truth, and the lenient 'closest trigger position' rule for FCD. That concern is valid and directly weakens the quantitative case. I elevate a related but more fundamental issue: the central claim requires that the generated motion be controlled by the input audio, yet no experiment manipulates the audio condition or compares against an audio-agnostic baseline. The existing Table 1 ablation only toggles the interactive contact losses, so even if the metrics were perfectly reliable, they would not establish that the audio signal, rather than the dataset prior, drives the output. The reader's concern and mine are complementary: the metric circularity makes it even harder to detect an audio-agnostic model. I therefore partially agree with the reader's weakest assumption. The paper does have independent support: the SPD-GEN dataset and code are released, the method is clearly described, and the paper honestly acknowledges limitations in Section 5, including occasional bow-string contact loss and limited long-context coherence. These acknowledgements do not resolve the missing audio-conditioning control. Since the missing control is precisely the kind of additional evidence the reader's CONDITIONAL verdict calls for, I keep the verdict at CONDITIONAL (UNCHANGED), with the condition made more specific: the authors should demonstrate, via shuffled-audio or unconditional baselines, that the audio condition materially changes the generated motion.","tokens_in":13397,"tokens_out":4726,"duration_ms":50607,"concrete_test":"Generate motions for a fixed set of test clips under three conditions: (1) correct audio, (2) audio from a different piece (shuffled condition), and (3) unconditional generation (drop the audio condition). Compute FCD, BSD, BF1, and additionally a string/pitch agreement rate comparing the string implied by the generated finger trigger position to the pitch present in the audio. If the shuffled or unconditional conditions match condition (1) within noise, or if string/pitch agreement is at chance, the audio condition is inert and the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ELGAR generates whole-body cello performance motion 'directly from audio signals' (Section 1). The paper's experiments never test whether the audio condition actually controls the output: Section 3.4 inserts Jukebox features via cross-attention and CFG (Eq. 4), but Table 1 only ablates HICL/BICL, and every quantitative evaluation uses the same audio that produced the ground-truth motion. A model that ignores the audio condition and outputs generic, dataset-typical cello gestures could still obtain the reported FCD and BSD values, because finger-contact distance (Section 4.2) resolves the string/finger ambiguity by picking the trigger position closest to the generated finger, and the contact losses (Eqs. 6-7) are training-time regularizers that do not certify inference-time audio sensitivity. The reported metrics therefore conflate motion plausibility with audio-to-motion alignment. The paper's own Discussion acknowledges residual artifacts ('bow occasionally loses contact with the strings', 'limited context awareness'), so the quantitative claims carry most of the burden; without an audio-conditioning control, the strongest claim is not demonstrably supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ELGAR, a diffusion-transformer framework that generates whole-body cello performance motion from raw audio using Jukebox audio features. The motion representation includes body and hand rotations plus a bow direction vector, and the framework uses classifier-free guidance. Two domain-specific losses are introduced: Hand Interactive Contact Loss (HICL) and Bow Interactive Contact Loss (BICL), which encourage contact between the left-hand fingers and the audio-derived string positions and between the bow and the active string. The authors contribute SPD-GEN, a normalized motion-capture dataset, and new evaluation metrics for string performance: finger-contact distance, bow-string distance, and bowing F1/cosine similarity. Experiments are presented as ablations over the interaction losses.","tokens_in":13656,"tokens_out":7358,"duration_ms":68385,"significance":"If the central claims were fully supported, ELGAR would be the first audio-only, full-body model for instrument performance motion, and the SPD-GEN dataset plus the contact-aware losses would be useful community assets. The authors are transparent about limitations (static cello, binary finger-string pressure, occasional bow-string contact loss) and promise code and data release. The main weakness is evidentiary: the headline metrics largely mirror the training losses, and no experiment isolates the audio-conditioning behavior, so the strongest claims currently outrun the evaluation.","major_comments":[{"comment":"The two headline metrics are essentially re-statements of the training losses. Finger-contact distance is exactly the note-playing-finger term in HICL (the first term of Eq. 6 with indicator I_f0), and bow-string distance is exactly the first term of BICL (Eq. 7). The reported improvements therefore mostly verify that the optimizer minimized the losses, not that the generated motions are more correct in an independent sense. I recommend adding distribution-level metrics (e.g., FID on pose features), penetration/collision counts between the hand or bow and the cello mesh, and a perceptual study, or at least reporting metrics computed under a protocol that is not directly optimized.","section":"Section 4.2, Table 1, Eqs. (6)-(7)"},{"comment":"The experiments do not test whether the audio condition causally controls generation. All test samples use the same audio that produced the ground-truth motion, the CFG scale w in Eq. (4) is not swept, and no unconditional or mismatched-audio condition is evaluated. A model that ignores audio and returns dataset-typical cello motions could plausibly achieve similar FCD and BSD values because the evaluation selects the trigger position closest to the generated finger. Please add an explicit control experiment: generate with mismatched audio (e.g., swapped between test pieces) and with the condition dropped or masked, and show clear degradation on all four metrics; also report a small sweep over w.","section":"Section 3.4 and Section 4.2"},{"comment":"The ambiguity-resolution rule, choosing the trigger position closest to the generated note-playing finger, makes FCD and BSD lenient and can conceal wrong-string or wrong-position playing. For example, a generated finger near an alternative string position for the same pitch is scored as correct even if it does not match the performer's actual string choice. Please report, in addition, the per-frame accuracy of the selected string/finger against the ground-truth annotation, or evaluate FCD and BSD with the ground-truth trigger positions fixed, so the metrics penalize actual fingering errors.","section":"Section 4.2"},{"comment":"No baselines, error bars, or significance tests are reported. The numbers in Table 1 are single values without variance or test-set size, and the 'state-of-the-art' claim in the abstract is not supported by any comparison to existing audio-to-motion or performance-motion methods. At minimum, compare with a deterministic regression baseline and an audio-agnostic diffusion variant, and report mean plus or minus standard deviation over multiple seeds or test splits.","section":"Table 1 and Section 1"}],"minor_comments":[{"comment":"The text says the bow frog is anchored between the middle finger, ring finger, and thumb of the left hand, but the bow is held in the right hand (Section 3.3.2 and Fig. 2). Please correct this typo, since the hand assignment is essential for reproducing the motion representation.","section":"Section 3.1"},{"comment":"The simple loss is written with a norm rather than a squared norm; if the squared L2 loss is intended, as in the cited works, please fix the notation.","section":"Equation (3)"},{"comment":"The terms 'bowing score' and 'bowing F1-score' are used interchangeably; define the exact quantity and clarify whether the reported BCS is the cosine similarity only.","section":"Section 4.2"},{"comment":"The loss weights in Eq. (8) and the CFG scale w in Eq. (4) are not reported; please include their values, as they are needed to reproduce the ablations.","section":"Section 4.1"},{"comment":"The number of test segments and the split procedure (e.g., whether overlapping 5-second slices are deduplicated) should be stated.","section":"Table 1"},{"comment":"The reference list contains duplicate entries (Karunratanakul et al. 2023a/b and Li et al. 2025a/b refer to the same papers); please consolidate.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and the authors have been transparent about limitations. My main concern is evidentiary: the evaluation does not yet demonstrate audio-conditioned control. I recommend major revision rather than rejection because the missing controls and metrics can be added without changing the method. I would also check that the 'first' claim is properly scoped against partial-body audio-to-performance methods such as Shlizerman et al. and Kao and Su."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is real work—new task framing, a cleaned-up dataset, and losses that improve contact plausibility—but the headline numbers don't yet prove that the motions are actually driven by the audio. The paper deserves a serious referee, and it will need a different evaluation design to support its central claim.\n\nWhat's genuinely new: ELGAR is the first system I know of that generates whole-body (torso + hands + bow) cello performance motion straight from raw audio, with no MIDI or physics sim. The SPD-GEN dataset—normalized to one body and one cello, with an arched bridge—is a useful contribution on its own. HICL and BICL are sensible domain losses, and the qualitative videos in the supplementary material show noticeably cleaner hand-string and bow-string interaction than the no-ICL variant. The paper is also honest about its limits: static cello, binary finger pressure, occasional bow-string separation.\n\nThe soft spots are in the evaluation. FCD and BSD are not independent of the training objectives: FCD is essentially the first term of HICL (Eq. 6), and BSD is the first term of BICL (Eq. 7). An ablation table that shows \"adding HICL/BICL improves FCD/BSD\" is, to a significant degree, a restatement of the loss design. The less directly optimized metrics—bowing F1 (0.4721 with ICL) and cosine similarity (0.7515)—are more informative, but there are no error bars or significance tests, and no baselines at all, neither prior partial-body methods nor a simple pose-only model. The stress-test concern holds: nothing in the experiments tests whether the audio condition actually controls the output. A model that ignores audio and emits dataset-typical cello gestures could plausibly achieve similar FCD/BSD numbers because the string/finger ambiguity is resolved by picking the trigger position nearest the generated finger. To support \"directly from audio,\" the authors need a conditioning control: shuffled audio, no-audio, or a different piece of music as input, reported across all metrics. The discussion acknowledges some artifacts, but the central claim lives or dies on the quantitative evidence.\n\nWho should read it: anyone working on audio-driven animation or human-object interaction generation. It's a useful system paper and the dataset is worth borrowing, but treat the evaluation as a proof-of-concept, not a validated result. If I were the editor, I would send it to review—there is enough new substance here—but I would push hard for the conditioning baseline and for reporting variance before accepting.","headline":"A genuinely new task and a useful dataset, but the evaluation is too circular and too under-powered to back the 'directly from audio' claim as it stands.","tokens_in":14205,"tokens_out":3306,"would_cite":true,"duration_ms":34144,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ELGAR generates whole-body cello performance motion with fine-grained hand and bow detail directly from raw audio.","keywords":["motion generation","cello performance","audio-conditioned generation","diffusion models","hand-object interaction","inverse kinematics","motion capture dataset","performance animation"],"falsifier":"Take a passage where the same pitch recurs on different strings and measure finger-contact and bow-string distance using the string choice fixed by the audio estimator alone, without the nearest-trigger selection. If the generated motions' error jumps while the paper's reported metric stays small, the correct conclusion is that ELGAR reproduces contact geometry only under a lenient, self-selected target; a human listener or a spectrogram check of whether the generated bow direction fits the recorded timbre would settle which string the audio truly implies.","tokens_in":13184,"feed_emoji":"🎻","tokens_out":8687,"duration_ms":78396,"temperature":0.7,"pith_summary":"The paper claims to open a task no earlier system handles end-to-end: turning a raw cello recording into a full-body animated performance—torso, both hands, and the bow—without MIDI, sheet music, or a physics simulator. Its central argument is that earlier audio-to-motion work failed because it modeled body pose while ignoring the performer-instrument interaction, so ELGAR adds two training losses, HICL and BICL, that explicitly pull the note-playing fingertip and the bow onto the strings the audio implies. To support this, the authors normalize the SPD motion-capture corpus into a new benchmark, SPD-GEN (about 7000 seconds of whole-body cello motion on a shared body and cello), and propose three string-specific evaluation metrics: finger-contact distance, bow-string distance, and bowing score. A sympathetic reader should care because, if the claim holds, expressive instrumental animation becomes a one-input problem, which changes how films, games, music education, and interactive art can be produced.","feed_headline":"From raw audio alone, ELGAR animates a whole cellist","feed_subtitle":"Audio-driven diffusion model adds hand-string and bow-string contact, fixing what earlier music-to-motion systems skipped.","key_machinery":"The load-bearing object is the Interactive Contact Loss, split into a Hand Interactive Contact Loss (HICL) and a Bow Interactive Contact Loss (BICL). Given the fundamental frequency of the audio, it computes a theoretical contact position on the cello fingerboard and identifies the activating string; HICL penalizes the distance between the generated note-playing fingertip and that contact position, BICL penalizes the distance between the generated bow and the activating string, and indicator functions switch the losses on only while a pitch is sounding. Around that core sits a diffusion transformer with DiT adaLN-Zero blocks, cross-attention to frozen Jukebox audio features, classifier-free guidance, and geometric losses (position, foot contact, rotation velocity, position velocity) for physical plausibility. The supporting dataset object is SPD-GEN, which normalizes all performers to one SMPL-X body and one shared cello with a restored arched bridge, using Kabsch alignment and two-stage inverse kinematics, so that audio-to-motion learning sees consistent geometry.","core_discovery":"ELGAR's claim, stated in the authors' terms, is that it is the first solution for generating whole-body instrument performance motion directly from audio, with the fine-grained hand gestures and intricate hand-string and bow-string interactions that prior work left out. The model represents a performance as a 309-dimensional motion vector (SMPL-X body and hand rotations plus a bow direction) and trains a diffusion transformer, conditioned by a frozen Jukebox audio encoder, to denoise that vector into clean motion. The two contact losses are the distinctive step: HICL supervises the left-hand note-playing finger to land on the audio-derived contact position while the other fingers keep a natural clearance from it, and BICL supervises the bow to reach the activating string with both endpoints respecting that string. The ablation results are reported as evidence for the core claim: adding BICL to HICL cuts average bow-string distance from 23.98 mm to 5.40 mm and lifts bowing F1-score from about 0.43 to 0.47, while the authors argue that general distribution metrics like FID are not appropriate here because the contact losses deliberately push generated motions away from the dataset distribution.","pith_inferences":["Because the same pitch can be played on more than one string, the evaluation's choice of the trigger position nearest the generated finger means a wrong string choice can still earn a small finger-contact error; a sharper test would fix the intended string from the audio first and then measure whether the generated hand and bow are on that string.","If contact supervision is the key, the same HICL/BICL recipe should transfer to the rest of the violin family, where the physics of vibrating string length is identical and only the fingerboard geometry and bow angles change.","The paper's own listed simplifications—binary finger-string pressure, a static cello, occasional bow-string lift-off, and small dataset size—suggest the next measurable step is force or pressure sensing and instrument-body tracking rather than more pose data alone.","A musician-listener study could test whether the generated motions read as musical: the current metrics measure physical contact distances, not whether the timing and bow direction feel like a real interpretation."],"forward_implications":["An animator could generate a full-body cellist, hands and bow included, directly from any audio clip, with no MIDI transcription or physics simulation in the loop.","SPD-GEN gives the research community a normalized benchmark for 3D instrument performance motion generation, filling a gap the paper identifies in existing datasets.","The ablations show the interaction losses are the decisive component for contact accuracy: adding BICL reduces bow-string distance from 23.98 mm to 5.40 mm.","The new metrics allow future string-performance generators to be judged by whether the fingers and bow actually meet the right places on the instrument, instead of only by distributional similarity to training data.","The overlapping-window long-form sampling strategy extends the method from five-second clips to full pieces, which is what practical use in animation demands."],"supporting_citations":[{"why":"Supplies the SPD motion-capture corpus and the audio-derived contact positions and activating strings that HICL and BICL use as targets.","marker":"[Jin et al. 2024a]"},{"why":"Jukebox is the frozen audio encoder whose features condition the diffusion denoiser.","marker":"[Dhariwal et al. 2020]"},{"why":"Provides the DiT architecture with adaLN-Zero blocks that form the denoising backbone.","marker":"[Peebles and Xie 2023]"},{"why":"Classifier-free guidance is the training and sampling recipe that lets the model trade diversity for audio adherence.","marker":"[Ho and Salimans 2022]"},{"why":"Source of the position, foot-contact, and rotation-velocity geometric losses, and of the argument that extra losses can hurt distribution metrics.","marker":"[Tevet et al. 2022]"},{"why":"Contributes the overlapping long-form sampling strategy and the small-test-set caveat for FID-style evaluation.","marker":"[Tseng et al. 2023]"},{"why":"SMPL-X and VPoser provide the body model and inverse-kinematics stages used to normalize all performers to one shape.","marker":"[Pavlakos et al. 2019]"},{"why":"Kabsch alignment rotates each frame's cello to the shared instrument so motion data share consistent geometry.","marker":"[Kabsch 1976]"},{"why":"Supplies the three-frame tolerance rule used to compute bowing attack F1-score.","marker":"[Kao and Su 2020]"}],"fun_headline_variants":["Audio to whole-body cellist: ELGAR adds hand and bow contact","ELGAR: First whole-body cello motion from audio, with string contact","ELGAR: Audio-driven whole-body cello motion with contact losses","Whole-body cello from audio: ELGAR nails string interactions","ELGAR: Audio to full cellist motion, contact-aware and whole-body"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the audio-derived contact position reliably identifies which finger stops which string, although the same pitch can be produced on more than one string; the evaluation softens this further by choosing the trigger position closest to the generated finger, so a motion that plays the wrong string can still score well.","fun_headline_variants_meta":{"raw":{"variants":["Audio to whole-body cellist: ELGAR adds hand and bow contact","ELGAR: First whole-body cello motion from audio, with string contact","ELGAR: Audio-driven whole-body cello motion with contact losses","Whole-body cello from audio: ELGAR nails string interactions","ELGAR: Audio to full cellist motion, contact-aware and whole-body"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001386,"raw_usage":{"total_tokens":5654,"prompt_tokens":1030,"completion_tokens":4624,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":4524}},"tokens_in":646,"tokens_out":4624,"duration_ms":30805,"temperature":1.0,"reasoning_tokens":4524,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:34:11.126079+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a passage where the same pitch recurs on different strings and measure finger-contact and bow-string distance using the string choice fixed by the audio estimator alone, without the nearest-trigger selection. If the generated motions' error jumps while the paper's reported metric stays small, the correct conclusion is that ELGAR reproduces contact geometry only under a lenient, self-selected target; a human listener or a spectrogram check of whether the generated bow direction fits the recorded timbre would settle which string the audio truly implies.","supporting_citations":[],"review_version":1}