{"id":"cca6501b-da7e-487e-92e1-c961b1f0997b","arxiv_id":"2502.10426","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A score follower that combines a Gaussian Process note classifier with a duration-aware HMM and windowed Viterbi decoder is demonstrated on solo piano and other instruments.","lead":"This thesis builds a musical score follower that first uses a Gaussian Process with a spectral mixture kernel to guess which notes are playing in each short audio frame, then feeds those guesses into a duration-aware hidden Markov model to track position in the score in real time. The author reports successful tracking on piano, flute, oboe, and violin recordings, but the evaluation is primarily qualitative.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative score-following accuracy is never measured, so the central claim of 'successful score following' is not yet established.","rationale":"The reader's weakest assumption concerns generalization of the hand-tuned GP likelihood; that is a real but downstream issue. The more fundamental blocker is that the paper's central claim is never quantified. The strongest claim is explicitly 'successful score following', and the only evidence is the author's qualitative judgment and screen recordings. The manuscript itself states in Sec. 6.2.1 that the benchmark is subjective expert judgment, but no expert evaluation method or results are reported; Sec. 8.6 uses visual inspection; Sec. 10.1 contains no numeric accuracy. Thus the claim is currently unfalsifiable from the paper. This does not mean the approach is wrong; it means the condition for the central claim, measured accurate tracking, is unverified. A conditional acceptance remains appropriate: the proof of concept is plausible, code is available, and a quantitative rerun would settle it. I partially agree with the reader's concern because the hand-tuned emission model may fail outside its tuning data, but that failure would only matter if success were first robustly established; conversely, if quantitative evaluation succeeds, the generalization concern can be tested directly on the non-piano recordings. Verdict unchanged: CONDITIONAL.","tokens_in":33731,"tokens_out":4052,"duration_ms":44890,"concrete_test":"Run the released Score-follower code (github.com/josephinecowley/Score-follower) on the Chapter 10 test recordings, using the corresponding MIDI files as ground truth; compute the per-frame score-position error in beats (or states) for each piece, including a no-pedal piano piece, the flute/oboe/violin pieces, and the sustain-pedal case. Then run a standard DTW alignment or a spectral-feature HMM baseline on the same audio. If the GP follower's median absolute error is not below one bar (or a pre-specified threshold) and no better than the baseline, the 'successful score following' claim is not supported; if it is, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the two-stage GP+HMM system 'achieves successful score following' (abstract; Sec. 11.1). For that claim to hold, the system must actually track score position accurately. The evaluation, however, is qualitative and self-assessed: Sec. 6.2.1 declares subjective expert judgment the primary benchmark, but Sec. 8.6 describes 'visual inspection (and some manual time-mapping)' and Chapter 10 reports screen recordings with prose assessments ('impressive', 'highly accurate') rather than a structured expert evaluation or quantitative alignment error. No per-frame or per-note accuracy numbers, no confusion statistics, no latency measurements, and no baseline comparison (e.g., DTW or an HMM with standard spectral features) are reported. Section 10.1 even acknowledges the follower 'rarely positions the marker on exactly the right state/note' for the multi-instrument test and can get 'lost' on repeated notes or ornaments. Because score following is by definition a mapping with a ground truth (the score and recorded onset times), the claimed success is checkable; the paper simply does not check it. This is independent of whether the hand-tuned GP likelihood generalizes: even a perfectly general likelihood could be paired with an evaluation that does not demonstrate the claim, and a miscalibrated likelihood could still be rescued by the duration HMM. The load-bearing gap is therefore the absence of any quantitative accuracy evidence for the headline assertion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a two-stage score-following system for solo piano (and, in tests, other monophonic instruments). In Stage 1, a Gaussian Process with a Spectral Mixture covariance kernel is used to compute a log marginal likelihood (LML) for each candidate note/chord state from 800–2000-sample audioframes; hyperparameters are selected so that the LML peaks at the true fundamental frequencies. In Stage 2, these LMLs are used as emission probabilities in a left-to-right HMM with a geometric state-duration model, and an approximate online 'Windowed' Viterbi algorithm tracks the most likely score position. The paper reports qualitative results on six recordings (piano, flute, oboe, violin, and a cello-piano duet) and claims successful score following, including on instruments the GP model was not designed for.","tokens_in":34056,"tokens_out":5488,"duration_ms":58731,"significance":"If the central claim were supported by quantitative evidence, the paper would be a useful proof-of-concept: it applies GP/Spectral-Mixture kernels to online score following, provides a stable Cholesky-based LML implementation, and contributes a modular open-source system with a real-time renderer. The cross-instrument demonstrations are intriguing and suggest the GP model may generalize beyond its piano tuning data. However, the evaluation is entirely qualitative and self-assessed; no alignment-error metrics, latency measurements, baselines, or ablation studies are reported. The scientific claim of 'successful score following' is therefore not yet established, and the paper currently reads as a detailed engineering report rather than a validated research contribution.","major_comments":[{"comment":"The headline claim of 'successful score following' is not supported by the reported evaluation. Section 8.6 states that results were analysed by 'visual inspection (and some manual time-mapping)', and Chapter 10 presents only prose descriptions of screen recordings (e.g., 'impressive', 'highly accurate', 'primarily achieves its goal') with no quantitative alignment error, no per-note or per-frame accuracy, no latency figures, and no comparison with any baseline score follower (e.g., DTW or an HMM with standard spectral features). Since score position has a ground truth (score events versus recorded audio), objective metrics are feasible. Section 10.1 itself concedes that in the Schumann duet the follower 'rarely positions the marker on exactly the right state/note' and that repeated notes or ornaments cause it to get lost. As it stands, the central claim in the abstract and in Section 11.1 is not established.","section":"§10 and §8.6"},{"comment":"The GP emission model is tuned to the intended output. The hyperparameters (M = 9, sigma_f = 0.005, T = 0.465, v = 2.37, and the per-key inharmonicity dictionary B) were selected by inspecting LML-versus-frequency plots to ensure that the LML peaks at the true fundamental frequencies for isolated piano frames (§7.4). These LMLs are then used directly as HMM emission probabilities (§8.3). No train/test split or cross-validation is described, so the piano results may partly reflect fitting to the tuning data rather than generalization. The cross-instrument results mitigate this concern, but they are only qualitative. Please add a held-out evaluation and a sensitivity analysis (e.g., varying sigma_f and M) to show that alignment performance does not depend on precisely tuned values.","section":"§7.4"},{"comment":"The abstract highlights a 'duration-dependent HMM', but two of the most successful non-piano tests were run with state_duration disabled. Section 10 states that for Syrinx 'we disabled the state_duration mode' because the local-tempo assumption failed, and for the Schumann duet state_duration was also turned off. The paper does not specify what transition model was used in these runs, nor does it provide an ablation quantifying the contribution of the duration model when it is enabled. This makes it impossible to attribute the reported performance to the two-stage GP+HMM design rather than to a simpler HMM with a flat transition model. Please report the fallback transition model and include an ablation with and without state_duration on the same pieces.","section":"§10 (Syrinx and Schumann tests)"},{"comment":"The 'real-time' claim is not quantified. The system is described as having 'low latency and no buffering' (§11.1), but no frame processing time, end-to-end delay, or CPU usage is measured. Real-time score following requires a hard bound on per-frame computation; the qualitative statement that the algorithm 'runs in real-time, at least for short recordings' (§8.6) is insufficient. Please report measured per-frame inference time (e.g., versus the 800-sample frame length and hop length) under the actual multiprocessing architecture, including worst-case behaviour.","section":"§9.3 and §11.1"}],"minor_comments":[{"comment":"The algorithm adds transition probabilities T^d_{k,k} and T^d_{k-1,k} directly to log-likelihoods LML(yn|sn = k). If these T^d values are probabilities, they should be log-transformed before addition; please clarify the notation.","section":"§8.5.2, Algorithm 1"},{"comment":"The spectral-envelope expression Em = 1/(1+T m^v) is ambiguous: the exponent v is introduced without a clear symbol for the harmonic index m, and the subscript m on Em is inconsistently used elsewhere. Please define all symbols explicitly in one place.","section":"§7.3.2"},{"comment":"There is a typo in the heading: 'Score Preprocesor' should be 'Score Preprocessor'.","section":"§9.3.1"},{"comment":"The statement that successful score following 'by definition' also means that GPs are capable of score alignment is not a definitional consequence; online tracking does not automatically imply that an offline alignment algorithm works. The claim should be softened or justified.","section":"§11.1"},{"comment":"Section 6.2.1 declares 'subjective evaluation of trained musicians' as the primary benchmark, but no structured expert evaluation is reported; the descriptions in Chapter 10 appear to be the author's self-assessment. Please either report a formal expert study or clearly label the results as informal demo observations.","section":"§6.2.1 and §10"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is an MEng thesis posted to arXiv. Its main contribution is a proof-of-concept for GP-based score following, and the methodological derivation is coherent. However, the central claim rests entirely on qualitative self-assessment, with no quantitative evaluation, baselines, or ablations. The hyperparameter-selection procedure in §7.4 is also not separated from the evaluation, which raises circularity concerns. These issues are fixable within the scope of the manuscript: the authors could add held-out quantitative alignment metrics, latency measurements, and ablations. The paper would then be a reasonable contribution to eess.AS. I recommend major revision rather than rejection because the core approach appears defensible and the missing evidence is obtainable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is an MEng thesis posted to arXiv, and it reads like one: carefully written, honest about limitations, but the central claim \"we have achieved successful score following\" (Sec 11.1) is supported mainly by prose descriptions of screen recordings. The stress-test note gets this right. If you are looking for a quantitative demonstration that a GP+HMM follower tracks score position accurately, this paper does not provide it. What it does provide is a genuinely new combination: a Spectral Mixture GP whose kernel is designed to model harmonic spectra with inharmonicity correction, used to produce per-frame note likelihoods that feed a duration-aware HMM with a windowed Viterbi decoder. I am not aware of prior GP-based score following, and the engineering is real: the author discusses covariance precomputation, Cholesky-based LML, multiprocessing queues, and an open-source rendering UI. The code appears to be available on GitHub, which helps reproducibility.\n\nThe paper's strengths: the GP model is physically motivated; hyperparameters like M, sigma_f, T, v are tied to observable spectral structure; the limitation sections (8.6, 10.1) openly discuss sustain-pedal failures, repeated notes, and ornamentation. That honesty is worth crediting. The author also correctly notes that score-following evaluation is not standardized, which justifies some qualitative assessment.\n\nThe soft spots are real and load-bearing. First, the evaluation is self-assessment: Section 8.6 says \"visual inspection (and some manual time-mapping)\" and Chapter 10 gives adjectives (\"impressive\", \"highly accurate\") rather than alignment-error numbers. No per-note or per-frame accuracy, no latency measurements, no baseline comparison. Second, the hyperparameters were explicitly selected in Section 7.4 to make LML peak at true fundamental frequencies. That is acceptable for a proof of concept, but it means the note-inference stage is fitted to the intended outcome, and the claim that the model generalizes across instruments is only supported anecdotally. Third, the scope is narrow: the HMM assumes the performer starts at the beginning and makes no significant deviations, and the author essentially avoids heavy sustain pedal in tests. These limitations are acknowledged, but they undercut the abstract's unqualified success claim.\n\nOverall: a promising, clearly written proof of concept that deserves to be seen by reviewers, but only after quantitative evaluation is added. I would accept it for peer review with major revision. For a reader working on GP-based MIR or score following, it's worth a look; for anyone wanting a reliable follower, not yet.","headline":"A genuine first proof-of-concept of GP-based score following, undermined by a purely qualitative evaluation that does not quantify the headline success claim.","tokens_in":34544,"tokens_out":2768,"would_cite":true,"duration_ms":29598,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Gaussian-process note-likelihood model plus a duration-aware HMM tracks a live performance through a score in real time.","keywords":["score following","Gaussian process","spectral mixture kernel","hidden Markov model","Viterbi algorithm","music information retrieval","real-time audio","pitch inference"],"falsifier":"Perform a controlled comparison of the same piano piece recorded with no pedal, moderate pedal, and heavy pedal, manually annotating the true score position for every audioframe. If per-frame accuracy is near-perfect without pedal but drops sharply under heavy pedal—as the paper's own results suggest—the claim that the GP likelihood generalizes across normal playing conditions is falsified.","tokens_in":33535,"feed_emoji":"🎵","tokens_out":10946,"duration_ms":104845,"temperature":0.7,"pith_summary":"Musical score following is the real-time mapping of a performance to a location in the score. This report argues that Gaussian processes (GPs) can provide the statistical core of that mapping: a GP with a spectral-mixture kernel scores every candidate set of notes for an audioframe, and a duration-aware hidden Markov model turns those frame scores into a running estimate of score position. The claim is that this two-stage pipeline follows simple keyboard pieces and, without retraining, solo flute, oboe, and violin recordings, including passages with heavy rubato. If true, this is a first proof of concept that GP audio models are efficient and flexible enough for online music information retrieval tasks like automatic page turning and accompaniment.","feed_headline":"GP-based follower tracks live music through a score in real time","feed_subtitle":"A spectral-mixture GP reads notes from 18 ms audioframes; a Viterbi tracker places them on the page.","key_machinery":"The load-bearing object is the spectral-mixture covariance function $$k(\\tau) = $e^{{-2\\pi^2\\sigma_f^2\\tau^2}}$\\sum_{q=1}^Q w_q \\sum_{m=1}^M E_m \\cos(2\\pi m f_q b_{m,f_q}\\tau),$$ obtained as the inverse Fourier transform of a mixture of Gaussians placed at the harmonic frequencies of each candidate note. Here $f_q$ encodes the candidate note, $E_m = 1/(1+T m^v)$ sets the harmonic envelope, $b_{m,f_q} = \\sqrt{1+B_{f_q}m^2}$ corrects inharmonicity, and $\\sigma_f$ governs the tolerated frequency spread. The log marginal likelihood of an audioframe under this kernel is computed with a Cholesky factorization and used as the HMM's emission probability. The HMM is left-to-right with a geometric state-duration model whose expected state length is estimated from MIDI durations and a moving-average tempo conversion; a windowed Viterbi search then keeps the maximum a posteriori path computable online.","core_discovery":"The central claim is that a stationary spectral-mixture Gaussian process, whose covariance function is constructed from the expected harmonic power spectrum of pitched notes, produces a sharply peaked log marginal likelihood at the true fundamental frequency of a piano audioframe, and that this likelihood can serve as the emission model of an HMM for real-time score alignment. The report demonstrates this on a monophonic keyboard piece where the follower never deviated by more than a semiquaver, on a two-part Minuet where it recovered from a missed left-hand note, on a four-part hymn arranged for keyboard, and on solo flute, oboe, and violin recordings despite heavy rubato. It also reports degradation under excessive sustain pedal, repeated notes, and heavy ornamentation, and identifies these as the main limitations.","pith_inferences":["Because the same piano-trained kernel followed flute, oboe, and violin, one can reasonably test the frame-level LML as a general pitch-observation model for instrument identification or source separation; that would be an extension beyond the paper's scope.","The failures under heavy sustain pedal and repeated notes both strain the left-to-right, no-backtracking HMM assumption, so replacing the transition model with a multi-resolution or jump-capable HMM would show whether those failures are structural or merely tuning artifacts.","The hyperparameters were chosen from LML plots on isolated piano frames, so a sensitivity sweep over those values on sustained-pedal recordings would reveal how much of the reported success depends on the particular tuning rather than on the GP framework itself."],"forward_implications":["Because the same piano-tuned likelihood followed wind and string instruments, one GP acoustic model can serve as the front end of a real-time score follower without per-instrument retraining.","Automatic page turning and computer-aided accompaniment become practical with only a MIDI score and a microphone, since the pipeline runs in real time on audioframes of about 18 ms.","The windowed Viterbi restricts the search to a sliding window, so runtime scales with window size rather than with the total length of the piece.","Rubato and local tempo changes are absorbed by the duration model's moving-average tempo conversion, but only as long as the performer substantially follows the score; ornaments and omitted notes can still break alignment."],"supporting_citations":[{"why":"Supplies the spectral-mixture kernel: a mixture of Gaussians in the frequency domain whose inverse transform gives the quasi-periodic covariance function used in every frame-level likelihood.","marker":"[53]"},{"why":"Provides the Gaussian-process definition, the log marginal likelihood expression, and the Cholesky-based stable computation used in the LML implementation.","marker":"[41]"},{"why":"Supplies the inharmonicity correction $b_{m,f_q} = \\sqrt{1+B_{f_q}m^2}$ that shifts predicted harmonic peaks to match real piano spectra.","marker":"[17]"},{"why":"Demonstrates Gaussian-process pitch estimation and missing-segment inference on polyphonic music, the immediate precursor that motivates applying GPs to score following.","marker":"[2]"},{"why":"Earlier HMM score follower using spectral features; the paper adopts the HMM-plus-emission-likelihood architecture for Stage 2.","marker":"[39]"},{"why":"Records the known difficulty of optimizing spectral-mixture kernels, which the paper uses to justify its hand-tuned hyperparameter selection.","marker":"[44]"}],"fun_headline_variants":["GP and HMM partner for real-time musical score following","Spectral-mixture GP powers real-time score follower for multiple instruments","Real-time score following from 18 ms audioframes via GP and HMM","Statistical inference keeps live music aligned to the score","GP-based score follower nails piano, flute, violin, and oboe"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The hand-tuned note-likelihood model, chosen on isolated piano frames, keeps giving reliable likelihoods for real performances on other instruments, at other tempi, and when the sustain pedal changes the sound.","fun_headline_variants_meta":{"raw":{"variants":["GP and HMM partner for real-time musical score following","Spectral-mixture GP powers real-time score follower for multiple instruments","Real-time score following from 18 ms audioframes via GP and HMM","Statistical inference keeps live music aligned to the score","GP-based score follower nails piano, flute, violin, and oboe"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000644,"raw_usage":{"total_tokens":2985,"prompt_tokens":994,"completion_tokens":1991,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":1903}},"tokens_in":610,"tokens_out":1991,"duration_ms":14392,"temperature":1.0,"reasoning_tokens":1903,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T00:26:06.614658+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Perform a controlled comparison of the same piano piece recorded with no pedal, moderate pedal, and heavy pedal, manually annotating the true score position for every audioframe. If per-frame accuracy is near-perfect without pedal but drops sharply under heavy pedal—as the paper's own results suggest—the claim that the GP likelihood generalizes across normal playing conditions is falsified.","supporting_citations":[{"cited_title":"Gaussian Process Kernels for Pattern Discovery and Extrapolation","cited_arxiv_id":"1302.4245","evidence_quote":"Supplies the spectral-mixture kernel: a mixture of Gaussians in the frequency domain whose inverse transform gives the quasi-periodic covariance function used in every frame-level likelihood."},{"cited_title":"Gaussian Processes for Machine Learning","cited_arxiv_id":null,"evidence_quote":"Provides the Gaussian-process definition, the log marginal likelihood expression, and the Cholesky-based stable computation used in the LML implementation."},{"cited_title":"Bayesian analysis of polyphonic western tonal music","cited_arxiv_id":null,"evidence_quote":"Supplies the inharmonicity correction $b_{m,f_q} = \\sqrt{1+B_{f_q}m^2}$ that shifts predicted harmonic peaks to match real piano spectra."},{"cited_title":"Gaussian processes for music audio modelling and content analysis","cited_arxiv_id":null,"evidence_quote":"Demonstrates Gaussian-process pitch estimation and missing-segment inference on polyphonic music, the immediate precursor that motivates applying GPs to score following."},{"cited_title":"Automatic segmentation of acoustic musical signals using hidden Markov models","cited_arxiv_id":null,"evidence_quote":"Earlier HMM score follower using spectral features; the paper adopts the HMM-plus-emission-likelihood architecture for Stage 2."}],"review_version":1}