{"id":"ecefe8ed-3e6a-4aaf-b7a8-7014548d1900","arxiv_id":"2505.05335","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FLAM trains an audio-language model with a frame-level contrastive goal and per-text logit adjustment, enabling open-vocabulary temporal localization of sound events while preserving global retrieval.","lead":"FLAM extends audio-language contrastive models with a frame-level training objective, so a text query can pinpoint when a sound occurs instead of only retrieving the whole clip. The method uses a one-million-sample synthetic dataset and per-text logit adjustment to keep detection calibrated across common and rare events.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix C.2 Eq. 17 silently equates pdata(x,l|y,z=1)/pdata(x,l|y,z=-1) with pdata(x,l|y)/pdata(x,l); this is false under the §4.2 generative model, so the unbiased-classifier derivation is unsupported.","rationale":"Reader and I identify the same soft spot: Eq. 17's unstated likelihood-ratio equality is the linchpin connecting the trained logit h* to the robust classifier. It is not a cosmetic issue: Eq. 8, Eq. 9, and the claim that the learned dot product approximates log(pdata(y|x,l)/pdata(y)) all consume this equality. The equality is demonstrably false for the synthetic data-generation process in §4.2, where z labels event presence and therefore changes the conditional distribution of the frame. I considered whether other concerns are more load-bearing. The closed-set SED evaluation is weakened by the lack of explicit train/test split separation for datasets also used in training, and the MGA-CLAP baseline is trained for fewer steps than FLAM, so the head-to-head margins are not perfectly compute-matched. These are reporting and control issues. However, even if both were fixed, the open-vocabulary localization claim rests on the frame-wise objective and its debiasing; an unsupported theoretical calibration step is the more fundamental risk. That said, the empirical AUROC gains on held-out and ASFX-SED are plausible and may survive the fix because AUROC is threshold-invariant and the frame-wise supervision itself provides signal. The conditional verdict is therefore appropriate; it should require either a corrected derivation or an explicit statement of the additional assumption, plus an empirical check of the exact inference rule. No verdict change.","tokens_in":19926,"tokens_out":12265,"duration_ms":119680,"concrete_test":"Independently re-derive Eq. 17 from the binary log-odds identity and check the disputed equality against the generative model in §4.2. A minimal discrete counterexample suffices: take y='dog barks', let z=1 mean the dog-bark event is active at frame l and z=-1 mean it is absent; then pdata(x,l|y,z=1) and pdata(x,l|y,z=-1) are concentrated on different acoustic states, so their ratio is not equal to pdata(x,l|y)/pdata(x,l). If the equality fails, keep training fixed and rerun ASFX-SED and Held-out with the exact robust score s(x,l,y)=sigma(h*)/(sigma(h*)+sigma(beta*)) from Eq. 22 instead of the Eq. 8 approximation; report AUROC and F1 to determine whether the unsupported approximation changes the headline localization numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The derivation of the robust logit in Appendix C.2 (Eq. 17) contains an unflagged replacement: it moves from log[pdata(x,l|y,z=1)/pdata(x,l|y,z=-1)] + log[pdata(z=1|y)/pdata(z=-1|y)] to log[pdata(x,l|y)/pdata(x,l)] + beta*(y). The first ratio is not equal to pdata(x,l|y)/pdata(x,l) in general. In the §4.2 generative process, z=1 means the event described by y is audible in frame (x,l), so the positive-frame and negative-frame conditional distributions are precisely what the detector must separate; their ratio is the event's per-frame detectability. The right-hand ratio pdata(x,l|y)/pdata(x,l) marginalizes over z and carries no information about the event-presence label. The equality holds only under a strong conditional-independence or balance condition that is neither stated nor implied by the augmentation pipeline. Since Eq. 9 and the practical approximation in Eq. 8 both rely on this step, the theoretical guarantee that alpha_t(y) ea,loc(x)_l · et(y) learns log pdata(y|x,l)/pdata(y) is not established. AUROC comparisons may be insensitive to this because AUROC is rank-based, but the paper's calibrated-inference and unbiased-classifier claims are directly affected.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FLAM, a contrastive audio-language model that augments a standard CLIP-style global objective with a frame-wise binary classification objective for open-vocabulary sound event detection. The model outputs both a global audio embedding and a temporal sequence of frame embeddings; each frame-text pair is scored by a logit that includes a per-text scale and a per-text bias, trained with a logit-adjustment motivation to handle label imbalance. The model is trained on a large synthetic dataset of 1M mixtures constructed by inserting text-labeled sound events into background audio, together with closed-set SED datasets. The paper reports large gains over a CLAP-style baseline and a retrained MGA-CLAP on held-out synthetic and external benchmarks, while showing comparable performance on retrieval and zero-shot classification, and presents a theoretical appendix claiming that the per-text bias yields a Bayes-optimal unbiased classifier.","tokens_in":20275,"tokens_out":9398,"duration_ms":87261,"significance":"If the empirical results are valid, FLAM is a useful contribution to open-vocabulary audio event detection: it demonstrates that explicit frame-level supervision, together with per-text logit scaling and biasing, can give substantial gains in temporal localization over clip-level contrastive models, and the synthetic data pipeline plus the released ASFX-SED benchmark are valuable resources. The held-out synthetic evaluations (Held-out, ASFX-SED) are well designed and provide credible evidence for the core localization claim. The ablations on global loss, initialization, and temporal granularity are informative. However, the paper's theoretical claims about calibration and unbiasedness rest on an unjustified equality in Appendix C.2, and the closed-set evaluation protocol is not documented; these issues must be resolved before the significance of the claims can be fully assessed.","major_comments":[{"comment":"The derivation silently replaces log[pdata(x,l|y,z=1)/pdata(x,l|y,z=-1)] with log[pdata(x,l|y)/pdata(x,l)] in the expression for pdata(z=1|x,l,y). This equality is not a consequence of Bayes rule and is generally false under the generative model of §4.2: the left-hand ratio is the per-frame likelihood ratio for event presence given the text y, while the right-hand ratio marginalizes over z and loses the event-presence information. Since Eq. (9), the approximation in Eq. (8), and the 'unbiased classifier' in Eqs. (7) and (20)-(22) all depend on this step, the claimed Bayes-optimality and calibration guarantees are not established. Please provide a correct derivation, or state and justify an explicit assumption, noting that the natural conditional-independence assumption pdata(x,l|y,z)=pdata(x,l|y) contradicts the detection task itself; alternatively, present the logit adjustment as a heuristic and support it with the empirical F1 results rather than a theoretical proof.","section":"Appendix C.2, Eq. (17)"},{"comment":"The paper states that FLAM is trained on AudioSet-Strong, DESED, and UrbanSED (Section 5.1, Appendix C.6), but Table 1 reports SED results on these same datasets without specifying the evaluation split. If the model was trained on the same recordings used for evaluation, the closed-set AUROC/PSDS numbers are in-sample and do not support the abstract's claim of outperforming prior models on closed-set SED. Please specify the exact train/validation/test partitions used for each dataset and confirm that no evaluation clips were included in training. This is essential for interpreting the closed-set results in Table 1 and the ablation in Table 4.","section":"Section 5.1 / 5.2, Table 1"},{"comment":"The claim that FLAM produces 'accurate and calibrated probabilities' and that the unbiased classifier is 'calibrated' is not supported by quantitative calibration metrics. The only quantitative evidence is the F1/precision/recall curve in Figure 3 and qualitative examples in Figures 2 and 4-9. Since the theoretical calibration argument in Appendix C.2/Eq. (17) is not established, the empirical calibration claim needs direct evidence. Please include reliability diagrams or expected calibration error on held-out data, or revise the wording to 'well-behaved scores' rather than 'calibrated probabilities'.","section":"Section 5.3"}],"minor_comments":[{"comment":"The table formatting is broken (e.g., '9.3791.66' and '29.52 93.62' run together), and the bold-number convention is unclear. In the DESED PSDS column, MGA-CLAP (reported) has 26.4, which is higher than FLAM's 9.37, so the table should not imply FLAM is best on every closed-set metric; the text acknowledges the DESED exception, but the table should be made consistent.","section":"Table 1"},{"comment":"Please state explicitly that the Clotho and AudioCaps training uses only the official training splits and that evaluation in Table 2 is on the official test splits. The current wording 'We sample data from our (1.1M) dataset, AudioCaps, and Clotho' leaves open the possibility of train/test overlap.","section":"Appendix C.5 / C.6"},{"comment":"The notation '∝_z' is confusing. Please define it as 'proportional as a function of z' or replace it with a clearer equivalence relation to avoid ambiguity.","section":"Appendix C.3, Eq. (19)"},{"comment":"The ring-based chunked loss computation is described at a high level. Please provide pseudocode or a precise description of how masks are used and how the per-GPU loss terms are accumulated, so that the memory-efficient strategy is reproducible.","section":"Section 3.3"},{"comment":"The term 'unbiased classifier' is potentially misleading in light of the derivation gap. Consider renaming it to 'bias-corrected classifier' throughout the paper.","section":"Section 3.2 and Contribution List"}],"recommendation":"major_revision","confidential_remarks":"The principal concern is the unproven equality in Appendix C.2, which underlies the theoretical claim of an unbiased, calibrated classifier. I believe this is fixable either by correcting the derivation or by reframing the logit adjustment as an empirically motivated heuristic, but it must be addressed explicitly. The second issue is the missing evaluation-split documentation for the closed-set datasets; if the numbers are in-sample, the paper would require substantial re-evaluation. Otherwise, the empirical setup with held-out synthetic benchmarks is sound, and the work is within the scope of an ML conference. No concerns about novelty or citation practice beyond what is noted in the report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FLAM is worth a serious referee's time. The empirical recipe is credible: frame-level contrastive training with per-text logit scale/bias on a million synthetic mixtures gives clear gains in open-vocabulary temporal localization over MGA-CLAP retrained on the same data. The synthetic pipeline with precise boundaries is a practical contribution, and the memory-efficient ring-based implementation is sensible engineering.\n\nThe novel part is explicit frame-level supervision with open-vocabulary text queries, unlike MGA-CLAP's sample-level labels or Text-Audio Grounding's fixed ontology. The per-text logit adjustment is a straightforward but effective adaptation.\n\nThe main soft spot is the theory in Appendix C.2. Equation 17 replaces log p(x,l|y,z=1)/p(x,l|y,z=-1) with log p(x,l|y)/p(x,l). That equality is not generally true; it amounts to a conditional-independence assumption that is neither stated nor derived. The claimed unbiased classifier and the approximation in Eq. 8 rest on this step. The empirical ablations show the bias/scale help, so the method may work without that theory, but the theoretical guarantee as written does not hold up. AUROC numbers are largely rank-based and probably unaffected.\n\nSecond, Table 1 reports closed-set SED results on DESED, AudioSet-Strong, and UrbanSED, which the paper says are used in training. The text does not clearly state that the evaluation splits are disjoint from training splits. That needs to be explicit.\n\nThird, reproducibility: no code, no weights, and the 1M-mixture training set is not released. The ASFX-SED test set is public, which helps, but without the pipeline the comparison is hard to reproduce.\n\nThe comparison to MGA-CLAP* trained on the same data is fair. Reported numbers from prior studies use different training sets, so the 'state-of-the-art' claim is only valid within this controlled setting.\n\nWho gets value: anyone working on audio-text retrieval, sound event detection, or multimodal grounding. The paper deserves peer review, but the authors need to fix the derivation or clearly state its assumptions, separate train/test for the closed-set datasets, and ideally release code and the synthetic pipeline. If the derivation holds after a proper justification or the empirical results are shown to be insensitive to it, this could be a useful addition to the literature.","headline":"FLAM presents a credible empirical advance in open-vocabulary frame-wise audio-text alignment, but the calibration theory in Appendix C.2 relies on an unjustified conditional-independence step and the closed-set evaluation overlaps with training data.","tokens_in":20803,"tokens_out":5929,"would_cite":true,"duration_ms":50654,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FLAM casts open-vocabulary sound event detection as frame-wise contrastive classification with event-dependent logit adjustment, and reports large AUROC gains over clip-level baselines while preserving retrieval.","keywords":["open-vocabulary sound event detection","frame-wise contrastive learning","audio-language model","logit adjustment","temporal localization","synthetic audio mixtures","text-audio retrieval","audio grounding"],"falsifier":"On the actual training mixtures, compute the ratio $p(x,l\\mid y,z=1)/p(x,l\\mid y,z=-1)$ and compare it with $p(x,l\\mid y)/p(x,l)$ for several event types; if the two diverge systematically, the robust-inference derivation in Appendix C.2 breaks. A direct behavioral test would construct a dataset pairing rare short events with frequent long events and check whether the model's 0.5 decision boundary still yields equal detection rates for both event types.","tokens_in":19730,"feed_emoji":"🎧","tokens_out":5843,"duration_ms":55740,"temperature":0.7,"pith_summary":"FLAM is an audio-language model trained to answer not just what sound is present but when in a clip a described sound occurs. The paper argues that existing contrastive audio-language models align whole clips to captions and therefore cannot pinpoint events, while traditional sound event detectors are locked to a fixed vocabulary. FLAM adds a frame-level contrastive objective that matches each audio frame to text embeddings, with event-dependent logit scales and biases that counter label imbalance and spurious event correlations. To get training labels, the authors synthesize a million 10-second mixtures of labeled events over background audio. If the central claim is right, the result is an open-vocabulary sound event detector with markedly better frame-level AUROC than clip-level baselines, without giving up global retrieval or zero-shot classification.","feed_headline":"Localizes arbitrary sound events in time: FLAM scores 91.0 AUROC","feed_subtitle":"A frame-wise audio-language contrastive model beats clip-level baselines at detection and keeps retrieval strong.","key_machinery":"The load-bearing object is the frame-wise logit $h(x,l,y)=\\alpha_t(y)\\,e^{\\mathrm{a,loc}}(x)_l \\cdot e^t(y)+\\beta_t(y)$, trained with a binary cross-entropy objective over all frame-text pairs in a batch. The per-text scale $\\alpha_t(y)$ and per-text bias $\\beta_t(y)$ come from lightweight MLPs on the text encoder, and the bias MLP is trained on the average per-prompt frame label to approximate the true log-prior ratio. At inference the unbiased score $s\\approx\\sigma(\\log(p(y\\mid x,l)/p(y)))$ converts raw frame-text similarity into calibrated detection probabilities. The machinery also includes a chunked ring-based training strategy that computes pairwise frame-text losses across GPUs without collecting all embeddings on one device.","core_discovery":"The central claim is that open-vocabulary sound event detection can be cast as a per-frame binary contrastive classification problem: for each frame embedding and text query, the model predicts whether the event described by the text is audible at that frame, using the logit $h(x,l,y)=\\alpha_t(y)\\,e^{\\mathrm{a,loc}}(x)_l \\cdot e^t(y)+\\beta_t(y)$. With a text-dependent logit bias trained by an auxiliary classifier to approximate $\\log[p(z=1\\mid y)/p(z=-1\\mid y)]$, and an unbiased inference score $s=p(z=1\\mid x,l,y)/(p(z=1\\mid x,l,y)+p(z=1\\mid y))\\approx\\sigma(\\log(p(y\\mid x,l)/p(y)))$, the paper argues that the model learns calibrated probabilities rather than dataset priors. On a held-out synthetic test set with unseen events it reports AUROC 91.0 versus 74.17 for a retrained clip-level baseline, and on ASFX-SED 81.23 versus 69.56, with retrieval and zero-shot classification approximately preserved.","pith_inferences":["The same recipe of frame-wise binary contrastive loss plus per-category logit adjustment and an unbiased score could transfer to other fine-grained grounding tasks, such as video-text frame grounding or image patch-text segmentation, where positive patches are also rare and imbalanced.","The approximation $s\\approx\\sigma(\\log(p(y\\mid x,l)/p(y)))$ relies on the per-text bias being strongly negative; in corpora where frequent events have priors closer to balanced, the calibrated boundary may drift and should be audited per event frequency.","The synthetic mixture pipeline places events with random overlap but does not deliberately control event co-occurrence statistics; an explicit test varying co-occurrence frequency would clarify whether the logit correction truly removes spurious dependencies or only re-weights marginal label imbalance."],"forward_implications":["A user can issue an arbitrary textual query for a sound event and receive a temporal boundary for when that event occurs, not just a clip-level match.","Closed-set sound event detection benchmarks improve or stay comparable: the paper reports gains on DESED, MAESTRO, AudioSet-Strong, and UrbanSED under AUROC and PSDS metrics.","Global retrieval and zero-shot classification do not degrade materially, showing that frame-level supervision and global alignment can be trained jointly.","Ablations without the per-text bias and per-text scale show lower F1 across thresholds, indicating that logit adjustment is what makes the outputs calibrated rather than mere frame-level matching.","The fixed 10-second input window and coarse 32-frame resolution bound the method; the authors acknowledge this limits longer or more temporally nuanced recordings."],"supporting_citations":[{"why":"Supplies the logit adjustment technique for long-tailed classification that the paper adapts to binary frame-level classification.","marker":"(Menon et al., 2021)"},{"why":"Provides the group-robust classification framework used to derive the unbiased robust classifier and the per-text bias correction.","marker":"(Tsirigotis et al., 2023)"},{"why":"Supplies the SigLIP binary-contrastive objective and the memory-efficient chunked training strategy that FLAM extends to frame-level data.","marker":"(Zhai et al., 2023)"},{"why":"Provides the LAION-CLAP architecture and contrastive audio-language pretraining setup that FLAM builds on.","marker":"(Wu* et al., 2023)"},{"why":"Defines the MGA-CLAP baseline, a temporal-aware contrastive model whose performance FLAM compares against on SED and retrieval.","marker":"(Li et al., 2024)"},{"why":"Supplies the HTSAT audio encoder that produces the frame-level embeddings used by FLAM.","marker":"(Chen et al., 2022)"},{"why":"Provides the Scaper soundscape synthesis approach that the authors extend to open-vocabulary synthetic SED data at scale.","marker":"(Salamon et al., 2017)"},{"why":"Supplies the DESED dataset used both as closed-set SED training data and as an evaluation benchmark.","marker":"(Serizel et al., 2020)"},{"why":"Supplies the AudioSet-Strong dataset with temporally strong labels used in FLAM training and evaluation.","marker":"(Hershey et al., 2021)"}],"fun_headline_variants":["FLAM pinpoints any sound event, 91 AUROC","Open-vocabulary event timing: FLAM scores 91 AUROC","FLAM localizes unseen sound events, 91 AUROC","91 AUROC: FLAM localizes arbitrary audio events","Frame-wise audio model localizes events: FLAM hits 91 AUROC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The derivation of the calibrated classifier assumes that, given a text description, the distribution of audio frames under the positive and negative labels is fully captured by the overall label frequencies and the marginal frame distribution, with no extra interaction between label and frame content; if that conditional-independence step fails, subtracting the learned per-text bias will not produce the claimed unbiased probabilities.","fun_headline_variants_meta":{"raw":{"variants":["FLAM pinpoints any sound event, 91 AUROC","Open-vocabulary event timing: FLAM scores 91 AUROC","FLAM localizes unseen sound events, 91 AUROC","91 AUROC: FLAM localizes arbitrary audio events","Frame-wise audio model localizes events: FLAM hits 91 AUROC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001104,"raw_usage":{"total_tokens":4601,"prompt_tokens":943,"completion_tokens":3658,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":3568}},"tokens_in":559,"tokens_out":3658,"duration_ms":25841,"temperature":1.0,"reasoning_tokens":3568,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:07:38.329208+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the actual training mixtures, compute the ratio $p(x,l\\mid y,z=1)/p(x,l\\mid y,z=-1)$ and compare it with $p(x,l\\mid y)/p(x,l)$ for several event types; if the two diverge systematically, the robust-inference derivation in Appendix C.2 breaks. A direct behavioral test would construct a dataset pairing rare short events with frequent long events and check whether the model's 0.5 decision boundary still yields equal detection rates for both event types.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SigLIP binary-contrastive objective and the memory-efficient chunked training strategy that FLAM extends to frame-level data."},{"cited_title":"Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection","cited_arxiv_id":null,"evidence_quote":"Supplies the HTSAT audio encoder that produces the frame-level embeddings used by FLAM."},{"cited_title":"Sound event detection in synthetic domestic environments","cited_arxiv_id":null,"evidence_quote":"Supplies the DESED dataset used both as closed-set SED training data and as an evaluation benchmark."},{"cited_title":"P., Fonseca, E., Jansen, A., Liu, C., Moore, R","cited_arxiv_id":null,"evidence_quote":"Supplies the AudioSet-Strong dataset with temporally strong labels used in FLAM training and evaluation."}],"review_version":1}