{"id":"58c0507f-fc8a-4533-8948-686c6a0e666b","arxiv_id":"2412.08161","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Co-Prop uses LLM-generated audio control points to split videos into consistent sound segments and propagates keyframe masks frame-by-frame with audio inserted, improving audio-visual segmentation scores.","lead":"This paper introduces Co-Prop, a two-stage audio-visual segmentation method that uses a large language model to detect when the sound source in a video changes, then propagates segmentation masks within each resulting time segment. The authors report consistent benchmark gains, especially on videos with multiple sound sources, and position the method as a plug-in for existing models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim leans on LLM control-point accuracy that is never measured; without a direct test, the reported gains cannot be attributed to temporal alignment.","rationale":"I read the paper in good faith. The two-stage design is coherent, and the ablations in Table 2 and Table 4 show that each module contributes relative to the chosen baselines. The consistent improvements on M3 and AVSS are also a positive sign. However, the decisive assumption is that the RCPG's LLM-generated control-point list actually marks the true transition frames where the sounding-object set changes. The manuscript validates this only indirectly: through aggregate MJ/MF and through a 17-video MOC alignment rate, which is small, self-selected, and reported without confidence intervals or error bars. The internal inconsistency between the length-5 reference lists and the length-10 output requested in Sec. 3.2/Fig. 2 further weakens confidence that the control-point signal is well-defined at the frame level. Table 6's noticeably lower keyframe performance (59.82 MJ on M3 key frames vs 65.19 MJ on normal frames) is consistent with the keyframe processor struggling at exactly the frames the RCPG selects, but the paper does not analyze transition-frame accuracy directly. Because a wrong boundary invalidates the premise of the propagation stage, while the model could still improve scores through AOT-Large pretraining and better contour propagation, the central temporal-alignment claim is not yet secured. The proposed concrete test, comparing LLM control points against oracle control points derived from GT masks and re-running the pipeline under both conditions, would settle whether the concern lands. I therefore maintain the reader's CONDITIONAL verdict; the needed conditions are direct control-point validation, release of code/benchmark, and variance reporting, as the reader states.","tokens_in":13655,"tokens_out":8157,"duration_ms":91130,"concrete_test":"Evaluate control-point accuracy directly: run RCPG on the M3 test set; derive oracle control points from GT masks using the same rule as Appendix A.1; report precision/recall and temporal IoU at tolerances of 0, 1, and 2 frames, plus the per-transition boundary error (predicted boundary frame minus GT boundary frame). Then run Co-Prop under three conditions: (a) LLM control points, (b) oracle control points from GT masks, and (c) no/random control points. If (a) is close to (b), the RCPG is validated and the concern is resolved; if (a) is close to (c) or clearly worse than (b), the reported gains arise from the propagation/backbone rather than from correct temporal alignment, and the central claim would need to be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The framework's central mechanism is Eq. (5): an LLM converts the audio into a binary control-point list that splits the video into sub-clips assumed to have a constant sounding-object set. If those boundaries are wrong, every downstream keyframe and propagation step operates on misaligned sub-clips, so the method's segmentation gains cannot be attributed to temporal alignment. The paper never measures control-point accuracy; it only reports aggregate MJ/MF and a self-built 17-video MOC alignment rate. This matters because (i) the training-set control points used for retrieval are derived from GT masks (Appendix A.1), so the reference examples encode oracle boundaries, while at inference the LLM sees only audio; (ii) Sec. 3.2's prompt requests a fixed 'ten frames' list even though the reference examples in Fig. 2 have length 5, and no mapping to the actual video frame count T is specified; and (iii) Table 6 shows key frames underperform normal frames (59.82 vs 65.19 MJ on M3), which is consistent with weak boundary anchoring but is not analyzed at transition frames. If the LLM mis-anchors, the method can still improve MJ/MF through AOT-Large pretraining and stronger contour propagation, so the headline 'temporal alignment' claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Audio-visual video segmentation (AVVS) pairs a video with its audio track to produce pixel-level masks of sound-producing objects. This paper argues that existing AVVS models suffer from temporal misalignment—they keep segmenting an object after it stops making sound—because they neglect the boundaries at which the sounding object changes. The proposed Co-Prop framework operates in two stages: (1) a Retrieval-Augmented Control Points Generation (RCPG) module uses the Qwen LLM, prompted in several steps with retrieved reference examples derived from training-set annotations, to output a binary control-point list marking frames at which the sounding object changes; the video is thereby split into sub-clips with consistent sounding objects; and (2) an Audio-Insert Propagation (AIP) module processes each sub-clip, combining a Keyframe Processor (an audio-guided single-frame segmentation model fine-tuned on a keyframe subset) with a frame-by-frame propagator built on AOT that embeds current-frame audio features (Eq. 6). The authors also curate a 17-video MOC test subset from the M3 test set and introduce an Alignment Rate metric. Reported results on M3, S4, and AVSS with ResNet-50 and PVT-v2 backbones show consistent improvements over prior methods (e.g., M3 MJ 63.58 vs. AVSegFormer 58.36 with PVT-v2), and plug-and-play integration with TPAVI, CATR, and AVSegFormer is validated.","tokens_in":13882,"tokens_out":16218,"duration_ms":151025,"significance":"Conditional on the results holding, this is a solid and useful contribution to AVVS. The gains on the multi-source benchmarks (M3, AVSS) are substantial and consistent across two backbones, the ablations in Tables 2 and 5 support the incremental value of each design choice, and the plug-and-play integration results in Fig. 4 suggest the module can be grafted onto existing AVVS models. The inclusion of a negative result (Table 4, text-guided propagation degrades performance) is a good-faith check on the design. The paper also ships testable artifacts: a publicly released MOC evaluation subset and a promise of code release that will let others reproduce the AOT-based propagation. The principal weakness is that the headline attribution—improved temporal alignment via LLM control points—is supported only indirectly: control-point accuracy is never measured, the alignment metric is informal, and the MOC set is small and hand-selected. I agree with the stress-test note that the circularity concern does not land at the level of the equations: final numbers are obtained on held-out test splits and no prediction reduces to a fitted value.","major_comments":[{"comment":"The conversion from the LLM's fixed-length output to a per-frame control-point list is never specified, and the prompt granularity is internally inconsistent. The §3.2 prompt asks Qwen to divide the audio into ten frames, the three reference examples in Fig. 2 are length-5 lists, and the 1-Step ablation prompt in §4.3 instructs 'divide the audio into five frames,' while Eq. (5) indexes the output over the full T-frame video. Since these control points determine the keyframe set and hence the entire sub-clip decomposition, the authors need to state (i) the actual frame counts T of the M3/S4/AVSS videos, (ii) how a fixed-length list is resampled or mapped onto T frames, and (iii) how sensitive the Table 2 results are to the chosen granularity and list length. Without this, Eq. (5) does not define a reproducible procedure.","section":"§3.2, Eq. (5); Fig. 2"},{"comment":"Control-point accuracy is never measured, so the reported gains cannot yet be attributed to temporal alignment. Appendix A.1 already defines an algorithmic derivation of control points from GT masks; applying that same procedure to the test splits and reporting precision/recall of the LLM's c_i would directly test the anchoring claim. This matters because the reference examples retrieved in Eq. (4) encode oracle boundaries derived from training-set GT masks, while at inference the LLM sees only audio; this is not circular in the equations, but it is a real gap. The indirect evidence is mixed: Table 6 reports 59.82 MJ on key frames versus 65.19 MJ on normal frames for M3, which is consistent with weak boundary anchoring, yet no analysis at transition frames is given, and the Limitations paragraph concedes reliance on the Keyframe Processor without addressing this deficit.","section":"§4.3, Table 2; Appendix A.3, Table 6; Limitations"},{"comment":"The provenance of the control-point annotations used to build the keyframe fine-tuning subset is described three different ways: §3.3 states the keyframe dataset was annotated by applying the RCPG method to the training set, while Appendix A.1 and the §4.3 ablation describe control-point lists derived algorithmically from GT masks, and §3.2 refers to manual annotation. These are different procedures with different error properties: training the Keyframe Processor on GT-derived keyframes while deploying it on LLM-derived keyframes creates a train/inference distribution shift, whereas training on LLM-derived keyframes risks reinforcing LLM anchoring errors. The authors should specify the exact procedure used and analyze its implications for the keyframe results in Table 6.","section":"§3.3 vs. Appendix A.1 vs. §4.3"},{"comment":"The temporal-alignment claim rests on the MOC set and the Alignment Rate metric, both of which need stronger support. MOC contains 17 hand-selected videos from the 64-video M3 test set; the selection protocol should be stated precisely, and the alignment-rate gaps in Fig. 3 (e.g., 77.65 for Co-Prop vs. 71.76 for AVSegFormer) need error bars or significance tests, as do the headline numbers in Table 1, which are reported from single runs. The Alignment Rate definition—'the proportion of predicted video frames where the identified object aligns with the ground truth'—is also informal: the paper should specify how the 'identified object' is determined for a binary M3 mask and for a semantic AVSS mask, and over which set of frames the rate is averaged.","section":"§4.2, Table 3, Fig. 3"}],"minor_comments":[{"comment":"The Conclusion contains a typo: 'Collaborative Hybrid Propgator' should read 'Propagator.'","section":"Conclusion"},{"comment":"Both panels are captioned '(a) Jaccard index on M3 dataset,' the caption text ('Pink denotes the model with Co-Prop as Keyframe Processor') is unclear, and the bar values 62.88 and 62.21 appear duplicated across panels; the figure should identify each compared method and distinguish the J and F panels.","section":"Fig. 4"},{"comment":"The indexing [ci]^T_{i=0} denotes T+1 control entries for a video stated to have T frames; the indexing should be fixed or c_0 explicitly defined.","section":"Eq. (1)"},{"comment":"The phrase 'the audio frames are evenly divided into ten frames' is ambiguous because audio is not naturally framed like video; clarify how audio features are synchronized to the video frame rate.","section":"§3.2"},{"comment":"The claimed memory reduction relative to simultaneous decoding is never quantified; a memory comparison on equal-length videos would substantiate this claim.","section":"Abstract; §4.1"},{"comment":"The retrieval implementation is underspecified: the number of reference samples per prompt and the matching criterion for 'samples of the same category' should be stated for reproducibility.","section":"Eq. (4)"},{"comment":"The cosine-similarity control-point baseline is not described: which audio features are compared, what threshold is used, and how the thresholded outputs are converted to a binary list are all unspecified, which limits the interpretability of the RCPG ablation.","section":"§4.3, Table 2(a)"},{"comment":"The alignment-rate values written on the bars (63.53, 68.23, 71.76, 77.65) are not reproduced in the text; given the 17-video MOC set, reporting the per-method rate with the number of videos in a small table would be more informative.","section":"Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this is a competent empirical paper whose main gap is evidential rather than computational: the authors can readily evaluate control-point accuracy using their own Appendix A.1 derivation on the test splits, so the requested revision is feasible within the manuscript's scope. Two context points for weighing the paper: the propagation backbone (AOT) comes from the authors' own prior work; although Table 2(a) ablates it, the absolute gains over AVSegFormer partly inherit AOT-Large pretraining, so the plug-and-play results deserve the most weight. Second, the MOC 'dataset' is only 17 videos; I would advise treating it as an evaluation subset rather than a new benchmark in publicity materials. The work fits a computer-vision/multimedia venue well; the LLM-prompting component is topical, but the paper's durable value rests on the AIP module and the controlled ablations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThis is a plausible, incremental but useful paper on audio-visual video segmentation. The core idea is to use an LLM to find the times when the sounding object changes, split the video into sub-clips, and then propagate keyframe masks through a video-object-segmentation-style propagator that inserts audio features frame-by-frame. The reported gains on M3 and AVSS are consistent and the ablations support the three main components.\n\nWhat's actually new: the specific pipeline of retrieval-augmented prompting (Qwen) for control points plus audio-informed propagation is not in the literature. The authors also show plug-and-play integration with three existing AVVS methods, which is useful. The MOC dataset, though small, is a reasonable attempt to measure alignment directly.\n\nThe soft spots are real but not disqualifying. The central load-bearing assumption is that the LLM control points are correct, but the paper never measures control-point accuracy directly. You only see downstream MJ/MF and the alignment rate on a 17-video test set, which is too small to be persuasive. The retrieval examples come from GT masks, so the LLM is shown perfect boundaries at inference time, while at test time it sees only audio; that distribution mismatch is never analyzed. There's also a prompt inconsistency: the instructions ask for ten frames but the reference examples in Fig. 2 have length five, and no mapping to the actual video length is given. Table 6 shows keyframes (the frames the control points mark) get worse segmentation than normal frames, which is consistent with weak boundary anchoring—though the paper attributes this to the keyframe processor being less mature.\n\nThe other weakness is that the gains over the cosine-similarity baseline are modest (about 1.6 MJ on M3), and the cosine baseline is a poor boundary detector. So the RCPG is helpful, but we don't know how well it finds true boundaries, and no error bars are reported.\n\nThat said, the paper is transparent about its limitations, and the method is a sensible modular improvement. With code, the MOC benchmark, and a direct evaluation of control-point accuracy (e.g., compare against GT transition times), I'd be reasonably convinced. It deserves a serious referee, and I'd send it to review with the request for those additions.\n\nRecommendation: send to peer review. It's not groundbreaking, but it's a solid contribution to a subfield, and the missing evidence is easy to supply.","headline":"Solid engineering contribution for audio-visual segmentation with a real soft spot: the LLM control-point mechanism is never directly validated.","tokens_in":14425,"tokens_out":2854,"would_cite":false,"duration_ms":28161,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that temporal misalignment in audio-visual segmentation can be fixed by first anchoring audio change points with an LLM and then propagating keyframe masks with audio inserted frame by frame.","keywords":["audio-visual segmentation","temporal alignment","control point generation","large language model prompting","retrieval-augmented prompting","video object segmentation","mask propagation","audio-guided segmentation"],"falsifier":"On a held-out set of videos where audio semantics change while the visible mask stays the same (e.g., one person switches from singing to speaking), compare RCPG's control points against manual audio-event annotations; if the control points miss these boundaries, Co-Prop's alignment gains cannot be attributed to boundary anchoring and would not transfer to such cases.","tokens_in":13437,"feed_emoji":"🔊","tokens_out":8691,"duration_ms":75850,"temperature":0.7,"pith_summary":"Audio-visual video segmentation aims to produce a pixel-level mask of the object currently making the sound in each frame. The paper argues that prior methods use audio mainly to identify what object is sounding but ignore when the sounding object changes, so predictions lag behind audio transitions, e.g., they keep segmenting a singer after she stops singing. The proposed Co-Prop framework addresses this in two stages: it first asks a large language model, prompted with retrieved examples, to mark the control points in the audio where the sounding object changes, then performs video segmentation separately on each resulting sub-clip by propagating keyframe masks with audio inserted frame by frame. Co-Prop reports improved Jaccard and F-scores over prior baselines on the M3, AVSS, and S4 benchmarks and higher alignment rates on a new MOC test set of videos with object conversions. If these results hold, the implication is that explicitly locating audio boundaries is a cheap and effective fix for temporal misalignment in audio-visual segmentation.","feed_headline":"LLM audio boundaries fix misaligned video segmentation","feed_subtitle":"Two-stage pipeline marks when the sounding object changes, then propagates keyframe masks with per-frame audio.","key_machinery":"The load-bearing machinery is the split of the pipeline into two coupled modules. The Retrieval-augmented Control Points Generation Module (RCPG) converts training ground-truth masks into control-point lists that serve as reference examples, retrieves examples of the same audio category, and prompts a large language model in multiple steps to output a binary control-point list for a new audio clip. The Audio-insert Propagator (AIP) then treats each sub-clip separately: a Keyframe Processor (an audio-guided image segmenter, fine-tuned on keyframe data) predicts the mask for the sub-clip's first frame, and a frame-by-frame propagator extends that mask while inserting the current audio features into the video features through a cross-attention block at every propagation step. The control point list is the interface between the two stages: its accuracy determines whether each sub-clip truly contains one consistent sounding object, and the per-frame audio insertion determines whether the propagation stays aligned to the sound as the mask moves.","core_discovery":"Co-Prop's central claim is that audio-visual segmentation should not fuse audio with all video frames at once, because that discards the times at which the sounding object changes. Instead, the framework first performs Preliminary Audio Boundary Anchoring: a retrieval-augmented module prompts a pretrained audio-capable large language model to mark control points — frames where the category, timbre, or quantity of the sound changes — producing a binary list that splits the audio into sub-clips with a consistent target object. Then Frame-by-Frame Audio-Insert Propagation segments each sub-clip: a Keyframe Processor produces masks for the first frame of every sub-clip, and an Audio-insert Propagator propagates those masks through the remaining frames while embedding the current frame's audio features via cross-attention at each step. On M3 and AVSS, the multi-source benchmarks where temporal misalignment is most severe, the paper reports the largest gains (M3: 63.58% MJ / 73.96% MF versus 58.36% / 69.3% for the strongest compared baseline with the same backbone; AVSS: 39.56% / 44.37% versus 37.3% / 42.8%), and it introduces a 17-video MOC test set plus an alignment-rate metric to measure synchronization directly.","pith_inferences":["Our inference: RCPG's reference lists are built from ground-truth mask changes, so the LLM is in effect taught to find 'mask-visible object switches.' On videos where the sound changes but the visible mask does not (one person singing then speaking), control points may be missed; a dedicated audio-event-boundary benchmark would reveal this.","Our inference: the same boundary-anchoring idea could transfer to other audio-visual grounding tasks — event localization, video captioning, or dialogue-scene segmentation — whenever a pretrained audio-capable LLM is available, though the paper does not test those tasks.","Our inference: the reported alignment-rate gain on MOC could be decomposed further by computing control-point precision/recall directly against manual audio-event annotations, which would separate the contribution of boundary anchoring from the contribution of propagation."],"forward_implications":["If control points are reliable, each sub-clip contains a single consistent sounding object, so the propagation stage never has to carry a mask across an object change — the exact situation where existing methods keep segmenting a stopped sound source.","Decoding keyframes only and propagating normal frames cuts memory compared with decoding every frame simultaneously, which the paper argues makes long-video processing more practical.","Because the Keyframe Processor is a pluggable component, existing audio-visual segmenters can be slotted into Co-Prop and gain from the audio-insert propagation stage; the paper demonstrates this by attaching Co-Prop to several prior models and observing improved scores.","On the authors' alignment-rate metric over the MOC videos, Co-Prop reports higher rates of frames whose predicted object matches ground truth, which is the direct measurement of temporal alignment the paper introduces."],"supporting_citations":[{"why":"Supplies the M3 and S4 benchmarks; its ground-truth masks are converted into the control-point reference lists that RCPG retrieves.","marker":"Zhou et al. 2022"},{"why":"Supplies the AVSS semantic-segmentation benchmark used for the main multi-source evaluation.","marker":"Zhou et al. 2023"},{"why":"Provides the AVSegFormer baseline that Co-Prop compares against and also wraps as a plug-and-play Keyframe Processor.","marker":"Gao et al. 2024"},{"why":"Supplies the audio-queried transformer decoding that the Keyframe Processor uses to turn fused audio-video features into keyframe masks.","marker":"Li et al. 2023"},{"why":"Supplies the video object segmentation propagator with identity-based association that the Audio-insert Propagator extends with audio insertion.","marker":"Yang et al. 2021"},{"why":"Represents the memory-based propagation line that previous methods rely on with no audio guidance, the limitation Co-Prop targets.","marker":"Oh et al. 2019"},{"why":"Supplies the cross-attention formulation used inside the Audio-Insert Block to embed audio features into video features.","marker":"Chen et al. 2021"},{"why":"Supplies the audio feature extractor used to encode audio for both stages.","marker":"Hershey et al. 2017"}],"fun_headline_variants":["LLM audio boundaries fix misaligned video segmentation","Audio boundary LLM anchors robust video mask propagation","Find audio change points to align video segmentation","Two-stage audio-insert propagation fixes time misalignment","LLM marks sound shifts, then propagates keyframe masks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline rests on the premise that the LLM-generated control-point list correctly identifies when the sounding object changes; if those boundaries are wrong, the sub-audio clips are cut in the wrong places and the downstream propagation cannot repair the misalignment.","fun_headline_variants_meta":{"raw":{"variants":["LLM audio boundaries fix misaligned video segmentation","Audio boundary LLM anchors robust video mask propagation","Find audio change points to align video segmentation","Two-stage audio-insert propagation fixes time misalignment","LLM marks sound shifts, then propagates keyframe masks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1586,"prompt_tokens":1072,"completion_tokens":514,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":439}},"tokens_in":688,"tokens_out":514,"duration_ms":5570,"temperature":1.0,"reasoning_tokens":439,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:08:15.441114+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set of videos where audio semantics change while the visible mask stays the same (e.g., one person switches from singing to speaking), compare RCPG's control points against manual audio-event annotations; if the control points miss these boundaries, Co-Prop's alignment gains cannot be attributed to boundary anchoring and would not transfer to such cases.","supporting_citations":[{"cited_title":"Audio--visual segmentation","cited_arxiv_id":null,"evidence_quote":"Supplies the M3 and S4 benchmarks; its ground-truth masks are converted into the control-point reference lists that RCPG retrieves."},{"cited_title":"Avsegformer: Audio-visual segmentation with transformer","cited_arxiv_id":null,"evidence_quote":"Provides the AVSegFormer baseline that Co-Prop compares against and also wraps as a plug-and-play Keyframe Processor."},{"cited_title":"Associating objects with transformers for video object segmentation","cited_arxiv_id":null,"evidence_quote":"Supplies the video object segmentation propagator with identity-based association that the Audio-insert Propagator extends with audio insertion."},{"cited_title":"Cnn architectures for large-scale audio classification","cited_arxiv_id":null,"evidence_quote":"Supplies the audio feature extractor used to encode audio for both stages."}],"review_version":1}