Collaborative Hybrid Propagator for Temporal Misalignment in Audio-Visual Segmentation

Jun Xiao; Kexin Li; Yi Yang; Zongxin Yang

REVIEW 4 major objections 8 minor 39 references

Collaborative Hybrid Propagator for Temporal Misalignment in Audio-Visual Segmentation

T0 review · 4 major / 8 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that temporal misalignment in audio-visual segmentation can be fixed by first anchoring audio change points with an LLM and then propagating keyframe masks with audio inserted frame by frame.

desk verdict Solid engineering contribution for audio-visual segmentation with a real soft spot: the LLM control-point mechanism is never directly validated. read the letter →

arxiv 2412.08161 v1 pith:FTJV7O5A submitted 2024-12-11 cs.CV cs.LGcs.MMcs.SDeess.AS

Kexin Li , Zongxin Yang , Yi Yang , Jun Xiao This is my paper · ORCID

classification cs.CVcs.LGcs.MMcs.SDeess.AS

keywords audio-visualsegmentationtemporalalignmentcontrolpointgenerationlargelanguagemodelpromptingretrieval-augmentedvideoobjectmaskpropagationaudio-guided

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Audio-visual video segmentation aims to produce a pixel-level mask of the object currently making the sound in each frame. The paper argues that prior methods use audio mainly to identify what object is sounding but ignore when the sounding object changes, so predictions lag behind audio transitions, e.g., they keep segmenting a singer after she stops singing. The proposed Co-Prop framework addresses this in two stages: it first asks a large language model, prompted with retrieved examples, to mark the control points in the audio where the sounding object changes, then performs video segmentation separately on each resulting sub-clip by propagating keyframe masks with audio inserted frame by frame. Co-Prop reports improved Jaccard and F-scores over prior baselines on the M3, AVSS, and S4 benchmarks and higher alignment rates on a new MOC test set of videos with object conversions. If these results hold, the implication is that explicitly locating audio boundaries is a cheap and effective fix for temporal misalignment in audio-visual segmentation.

What carries the argument

The load-bearing machinery is the split of the pipeline into two coupled modules. The Retrieval-augmented Control Points Generation Module (RCPG) converts training ground-truth masks into control-point lists that serve as reference examples, retrieves examples of the same audio category, and prompts a large language model in multiple steps to output a binary control-point list for a new audio clip. The Audio-insert Propagator (AIP) then treats each sub-clip separately: a Keyframe Processor (an audio-guided image segmenter, fine-tuned on keyframe data) predicts the mask for the sub-clip's first frame, and a frame-by-frame propagator extends that mask while inserting the current audio features into the video features through a cross-attention block at every propagation step. The control point list is the interface between the two stages: its accuracy determines whether each sub-clip truly contains one consistent sounding object, and the per-frame audio insertion determines whether the propagation stays aligned to the sound as the mask moves.

What would settle it

On a held-out set of videos where audio semantics change while the visible mask stays the same (e.g., one person switches from singing to speaking), compare RCPG's control points against manual audio-event annotations; if the control points miss these boundaries, Co-Prop's alignment gains cannot be attributed to boundary anchoring and would not transfer to such cases.

Watch

Extended reading notes

Core claim

Co-Prop's central claim is that audio-visual segmentation should not fuse audio with all video frames at once, because that discards the times at which the sounding object changes. Instead, the framework first performs Preliminary Audio Boundary Anchoring: a retrieval-augmented module prompts a pretrained audio-capable large language model to mark control points — frames where the category, timbre, or quantity of the sound changes — producing a binary list that splits the audio into sub-clips with a consistent target object. Then Frame-by-Frame Audio-Insert Propagation segments each sub-clip: a Keyframe Processor produces masks for the first frame of every sub-clip, and an Audio-insert Propagator propagates those masks through the remaining frames while embedding the current frame's audio features via cross-attention at each step. On M3 and AVSS, the multi-source benchmarks where temporal misalignment is most severe, the paper reports the largest gains (M3: 63.58% MJ / 73.96% MF versus 58.36% / 69.3% for the strongest compared baseline with the same backbone; AVSS: 39.56% / 44.37% versus 37.3% / 42.8%), and it introduces a 17-video MOC test set plus an alignment-rate metric to measure synchronization directly.

Load-bearing premise

The pipeline rests on the premise that the LLM-generated control-point list correctly identifies when the sounding object changes; if those boundaries are wrong, the sub-audio clips are cut in the wrong places and the downstream propagation cannot repair the misalignment.

Editorial extensions

If this is right

If control points are reliable, each sub-clip contains a single consistent sounding object, so the propagation stage never has to carry a mask across an object change — the exact situation where existing methods keep segmenting a stopped sound source.
Decoding keyframes only and propagating normal frames cuts memory compared with decoding every frame simultaneously, which the paper argues makes long-video processing more practical.
Because the Keyframe Processor is a pluggable component, existing audio-visual segmenters can be slotted into Co-Prop and gain from the audio-insert propagation stage; the paper demonstrates this by attaching Co-Prop to several prior models and observing improved scores.
On the authors' alignment-rate metric over the MOC videos, Co-Prop reports higher rates of frames whose predicted object matches ground truth, which is the direct measurement of temporal alignment the paper introduces.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Our inference: RCPG's reference lists are built from ground-truth mask changes, so the LLM is in effect taught to find 'mask-visible object switches.' On videos where the sound changes but the visible mask does not (one person singing then speaking), control points may be missed; a dedicated audio-event-boundary benchmark would reveal this.
Our inference: the same boundary-anchoring idea could transfer to other audio-visual grounding tasks — event localization, video captioning, or dialogue-scene segmentation — whenever a pretrained audio-capable LLM is available, though the paper does not test those tasks.
Our inference: the reported alignment-rate gain on MOC could be decomposed further by computing control-point precision/recall directly against manual audio-event annotations, which would separate the contribution of boundary anchoring from the contribution of propagation.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear [Colleague],

This is a plausible, incremental but useful paper on audio-visual video segmentation. The core idea is to use an LLM to find the times when the sounding object changes, split the video into sub-clips, and then propagate keyframe masks through a video-object-segmentation-style propagator that inserts audio features frame-by-frame. The reported gains on M3 and AVSS are consistent and the ablations support the three main components.

What's actually new: the specific pipeline of retrieval-augmented prompting (Qwen) for control points plus audio-informed propagation is not in the literature. The authors also show plug-and-play integration with three existing AVVS methods, which is useful. The MOC dataset, though small, is a reasonable attempt to measure alignment directly.

The soft spots are real but not disqualifying. The central load-bearing assumption is that the LLM control points are correct, but the paper never measures control-point accuracy directly. You only see downstream MJ/MF and the alignment rate on a 17-video test set, which is too small to be persuasive. The retrieval examples come from GT masks, so the LLM is shown perfect boundaries at inference time, while at test time it sees only audio; that distribution mismatch is never analyzed. There's also a prompt inconsistency: the instructions ask for ten frames but the reference examples in Fig. 2 have length five, and no mapping to the actual video length is given. Table 6 shows keyframes (the frames the control points mark) get worse segmentation than normal frames, which is consistent with weak boundary anchoring—though the paper attributes this to the keyframe processor being less mature.

The other weakness is that the gains over the cosine-similarity baseline are modest (about 1.6 MJ on M3), and the cosine baseline is a poor boundary detector. So the RCPG is helpful, but we don't know how well it finds true boundaries, and no error bars are reported.

That said, the paper is transparent about its limitations, and the method is a sensible modular improvement. With code, the MOC benchmark, and a direct evaluation of control-point accuracy (e.g., compare against GT transition times), I'd be reasonably convinced. It deserves a serious referee, and I'd send it to review with the request for those additions.

Recommendation: send to peer review. It's not groundbreaking, but it's a solid contribution to a subfield, and the missing evidence is easy to supply.

Referee Report

4 major / 8 minor

Summary. Audio-visual video segmentation (AVVS) pairs a video with its audio track to produce pixel-level masks of sound-producing objects. This paper argues that existing AVVS models suffer from temporal misalignment—they keep segmenting an object after it stops making sound—because they neglect the boundaries at which the sounding object changes. The proposed Co-Prop framework operates in two stages: (1) a Retrieval-Augmented Control Points Generation (RCPG) module uses the Qwen LLM, prompted in several steps with retrieved reference examples derived from training-set annotations, to output a binary control-point list marking frames at which the sounding object changes; the video is thereby split into sub-clips with consistent sounding objects; and (2) an Audio-Insert Propagation (AIP) module processes each sub-clip, combining a Keyframe Processor (an audio-guided single-frame segmentation model fine-tuned on a keyframe subset) with a frame-by-frame propagator built on AOT that embeds current-frame audio features (Eq. 6). The authors also curate a 17-video MOC test subset from the M3 test set and introduce an Alignment Rate metric. Reported results on M3, S4, and AVSS with ResNet-50 and PVT-v2 backbones show consistent improvements over prior methods (e.g., M3 MJ 63.58 vs. AVSegFormer 58.36 with PVT-v2), and plug-and-play integration with TPAVI, CATR, and AVSegFormer is validated.

Significance. Conditional on the results holding, this is a solid and useful contribution to AVVS. The gains on the multi-source benchmarks (M3, AVSS) are substantial and consistent across two backbones, the ablations in Tables 2 and 5 support the incremental value of each design choice, and the plug-and-play integration results in Fig. 4 suggest the module can be grafted onto existing AVVS models. The inclusion of a negative result (Table 4, text-guided propagation degrades performance) is a good-faith check on the design. The paper also ships testable artifacts: a publicly released MOC evaluation subset and a promise of code release that will let others reproduce the AOT-based propagation. The principal weakness is that the headline attribution—improved temporal alignment via LLM control points—is supported only indirectly: control-point accuracy is never measured, the alignment metric is informal, and the MOC set is small and hand-selected. I agree with the stress-test note that the circularity concern does not land at the level of the equations: final numbers are obtained on held-out test splits and no prediction reduces to a fitted value.

major comments (4)

[§3.2, Eq. (5); Fig. 2] The conversion from the LLM's fixed-length output to a per-frame control-point list is never specified, and the prompt granularity is internally inconsistent. The §3.2 prompt asks Qwen to divide the audio into ten frames, the three reference examples in Fig. 2 are length-5 lists, and the 1-Step ablation prompt in §4.3 instructs 'divide the audio into five frames,' while Eq. (5) indexes the output over the full T-frame video. Since these control points determine the keyframe set and hence the entire sub-clip decomposition, the authors need to state (i) the actual frame counts T of the M3/S4/AVSS videos, (ii) how a fixed-length list is resampled or mapped onto T frames, and (iii) how sensitive the Table 2 results are to the chosen granularity and list length. Without this, Eq. (5) does not define a reproducible procedure.
[§4.3, Table 2; Appendix A.3, Table 6; Limitations] Control-point accuracy is never measured, so the reported gains cannot yet be attributed to temporal alignment. Appendix A.1 already defines an algorithmic derivation of control points from GT masks; applying that same procedure to the test splits and reporting precision/recall of the LLM's c_i would directly test the anchoring claim. This matters because the reference examples retrieved in Eq. (4) encode oracle boundaries derived from training-set GT masks, while at inference the LLM sees only audio; this is not circular in the equations, but it is a real gap. The indirect evidence is mixed: Table 6 reports 59.82 MJ on key frames versus 65.19 MJ on normal frames for M3, which is consistent with weak boundary anchoring, yet no analysis at transition frames is given, and the Limitations paragraph concedes reliance on the Keyframe Processor without addressing this deficit.
[§3.3 vs. Appendix A.1 vs. §4.3] The provenance of the control-point annotations used to build the keyframe fine-tuning subset is described three different ways: §3.3 states the keyframe dataset was annotated by applying the RCPG method to the training set, while Appendix A.1 and the §4.3 ablation describe control-point lists derived algorithmically from GT masks, and §3.2 refers to manual annotation. These are different procedures with different error properties: training the Keyframe Processor on GT-derived keyframes while deploying it on LLM-derived keyframes creates a train/inference distribution shift, whereas training on LLM-derived keyframes risks reinforcing LLM anchoring errors. The authors should specify the exact procedure used and analyze its implications for the keyframe results in Table 6.
[§4.2, Table 3, Fig. 3] The temporal-alignment claim rests on the MOC set and the Alignment Rate metric, both of which need stronger support. MOC contains 17 hand-selected videos from the 64-video M3 test set; the selection protocol should be stated precisely, and the alignment-rate gaps in Fig. 3 (e.g., 77.65 for Co-Prop vs. 71.76 for AVSegFormer) need error bars or significance tests, as do the headline numbers in Table 1, which are reported from single runs. The Alignment Rate definition—'the proportion of predicted video frames where the identified object aligns with the ground truth'—is also informal: the paper should specify how the 'identified object' is determined for a binary M3 mask and for a semantic AVSS mask, and over which set of frames the rate is averaged.

minor comments (8)

[Conclusion] The Conclusion contains a typo: 'Collaborative Hybrid Propgator' should read 'Propagator.'
[Fig. 4] Both panels are captioned '(a) Jaccard index on M3 dataset,' the caption text ('Pink denotes the model with Co-Prop as Keyframe Processor') is unclear, and the bar values 62.88 and 62.21 appear duplicated across panels; the figure should identify each compared method and distinguish the J and F panels.
[Eq. (1)] The indexing [ci]^T_{i=0} denotes T+1 control entries for a video stated to have T frames; the indexing should be fixed or c_0 explicitly defined.
[§3.2] The phrase 'the audio frames are evenly divided into ten frames' is ambiguous because audio is not naturally framed like video; clarify how audio features are synchronized to the video frame rate.
[Abstract; §4.1] The claimed memory reduction relative to simultaneous decoding is never quantified; a memory comparison on equal-length videos would substantiate this claim.
[Eq. (4)] The retrieval implementation is underspecified: the number of reference samples per prompt and the matching criterion for 'samples of the same category' should be stated for reproducibility.
[§4.3, Table 2(a)] The cosine-similarity control-point baseline is not described: which audio features are compared, what threshold is used, and how the thresholded outputs are converted to a binary list are all unspecified, which limits the interpretability of the RCPG ablation.
[Fig. 3] The alignment-rate values written on the bars (63.53, 68.23, 71.76, 77.65) are not reproduced in the text; given the 17-video MOC set, reporting the per-method rate with the number of videos in a small table would be more informative.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's comparisons use held-out standard benchmarks, and its self-citations (AOT, CATR) are published, reproducible components rather than load-bearing justifications.

full rationale

The paper's central claims are supported by held-out evaluations on the standard M3, S4, and AVSS test splits (Table 1), and the ablation study (Table 2) decomposes the gains into keyframe processing, retrieval-augmented control point generation, and the audio-insert propagator. No equation reduces a prediction to a fitted value or to a ground-truth-derived quantity at test time. The RCPG module retrieves control-point reference examples from the training set only (Appendix A.1, Algorithm 1); the final control-point list for a test video is the LLM's output on that video's audio, not a copy of the retrieved examples, so there is no self-definitional reduction. The MOC test set and Alignment Rate metric are defined from ground-truth masks, but they are evaluation instruments, not inputs into the model's predictions. The self-citations to AOT (Yang et al. 2021) and CATR (Li et al. 2023) are used as published, code-reproduced building blocks (propagation backbone and audio-queried decoding, respectively); they do not themselves encode the paper's temporal-alignment claim. The main unaddressed risk is that LLM control-point accuracy is not directly measured, so the reported gains cannot be rigorously attributed to temporal alignment rather than to AOT-Large pretraining or stronger propagation. This is a correctness/robustness concern, not circularity, because the downstream metric is not constructed to force the result. Overall, the derivation chain is self-contained against external benchmarks.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central method depends on the reliability of an external LLM for boundary detection, on an assumed equivalence between ground-truth mask changes and audio semantic changes, on transfer of AOT-based propagation, and on a small curated test set. These are not free parameters in the classical sense, but they are unverified dependencies that a replication study would need to isolate.

free parameters (3)

Number of video feature layers for audio insertion = 4
Appendix A.2 Table 5 shows 4-layer interaction beats 1-layer; this design choice is tuned on the same M3 and S4 benchmarks.
Control point list granularity = Not stated; prompt example uses 10 frames
Sec 3.2 prompt says audio frames are evenly divided into ten frames, while Eq. 1 indexes over T frames; the actual granularity used in experiments is not specified.
Number of retrieval reference examples per prompt = Not stated; three shown in Fig 2
Fig 2(a) shows Ref1 to Ref3, but the fixed number of retrieved samples used in practice is not reported, so this prompt-design parameter is under-specified.

assumptions (5)

domain assumption Qwen can reliably map audio to semantic category transitions from retrieval-augmented prompts.
Sec 3.2, Eq. 5: the entire RCPG stage is an untrained LLM call; no direct accuracy measure for control points is provided.
domain assumption Ground-truth mask changes are a valid proxy for audio semantic boundaries.
Appendix A.1 builds control point lists D by comparing semantic content of GT masks across consecutive frames; this assumes visual mask transitions coincide with audio transitions.
domain assumption AOT-based mask propagation transfers to audio-visual segmentation after fine-tuning.
Sec 4.2 second stage initializes from AOT-Large and fine-tunes on AVS datasets; the paper does not ablate other propagation backbones.
domain assumption The MOC subset is representative of temporal misalignment.
Sec 4.2 selects 17 of 64 M3 test videos with multiple changing targets; results on MOC underpin the alignment-rate claim.
domain assumption Standard supervised splits of M3, S4, and AVSS are used without data leakage.
The paper does not describe train/test split handling for RCPG retrieval examples, but retrieval is stated to use the training set only.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Collaborative Hybrid Propagator for Temporal Misalignment in Audio-Visual Segmentation." pith.science (2026). https://pith.science/paper/FTJV7O5A

@misc{pith2026241208161,
  author       = {Pith},
  title        = {Pith review of: Collaborative Hybrid Propagator for Temporal Misalignment in Audio-Visual Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FTJV7O5A}},
  note         = {Machine review of arXiv:2412.08161}
}

read the original abstract

Audio-visual video segmentation (AVVS) aims to generate pixel-level maps of sound-producing objects that accurately align with the corresponding audio. However, existing methods often face temporal misalignment, where audio cues and segmentation results are not temporally coordinated. Audio provides two critical pieces of information: i) target object-level details and ii) the timing of when objects start and stop producing sounds. Current methods focus more on object-level information but neglect the boundaries of audio semantic changes, leading to temporal misalignment. To address this issue, we propose a Collaborative Hybrid Propagator Framework~(Co-Prop). This framework includes two main steps: Preliminary Audio Boundary Anchoring and Frame-by-Frame Audio-Insert Propagation. To Anchor the audio boundary, we employ retrieval-assist prompts with Qwen large language models to identify control points of audio semantic changes. These control points split the audio into semantically consistent audio portions. After obtaining the control point lists, we propose the Audio Insertion Propagator to process each audio portion using a frame-by-frame audio insertion propagation and matching approach. We curated a compact dataset comprising diverse source conversion cases and devised a metric to assess alignment rates. Compared to traditional simultaneous processing methods, our approach reduces memory requirements and facilitates frame alignment. Experimental results demonstrate the effectiveness of our approach across three datasets and two backbones. Furthermore, our method can be integrated with existing AVVS approaches, offering plug-and-play functionality to enhance their performance.

Figures

Figures reproduced from arXiv: 2412.08161 by the authors.

**Figure 2.** Overview. Our Collaborative Hybrid Propagator Framework (Co-Prop) comprises RetrievalAugmented Control Points Generation and Audio Insertion Propagation. Retrieval-augmented Control Points Generation Module aims to anchor key points during audio category transitions preemptively. Additionally, the Audio-insert Propagator aims to embed audio frame by frame, reducing memory demands while facilitating frame-aligned in… view at source ↗

**Figure 3.** Comparison Alignment Rate on MOC Test Dataset. 1) Mitigating the temporal misalignment issue. Our study demonstrates that our model exhibits more substantial performance enhancements on the M3 and AVSS datasets than the S4 dataset. This discrepancy arises from the multi-source audio nature of the M3 and AVSS datasets, which encompass diverse sound sources, exacerbating temporal misalignment between audio and predi… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Comparison w/o Co-Prop. Pink denotes the model with Co-Prop as Keyframe Processor. Ablation Study of Main Modules [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Comparative analysis of the AVSeg method and our proposed model. We present three [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 24 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Crossvit: Cross-attention multi-scale vision transformer for image classification

Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 357--366, 2021

work page 2021
[3]

Unraveling instance associations: A closer look for audio-visual segmentation, 2024

Yuanhong Chen, Yuyuan Liu, Hu Wang, Fengbei Liu, Chong Wang, Helen Frazer, and Gustavo Carneiro. Unraveling instance associations: A closer look for audio-visual segmentation, 2024

work page 2024
[4]

Blazingly fast video object segmentation with pixel-wise metric learning

Yuhua Chen, Jordi Pont - Tuset, Alberto Montes, and Luc Van Gool. Blazingly fast video object segmentation with pixel-wise metric learning. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pp.\ 1189--1198. Computer Vision Foundation / IEEE Computer Society, 2018. doi:10.1109/CVPR.20...

arXiv 2018
[5]

The pascal visual object classes (voc) challenge

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 0 303--338, 2010

2010
[6]

Semi-supervised video object segmentation via learning object-aware global-local correspondence

Jiaqing Fan, Bo Liu, Kaihua Zhang, and Qingshan Liu. Semi-supervised video object segmentation via learning object-aware global-local correspondence. IEEE Transactions on Circuits and Systems for Video Technology, 32 0 (12): 0 8153--8164, 2021

work page 2021
[7]

Avsegformer: Audio-visual segmentation with transformer

Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, and Tong Lu. Avsegformer: Audio-visual segmentation with transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 12155--12163, 2024

work page 2024
[8]

Improving audio-visual segmentation with bidirectional generation, 2023

Dawei Hao, Yuxin Mao, Bowen He, Xiaodong Han, Yuchao Dai, and Yiran Zhong. Improving audio-visual segmentation with bidirectional generation, 2023

work page 2023

Show all 39 references

[9]

Improving audio-visual segmentation with bidirectional generation

Dawei Hao, Yuxin Mao, Bowen He, Xiaodong Han, Yuchao Dai, and Yiran Zhong. Improving audio-visual segmentation with bidirectional generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 2067--2075, 2024

2024
[10]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

2016
[11]

Interactive video object segmentation using global and local transfer modules

Yuk Heo, Yeong Jun Koh, and Chang - Su Kim. Interactive video object segmentation using global and local transfer modules. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan - Michael Frahm (eds.), ECCV, volume 12362 of Lecture Notes in Computer Science, pp.\ 297--313. Spr...

2020 doi
[12]

Guided interactive video object segmentation using reliability-based attention maps

Yuk Heo, Yeong Jun Koh, and Chang-Su Kim. Guided interactive video object segmentation using reliability-based attention maps. In CVPR, pp.\ 7322--7330, 2021

2021
[13]

Cnn architectures for large-scale audio classification

Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speech ...

2017
[14]

Discovering sounding objects by audio queries for audio visual segmentation, 2023

Shaofei Huang, Han Li, Yuqing Wang, Hongji Zhu, Jiao Dai, Jizhong Han, Wenge Rong, and Si Liu. Discovering sounding objects by audio queries for audio visual segmentation, 2023. URL https://arxiv.org/abs/2309.09501

2023 arXiv
[15]

Siamese network with interactive transformer for video object segmentation

Meng Lan, Jing Zhang, Fengxiang He, and Lefei Zhang. Siamese network with interactive transformer for video object segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 1228--1236, 2022

2022
[16]

Catr: Combinatorial-dependence audio-queried transformer for audio-visual video segmentation

Kexin Li, Zongxin Yang, Lei Chen, Yi Yang, and Jun Xiao. Catr: Combinatorial-dependence audio-queried transformer for audio-visual video segmentation. In Proceedings of the 31st ACM International Conference on Multimedia, pp.\ 1485--1494, 2023

2023
[17]

Idpro: Flexible interactive video object segmentation by id-queried concurrent propagation

Kexin Li, Tao Jiang, Zongxin Yang, Yi Yang, Yueting Zhuang, and Jun Xiao. Idpro: Flexible interactive video object segmentation by id-queried concurrent propagation. IEEE Transactions on Circuits and Systems for Video Technology, 2024

2024
[18]

Audio-visual segmentation by exploring cross-modal mutual semantics

Chen Liu, Peike Patrick Li, Xingqun Qi, Hu Zhang, Lincheng Li, Dadong Wang, and Xin Yu. Audio-visual segmentation by exploring cross-modal mutual semantics. In Proceedings of the 31st ACM International Conference on Multimedia, pp.\ 7590--7598, 2023 a

2023
[19]

Bavs: Bootstrapping audio-visual segmentation by integrating foundation knowledge

Chen Liu, Peike Li, Hu Zhang, Lincheng Li, Zi Huang, Dadong Wang, and Xin Yu. Bavs: Bootstrapping audio-visual segmentation by integrating foundation knowledge. IEEE Transactions on Multimedia, pp.\ 1--13, 2024 a . doi:10.1109/TMM.2024.3405622

2024
[20]

Bavs: bootstrapping audio-visual segmentation by integrating foundation knowledge

Chen Liu, Peike Li, Hu Zhang, Lincheng Li, Zi Huang, Dadong Wang, and Xin Yu. Bavs: bootstrapping audio-visual segmentation by integrating foundation knowledge. IEEE Transactions on Multimedia, 2024 b

2024
[21]

Audio-aware query-enhanced transformer for audio-visual segmentation, 2023 b

Jinxiang Liu, Chen Ju, Chaofan Ma, Yanfeng Wang, Yu Wang, and Ya Zhang. Audio-aware query-enhanced transformer for audio-visual segmentation, 2023 b . URL https://arxiv.org/abs/2307.13236

2023 arXiv
[22]

Annotation-free audio-visual segmentation

Jinxiang Liu, Yu Wang, Chen Ju, Chaofan Ma, Ya Zhang, and Weidi Xie. Annotation-free audio-visual segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.\ 5604--5614, 2024 c

2024
[23]

Contrastive conditional latent diffusion for audio-visual segmentation

Yuxin Mao, Jing Zhang, Mochu Xiang, Yunqiu Lv, Yiran Zhong, and Yuchao Dai. Contrastive conditional latent diffusion for audio-visual segmentation. arXiv preprint arXiv:2307.16579, 2023 a

2023 arXiv
[24]

Multimodal variational auto-encoder based audio-visual segmentation

Yuxin Mao, Jing Zhang, Mochu Xiang, Yiran Zhong, and Yuchao Dai. Multimodal variational auto-encoder based audio-visual segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 954--965, 2023 b

2023
[25]

Weakly-supervised audio-visual segmentation

Shentong Mo and Bhiksha Raj. Weakly-supervised audio-visual segmentation. Advances in Neural Information Processing Systems, 36, 2024

2024
[26]

Video object segmentation using space-time memory networks

Seoung Wug Oh, Joon - Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019 , pp.\ 9225--9234. IEEE , 2019. ...

2019
[27]

Segment anything meets point tracking

Frano Raji c , Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, and Fisher Yu. Segment anything meets point tracking. arXiv preprint arXiv:2307.01197, 2023

2023 arXiv
[28]

Self-supervised audio-visual co-segmentation

Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh McDermott, and Antonio Torralba. Self-supervised audio-visual co-segmentation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 2357--2361. IEEE, 2019

2019
[29]

Revisiting click-based interactive video object segmentation

Stephane Vujasinovic, Sebastian Bullinger, Stefan Becker, Norbert Scherer-Negenborn, Michael Arens, and Rainer Stiefelhagen. Revisiting click-based interactive video object segmentation. In 2022 IEEE International Conference on Image Processing (ICIP). IEEE, October 2022. doi:...

2022
[30]

Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, pp...

2021
[31]

Cooperation does matter: Exploring multi-order bilateral relations for audio-visual segmentation

Qi Yang, Xing Nie, Tong Li, Pengfei Gao, Ying Guo, Cheng Zhen, Pengfei Yan, and Shiming Xiang. Cooperation does matter: Exploring multi-order bilateral relations for audio-visual segmentation. arXiv preprint arXiv:2312.06462, 2023

2023 arXiv
[32]

Associating objects with transformers for video object segmentation

Zongxin Yang, Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmentation. Advances in Neural Information Processing Systems, 34: 0 2491--2502, 2021

2021
[33]

Learning to recommend frame for interactive video object segmentation in the wild, 2021

Zhaoyuan Yin, Jia Zheng, Weixin Luo, Shenhan Qian, Hanling Zhang, and Shenghua Gao. Learning to recommend frame for interactive video object segmentation in the wild, 2021

2021
[34]

Audio--visual segmentation

Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio--visual segmentation. In European Conference on Computer Vision, pp.\ 386--403. Springer, 2022

2022
[35]

Audio-visual segmentation with semantics

Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, et al. Audio-visual segmentation with semantics. arXiv preprint arXiv:2301.13190, 2023

2023 arXiv
[36]

Separable structure modeling for semi-supervised video object segmentation

Wencheng Zhu, Jiahao Li, Jiwen Lu, and Jie Zhou. Separable structure modeling for semi-supervised video object segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 32 0 (1): 0 330--344, 2021

2021
[37]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[38]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[39]

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Crossvit: Cross-attention multi-scale vision transformer for image classification

Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 357--366, 2021

work page 2021

[3] [3]

Unraveling instance associations: A closer look for audio-visual segmentation, 2024

Yuanhong Chen, Yuyuan Liu, Hu Wang, Fengbei Liu, Chong Wang, Helen Frazer, and Gustavo Carneiro. Unraveling instance associations: A closer look for audio-visual segmentation, 2024

work page 2024

[4] [4]

Blazingly fast video object segmentation with pixel-wise metric learning

Yuhua Chen, Jordi Pont - Tuset, Alberto Montes, and Luc Van Gool. Blazingly fast video object segmentation with pixel-wise metric learning. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pp.\ 1189--1198. Computer Vision Foundation / IEEE Computer Society, 2018. doi:10.1109/CVPR.20...

arXiv 2018

[5] [5]

The pascal visual object classes (voc) challenge

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 0 303--338, 2010

2010

[6] [6]

Semi-supervised video object segmentation via learning object-aware global-local correspondence

Jiaqing Fan, Bo Liu, Kaihua Zhang, and Qingshan Liu. Semi-supervised video object segmentation via learning object-aware global-local correspondence. IEEE Transactions on Circuits and Systems for Video Technology, 32 0 (12): 0 8153--8164, 2021

work page 2021

[7] [7]

Avsegformer: Audio-visual segmentation with transformer

Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, and Tong Lu. Avsegformer: Audio-visual segmentation with transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 12155--12163, 2024

work page 2024

[8] [8]

Improving audio-visual segmentation with bidirectional generation, 2023

Dawei Hao, Yuxin Mao, Bowen He, Xiaodong Han, Yuchao Dai, and Yiran Zhong. Improving audio-visual segmentation with bidirectional generation, 2023

work page 2023