Pith. sign in

REVIEW 4 major objections 4 minor 4 cited by

ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectives

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ObjectRelator claims that adding a text-condition fusion module and a cross-view embedding alignment loss to the PSALM segmenter produces state-of-the-art ego-exo object correspondence masks, raising IoU to 50.9 on Ego-Exo4D with only…

desk verdict Solid engineering for an emerging task, but the XObjAlign ablation is confounded and the 'self-supervised' label is wrong; worth reviewing, but the authors need to isolate the alignment loss. read the letter →

arxiv 2411.19083 v2 pith:3I5FQA5L submitted 2024-11-28 cs.CV cs.AI

Yuqian Fu , Runze Wang , Bin Ren , Guolei Sun , Biao Gong , Yanwei Fu , Danda Pani Paudel , Xuanjing Huang
show 1 more author
Luc Van Gool
This is my paper · ORCID
classification cs.CVcs.AI
keywords ego-exoobjectcorrespondencecross-viewsegmentationmultimodalconditionfusionself-supervisedalignmentvision-languageobject-levelconsistencyEgo-Exo4DHANDAL-X
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the ego-exo object correspondence task—matching and segmenting the same object across first-person and third-person views—can be solved far better than a strong zero-shot baseline by a small adaptation of a universal segmentation model. It claims two additions to PSALM do the work: MCFuse, which fuses a text description of the query object with the visual mask prompt, and XObjAlign, which pulls the ego and exo visual embeddings of the same object together with a simple Euclidean-distance loss. On Ego-Exo4D, ObjectRelator raises IoU from 41.3 to 45.4 (Ego2Exo) and from 47.3 to 50.9 (Exo2Ego) on the Full TrainSet, at a cost of only 0.26M extra parameters. If correct, the result matters because it shows that language and cross-view consistency are complementary cues for a task where pure mask prompts fail under large viewpoint change.

What carries the argument

Two lightweight modules carry the argument. MCFuse fuses the ego text embedding and ego visual embedding via cross-attention (text as query, visual as key/value) followed by a residual connection with a learnable weight: $E^*_{con} = k_{lea} \cdot E^*_{vis} + (1 - k_{lea}) \cdot CA_{fuse}$, so the visual prompt remains the primary cue while language refines it. XObjAlign enforces object-level cross-view consistency with a self-supervised loss $L_{Xobj} = \mathrm{Dist}(E^*_{vis}, E_{vis})$, the Euclidean distance between ego and exo visual embeddings of the same object, computed during training using the ground-truth exo mask prompt and removed at inference. The final training loss is $L = L_{mask} + L_{Xobj}$, with a two-stage schedule that first initializes MCFuse before joint training.

What would settle it

Measure the Euclidean distance between ego and exo visual embeddings on the Ego-Exo4D val set for same-object pairs and different-object pairs. If, after training with XObjAlign, the separation between these two distance distributions does not increase relative to the retrained PSALM baseline, the claim that the alignment loss drives the improvement is falsified. A more direct test: retrain with deliberately mismatched alignment pairs (ego object A aligned with exo object B); if IoU does not drop substantially, the alignment term is not playing the causal role the paper attributes to it.

Watch

Extended reading notes

Core claim

We show that a universal segmentation model (PSALM) can be turned into a state-of-the-art ego-exo object correspondence segmenter by combining two modest modules: MCFuse, which fuses generated text descriptions with the visual query mask through cross-attention with a learned residual weight, and XObjAlign, which minimizes the Euclidean distance between ego and exo object embeddings in the LLM space during training. With these modules ObjectRelator achieves the best IoU on Ego-Exo4D (45.4 Ego2Exo / 50.9 Exo2Ego on Full TrainSet) and on the adapted HANDAL-X benchmark (84.7 vs 83.4 for retrained PSALM), adding only 0.2632M parameters.

Load-bearing premise

The load-bearing premise is that the shared LLM embedding space represents object identity faithfully enough across ego and exo views that minimizing the Euclidean distance between the two visual embeddings improves correspondence; if that distance is not a faithful proxy under large viewpoint change, the alignment loss could distort the conditioning signal without helping inference.

Editorial extensions

If this is right

  • ObjectRelator's gains come with almost no parameter overhead, so the recipe of text-condition fusion plus embedding alignment is portable to other universal segmentation backbones.
  • Jointly training both directions (Ego2Exo and Exo2Ego) in a single model matches or beats separately trained models, suggesting a shared cross-view representation is learnable.
  • The model keeps most of its improvement when the text condition is dropped at inference (43.3 vs 44.3 Ego2Exo on Small TrainSet), implying a joint vision-language condition space that is robust to missing text at test time.
  • Training on Ego-Exo4D transfers zero-shot to HANDAL-X (42.8 vs 39.9 IoU for PSALM), indicating that cross-view training data confers generalization to new cross-view settings.
  • The method is frame-level; adding temporal modeling, which the paper explicitly leaves to future work, is a natural next step that may compound the gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • XObjAlign as formulated uses only positive pairs (same object across views); adding explicit negative pairs or a margin could prevent potential embedding collapse and likely improve the alignment further—a testable extension the paper does not explore.
  • Because the alignment operates in the LLM embedding space shared with the mask generator, the same loss could serve as a lightweight cross-view adapter for other prompt-decoder segmentation models (e.g., SEEM, UNINEXT), not just PSALM.
  • The ~80% accuracy of LLaVA-generated descriptions still yields most of the gain, suggesting the fusion mechanism is tolerant to caption noise; a stronger or self-supervised description generator could push MCFuse further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper tackles the Ego-Exo object correspondence task, where an object mask in one view (ego or exo) must be mapped to the corresponding mask in the other view. The authors build on the PSALM segmentation model and propose two additions: MCFuse, which fuses a visual mask prompt with an automatically generated text description of the object, and XObjAlign, a training-only loss that pulls together the ego and exo visual embeddings of the same object. Experiments on Ego-Exo4D (with a modified protocol) and on a newly introduced HANDAL-X dataset report consistent IoU gains over retrained PSALM baselines, with the largest single gain attributed to XObjAlign in the ablations.

Significance. If the reported gains are real and correctly attributed, the paper provides a useful building block for cross-view object correspondence, a relatively underexplored task. The design is simple, adds few parameters (about 0.26M), and the code is promised. The construction of HANDAL-X as an additional cross-view segmentation testbed is a useful contribution, and the authors are transparent about the altered Ego-Exo4D protocol and the fact that the proposed modules are removed or partially removed at inference. The main weakness is that the central component, XObjAlign, is not isolated from the use of ground-truth target masks in the ablations, so the causal contribution of the alignment objective remains unclear.

major comments (4)
  1. [§3.1–3.3, Eq. (6); Table 3 and Supp. Table E] This comment is repeated; please remove duplicates.
  2. [§4, 'Datasets' paragraph and Table 2]
  3. [Reproducibility and statistical significance]
  4. [§3.3 and Abstract]
minor comments (4)
  1. [§3.1, Eq. (1)–(3)] The equations and surrounding text are garbled in the provided manuscript, e.g., Eq. (2) contains a stray comma in the left-hand side, and the definition of which outputs correspond to which forward pass is unclear. Please rewrite this part carefully so that it is unambiguous which forward pass produces which embedding.
  2. [§4.1, Table 2] The notation for ZSL results is inconsistent: the caption says '◦ means results from PSALM [75]' but the text says these are from the original paper. Please clarify in the caption which numbers are from prior work and which are newly reported, and also indicate that the PSALM ZSL row is evaluated on the filtered setup.
  3. [Supp. §1.3, Table B] In the Ambiguity Robustness rows, the metric is IoU, but the table does not say this explicitly in the table heading; please state the units in the row labels or in the caption.
  4. [§4.4, HANDAL-X] Please provide, either in the main text or the supplementary, statistics on HANDAL-X analogous to those in Supp. Table A (number of images, objects, pair construction details, etc.) and clarify the license and release plan for the dataset and the exact code used to generate it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ObjectRelator's reported gains rest on an external baseline (PSALM) and supervised auxiliary losses, not on definitional reuse of the predicted quantity.

full rationale

The derivation chain is self-contained. The central prediction is the target-view mask produced by the Mask Generator L_mask = G(f'''_I, E*_con, E_M) (Eq. 3), using conditioning embeddings obtained from the query view and the fused text/visual condition. MCFuse (Eqs. 4-5) is a learned fusion of E*_txt and E*_vis; XObjAlign (Eq. 6) minimizes Euclidean distance between the ego and exo visual embeddings, but the exo embedding is formed from the ground-truth target mask during training only and is removed at inference, so it acts as an auxiliary supervisory signal rather than as the predicted output. The gains over PSALM are measured on held-out Val masks and on HANDAL-X, and the main baseline PSALM [75] is external work with released checkpoints, not a same-author result. The self-citations in the bibliography (Cafuser [1], Xtrack [61], and other author papers) appear only in contextual related-work lists and are not load-bearing premises. The paper's own limitation statement (Supp. Sec. 4) acknowledges remaining failure cases, which does not indicate circularity. The closest concern is that the +XObjAlign ablation row in Table 3 also includes a second LLM forward pass conditioned on the GT exo mask, so the isolated contribution of L_Xobj is not fully controlled; that is an experimental confound, not a circular reduction of the predicted mask to its input. No equation or claim equates a reported prediction with a fitted parameter or with the target ground truth by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central method rests on standard pretrained components (PSALM, LLaVA) plus four tuned or learned quantities: the MCFuse residual weight, cross-attention weights, the XObjAlign loss weight, and the learning rate choice. No physical entities are postulated; HANDAL-X is a new benchmark artifact. The main unproven background assumptions are about data reliability, text description quality, and the semantic validity of Euclidean distance in the LLM embedding space.

free parameters (4)
  • k_lea (learnable residual fusion weight in MCFuse) = learned during training, exact value not reported
    In Eq. 5, E*_con = k_lea * E*_vis + (1 - k_lea) * CA_fuse. The weight balances visual and text conditions; ablations in Supp. Tab. D compare fixed values 0.2, 0.5, 0.8 and find the learned version best.
  • MCFuse cross-attention parameters W_Q, W_K, W_V = learned; module adds 0.2632M parameters
    Standard attention projections in Eq. 4, introduced by MCFuse, trained on data.
  • XObjAlign loss weight = 1.0
    The total loss is L = L_mask + L_XObj. Supp. Tab. E ablated weights 0.2, 0.5, 1.0, 10 and selected 1.0; this is a hand-chosen hyperparameter.
  • learning rate selection = one of 6e-5 or 2e-4, not specified per run
    Supp. 1.4 says the learning rate is 'select from 6e-5, 2e-4' with cosine decay; the exact value used for the main runs is not stated.
assumptions (4)
  • domain assumption Ego-Exo4D provides temporally aligned ego/exo frames with reliable ground-truth masks.
    The method relies on synchronized frames and accurate masks for both the mask loss and the XObjAlign loss; Section 3, Task Formulation and Supp. 1.1.
  • domain assumption LLaVA-generated text descriptions are accurate enough to help localization.
    MCFuse uses LLaVA descriptions of the masked query object as the text condition; Supp. Tab. B reports about 22% error rate by human judge, yet the residual design keeps visual cues primary.
  • domain assumption Euclidean distance in the LLM embedding space is a valid proxy for cross-view object identity.
    XObjAlign minimizes Dist(E*_vis, E_vis) in Eq. 6; this only helps if the embeddings of the same object from different views are closer than embeddings of different objects.
  • domain assumption PSALM's pretrained weights, including the Mask2Former-style mask generator, are a good initialization for the ego-exo task.
    All models, including the retrained baseline, start from the pretrained PSALM checkpoint (Supp. 1.4), so the reported gains are relative to that initialization.
invented entities (1)
  • HANDAL-X dataset independent evidence
    purpose: A new cross-view object segmentation benchmark adapted from HANDAL by pairing frames of the same object at 100-frame intervals; serves as an additional testbed for ego-exo style segmentation.
    The dataset can be released and used by other groups to test cross-view segmentation methods, giving it a falsifiable handle outside this paper. It is an artifact, not a physical entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectives." pith.science (2026). https://pith.science/paper/3I5FQA5L

@misc{pith2026241119083,
  author       = {Pith},
  title        = {Pith review of: ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectives},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3I5FQA5L}},
  note         = {Machine review of arXiv:2411.19083}
}
read the original abstract

Bridging the gap between ego-centric and exo-centric views has been a long-standing question in computer vision. In this paper, we focus on the emerging Ego-Exo object correspondence task, which aims to understand object relations across ego-exo perspectives through segmentation. While numerous segmentation models have been proposed, most operate on a single image (view), making them impractical for cross-view scenarios. PSALM, a recently proposed segmentation method, stands out as a notable exception with its demonstrated zero-shot ability on this task. However, due to the drastic viewpoint change between ego and exo, PSALM fails to accurately locate and segment objects, especially in complex backgrounds or when object appearances change significantly. To address these issues, we propose ObjectRelator, a novel approach featuring two key modules: Multimodal Condition Fusion (MCFuse) and SSL-based Cross-View Object Alignment (XObjAlign). MCFuse introduces language as an additional cue, integrating both visual masks and textual descriptions to improve object localization and prevent incorrect associations. XObjAlign enforces cross-view consistency through self-supervised alignment, enhancing robustness to object appearance variations. Extensive experiments demonstrate ObjectRelator's effectiveness on the large-scale Ego-Exo4D benchmark and HANDAL-X (an adapted dataset for cross-view segmentation) with state-of-the-art performance. Code is made available at: http://yuqianfu.com/ObjectRelator.

Figures

Figures reproduced from arXiv: 2411.19083 by the authors.

Figure 1
Figure 1. Illustration of the Ego-Exo Object Correspondence Task [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of ObjectRelator. Ego2Exo is used as an example. Our method builds on the PSALM baseline (pink blocks) and tailors it for Ego-Exo Object Correspondence with two novel modules: Multimodal Condition Fusion and Cross-View Object Alignment. ego mask prompt P ∗ vis and the instruction hint is the text instruction prompt P ∗ txt. Specifically, P ∗ vis is formed by combing the ego image I ∗ and its object mask m∗ … view at source ↗
Figure 3
Figure 3. Architecture of our Multimodal Condition Fusion (MC [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: ObjectRelator vs. PSALM Visualization Results. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 4
Figure 4. Figure 4: ObjectRelator Visualization for Ego2Exo and Exo2Ego. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The First EgoCross Challenge at EgoVis 2026: Cross-Domain Egocentric Video Question Answering

    cs.CV 2026-08 accept novelty 5.0 of 10

    At CVPR 2026, the EgoCross Challenge evaluated cross-domain egocentric video QA across four specialist domains, with the best system reaching 66.98% accuracy.

  2. DOMR: Establishing Cross-View Segmentation via Dense Object Matching

    cs.CV 2025-08 conditional novelty 5.0 of 10

    DOMR jointly matches and refines multiple object masks across ego and exo views, reaching 49.7% and 55.2% mean IoU on Ego-Exo4D.

  3. Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision

    cs.CV 2025-06 accept novelty 3.0 of 10

    A comprehensive review of cross-view video understanding that uses both first-person and third-person cameras, organized into a three-direction taxonomy with a dataset catalog and future research gaps.

  4. Cross-View Multi-Modal Segmentation @ Ego-Exo4D Challenges 2025

    cs.CV 2025-06 conditional novelty 2.0 of 10

    A multimodal segmentation system using visual masks plus auto-generated text and a cross-view alignment loss ranks second in the Ego-Exo4D object correspondence benchmark.

Reference graph

Works this paper leans on

82 extracted references · 70 canonical work pages · cited by 4 Pith papers

  1. [1]

    Cafuser: Condition-aware multimodal fusion for robust semantic perception of driving scenes.IEEE Robotics and Automation Letters, 2025

    Tim Br ¨odermann, Christos Sakaridis, Yuqian Fu, and Luc Van Gool. Cafuser: Condition-aware multimodal fusion for robust semantic perception of driving scenes.IEEE Robotics and Automation Letters, 2025. 2

    2025

  2. [2]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. InIEEE/CVF conference on computer vision and pattern recognition,

  3. [3]

    4diff: 3d- aware diffusion model for third-to-first viewpoint translation

    Feng Cheng, Mi Luo, Huiyu Wang, Alex Dimakis, Lorenzo Torresani, Gedas Bertasius, and Kristen Grauman. 4diff: 3d- aware diffusion model for third-to-first viewpoint translation. InEuropean Conference on Computer Vision, 2025. 2

    work page 2025

  4. [4]

    Xmem: Long- term video object segmentation with an atkinson-shiffrin memory model

    Ho Kei Cheng and Alexander G Schwing. Xmem: Long- term video object segmentation with an atkinson-shiffrin memory model. InEuropean Conference on Computer Vi- sion, 2022. 3, 6

    work page 2022

  5. [5]

    Tracking anything with decoupled video segmentation

    Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexan- der Schwing, and Joon-Young Lee. Tracking anything with decoupled video segmentation. InProceedings of the IEEE/CVF International Conference on Computer Vision,

    work page

  6. [6]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. InEu- ropean conference on computer vision, 2018. 1, 2

    work page 2018

  7. [7]

    Guide to the carnegie mellon university multimodal activity (cmu-mmac) database

    Fernando De la Torre, Jessica Hodgins, Adam Bargteil, Xavier Martin, Justin Macey, Alex Collado, and Pep Beltran. Guide to the carnegie mellon university multimodal activity (cmu-mmac) database. 2009. 2

    work page 2009

  8. [8]

    Open- vocabulary universal image segmentation with maskclip

    Zheng Ding, Jieke Wang, and Zhuowen Tu. Open- vocabulary universal image segmentation with maskclip. arXiv preprint arXiv:2208.08984, 2022. 6

    arXiv 2022

Show all 82 references
  1. [9]

    Understand- ing egocentric activities

    Alireza Fathi, Ali Farhadi, and James M Rehg. Understand- ing egocentric activities. In2011 international conference on computer vision, 2011. 2

    2011

  2. [10]

    Embodied one-shot video recognition: Learning from actions of a vir- tual embodied agent

    Yuqian Fu, Chengrong Wang, Yanwei Fu, Yu-Xiong Wang, Cong Bai, Xiangyang Xue, and Yu-Gang Jiang. Embodied one-shot video recognition: Learning from actions of a vir- tual embodied agent. InACM international conference on multimedia, 2019. 2

    2019

  3. [11]

    Depth guided adaptive meta-fusion network for few- shot video recognition

    Yuqian Fu, Li Zhang, Junke Wang, Yanwei Fu, and Yu-Gang Jiang. Depth guided adaptive meta-fusion network for few- shot video recognition. In28th ACM International Confer- ence on Multimedia, 2020. 2

    2020

  4. [12]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jack- son Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InIEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2022

  5. [13]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. InIEEE/CVF Con...

    2024

  6. [14]

    HAN- DAL: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions

    Andrew Guo, Bowen Wen, Jianhe Yuan, Jonathan Tremblay, Stephen Tyree, Jeffrey Smith, and Stan Birchfield. HAN- DAL: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions. In IROS, 2023. 5, 1

    2023

  7. [15]

    A sur- vey on instance segmentation: state of the art.International journal of multimedia information retrieval, 2020

    Abdul Mueed Hafiz and Ghulam Mohiuddin Bhat. A sur- vey on instance segmentation: state of the art.International journal of multimedia information retrieval, 2020. 2

    2020

  8. [16]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. InIEEE international conference on computer vision, 2017. 2, 3

    2017

  9. [17]

    Cvm-net: Cross-view matching network for image- based ground-to-aerial geo-localization

    Sixing Hu, Mengdan Feng, Rang MH Nguyen, and Gim Hee Lee. Cvm-net: Cross-view matching network for image- based ground-to-aerial geo-localization. InIEEE Conference on Computer Vision and Pattern Recognition, 2018. 3

    2018

  10. [18]

    Learning disentangled identifiers for action-customized text-to-image generation

    Siteng Huang, Biao Gong, Yutong Feng, Xi Chen, Yuqian Fu, Yu Liu, and Donglin Wang. Learning disentangled identifiers for action-customized text-to-image generation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2

    2024

  11. [19]

    Egoexolearn: A dataset for bridg- ing asynchronous ego- and exo-centric view of procedural activities in real world

    Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Li- jin Yang, Baoqi Pei, Hongjie Zhang, Lu Dong, Yali Wang, Limin Wang, and Yu Qiao. Egoexolearn: A dataset for bridg- ing asynchronous ego- and exo-centric view of procedural activities in real world. InIEEE/CVF Conference on C...

    2024

  12. [20]

    Multi-modal dense video captioning

    Vladimir Iashin and Esa Rahtu. Multi-modal dense video captioning. InIEEE/CVF conference on computer vision and pattern recognition workshops, 2020. 1, 2

    2020

  13. [21]

    Image-to-image translation with conditional adver- sarial networks

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adver- sarial networks. InIEEE conference on computer vision and pattern recognition, 2017. 3

    2017

  14. [22]

    A survey of methods for addressing the challenges of referring image segmentation.Neurocomputing, 2024

    Lixia Ji, Yunlong Du, Yiping Dang, Wenzhao Gao, and Han Zhang. A survey of methods for addressing the challenges of referring image segmentation.Neurocomputing, 2024. 2

    2024

  15. [23]

    Lemma: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivities

    Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-chun Zhu. Lemma: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivities. InEuropean Conference on Computer Vision, 2020. 2

    2020

  16. [24]

    Egotaskqa: Understanding human tasks in egocentric videos

    Baoxiong Jia, Ting Lei, Song-Chun Zhu, and Siyuan Huang. Egotaskqa: Understanding human tasks in egocentric videos. Advances in Neural Information Processing Systems, 2022. 2

    2022

  17. [25]

    An unsupervised person re-identification approach based on cross-view distribution alignment.IET Image Processing, 2021

    Xibin Jia, Xing Wang, and Qing Mi. An unsupervised person re-identification approach based on cross-view distribution alignment.IET Image Processing, 2021. 3

    2021

  18. [26]

    Large-scale video classification with convolutional neural networks

    Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. InIEEE conference on Computer Vision and Pattern Recognition,

  19. [27]

    Panoptic feature pyramid networks

    Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Doll´ar. Panoptic feature pyramid networks. InIEEE/CVF conference on computer vision and pattern recognition,

  20. [28]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InIEEE/CVF International Conference on Computer Vision, 2023. 2, 3

    2023

  21. [29]

    Fast unsupervised ego-action learning for first- person sports videos

    Kris M Kitani, Takahiro Okabe, Yoichi Sato, and Akihiro Sugimoto. Fast unsupervised ego-action learning for first- person sports videos. InCVPR 2011, 2011. 1, 2

    2011

  22. [30]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In IEEE international conference on computer vision, 2017. 1, 2

    2017

  23. [31]

    H2o: Two hands manipulating objects for first person interaction recognition

    Taein Kwon, Bugra Tekin, Jan St ¨uhmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. InIEEE/CVF Interna- tional Conference on Computer Vision, 2021. 2

    2021

  24. [32]

    Lisa: Reasoning segmenta- tion via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmenta- tion via large language model. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 1, 2, 3

    2024

  25. [33]

    Survey on semantic seg- mentation using deep learning techniques.Neurocomputing,

    Fahad Lateef and Yassine Ruichek. Survey on semantic seg- mentation using deep learning techniques.Neurocomputing,

  26. [34]

    Crossviewdiff: A cross-view diffusion model for satellite- to-street view synthesis.arXiv preprint arXiv:2408.14765,

    Weijia Li, Jun He, Junyan Ye, Huaping Zhong, Zhi- meng Zheng, Zilong Huang, Dahua Lin, and Conghui He. Crossviewdiff: A cross-view diffusion model for satellite- to-street view synthesis.arXiv preprint arXiv:2408.14765,

    arXiv

  27. [35]

    Omg-seg: Is one model good enough for all segmentation? InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. Omg-seg: Is one model good enough for all segmentation? InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2, 3

    2024

  28. [36]

    Textbooks are all you need ii: phi-1.5 technical report (2023).https://arxiv

    Yuanzhi Li, S ´ebastien Bubeck, Ronen Eldan, AD Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report (2023).https://arxiv. org/abs/2309.05463. 5

    2023 arXiv

  29. [37]

    Delving into ego- centric actions

    Yin Li, Zhefan Ye, and James M Rehg. Delving into ego- centric actions. InIEEE conference on computer vision and pattern recognition, 2015. 2

    2015

  30. [38]

    Egocentric video-language pretraining.Advances in Neural Information Processing Sys- tems, 2022

    Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z Xu, Difei Gao, Rong-Cheng Tu, Wen- zhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining.Advances in Neural Information Processing Sys- tems, 2022. 1, 2

    2022

  31. [39]

    Multi-task mid-level feature alignment network for unsupervised cross-dataset person re-identification.arXiv preprint arXiv:1807.01440, 2018

    Shan Lin, Haoliang Li, Chang-Tsun Li, and Alex Chichung Kot. Multi-task mid-level feature alignment network for unsupervised cross-dataset person re-identification.arXiv preprint arXiv:1807.01440, 2018. 3

    2018 arXiv

  32. [40]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 2, 4, 1

    2023

  33. [41]

    Exocentric-to-egocentric video genera- tion

    Jia-Wei Liu, Weijia Mao, Zhongcong Xu, Jussi Keppo, and Mike Zheng Shou. Exocentric-to-egocentric video genera- tion. InThe Thirty-eighth Annual Conference on Neural In- formation Processing Systems. 2

  34. [42]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF International Conference on Computer Vision,

  35. [43]

    Intention-driven ego-to-exo video generation.arXiv preprint arXiv:2403.09194, 2024

    Hongchen Luo, Kai Zhu, Wei Zhai, and Yang Cao. Intention-driven ego-to-exo video generation.arXiv preprint arXiv:2403.09194, 2024. 2

    2024 arXiv

  36. [44]

    Put myself in your shoes: Lifting the egocentric perspective from exocentric videos

    Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman. Put myself in your shoes: Lifting the egocentric perspective from exocentric videos. InEuropean Conference on Com- puter Vision, 2025. 1, 2

    2025

  37. [45]

    Image seg- mentation using deep learning: A survey.IEEE transactions on pattern analysis and machine intelligence, 2021

    Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image seg- mentation using deep learning: A survey.IEEE transactions on pattern analysis and machine intelligence, 2021. 2

    2021

  38. [46]

    Codef: Content deformation fields for tempo- rally consistent video processing

    Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen. Codef: Content deformation fields for tempo- rally consistent video processing. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 1, 2

    2024

  39. [47]

    The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Ar- bel´aez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017. 6

    2017 arXiv

  40. [48]

    Egovlpv2: Egocentric video-language pre-training with fusion in the backbone

    Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pre-training with fusion in the backbone. InIEEE/CVF International Conference on Computer Vision, 2023. 2

    2023

  41. [49]

    Home action genome: Cooperative compositional action understanding

    Nishant Rai, Haofeng Chen, Jingwei Ji, Rishi Desai, Kazuki Kozuka, Shun Ishizaka, Ehsan Adeli, and Juan Carlos Niebles. Home action genome: Cooperative compositional action understanding. InIEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2021. 2

    2021

  42. [50]

    A sur- vey of recent interactive image segmentation methods.Com- putational visual media, 2020

    Hiba Ramadan, Chaymae Lachqar, and Hamid Tairi. A sur- vey of recent interactive image segmentation methods.Com- putational visual media, 2020. 2

    2020

  43. [51]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2, 3

    2024

  44. [52]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...

    2024 arXiv

  45. [53]

    Cross-view image synthesis using geometry-guided conditional gans.Computer Vision and Image Understanding, 2019

    Krishna Regmi and Ali Borji. Cross-view image synthesis using geometry-guided conditional gans.Computer Vision and Image Understanding, 2019. 1

    2019

  46. [54]

    Cascaded cross mlp- mixer gans for cross-view image translation

    Bin Ren, Hao Tang, Nicu Sebe, et al. Cascaded cross mlp- mixer gans for cross-view image translation. InBritish Ma- chine Vision Conference, 2021. 1

    2021

  47. [55]

    Pixellm: Pixel reasoning with large multimodal model

    Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. InIEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2024. 2, 3

    2024

  48. [56]

    As- sembly101: A large-scale multi-view video dataset for un- derstanding procedural activities

    Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. As- sembly101: A large-scale multi-view video dataset for un- derstanding procedural activities. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 2

    2022

  49. [57]

    Learning co-segmentation by segment swapping for retrieval and discovery.arXiv, 2021

    Xi Shen, Alexei A Efros, Armand Joulin, and Mathieu Aubry. Learning co-segmentation by segment swapping for retrieval and discovery.arXiv, 2021. 3, 6

    2021

  50. [58]

    Beyond cross-view image re- trieval: Highly accurate vehicle localization using satellite image

    Yujiao Shi and Hongdong Li. Beyond cross-view image re- trieval: Highly accurate vehicle localization using satellite image. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 3

    2022

  51. [59]

    Charades-ego: A large-scale dataset of paired third and first person videos.arXiv preprint arXiv:1804.09626, 2018

    Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large-scale dataset of paired third and first person videos.arXiv preprint arXiv:1804.09626, 2018. 2

    2018 arXiv

  52. [60]

    Make-a-video: Text-to-video generation without text-video data.arXiv preprint arXiv:2209.14792,

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data.arXiv preprint arXiv:2209.14792,

    arXiv

  53. [61]

    Xtrack: Multimodal training boosts rgb-x video object trackers.arXiv preprint arXiv:2405.17773, 2024

    Yuedong Tan, Zongwei Wu, Yuqian Fu, Zhuyun Zhou, Guolei Sun, Eduard Zamfi, Chao Ma, Danda Pani Paudel, Luc Van Gool, and Radu Timofte. Xtrack: Multimodal training boosts rgb-x video object trackers.arXiv preprint arXiv:2405.17773, 2024. 2

    2024 arXiv

  54. [62]

    Corso, and Yan Yan

    Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang, Jason J. Corso, and Yan Yan. Multi-channel attention selection gan with cas- caded semantic guidance for cross-view image translation. In CVPR, 2019. 1, 2, 3

    2019

  55. [63]

    Cross-view image matching for geo-localization in urban environments

    Yicong Tian, Chen Chen, and Mubarak Shah. Cross-view image matching for geo-localization in urban environments. InIEEE Conference on Computer Vision and Pattern Recog- nition, 2017. 3

    2017

  56. [64]

    Mocogan: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. InIEEE conference on computer vision and pattern recognition, 2018. 2

    2018

  57. [65]

    Holoassist: an egocen- tric human interaction dataset for interactive ai assistants in the real world

    Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bu- gra Tekin, Felipe Vieira Frujeri, et al. Holoassist: an egocen- tric human interaction dataset for interactive ai assistants in the real world. InIEEE/CVF Internationa...

    2023

  58. [66]

    Zero-shot person re-identification via cross-view consistency.IEEE Transac- tions on Multimedia, 2015

    Zheng Wang, Ruimin Hu, Chao Liang, Yi Yu, Junjun Jiang, Mang Ye, Jun Chen, and Qingming Leng. Zero-shot person re-identification via cross-view consistency.IEEE Transac- tions on Multimedia, 2015. 3

    2015

  59. [67]

    Assistq: Affordance-centric question-driven task completion for ego- centric assistant

    Benita Wong, Joya Chen, You Wu, Stan Weixian Lei, Dongxing Mao, Difei Gao, and Mike Zheng Shou. Assistq: Affordance-centric question-driven task completion for ego- centric assistant. InEuropean Conference on Computer Vi- sion, 2022. 2

    2022

  60. [68]

    Tune-a-video: One-shot tun- ing of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tun- ing of image diffusion models for text-to-video generation. InIEEE/CVF International Conference on Computer Vision,

  61. [69]

    Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification

    Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. InEuro- pean conference on computer vision, 2018. 1, 2

    2018

  62. [70]

    Universal instance perception as object discovery and retrieval

    Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Ze- huan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 2, 3

    2023

  63. [71]

    Vid2seq: Large-scale pretraining of a visual language model for dense video captioning

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, An- toine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. InIEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2023

  64. [72]

    Helping hands: An object-aware ego-centric video recogni- tion model

    Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Helping hands: An object-aware ego-centric video recogni- tion model. InIEEE/CVF International Conference on Com- puter Vision, 2023. 1, 2

    2023

  65. [73]

    Object-centric video representation for long-term action anticipation

    Ce Zhang, Changcheng Fu, Shijie Wang, Nakul Agarwal, Kwonjoon Lee, Chiho Choi, and Chen Sun. Object-centric video representation for long-term action anticipation. In IEEE/CVF Winter Conference on Applications of Computer Vision, 2024. 2

    2024

  66. [74]

    Multi-scale image-and feature-level align- ment for cross-resolution person re-identification.Remote Sensing, 2024

    Guoqing Zhang, Zhun Wang, Jiangmei Zhang, Zhiyuan Luo, and Zhihao Zhao. Multi-scale image-and feature-level align- ment for cross-resolution person re-identification.Remote Sensing, 2024. 3

    2024

  67. [75]

    Psalm: Pixelwise segmentation with large multi-modal model

    Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large multi-modal model. InEuropean Conference on Computer Vision, 2024. 1, 2, 3, 5, 6, 7, 8

    2024

  68. [76]

    Camsam2: Segment any- thing accurately in camouflaged videos.arXiv preprint arXiv:2503.19730, 2025

    Yuli Zhou, Guolei Sun, Yawei Li, Yuqian Fu, Luca Benini, and Ender Konukoglu. Camsam2: Segment any- thing accurately in camouflaged videos.arXiv preprint arXiv:2503.19730, 2025. 2

    2025

  69. [77]

    Segment everything everywhere all at once.Advances in Neural Information Processing Systems, 2024

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once.Advances in Neural Information Processing Systems, 2024. 1, 2, 3, 6, 8 ObjectRelator: Enabling Cross-View Object Relat...

    2024

  70. [78]

    Identify the single object covered by the green mask without describing it. Note that it is not a hand. Format your answer as follows: The object covered by the green mask is

    Implementation Details 1.1. Data Processing on Ego-Exo4D Frame Extraction.We follow the same frame extraction process as the baselines, i.e., XSegTx, XView-Xmem, as provided by Ego-Exo4D [13]. Specifically, for both ego and exo views, we sample one frame every 30 frames in chr...

  71. [79]

    Add”, “CA w/o Params

    More Results 2.1. More Ablations on MCFuse To thoroughly assess the optimality of our design, we con- duct additional ablations applying various options to MC- Fuse. These include testing: whether cross attention (CA) is the best method for fusing conditions; the impact of ado...

  72. [80]

    CA, S2” to “CA + LearnResid- ual, S2

    The improvement from “CA, S2” to “CA + LearnResid- ual, S2” supports protecting the more reliable visual prompt condition. For theresidual weightexperiments, we com- pare our learnable approach with fixed weightsk lea (0.2, 0.5, 0.8). Results indicate that learnable weightklea...

  73. [81]

    Failure Cases We also analyze the failure cases produced by our method, with typical examples summarized in Fig. B. Results indi- cate that our method struggles in several scenarios: it fails to generate a complete mask when the object’s surface is discontinuous or blends clos...

  74. [82]

    The approach primarily consists of multimodal condition fusion and SSL- based cross-view object alignment, built on top of a frame- level multimodal segmentation model

    Limitations and Future Work In this paper, we propose ObjectRelator, a method de- signed to understand cross-view object relationships in terms of segmentation, validated on ego-exo perspectives and a relatively easier cross-view dataset. The approach primarily consists of mul...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.