{"id":"20d59251-730a-49a5-a6b7-bb23ab88f20c","arxiv_id":"2411.19083","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ObjectRelator adds multimodal condition fusion and cross-view embedding alignment to PSALM, improving ego-exo object correspondence IoU by about 4 points on Ego-Exo4D and setting SOTA on HANDAL-X.","lead":"This paper presents ObjectRelator, a method that segments corresponding objects across first-person (ego) and third-person (exo) videos by combining visual masks with text descriptions in a large multimodal segmentation model. It reports state-of-the-art results on the Ego-Exo4D correspondence benchmark and on a new HANDAL-X testbed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"XObjAlign's reported gain is confounded: the ablation adds a second LLM forward pass with ground-truth target-mask conditioning along with the distance loss, so the +4.1/+4.2 IoU may not be caused by the alignment objective.","rationale":"The paper is a clear engineering contribution with consistent tables and a detailed supplement; the ablation structure is the most load-bearing weak point. The reader's identified assumption concerned whether Euclidean distance in the LLM embedding space is a faithful proxy for object identity under viewpoint change. My concern is adjacent but distinct: even if the distance were a faithful proxy, the +XObjAlign row does not prove the distance loss is the operative mechanism, because it also introduces a second LLM forward pass and GT target-mask conditioning absent from the base PSALM. This is a testable confound rather than an external-consensus disagreement. The paper does ablate the loss metric and weight, but the missing control is the no-loss version of the same two-pass architecture. The overall verdict should remain CONDITIONAL: MCFuse alone shows a solid gain, and the full system may be valid, but the key second module's benefit is unproven until the control is run.","tokens_in":20730,"tokens_out":5899,"duration_ms":53418,"concrete_test":"On the Ego2Exo Small TrainSet, add a control that keeps the two-pass architecture and the GT exo-mask prompt exactly as in ObjectRelator but sets L_Xobj = 0 (or detaches E_vis from the graph so no alignment gradient flows), with all other hyperparameters unchanged. Compare its IoU with the 43.8 reported for '+XObjAlign' and the 39.7 base. If the control reaches about 43.8, the gain is not from the distance objective; if it falls back toward 39.7, L_Xobj is causal. Repeat on Exo2Ego to confirm the pattern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The largest single gain in Table 3 is attributed to XObjAlign (+4.1 Ego2Exo, +4.2 Exo2Ego), but the ablation row does not isolate the alignment objective. According to Sec. 3.1, ObjectRelator training runs two LLM forward passes: Eq. (1) with the ego visual mask prompt and Eq. (2) with the exo visual mask prompt formed from the ground-truth target mask. The '+XObjAlign' row includes both this second forward pass and the Euclidean loss L_Xobj from Eq. (6), whereas the base PSALM is trained with a single forward pass. Therefore the reported improvement could come from the extra LLM forward pass, the extra conditioning on the GT exo mask during training, or the distance loss itself. The supplementary ablations vary the distance metric (Euclidean vs cosine) and loss weight (Table E) but never remove L_Xobj while keeping the second forward pass, so the causal contribution of the alignment loss is unidentified. Since XObjAlign is removed at inference and is claimed as a key contribution, this confound directly affects the central claim that ObjectRelator's improvements are due to cross-view embedding alignment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper tackles the Ego-Exo object correspondence task, where an object mask in one view (ego or exo) must be mapped to the corresponding mask in the other view. The authors build on the PSALM segmentation model and propose two additions: MCFuse, which fuses a visual mask prompt with an automatically generated text description of the object, and XObjAlign, a training-only loss that pulls together the ego and exo visual embeddings of the same object. Experiments on Ego-Exo4D (with a modified protocol) and on a newly introduced HANDAL-X dataset report consistent IoU gains over retrained PSALM baselines, with the largest single gain attributed to XObjAlign in the ablations.","tokens_in":20941,"tokens_out":7896,"duration_ms":79964,"significance":"If the reported gains are real and correctly attributed, the paper provides a useful building block for cross-view object correspondence, a relatively underexplored task. The design is simple, adds few parameters (about 0.26M), and the code is promised. The construction of HANDAL-X as an additional cross-view segmentation testbed is a useful contribution, and the authors are transparent about the altered Ego-Exo4D protocol and the fact that the proposed modules are removed or partially removed at inference. The main weakness is that the central component, XObjAlign, is not isolated from the use of ground-truth target masks in the ablations, so the causal contribution of the alignment objective remains unclear.","major_comments":[{"comment":"This comment is repeated; please remove duplicates.","section":"§3.1–3.3, Eq. (6); Table 3 and Supp. Table E"},{"comment":"","section":"§4, 'Datasets' paragraph and Table 2"},{"comment":"","section":"Reproducibility and statistical significance"},{"comment":"","section":"§3.3 and Abstract"}],"minor_comments":[{"comment":"The equations and surrounding text are garbled in the provided manuscript, e.g., Eq. (2) contains a stray comma in the left-hand side, and the definition of which outputs correspond to which forward pass is unclear. Please rewrite this part carefully so that it is unambiguous which forward pass produces which embedding.","section":"§3.1, Eq. (1)–(3)"},{"comment":"The notation for ZSL results is inconsistent: the caption says '◦ means results from PSALM [75]' but the text says these are from the original paper. Please clarify in the caption which numbers are from prior work and which are newly reported, and also indicate that the PSALM ZSL row is evaluated on the filtered setup.","section":"§4.1, Table 2"},{"comment":"In the Ambiguity Robustness rows, the metric is IoU, but the table does not say this explicitly in the table heading; please state the units in the row labels or in the caption.","section":"Supp. §1.3, Table B"},{"comment":"Please provide, either in the main text or the supplementary, statistics on HANDAL-X analogous to those in Supp. Table A (number of images, objects, pair construction details, etc.) and clarify the license and release plan for the dataset and the exact code used to generate it.","section":"§4.4, HANDAL-X"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically sound in its internal comparisons, and the authors are transparent about the modified Ego-Exo4D protocol. The main issue is that the key module XObjAlign is not cleanly ablated, which weakens the paper's central claim. The solution is additional experiments, so I do not recommend rejection. I also note that the abstract and conclusion claim 'state-of-the-art' on Ego-Exo4D despite the protocol alteration; this should be tempered in revision. The self-references in the bibliography are not problematic in themselves, but the novelty relative to closely related concurrent cross-view segmentation work might deserve a broader comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is a solid, well-engineered contribution to a genuinely new task: cross-view ego-exo object correspondence by segmentation. The authors build on PSALM, add text-conditioned fusion (MCFuse) and an embedding-distance consistency loss (XObjAlign), and report consistent IoU gains on Ego-Exo4D plus a new HANDAL-X benchmark. The extra parameters are tiny, the ablations are mostly careful, and the writing is clear. The new dataset and the first text-conditioned treatment of this task are useful for anyone working on ego-exo perception for robotics or VR.\n\nThe soft spots are real but not fatal. The stress-test concern is valid: the +XObjAlign row in Table 3 adds not only the Euclidean loss but also a second LLM forward pass with the ground-truth exo mask as conditioning. The base PSALM is trained with a single forward pass, so the +4.1/+4.2 IoU could come from the extra conditioning, the extra forward pass, or the loss itself. The supplementary ablations vary the distance metric and loss weight but never keep the second forward pass while removing the loss, so the causal contribution of the alignment objective is unidentified. That is a load-bearing gap for the paper's central claim about cross-view alignment. The authors should add that ablation; if the loss is truly what matters, the number should survive.\n\nAlso, the 'self-supervised' label is wrong. XObjAlign uses the ground-truth target mask to form the exo embedding, so the supervision comes from annotated masks, not from a self-supervised signal. The loss itself is supervised auxiliary alignment, and the paper should say so. This is a characterisation issue but an important one for how readers interpret the method.\n\nOther concerns are minor: results are single runs without error bars, and the evaluation protocol retains only objects visible in both views and tests on the validation set. These are acknowledged in the paper and are common in early benchmark work; they do not undermine the main results. The embedding-distance assumption (that Euclidean closeness in LLM space tracks object identity under viewpoint change) is plausible given the empirical gains, but the confound makes it hard to tell how much credit the distance loss deserves.\n\nWho is this for? Researchers working on ego-exo correspondence, cross-view segmentation, or robotics/VR applications that need object grounding across viewpoints. It is a useful recipe and a new testbed, not a fundamental breakthrough. It deserves serious peer review: the task is new, the engineering is honest, and the results are reproducible in principle. I would recommend sending it out, with the expectation that the authors will fix the XObjAlign ablation and correct the supervision terminology before acceptance.","headline":"Solid engineering for an emerging task, but the XObjAlign ablation is confounded and the 'self-supervised' label is wrong; worth reviewing, but the authors need to isolate the alignment loss.","tokens_in":21553,"tokens_out":2248,"would_cite":true,"duration_ms":22563,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ObjectRelator claims that adding a text-condition fusion module and a cross-view embedding alignment loss to the PSALM segmenter produces state-of-the-art ego-exo object correspondence masks, raising IoU to 50.9 on Ego-Exo4D with only…","keywords":["ego-exo object correspondence","cross-view segmentation","multimodal condition fusion","self-supervised alignment","vision-language segmentation","object-level consistency","Ego-Exo4D","HANDAL-X"],"falsifier":"Measure the Euclidean distance between ego and exo visual embeddings on the Ego-Exo4D val set for same-object pairs and different-object pairs. If, after training with XObjAlign, the separation between these two distance distributions does not increase relative to the retrained PSALM baseline, the claim that the alignment loss drives the improvement is falsified. A more direct test: retrain with deliberately mismatched alignment pairs (ego object A aligned with exo object B); if IoU does not drop substantially, the alignment term is not playing the causal role the paper attributes to it.","tokens_in":20464,"feed_emoji":"🎯","tokens_out":5346,"duration_ms":66503,"temperature":0.7,"pith_summary":"The paper tries to establish that the ego-exo object correspondence task—matching and segmenting the same object across first-person and third-person views—can be solved far better than a strong zero-shot baseline by a small adaptation of a universal segmentation model. It claims two additions to PSALM do the work: MCFuse, which fuses a text description of the query object with the visual mask prompt, and XObjAlign, which pulls the ego and exo visual embeddings of the same object together with a simple Euclidean-distance loss. On Ego-Exo4D, ObjectRelator raises IoU from 41.3 to 45.4 (Ego2Exo) and from 47.3 to 50.9 (Exo2Ego) on the Full TrainSet, at a cost of only 0.26M extra parameters. If correct, the result matters because it shows that language and cross-view consistency are complementary cues for a task where pure mask prompts fail under large viewpoint change.","feed_headline":"Ego-exo object IoU climbs to 50.9 with two small add-ons","feed_subtitle":"Text cues plus cross-view embedding alignment beat the PSALM baseline with only 0.26M extra parameters.","key_machinery":"Two lightweight modules carry the argument. MCFuse fuses the ego text embedding and ego visual embedding via cross-attention (text as query, visual as key/value) followed by a residual connection with a learnable weight: $E^*_{con} = k_{lea} \\cdot E^*_{vis} + (1 - k_{lea}) \\cdot CA_{fuse}$, so the visual prompt remains the primary cue while language refines it. XObjAlign enforces object-level cross-view consistency with a self-supervised loss $L_{Xobj} = \\mathrm{Dist}(E^*_{vis}, E_{vis})$, the Euclidean distance between ego and exo visual embeddings of the same object, computed during training using the ground-truth exo mask prompt and removed at inference. The final training loss is $L = L_{mask} + L_{Xobj}$, with a two-stage schedule that first initializes MCFuse before joint training.","core_discovery":"We show that a universal segmentation model (PSALM) can be turned into a state-of-the-art ego-exo object correspondence segmenter by combining two modest modules: MCFuse, which fuses generated text descriptions with the visual query mask through cross-attention with a learned residual weight, and XObjAlign, which minimizes the Euclidean distance between ego and exo object embeddings in the LLM space during training. With these modules ObjectRelator achieves the best IoU on Ego-Exo4D (45.4 Ego2Exo / 50.9 Exo2Ego on Full TrainSet) and on the adapted HANDAL-X benchmark (84.7 vs 83.4 for retrained PSALM), adding only 0.2632M parameters.","pith_inferences":["XObjAlign as formulated uses only positive pairs (same object across views); adding explicit negative pairs or a margin could prevent potential embedding collapse and likely improve the alignment further—a testable extension the paper does not explore.","Because the alignment operates in the LLM embedding space shared with the mask generator, the same loss could serve as a lightweight cross-view adapter for other prompt-decoder segmentation models (e.g., SEEM, UNINEXT), not just PSALM.","The ~80% accuracy of LLaVA-generated descriptions still yields most of the gain, suggesting the fusion mechanism is tolerant to caption noise; a stronger or self-supervised description generator could push MCFuse further."],"forward_implications":["ObjectRelator's gains come with almost no parameter overhead, so the recipe of text-condition fusion plus embedding alignment is portable to other universal segmentation backbones.","Jointly training both directions (Ego2Exo and Exo2Ego) in a single model matches or beats separately trained models, suggesting a shared cross-view representation is learnable.","The model keeps most of its improvement when the text condition is dropped at inference (43.3 vs 44.3 Ego2Exo on Small TrainSet), implying a joint vision-language condition space that is robust to missing text at test time.","Training on Ego-Exo4D transfers zero-shot to HANDAL-X (42.8 vs 39.9 IoU for PSALM), indicating that cross-view training data confers generalization to new cross-view settings.","The method is frame-level; adding temporal modeling, which the paper explicitly leaves to future work, is a natural next step that may compound the gains."],"supporting_citations":[{"why":"Supplies the base PSALM architecture (LLM + Mask2Former) that ObjectRelator builds on and is the main baseline it must beat.","marker":"[75]"},{"why":"Defines the ego-exo object correspondence task, provides the Ego-Exo4D benchmark data, and contributes the XSegTx/XView-XMem baselines.","marker":"[13]"},{"why":"Provides the Mask2Former pixel decoder and mask generator that produce the final masks inside PSALM.","marker":"[2]"},{"why":"The vision-language model used to generate the text descriptions that MCFuse fuses with the visual mask prompt.","marker":"[40]"},{"why":"Source dataset adapted into HANDAL-X, the additional cross-view segmentation testbed used for validation.","marker":"[14]"}],"fun_headline_variants":["Text + cross-view alignment boost ego-exo object IoU to 50.9","Two tiny modules give ego-exo segmentation a 50.9 IoU","ObjectRelator: state-of-the-art ego-exo object correspondence","Ego-exo object IoU hits 50.9 with just 0.26M extra params","New method wins ego-exo segmentation with text cues and SSL alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the shared LLM embedding space represents object identity faithfully enough across ego and exo views that minimizing the Euclidean distance between the two visual embeddings improves correspondence; if that distance is not a faithful proxy under large viewpoint change, the alignment loss could distort the conditioning signal without helping inference.","fun_headline_variants_meta":{"raw":{"variants":["Text + cross-view alignment boost ego-exo object IoU to 50.9","Two tiny modules give ego-exo segmentation a 50.9 IoU","ObjectRelator: state-of-the-art ego-exo object correspondence","Ego-exo object IoU hits 50.9 with just 0.26M extra params","New method wins ego-exo segmentation with text cues and SSL alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000722,"raw_usage":{"total_tokens":3254,"prompt_tokens":978,"completion_tokens":2276,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":2171}},"tokens_in":594,"tokens_out":2276,"duration_ms":13689,"temperature":1.0,"reasoning_tokens":2171,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:33:21.383873+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the Euclidean distance between ego and exo visual embeddings on the Ego-Exo4D val set for same-object pairs and different-object pairs. If, after training with XObjAlign, the separation between these two distance distributions does not increase relative to the retrained PSALM baseline, the claim that the alignment loss drives the improvement is falsified. A more direct test: retrain with deliberately mismatched alignment pairs (ego object A aligned with exo object B); if IoU does not drop substantially, the alignment term is not playing the causal role the paper attributes to it.","supporting_citations":[{"cited_title":"Psalm: Pixelwise segmentation with large multi-modal model","cited_arxiv_id":null,"evidence_quote":"Supplies the base PSALM architecture (LLM + Mask2Former) that ObjectRelator builds on and is the main baseline it must beat."},{"cited_title":"Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives","cited_arxiv_id":null,"evidence_quote":"Defines the ego-exo object correspondence task, provides the Ego-Exo4D benchmark data, and contributes the XSegTx/XView-XMem baselines."},{"cited_title":"Visual instruction tuning, 2023","cited_arxiv_id":null,"evidence_quote":"The vision-language model used to generate the text descriptions that MCFuse fuses with the visual mask prompt."},{"cited_title":"HAN- DAL: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions","cited_arxiv_id":null,"evidence_quote":"Source dataset adapted into HANDAL-X, the additional cross-view segmentation testbed used for validation."}],"review_version":1}