Pith. sign in

REVIEW 5 major objections 5 minor 53 references

Discovering and using Spelke segments

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A self-supervised world model finds physically co-moving segments in static images, outperforming supervised SAM on a new Spelke benchmark and improving 3D edits.

desk verdict A solid extension of the same lab's CWM/LRAS line into probabilistic counterfactual segmentation, with a useful new benchmark, but the physical-grounding claim needs independent validation before it should be taken at face value. read the letter →

arxiv 2507.16038 v1 pith:RAF7B3CY submitted 2025-07-21 cs.CV cs.AI

Rahul Venkatesh , Klemen Kotar , Lilian Naing Chen , Seungwoo Kim , Luca Thomas Wheeler , Jared Watrous , Ashley Xu , Gia Ancone
show 5 more authors
Wanhee Lee Honglin Chen Daniel Bear Stefan Stojanov Daniel Yamins
This is my paper
classification cs.CVcs.AI
keywords Spelkeobjectsmotion-basedsegmentationself-supervisedlearningvisualworldmodelsopticalflowcompletioncounterfactualprobingobjectmanipulationBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most segmentation systems label pixels by semantic category or appearance, so their masks can split objects into pieces or glue them to backgrounds. This paper argues for a different unit: Spelke objects, groups of pixels that would move together when a force is applied, and that this unit can be extracted from a single static image without segmentation labels. The authors build SpelkeNet, an autoregressive visual world model trained on internet video to predict future optical flows, and probe it with sparse virtual pokes under a static camera. Averaging the model's imagined flow fields across many pokes yields segments of co-moving pixels. On the new SpelkeBench benchmark SpelkeNet reports higher average recall and mIoU than SAM2 and DINO baselines, and using these segments instead of SAM masks improves physical edit quality on 3DEditBench across four editing pipelines.

What carries the argument

The load-bearing machinery is statistical counterfactual probing: a procedure that turns a trained flow-completion model into a segmenter by asking, for many random virtual pokes at a sampled location, which pixels would move in the same direction as the poke. Two auxiliary maps carry the computation: the motion affordance map $p_{\mathrm{motion}}$, the probability that each location moves under external force, and the expected displacement map $E_{\mathrm{disp}}$, the probability-weighted average flow at every pixel conditioned on a poke. Averaging the dot product $\langle f^{(r)}, E_{\mathrm{disp}}^{(r)}[u]\rangle$ over pokes and thresholding with Otsu's method yields the segment; for unassisted discovery the same correlations are assembled into a pixel-to-pixel affinity matrix and clustered by an iterative select-threshold-refine loop. The model itself is an LRAS-style autoregressive transformer over quantized RGB and flow patches, which matters because sparse flow tokens can be appended to the sequence as localized interventions, and autoregressive decoding lets later motion tokens depend on earlier ones.

What would settle it

Run SpelkeNet on an image with two visually identical but physically separate objects (e.g. two identical cups side by side) and poke one: a segment that includes the un-poked twin, or a displacement field that moves both cups, would show the correlation signal is driven by appearance similarity, not physical co-movement. Conversely, comparing predicted segments against measured motion in a real robot poke experiment on the same scene would directly test whether the virtual-poke flow completions match physical response.

Watch

Extended reading notes

Core claim

The central claim is that the Spelke object—a category-agnostic grouping of pixels that reliably co-move under physical force—can be discovered self-supervised from static images and is the right segmentation primitive for physical manipulation. SpelkeNet is an instance of LRAS, a next-token autoregressive transformer over locally quantized RGB and optical-flow tokens; trained on internet video, it learns a distribution over plausible future flow fields. To segment, the paper conditions the model on a zero camera-pose token plus a sparse flow token at a sampled location (the virtual poke), decodes the flow field many times, and averages the dot product of the poke vector with the expected displacement at each pixel; Otsu thresholding of this average isolates the co-moving region. The paper reports that this statistical counterfactual probing reaches mIoU 0.681 and AR 0.541 on SpelkeBench, versus 0.623 and 0.482 for SAM2, and that on 3DEditBench its segments improve Edit Adherence in all four tested editing pipelines (for LRAS-3D, EA 0.776 with SpelkeNet versus 0.633 with SAM masks).

Load-bearing premise

The load-bearing premise is that SpelkeNet's flow completions, learned from naturally occurring video motion, faithfully predict how the scene would move under a physical poke; if the completions are instead driven by appearance or texture heuristics, the discovered segments are artifacts of the model rather than physical objects.

Editorial extensions

If this is right

  • Point-prompted Spelke segmentation outperforms the supervised SAM2 baseline and the self-supervised DINO and CWM baselines on SpelkeBench, suggesting that motion causality is a learnable, category-agnostic segmentation signal.
  • Swapping SpelkeNet segments for SAM masks raises Edit Adherence in all four editing pipelines tested on 3DEditBench, so the choice of segment, not just the editor, determines the physical plausibility of an edit.
  • The unprompted auto-discovery variant finds Spelke segments without any point prompts and beats self-supervised CutLER and ProMerge on F1 score, though it trails SAM2 in recall and mIoU, which the paper attributes to sub-optimal poke sampling and clustering.
  • Because SpelkeNet is never trained on segmentation labels or physical poke interventions, the results imply that object structure can emerge from video prediction alone.
  • The extracted segments include objects supported by the poked object, indicating the probing procedure implicitly encodes support relations in the scene.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the flow completions are physically faithful, the same probing procedure could serve directly as a pre-contact planner for a robot, estimating which clusters of pixels—and which attached objects—will move for a candidate push.
  • Editorial inference: because the segments are defined through a particular world model, a stress test is whether two independently trained flow models produce the same segments on the same image; true physical co-movement should be model-independent.
  • Editorial inference: the observation that $p_{\mathrm{motion}}$ is uniform for rigid objects and localized near the poke for deformable ones could be turned into a quantitative rigidity or deformability descriptor, a step the paper leaves qualitative.
  • Editorial inference: the probe-a-predictive-model-and-aggregate-correlations recipe may transfer beyond vision, e.g. to time-lapse microscopy or cosmological simulations, where meaningful units are also sets of degrees of freedom that respond coherently to perturbations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces SpelkeBench, a curated benchmark of 500 (or 550) images with ground-truth segments defined according to the authors' interpretation of Spelke's principles of objecthood, and SpelkeNet, a 7-billion-parameter autoregressive visual world model built on the LRAS framework. SpelkeNet is trained on internet video and standard video/3D datasets to complete optical-flow fields conditioned on an RGB image, a zero camera token, and a sparse 'virtual poke' flow token. The paper defines Spelke objects operationally as groups of pixels that consistently move together across multiple sampled flow completions under different virtual pokes. It evaluates SpelkeNet on SpelkeBench for point-prompted segmentation (Table 1: mIoU 0.681 vs SAM2 0.623) and for automatic discovery (Table 2: AR 0.46, mIoU 0.57 vs SAM2 0.62 and 0.68), and applies the discovered segments to four image-editing pipelines on 3DEditBench (Table 3: higher Edit Adherence in all four cases). The central claims are that Spelke objects can be discovered self-supervised from static images and that they are practically useful for physically plausible manipulation.

Significance. If the physical-grounding claim holds, the paper offers a genuinely useful category-agnostic segmentation basis for manipulation and planning, and the LRAS-based flow-completion architecture with probabilistic counterfactual pokes is a creative and potentially influential design. The introduction of a dedicated benchmark for motion-defined objecthood is a valuable contribution, and the consistent downstream improvements across four independent editing pipelines are encouraging. However, the current evidence is partly self-referential: SpelkeBench annotations are produced by the authors' own Spelke criteria, SpelkeNet's objects are defined through its own flow completions, and no independent physical measurements of co-movement are provided. The point-prompted result is a clear and interesting demonstration of matching the benchmark criterion, but it does not by itself establish physical grounding. The paper would be strengthened substantially by external validation, statistical reporting, and a more honest treatment of the automatic-discovery results.

major comments (5)
  1. [Section 3.1, Section 3.2, Table 1] The central physical-grounding claim is not independently validated. SpelkeNet's objects are defined operationally through the model's own flow completions ('groups of pixels that consistently move together across multiple plausible outcomes of a world model'), and SpelkeBench ground truth is constructed by the authors manually filtering segments according to their own Spelke criteria (Stages 1-3 in Section 3.1). The agreement in Table 1 may therefore reflect self-consistency between the model and the annotators' motion-based intuitions rather than agreement with measured physical co-movement. I would rate the point-prompted result as a strong demonstration of matching the authors' criterion, but not as evidence of mechanical cohesion. Please add external validation—for example, a controlled physical-poke dataset with tracked point trajectories, or independent annotators with inter-annotator agreement statistics—and report results separately for the EntitySeg and OpenX portions of the benchmark.
  2. [Section 7.4 vs Section 3.1] OpenX-Embodiment appears both as a pretraining source (Section 7.4: 'standard video datasets such as Kinetics, SomethingSomethingv2 and OpenX embodiment') and as a source of 50 SpelkeBench images (Section 3.1). The paper does not state that these 50 images were withheld from SpelkeNet's training set. Because the reported gain over SAM2 in Table 1 is modest (mIoU 0.681 vs 0.623), even partial train/eval overlap could inflate the result. Please describe the exact exclusion procedure and report SpelkeBench results both with and without the OpenX subset.
  3. [Tables 1 and 3] No error bars, confidence intervals, or significance tests are reported. The headline comparisons (mIoU 0.681 vs 0.623 on 500 images; EA gaps of 0.081-0.143 across four editing pipelines on 100 scenes) are presented as decisive, but there is no variance estimate. Please add per-image paired statistics, bootstrap confidence intervals, or at least standard errors, so that the robustness of the improvements can be assessed.
  4. [Section 4.2, Table 2] The claim to 'Automatically discovering every Spelke object in a scene' (Section 3.2) is not supported by the reported metrics. In the automatic setting, SpelkeNet has lower AR (0.46 vs 0.62) and lower mIoU (0.57 vs 0.68) than SAM2; it surpasses SAM2 only on AP (0.35 vs 0.11) and F1 (0.38 vs 0.17). The text's explanation that SAM over-segments does not change the fact that SpelkeNet misses a substantially larger fraction of ground-truth Spelke segments. Please improve the iterative clustering/refinement, or explicitly re-scope the contribution as point-prompted segmentation with automatic discovery as a preliminary result.
  5. [Section 3.2, Section 7.4] SpelkeNet is trained on naturally occurring video motion and never observes a localized poke applied to a static scene. The virtual poke is encoded as a sparse flow token appended with a zero-camera token, which is an out-of-distribution intervention; the paper provides no direct evidence that the completed flow reflects mechanical cohesion rather than appearance or semantic priors. I would like a diagnostic that separates these explanations—for example, poking texture boundaries versus physical object boundaries, or comparing predicted flow against real force-response data—to establish that the model is performing causal inference rather than flow hallucination.
minor comments (5)
  1. [Section 4.1] The definition of Average Recall says a ground-truth segment is 'detected if the predicted segment obtains an IoU less than some threshold τ'; this should be 'greater than', otherwise low-overlap predictions would be counted as detections.
  2. [Section 3.2] In the formula for Eseq_disp[ik] = (1/T) Σ_t fhat_t^(seq)[ik]·v_j, the term v_j is not defined in the equation, and the averaging over rollouts appears to refer to a single token rather than the decoded flow vector. Please rewrite with explicit notation, e.g., vhat_t[ik] for the decoded flow vector at location ik in rollout t.
  3. [Section 3.1] The text first describes 'a curated set of 500 images with ground-truth Spelke segment annotations' and then describes manually annotating a subset of 50 OpenX images; Table 1 and the abstract refer to 'our 500-image benchmark'. Please clarify whether SpelkeBench contains 500 or 550 images and describe the OpenX split explicitly.
  4. [Section 7.5] The paper does not state whether model weights, code, or the SpelkeBench annotations will be released. Given that the benchmark is a core contribution, a release statement would improve reproducibility and facilitate independent verification.
  5. [Figure 7 caption] The caption contains the typo 'supplementry'; it should be 'supplementary'.

Circularity Check

2 steps flagged · score 4.0 of 10

Partial circularity: Spelke objects are operationalized as SpelkeNet's own flow-completion aggregates, and SpelkeBench ground truth is the authors' manual application of the same co-movement criterion; the benchmark still uses human annotation and external editing pipelines, so the core comparison is not fully forced.

  1. self definitional [Section 3.2, 'Defining Spelke objects using SpelkeNet']
    "We operationalize Spelke objects as groups of pixels that consistently move together across multiple plausible outcomes of a world model, under different virtual pokes."

    In this sentence the target concept is defined as the statistical output of a world model. SpelkeNet is that world model, and the extraction procedure in the same section computes the segment as an Otsu threshold of the model's own expected-displacement aggregates ('Finally, Otsu thresholding of ¯dot yields our desired Spelke segment'). Therefore SpelkeNet's segments satisfy the operational definition by construction; the statement that SpelkeNet 'discovers' Spelke objects is partly a restatement of the definition. The independent content must come from SpelkeBench, but that benchmark is annotated by the same authors using the same Spelke co-movement criterion, so it does not fully break the self-reference.

  2. other [Section 3.1 / Figure 2 caption]
    "Our SpelkeBench benchmark is constructed by manually filtering out such segments (as described in Section 3.1), retaining only those that correspond to physically grounded entities defined by correlated motion in response to applied forces."

    The evaluation set is curated by the authors using the same notion of correlated motion under applied forces that SpelkeNet operationalizes, rather than by any measured physical response (e.g., actual pokes with observed co-movement). Thus the reported mIoU advantage over SAM (Table 1, 0.681 vs 0.623) measures agreement with the authors' Spelke judgments; since SpelkeNet was trained on natural motion statistics, a high score can reflect shared priors about what moves together instead of verified physical cohesion. This makes the headline comparison partly a self-consistency check, though the human annotation and external editing pipelines prevent it from being a complete tautology.

full rationale

The paper contains a genuinely useful benchmark and a nontrivial self-supervised segmenter; most comparisons against SAM, DINO, and CWM are conducted on fixed datasets with standard metrics. However, the central 'physical grounding' claim is weakened by two self-referential moves. First, the method section defines Spelke objects as the statistical aggregates of a world model's rollout distributions, and SpelkeNet is that world model; the segment extraction (Otsu thresholding of the model's own expected-displacement dot products) therefore matches the operational definition by construction. Second, the SpelkeBench ground truth is manually curated by the authors using the same Spelke co-movement criterion, not by measured physical pokes; the benchmark can validate agreement with the authors' judgment but not independently establish physical correctness. These steps make the headline 'outperforms SAM' partly a self-consistency check. The paper would be substantially less circular if SpelkeBench included inter-annotator agreement, physical poke experiments, or independent labels of observed co-movement, and if the counterfactual poke completions were validated against real force-response data. The self-citations to LRAS [9] and 3DEditBench [9] are not load-bearing because external editing pipelines (LightningDrag, DiffusionHandles, DiffusionAsShader) are also used. Overall score 4: partial circularity in the central definition/evaluation loop, not a fully forced derivation.

Assumptions & free parameters 5 free parameters · 6 assumptions · 1 invented entities

The central claim depends on the training video distribution capturing real physical co-movement, on the zero-camera-pose conditioning isolating object motion, on SeaRAFT flow as a training target, and on human Spelke annotations as a proxy for true physics. These are stated or implied in Sections 3.1, 3.2, and 7.4, but none are validated against direct physical poke measurements. Five hand-set hyperparameters (tau, tau_p, R, T, N) are used without sensitivity analysis.

free parameters (5)
  • tau (motion threshold) = unreported
    Threshold on flow-token magnitude defining Fmotion for the pmotion map (Section 3.2). No sensitivity analysis given.
  • tau_p (poke sampling threshold) = unreported
    Minimum pmotion value for sampling virtual poke locations (Section 3.2). Chosen by hand.
  • R (number of poke directions) = 8 (evaluation)
    Number of diverse virtual pokes per location. Set to 8 in the point-prompted evaluation; no ablation.
  • T (number of rollouts) = 3 (evaluation)
    Number of stochastic sequential flow completions averaged per poke. Set to 3 in evaluation.
  • N (number of poke points for affinity) = unreported
    Number of sampled poke locations in automatic discovery affinity-matrix construction (Section 3.2).
assumptions (6)
  • domain assumption Training on natural videos conveys physical co-movement of objects, so flow completions under pokes are physically meaningful.
    SpelkeNet is trained on BVD and standard video datasets (Section 7.4) and is used as a physical simulator without any poke supervision.
  • domain assumption A zero camera pose token fully attributes input motion to object motion rather than camera motion.
    Invoked in Section 3.2 ('Disentangling object motion from camera motion') to justify static-camera conditioning.
  • domain assumption SeaRAFT optical flow is a sufficiently accurate training target for learning flow completions.
    Optical flow for every training frame pair is computed with SeaRAFT (Section 7.4) and quantized as supervision.
  • domain assumption Human Spelke judgments on static images are a valid proxy for true physical co-movement.
    SpelkeBench ground truth is produced by manual filtering and annotation (Section 3.1) without physical measurement.
  • domain assumption Otsu thresholding of averaged dot-product maps yields correct Spelke segment boundaries.
    Used throughout Section 3.2 for both point-prompted and automatic segmentation; no validation of the thresholding steps.
  • domain assumption Averaging over R pokes and T rollouts gives a faithful estimate of co-movement marginals.
    The statistical counterfactual probing definition in Section 3.2 relies on the convergence of these averages.
invented entities (1)
  • Operationalized Spelke segment (world-model-defined) independent evidence
    purpose: Defines the target concept as groups of pixels that move together across plausible world-model rollouts under virtual pokes.
    The construct is evaluated against human-annotated SpelkeBench rather than against measured physics, but it is not fitted to the benchmark results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Discovering and using Spelke segments." pith.science (2026). https://pith.science/paper/RAF7B3CY

@misc{pith2026250716038,
  author       = {Pith},
  title        = {Pith review of: Discovering and using Spelke segments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RAF7B3CY}},
  note         = {Machine review of arXiv:2507.16038}
}
read the original abstract

Segments in computer vision are often defined by semantic considerations and are highly dependent on category-specific conventions. In contrast, developmental psychology suggests that humans perceive the world in terms of Spelke objects--groupings of physical things that reliably move together when acted on by physical forces. Spelke objects thus operate on category-agnostic causal motion relationships which potentially better support tasks like manipulation and planning. In this paper, we first benchmark the Spelke object concept, introducing the SpelkeBench dataset that contains a wide variety of well-defined Spelke segments in natural images. Next, to extract Spelke segments from images algorithmically, we build SpelkeNet, a class of visual world models trained to predict distributions over future motions. SpelkeNet supports estimation of two key concepts for Spelke object discovery: (1) the motion affordance map, identifying regions likely to move under a poke, and (2) the expected-displacement map, capturing how the rest of the scene will move. These concepts are used for "statistical counterfactual probing", where diverse "virtual pokes" are applied on regions of high motion-affordance, and the resultant expected displacement maps are used define Spelke segments as statistical aggregates of correlated motion statistics. We find that SpelkeNet outperforms supervised baselines like SegmentAnything (SAM) on SpelkeBench. Finally, we show that the Spelke concept is practically useful for downstream applications, yielding superior performance on the 3DEditBench benchmark for physical object manipulation when used in a variety of off-the-shelf object manipulation models.

Figures

Figures reproduced from arXiv: 2507.16038 by the authors.

Figure 1
Figure 1. Overview of SpelkeNet’s capabilities. Our model first predicts a probability of motion map, indicating regions likely to undergo movement independent of camera motion—i.e. candidate movable objects. We sample a point from this map and apply a virtual poke. Conditioned on this intervention, our model completes the flow field. From this, we extract a grouping of pixels, or a “segment” corresponding to an entity that w… view at source ↗
Figure 2
Figure 2. Benchmarking Spelke Segmentation: comparing Spelke segments with [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. SpelkeNet Architecture. The left panel illustrates SpelkeNet—an instance of the LRAS [9] framework applied to the task of optical flow completion for Spelke object discovery. The input consists of a tokenized RGB image ({xk, k ∈ I}) and a sparse virtual poke indicated by a flow token, f. Each token is paired with a pointer token indicating its spatial location, forming a 1D sequence of (pointer, content) pairs. The … view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Spelke object discovery using statistical counterfactual probing [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Automatic discovery of Spelke segments. We extract probability of motion maps from an image, and use it to sample candidate poke points (top left). We apply an optical flow vector “poke” to the image at the sampled points and obtain dense flow fields conditioned on the…
Figure 6
Figure 6. Figure 6: Standard pipeline for object editing using segmentation masks: [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Qualitative results for point-promoted segmentation across models. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: CWM Segmentation failure modes in complex scenes. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Illustration of unprompted Spelke segment discovery using [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Qualitative comparisons of scene edits using SAM masks versus [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Support relationship understanding capabilities. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Emergent material property understanding capabilities. [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: More illustrations of our Spelke segment discovery algorithm [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Additional qualitative results for point-promoted segmentation across [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Additional qualitative comparisons of scene edits using SAM masks versus [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 15 canonical work pages

  1. [1]

    Principles of object perception.Cognitive science, 14(1):29–56, 1990

    Elizabeth S Spelke. Principles of object perception.Cognitive science, 14(1):29–56, 1990

    1990

  2. [2]

    Development of perceptual completion in infancy.Psychological Science, 15 (11):769–775, 2004

    Scott P Johnson. Development of perceptual completion in infancy.Psychological Science, 15 (11):769–775, 2004

    work page 2004

  3. [3]

    The role of objects in perceptual grouping

    Jacob Feldman. The role of objects in perceptual grouping. Acta Psychologica, 102(2-3): 137–163, 1999

    work page 1999

  4. [4]

    Object segmentation by long term analysis of point trajectories

    Thomas Brox and Jitendra Malik. Object segmentation by long term analysis of point trajectories. In European conference on computer vision, pages 282–295. Springer, 2010

    work page 2010

  5. [5]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015. URLhttps://arxiv.org/abs/1405.0312

    arXiv 2015

  6. [6]

    Semantic understanding of scenes through the ade20k dataset, 2018

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset, 2018. URLhttps: //arxiv.org/abs/1608.05442

    arXiv 2018

  7. [7]

    High-quality entity segmentation

    Lu Qi, Jason Kuen, Tiancheng Shen, Jiuxiang Gu, Weidong Guo, Jiaya Jia, Zhe Lin, and Ming-Hsuan Yang. High-quality entity segmentation. InInternational Conference on Computer Vision (ICCV), October 2023

    work page 2023

  8. [8]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.007...

    arXiv 2024

Show all 53 references
  1. [9]

    3d scene understanding through local random access sequence modeling.arXiv preprint arXiv:2504.03875, 2025

    Wanhee Lee, Klemen Kotar, Rahul Mysore Venkatesh, Jared Watrous, Honglin Chen, Khai Loong Aw, and Daniel LK Yamins. 3d scene understanding through local random access sequence modeling.arXiv preprint arXiv:2504.03875, 2025

    2025 arXiv

  2. [10]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  3. [11]

    Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas B...

    2023

  4. [12]

    LVIS: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019

    2019

  5. [13]

    Panoptic segmentation

    Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9404–9413, 2019. 28 SpelkeNet Stanford NeuroAI Lab

    2019

  6. [14]

    A review of robot learning for manipu- lation: Challenges, representations, and algorithms.Journal of machine learning research, 22 (30):1–82, 2021

    Oliver Kroemer, Scott Niekum, and George Konidaris. A review of robot learning for manipu- lation: Challenges, representations, and algorithms.Journal of machine learning research, 22 (30):1–82, 2021

    2021

  7. [15]

    Mask r-cnn, 2017

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn, 2017

    2017

  8. [16]

    Schwing, and Alexander Kirillov

    Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. InProceedings of the conference on Neural Information Processing Systems, 2021

    2021

  9. [17]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023

    2023

  10. [18]

    Segment everything everywhere all at once, 2023

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once, 2023. URL https://arxiv.org/abs/2304.06718

    2023 arXiv

  11. [19]

    One- Former: One Transformer to Rule Universal Image Segmentation

    Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. One- Former: One Transformer to Rule Universal Image Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  12. [20]

    Image segmentation using text and image prompts

    Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7086–7096, June 2022

    2022

  13. [21]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the International Conference on Computer Vision (ICCV), 2021

    2021

  14. [22]

    Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce

    Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and no labels, 2021. URLhttps://arxiv.org/abs/2109.14279

    2021 arXiv

  15. [23]

    Tokencut: Segmenting objects in images and videos with self- supervised transformer and normalized cut, 2023

    Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L Crowley, and Dominique Vaufreydaz. Tokencut: Segmenting objects in images and videos with self- supervised transformer and normalized cut, 2023. URLhttps://arxiv.org/abs/2209.00383

    2023 arXiv

  16. [24]

    Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. Unsupervised semantic segmentation by distilling feature correspondences, 2022. URLhttps: //arxiv.org/abs/2203.08414

    2022 arXiv

  17. [25]

    Groupvit: Semantic segmentation emerges from text supervision, 2022

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision, 2022. URL https://arxiv.org/abs/2202.11094

    2022 arXiv

  18. [26]

    Discovering object masks with transformers for unsupervised semantic segmentation, 2022

    Wouter Van Gansbeke, Simon Vandenhende, and Luc Van Gool. Discovering object masks with transformers for unsupervised semantic segmentation, 2022. URLhttps://arxiv.org/ abs/2206.06363

    2022 arXiv

  19. [27]

    Cut and learn for unsupervised object detection and instance segmentation

    Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3124–3134, 2023. 29 SpelkeNet Stanford NeuroAI Lab

    2023

  20. [28]

    Promerge: Prompt and merge for unsupervised instance segmen- tation

    Dylan Li and Gyungin Shin. Promerge: Prompt and merge for unsupervised instance segmen- tation. In European Conference on Computer Vision (ECCV), 2024

    2024

  21. [29]

    Monet: Unsupervised scene decomposition and representa- tion

    Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint arXiv:1901.11390, 2019

    1901 arXiv

  22. [30]

    Object-centric learning with slot attention

    Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in neural information processing systems, 33:11525–11538, 2020

    2020

  23. [31]

    Conditional object-centric learning from video.arXiv preprint arXiv:2111.12594, 2021

    Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video.arXiv preprint arXiv:2111.12594, 2021

    2021 arXiv

  24. [32]

    Contrastive learning of structured world models

    Thomas Kipf, Elise Van der Pol, and Max Welling. Contrastive learning of structured world models. arXiv preprint arXiv:1911.12247, 2019

    1911 arXiv

  25. [33]

    Bear, Kevin Feigelis, Honglin Chen, Wanhee Lee, Rahul Venkatesh, Klemen Kotar, Alex Durango, and Daniel L

    Daniel M. Bear, Kevin Feigelis, Honglin Chen, Wanhee Lee, Rahul Venkatesh, Klemen Kotar, Alex Durango, and Daniel L. K. Yamins. Unifying (machine) vision via counterfactual world modeling, 2023. URL https://arxiv.org/abs/2306.01828

    2023 arXiv

  26. [34]

    Understanding physical dynamics with counterfactual world modeling

    Rahul Venkatesh, Honglin Chen, Kevin Feigelis, Daniel M Bear, Khaled Jedoui, Klemen Kotar, Felix Binder, Wanhee Lee, Sherry Liu, Kevin A Smith, et al. Understanding physical dynamics with counterfactual world modeling. In European Conference on Computer Vision, pages 368–387. ...

    2024

  27. [35]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  28. [36]

    Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. InInternational conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  29. [37]

    Causalvlbench: Benchmarking visual causal reasoning in large vision-language models.arXiv preprint arXiv:2506.11034, 2025

    Aneesh Komanduri, Karuna Bhaila, and Xintao Wu. Causalvlbench: Benchmarking visual causal reasoning in large vision-language models.arXiv preprint arXiv:2506.11034, 2025

    2025

  30. [38]

    Cello: Causal evaluation of large vision-language models

    Meiqi Chen, Bo Peng, Yan Zhang, and Chaochao Lu. Cello: Causal evaluation of large vision-language models. arXiv preprint arXiv:2406.19131, 2024

    2024 arXiv

  31. [39]

    Draganything: Motion control for anything using entity representation, 2024

    Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation, 2024

    2024

  32. [40]

    Yujun Shi, Jun Hao Liew, Hanshu Yan, Vincent Y. F. Tan, and Jiashi Feng. Lightningdrag: Lightning fast and accurate drag-based image editing emerging from videos, 2024. URL https://arxiv.org/abs/2405.13722

    2024 arXiv

  33. [41]

    Karran Pandey, Paul Guerrero, Metheus Gadelha, Yannick Hold-Geoffroy, Karan Singh, and Niloy J. Mitra. Diffusion handles: Enabling 3d edits for diffusion models by lifting activations to 3d. CVPR, 2024. 30 SpelkeNet Stanford NeuroAI Lab

    2024

  34. [42]

    Transferring foundation models for generalizable robotic manipulation, 2025

    Jiange Yang, Wenhui Tan, Chuhao Jin, Keling Yao, Bei Liu, Jianlong Fu, Ruihua Song, Gangshan Wu, and Limin Wang. Transferring foundation models for generalizable robotic manipulation, 2025. URL https://arxiv.org/abs/2306.05716

    2025 arXiv

  35. [43]

    Diffusion as shader: 3d-aware video diffusion for versatile video generation control, 2025

    Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, Wenping Wang, and Yuan Liu. Diffusion as shader: 3d-aware video diffusion for versatile video generation control, 2025. URLhttps://arxiv.org/abs/2501.03847

    2025 arXiv

  36. [44]

    Open X-Embodiment Collaboration, Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexan- der Khazatsky, Ana...

    arXiv

  37. [45]

    A threshold selection method from gray-level histograms.IEEE Transactions on Systems, Man, and Cybernetics, 9(1):62–66, 1979

    Nobuyuki Otsu. A threshold selection method from gray-level histograms.IEEE Transactions on Systems, Man, and Cybernetics, 9(1):62–66, 1979. doi: 10.1109/TSMC.1979.4310076

    1979

  38. [46]

    SEA-RAFT: Simple, efficient, accurate raft for optical flow

    Yihan Wang, Lahav Lipson, and Jia Deng. SEA-RAFT: Simple, efficient, accurate raft for optical flow. InProceedings of the European Conference on Computer Vision (ECCV), 2024. URL https://arxiv.org/abs/2405.14793

    2024 arXiv

  39. [47]

    The hungarian method for the assignment problem.Naval research logistics quarterly, 2(1-2):83–97, 1955

    Harold W Kuhn. The hungarian method for the assignment problem.Naval research logistics quarterly, 2(1-2):83–97, 1955

    1955

  40. [48]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023 arXiv

  41. [49]

    ScanNet++: A high-fidelity dataset of 3d indoor scenes

    Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. ScanNet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. URL https://arxiv.org/abs/2308.11417. Dataset: ScanNet++

    2023 arXiv

  42. [50]

    Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction

    Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. InProceedings of the IEEE/CVF International Conference on Computer Visio...

    2021 arXiv

  43. [51]

    Stereo magnifi- cation: Learning view synthesis using multiplane images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnifi- cation: Learning view synthesis using multiplane images. InACM SIGGRAPH Conference Proceedings, 2018. URL https://arxiv.org/abs/1805.09817. Dataset: RealEstate-10K

    2018 arXiv

  44. [53]

    SomethingSomething

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz MuellerFreitag, FlorianHoppe, ChristianThurau, IngoBax, andRolandMemisevic. The“SomethingSomething” Video Database for ...

    2017

  45. [2017]

    Dataset: Kinetics-400

    URL https://arxiv.org/abs/1705.06950. Dataset: Kinetics-400

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.