Pith. sign in

REVIEW 4 major objections 6 minor 89 references

Learning segmentation from point trajectories

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Long-term point trajectories, grouped by a loss that suppresses trailing singular values of each segment's trajectory matrix, are enough to supervise a segmentation network and beat prior motion-based methods.

desk verdict A genuinely new low-rank trajectory loss for motion segmentation; the results look strong, but the benchmark numbers need multi-seed runs and code before the SOTA claim holds up. read the letter →

arxiv 2501.12392 v1 pith:OGRUROLR submitted 2025-01-21 cs.CV cs.AIcs.LG

Laurynas Karazija , Iro Laina , Christian Rupprecht , Andrea Vedaldi This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords unsupervisedvideoobjectsegmentationpointtrajectorieslow-ranktrajectorymatrixsingularvaluelossmotioncommonfateopticalflowsubspaceclustering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether long-term point trajectories, rather than only instantaneous optical flow, can supervise an image segmentation network without any manual labels. It proposes a loss, $L_t = \sum_k \sum_{i=r}^{\min(2T,N)} \sigma_i(P_k)$, that drives the trailing singular values of each mask's trajectory matrix to zero, enforcing that points on one object lie near a low-rank subspace. On three standard benchmarks the method outperforms prior motion-based segmentation, reaching 82.2 J on DAVIS 2016, 81.2 J on SegTrackv2, and 79.6 J on FBMS. The point is that long-term motion carries object information that instantaneous flow misses, and the low-rank loss is a practical way to harvest it.

What carries the argument

The central object is the masked trajectory matrix $P_k = P \odot \hat{M}_k$, whose columns are the 2D positions over time of the tracked points assigned to segment $k$. The mechanism is a truncated-SVD loss: instead of reconstructing the matrix, the loss sums the singular values from rank $r$ onward, so minimizing it pushes the group of trajectories toward rank $r$. The paper argues this is more stable than full SVD reconstruction because gradients flow only through the singular values, not through $U$ and $V$, and it enforces low-rankness without insisting on rigid motion. Alongside it, the flow loss and temporal-smoothing loss combine dense instantaneous motion with sparse long-term trajectories.

What would settle it

Take a synthetic video of a non-rigid object whose ground-truth trajectory matrix has rank substantially greater than 5 and compare the loss on the true mask versus masks that merge it with background; if a merged mask has lower or equal loss, the low-rank signal is not selecting the object.

Watch

Extended reading notes

Core claim

The paper's central claim is that a segmentation network can be trained end-to-end from point trajectories, provided the loss is built on the tail singular values of the masked trajectory matrix. For each predicted segment $k$, the loss $L_t(M|P) = \sum_k \sum_{i=r}^{\min(2T,N)} \sigma_i(P_k)$ pushes the $r$-th and later singular values of $P_k$ toward zero, forcing the selected trajectories to lie near an $r$-dimensional subspace; the paper chooses $r=5$ empirically. Under a rigid-body, perspective-camera model the trajectory matrix factorises into camera motion and object geometry, making rank roughly 4, and a slightly higher rank tolerates depth variation and non-rigidity. Combined with a flow-based loss and temporal smoothing, this yields the reported improvements over prior work, including multi-stage methods that use higher resolution and pseudo-labels.

Load-bearing premise

The load-bearing premise is that trajectories of points on one object are well approximated by a rank-5 subspace, so suppressing singular values above rank 5 separates objects; if object motion needs more independent components, or if mixed-object groups can also be low-rank, the loss will not select true objects.

Editorial extensions

If this is right

  • Unsupervised video object segmentation can be improved by adding long-term point trajectories to optical flow, without changing the segmentation architecture.
  • A single end-to-end trained network, without per-video optimisation, pseudo-labelling, or CRF post-processing, can surpass multi-stage methods that use those extra steps.
  • The SVD-tail loss outperforms classical subspace clustering pipelines, suggesting it is a scalable replacement for affinity-matrix construction and spectral clustering.
  • The choice of rank matters: $r=5$ works best on DAVIS, with $r=4$ and $r=6$ both worse, so the method implicitly assumes about five degrees of freedom in the trajectory data.
  • Stronger point trackers translate into stronger segmentation, since replacing the default tracker with alternatives lowers DAVIS J from $78.9$ to values between $73.4$ and $76.8$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test is adaptive rank per segment: if $r$ is chosen per object or per video, articulated and non-rigid objects with more degrees of freedom might be recovered without over-segmenting rigid objects.
  • Because the loss only needs masks and trajectory coordinates, it could be applied to other dense prediction tasks wherever a trajectory-like correspondence can be defined and low-rank groups are expected.
  • The results suggest a stronger statement than the paper makes explicit: long-term motion alone may be sufficient for object discovery, with optical flow acting mainly as a densifier; a test would train on the trajectory loss plus temporal smoothing with no flow term at higher trajectory density.
  • The fixed context window of 41 frames limits temporal integration, so caching trajectories across the full video and handling occlusions explicitly, which the paper flags as limitations, would likely push the numbers further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a self-supervised loss for video object segmentation that uses long-term point trajectories. The core idea is to encourage predicted masks to define groups of trajectories whose masked trajectory matrix has small trailing singular values (Eq. 4), complementing an existing optical-flow parametric loss. The method is evaluated on MOVi-F for feasibility, on DAVIS 2016, SegTrackv2, and FBMS against prior unsupervised methods, and through ablations of the rank r, the number of components k, the context length, the tracker, and the architecture. The authors report DAVIS J=82.2, STv2=81.2, FBMS=79.6, and state that code and models will be released upon acceptance.

Significance. If the results hold, the paper makes a useful contribution: it shows that long-term trajectory information can be injected into an image segmentation network through a simple, architecture-agnostic loss, providing a principled alternative to flow-only losses. The feasibility study with ground-truth trajectories, the systematic ablations (Tables 3, 5-9), and the comparison against subspace clustering are valuable; the paper is also honest about its limitations and about the absence of code and confidence intervals. However, the headline claims are currently supported by a single run per configuration and depend on hyperparameters (r, k, context length, loss weights) selected on the same benchmarks used for the main comparison, so the significance is conditional on additional validation.

major comments (4)
  1. [NeurIPS checklist item 7; Section 5.2, Table 2] The central comparative claims are reported without any measure of variability. The checklist explicitly states that confidence intervals are omitted, and Table 2 is based on a single run per configuration. The claimed DAVIS gains over the flow-only baseline (82.2 vs 78.5) and over RCF (82.2 vs 80.9) are small enough that run-to-run variance could change the conclusion. I ask for multi-seed results with mean and standard deviation (or bootstrap intervals) for the main configuration and the closest baselines, or a clear statement of which results are known to be stable.
  2. [Section 3.2 (Choice of r); Appendix B, Table 5] The rank r=5 is selected empirically on DAVIS, the same benchmark on which the headline number is reported. Table 5 shows a sharp peak at r=5 (82.2) with drops to 79.6 at r=4 and 80.9 at r=6, so the improvement over flow-only (78.5) and over the r=4 variant may be an artifact of benchmark-specific tuning rather than evidence for the low-rank assumption. Please validate the rank choice on a held-out split or on at least one other dataset without retuning, and state the tuning criterion (e.g., grid search on a validation set). The same concern applies to k (Table 6) and context length (Table 7), although those show broader plateaus.
  3. [Section 4, Fig. 3; Eq. (4)] The loss (4) is not by construction an object-selection loss: any grouping whose masked trajectory matrix has small tail singular values will be favored, including a single cluster containing all trajectories in a static or rigidly moving scene, or a grouping that mixes points from different objects with similar motion. The feasibility study shows that under-segmentation is penalized more than over-segmentation (Section 4), but it does not characterize when the global minimum of Eq. (4) corresponds to semantic objects. I would like to see an experiment (e.g., loss values for merge/split perturbations on real sequences with noisy tracks) that addresses this degeneracy, or a discussion of why the combination with the flow loss removes it.
  4. [Section 5.1, Table 1; Appendix E.4] The table caption says the comparison with K-means, SSC, and LRR is 'fair', but the baselines are tuned with oracle knowledge of the number of clusters ('optimal number of clusters determined by an oracle', Appendix E.4), while LRTL uses a fixed k=25 with a randomly initialized U-Net. This asymmetry should be disclosed in the main text, and if possible the baselines should also be evaluated without oracle cluster counts, since the central claim about outperforming subspace clustering depends on this comparison.
minor comments (6)
  1. [Table 4] The row 'λtLf + λtLt' should read 'λf Lf + λt Lt'; the duplicate λt appears to be a typo.
  2. [Section 4, Fig. 3] The text says 'All three plots show the loss value as a function of structural corruption', but the first two plots vary mask noise and temperature, not structural corruption; please rephrase.
  3. [Section 3.3, Eq. (7)] Clarify whether Pt denotes the full trajectory set or the point positions at time t; as written, π(Φ(It), (Pt)t) is ambiguous because Pt is indexed twice.
  4. [Appendix E.3] State explicitly whether every reported number is a single run, and if so, say so in the main text when discussing results.
  5. [Table 3; Section 5.3] Define 'tracks-as-flow' more precisely; Eq. (2) is for optical flow, and it is not immediately clear how T-frame trajectories are converted to per-frame flow without introducing temporal indexing.
  6. [Appendix B, Table 9] Report the number of parameters or FLOPs for the compared segmenters, since the comparison across architectures is otherwise hard to interpret.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity: the low-rank trajectory loss is supervised by measured trajectories and evaluated against independent ground-truth masks, so Eq. (4) does not reduce to its inputs.

full rationale

The central loss, Eq. (4), is L_t(M|P) = sum_k sum_{i=r}^{...} sigma_i(P ⊙ M_k), a differentiable function of the predicted masks M and the measured trajectory matrix P; no ground-truth mask or evaluation target appears in the loss definition, so the supervision is not defined in terms of the quantity being predicted. The low-rank assumption is explicitly stated as a posited inductive bias ('we posit that the set of trajectories should be low-rank') and its validity is tested against independent human-annotated benchmarks (DAVIS, SegTrackv2, FBMS) and synthetic ground truth (MOVi-F), so successful segmentation is not guaranteed by construction. The rank-4 factorization motivating Lper is a standard SVD/factorization identity from [23] and is not used as the main result (Lper scores 18.2 on DAVIS versus 82.2 for Eq. (4)). The author-overlapping inputs are the GWM flow loss/architecture [10] and CoTracker [28]; both are off-the-shelf components with external evaluations and are not invoked as uniqueness theorems or as the justification for the low-rank loss. The rank r=5 is 'empirically determined' on DAVIS and Table 5 shows sensitivity (76.0/79.6/82.2/80.9 for r=3/4/5/6), and the checklist states that error bars and code are not provided; these are reproducibility and robustness concerns about the stability of the 82.2 versus 78.5 flow-only gain, not equation-level circularity. No predicted quantity is reduced to a fitted parameter by construction, and no load-bearing claim rests on a self-citation chain.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a small set of hand-tuned quantities. The most important is the rank r=5 used in Eq. (4), which is set on DAVIS by sweeping (Table 5) and not derived from scene statistics. The number of components k=4 and context window f=20 are also tuned on DAVIS (Tables 6 and 7). Loss weights are set by hand. The assumptions are mostly standard (SVD optimality, common fate) plus domain assumptions about tracker reliability. The method introduces no new physical entities.

free parameters (5)
  • Rank r of trajectory matrix = 5
    Chosen by sweeping r=3..6 on DAVIS (Table 5); main experiments all use r=5.
  • Number of predicted components k = 4
    Chosen on DAVIS via Table 6; k=4 before merging to binary masks.
  • Context window f (trajectory length T = 2f+1) = f = 20 (T = 41)
    Chosen by context length sweep on DAVIS (Table 7).
  • Loss weights lambda_f, lambda_t, lambda_tau = 0.03, 5e-5, 0.1
    Set by hand in Appendix E.3 to put losses in a similar numerical range; no sensitivity analysis is reported.
  • Temporal smoothing offset Delta t = 5
    Chosen in Appendix E.3 without an ablation.
assumptions (5)
  • standard math Eckart-Young theorem: truncating SVD at rank r gives the best rank-r approximation in Frobenius norm.
    Used in Section 3.2 to justify Lrec@r and to argue that minimizing tail singular values lowers reconstruction error.
  • domain assumption Points on the same object have strongly correlated motion (principle of common fate).
    Grounds both the flow and trajectory losses; stated in Sections 1 and 3 as the core inductive bias.
  • domain assumption CoTracker point trajectories are accurate enough, including occluded points, to serve as supervision.
    All real-world experiments rely on CoTracker predictions (Appendix E.2); the paper does not verify trajectory accuracy on real data.
  • domain assumption Perspective camera trajectory matrix is approximately rank 4 under constant projective depth.
    Equation (5) and the surrounding discussion in Section 3.2, with the authors noting that depth is not constant in real settings, so the decomposition is approximate.
  • ad hoc to paper A rank-5 subspace is sufficient to capture coherent object motion despite non-rigidity, parallax, and tracking errors.
    r=5 is chosen empirically on DAVIS, not derived from scene statistics; this is the main modeling choice behind Eq. (4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning segmentation from point trajectories." pith.science (2026). https://pith.science/paper/OGRUROLR

@misc{pith2026250112392,
  author       = {Pith},
  title        = {Pith review of: Learning segmentation from point trajectories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OGRUROLR}},
  note         = {Machine review of arXiv:2501.12392}
}
read the original abstract

We consider the problem of segmenting objects in videos based on their motion and no other forms of supervision. Prior work has often approached this problem by using the principle of common fate, namely the fact that the motion of points that belong to the same object is strongly correlated. However, most authors have only considered instantaneous motion from optical flow. In this work, we present a way to train a segmentation network using long-term point trajectories as a supervisory signal to complement optical flow. The key difficulty is that long-term motion, unlike instantaneous motion, is difficult to model -- any parametric approximation is unlikely to capture complex motion patterns over long periods of time. We instead draw inspiration from subspace clustering approaches, proposing a loss function that seeks to group the trajectories into low-rank matrices where the motion of object points can be approximately explained as a linear combination of other point tracks. Our method outperforms the prior art on motion-based segmentation, which shows the utility of long-term motion and the effectiveness of our formulation.

Figures

Figures reproduced from arXiv: 2501.12392 by the authors.

Figure 1
Figure 1. Illustrative 2D example for the low-rank nature of Pk. A triangle undergoes rigid rotation over three frames. As the rate of rotation is not constant, the flow vectors and point positions are difficult to model. However, the point p is part of the triangle and can be expressed as a combination of the three vertices at an appropriate time. Thus, the last column of Pk is linearly dependent, and Pk is rank deficient. A… view at source ↗
Figure 2
Figure 2. Overview of our approach. We self-supervise a segmentation network, i.e., without access to mask annotations, using both short-term motion information (optical flow) and long-term motion (point trajectories). We design a loss function that encourages the segmentation network to cluster regions where trajectories form low-rank-r groups, which should align well with objects. Off-the-shelf methods are used to estimate … view at source ↗
Figure 3
Figure 3. Feasibility analysis of Lt. Using a synthetic sequence (left), we vary the amount of noise η injected into the mask, the temperature τ of the mask logits and plot the loss value as a function of the mask under/over segmentation. The plots show that the loss is reduced in low-noise, low-entropy settings and penalises both over- and under-segmentation. 5 Experiments In this section, we evaluate our approach for unsupe… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of our results on DAVIS with RCF which uses higher resolution [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison of our results on SegTrackv2 with RCF which uses higher resolution [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Example noise mask alteration. The parameter η is the probability of assigning a mask pixel at random. controlled way to enable studying the effect this has on the loss. For this purpose, we use synthetic data from MOVi-F sequences of the Kubric [20] dataset suite, whi…
Figure 7
Figure 7. Figure 7 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 9
Figure 9. Figure 9: Example structural mask alteration modelling over-segmentation. The parameter s controls the number if objects split into two at random. The three types of alteration are composed to generate a synthetic prediction mask that can be used to investigate how the trajector…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

89 extracted references · 72 canonical work pages

  1. [1]

    Determining three-dimensional motion and structure from optical flow generated by several moving objects

    Gilad Adiv. Determining three-dimensional motion and structure from optical flow generated by several moving objects. IEEE transactions on pattern analysis and machine intelligence, (4):384–401, 1985. 4

    1985

  2. [2]

    Self-supervised object-centric learning for videos

    Görkay Aydemir, Weidi Xie, and Fatma Guney. Self-supervised object-centric learning for videos. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  3. [3]

    It’s moving! a probabilistic model for causal motion segmentation in moving camera videos

    Pia Bideau and Erik Learned-Miller. It’s moving! a probabilistic model for causal motion segmentation in moving camera videos. In European Conference on Computer Vision, pages 433–449. Springer, 2016. 2

    2016

  4. [4]

    The best of both worlds: Combining cnns and geometric constraints for hierarchical motion segmentation

    Pia Bideau, Aruni RoyChowdhury, Rakesh R Menon, and Erik Learned-Miller. The best of both worlds: Combining cnns and geometric constraints for hierarchical motion segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 508–517, 2018. 2

    2018

  5. [5]

    Large displacement optical flow: descriptor matching in variational motion estimation

    Thomas Brox and Jitendra Malik. Large displacement optical flow: descriptor matching in variational motion estimation. IEEE transactions on pattern analysis and machine intelligence, 33(3):500–513, 2010. 8

    2010

  6. [6]

    Object segmentation by long term analysis of point trajectories

    Thomas Brox and Jitendra Malik. Object segmentation by long term analysis of point trajectories. In Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part V 11, pages 282–295. Springer, 2010. 3

    2010

  7. [7]

    One-shot video object segmentation

    Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. One-shot video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 221–230, 2017. 2

    2017

  8. [8]

    Fisher III

    Jason Chang and John W. Fisher III. Topology-constrained layered tracking with latent flow. 2013 IEEE International Conference on Computer Vision, pages 161–168, 2013. 2

    2013

Show all 89 references
  1. [9]

    Non-negative matrix factorization of partial track data for motion segmentation

    Anil M Cheriyadat and Richard J Radke. Non-negative matrix factorization of partial track data for motion segmentation. In 2009 ieee 12th international conference on computer vision, pages 865–872. IEEE, 2009. 3

    2009

  2. [10]

    Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion

    Subhabrata Choudhury, Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion. In British Machine Vision Conference (BMVC), 2022. 2, 3, 4, 6, 7, 8, 14, 17, 18

    2022

  3. [11]

    A multi-body factorization method for motion analysis

    Joao Costeira and Takeo Kanade. A multi-body factorization method for motion analysis. In Proceedings of IEEE International Conference on Computer Vision, pages 1071–1076. IEEE, 1995. 3

    1995

  4. [12]

    A multibody factorization method for independently moving objects

    Joao Paulo Costeira and Takeo Kanade. A multibody factorization method for independently moving objects. International Journal of Computer Vision, 29:159–179, 1998. 5

    1998

  5. [13]

    Motion-inductive self-supervised object discovery in videos

    Shuangrui Ding, Weidi Xie, Yabo Chen, Rui Qian, Xiaopeng Zhang, Hongkai Xiong, and Qi Tian. Motion-inductive self-supervised object discovery in videos. arXiv preprint arXiv:2210.00221, 2022. 2

    2022 arXiv

  6. [14]

    Tap-vid: A benchmark for tracking any point in a video

    Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens Continente, Kucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point in a video. In NeurIPS Datasets Track, 2022. 1, 3

    2022

  7. [15]

    Tapir: Tracking any point with per-frame initialization and temporal refinement

    Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. arXiv preprint arXiv:2306.08637, 2023. 1, 3, 14, 16

    2023 arXiv

  8. [16]

    Bootstap: Bootstrapped training for tracking-any-point

    Carl Doersch, Yi Yang, Dilara Gokay, Pauline Luc, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ross Goroshin, João Carreira, and Andrew Zisserman. Bootstap: Bootstrapped training for tracking-any-point. arXiv preprint arXiv:2402.00847, 2024. 14

    2024 arXiv

  9. [17]

    Sparse subspace clustering: Algorithm, theory, and applications

    Ehsan Elhamifar and René Vidal. Sparse subspace clustering: Algorithm, theory, and applications. IEEE transactions on pattern analysis and machine intelligence, 35(11):2765–2781, 2013. 2, 3, 7, 18 10

    2013

  10. [18]

    Video segmentation by non-local consensus voting

    Alon Faktor and Michal Irani. Video segmentation by non-local consensus voting. In Proceedings of the British Machine Vision Conference. BMV A Press, 2014. 2

    2014

  11. [19]

    Clustering point trajectories with various life-spans

    Matthieu Fradet, Philippe Robert, and Patrick Pérez. Clustering point trajectories with various life-spans. In 2009 Conference for Visual Media Production, pages 7–14. IEEE, 2009. 3

    2009

  12. [20]

    Kubric: A scalable dataset generator

    Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

  13. [21]

    A critique of self-expressive deep subspace clustering

    Benjamin David Haeffele, Chong You, and Rene Vidal. A critique of self-expressive deep subspace clustering. In International Conference on Learning Representations, 2020. 3

    2020

  14. [22]

    Particle video revisited: Tracking through occlusions using point trajectories

    Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vision, pages 59–75. Springer,

  15. [23]

    R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, ISBN: 0521540518, second edition, 2004. 5

    2004

  16. [24]

    Flowformer: A transformer architecture for optical flow

    Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer architecture for optical flow. In European Conference on Computer Vision, pages 668–685. Springer, 2022. 1

    2022

  17. [25]

    Fusionseg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos

    Suyog Jain, Bo Xiong, and Kristen Grauman. Fusionseg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos. arXiv preprint arXiv:1701.05384, 2017. 2

    2017 arXiv

  18. [26]

    Jepson and Michael J

    Allan D. Jepson and Michael J. Black. Mixture models for optical flow computation. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 760–761, 1993. 2

    1993

  19. [27]

    Jojic and B.J

    N. Jojic and B.J. Frey. Learning flexible sprites in video layers. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 1, pages I–I, 2001. 2

    2001

  20. [28]

    Cotracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. arXiv preprint arXiv:2307.07635, 2023. 1, 3, 7, 8, 14, 16, 17

    2023 arXiv

  21. [29]

    Unsuper- vised multi-object segmentation by predicting probable motion patterns

    Laurynas Karazija, Subhabrata Choudhury, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Unsuper- vised multi-object segmentation by predicting probable motion patterns. Advances in Neural Information Processing Systems, 35:2128–2141, 2022. 2

    2022

  22. [30]

    Higher-order minimum cost lifted multicuts for motion segmentation

    Margret Keuper. Higher-order minimum cost lifted multicuts for motion segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pages 4242–4250, 2017. 3

    2017

  23. [31]

    Motion trajectory segmentation via minimum cost multicuts

    Margret Keuper, Bjoern Andres, and Thomas Brox. Motion trajectory segmentation via minimum cost multicuts. In Proceedings of the IEEE international conference on computer vision, pages 3271–3279,

  24. [32]

    A multi-cut formulation for joint segmentation and tracking of multiple objects

    Margret Keuper, Siyu Tang, Yu Zhongjie, Bjoern Andres, Thomas Brox, and Bernt Schiele. A multi-cut formulation for joint segmentation and tracking of multiple objects. arXiv preprint arXiv:1607.06317,

    arXiv

  25. [33]

    Segmenting invisible moving objects

    Hala Lamdouar, Weidi Xie, and Andrew Zisserman. Segmenting invisible moving objects. InBMVC, 2021. 2

    2021

  26. [34]

    Divided attention: Unsupervised multi-object discovery with contextually separated slots

    Dong Lao, Zhengyang Hu, Francesco Locatello, Yanchao Yang, and Stefano Soatto. Divided attention: Unsupervised multi-object discovery with contextually separated slots. arXiv preprint arXiv:2304.01430,

    arXiv

  27. [35]

    Fuxin Li, Taeyoung Kim, Ahmad Humayun, David Tsai, and James M. Rehg. Video segmentation by tracking many figure-ground segments. 2013 IEEE International Conference on Computer Vision, pages 2192–2199, 2013. 2, 7

    2013

  28. [36]

    Instance embedding transfer to unsupervised video object segmentation

    Siyang Li, Bryan Seybold, Alexey V orobyov, Alireza Fathi, Qin Huang, and C-C Jay Kuo. Instance embedding transfer to unsupervised video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6526–6535, 2018. 2 11

    2018

  29. [37]

    Bootstrapping objectness from videos by relaxed common fate and visual grouping

    Long Lian, Zhirong Wu, and Stella X Yu. Bootstrapping objectness from videos by relaxed common fate and visual grouping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14582–14591, 2023. 2, 8, 15

    2023

  30. [38]

    Robust recovery of subspace structures by low-rank representation

    Guangcan Liu, Zhouchen Lin, Shuicheng Yan, Ju Sun, Yong Yu, and Yi Ma. Robust recovery of subspace structures by low-rank representation. IEEE transactions on pattern analysis and machine intelligence, 35 (1):171–184, 2012. 2, 3, 7

    2012

  31. [39]

    Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation

    Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang, Ying Tai, Donghao Luo, Chengjie Wang, Jilin Li, and Feiyue Huang. Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation. In Proceedings of the IEEE/CVF Conference on...

    2020

  32. [40]

    The emergence of objectness: Learning zero-shot segmentation from videos

    Runtao Liu, Zhirong Wu, Stella Yu, and Stephen Lin. The emergence of objectness: Learning zero-shot segmentation from videos. Advances in Neural Information Processing Systems, 34, 2021. 2

    2021

  33. [41]

    Robust and efficient subspace segmentation via least squares regression

    Can-Yi Lu, Hai Min, Zhong-Qiu Zhao, Lin Zhu, De-Shuang Huang, and Shuicheng Yan. Robust and efficient subspace segmentation via least squares regression. In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings...

    2012

  34. [42]

    See more, know more: Unsupervised video object segmentation with co-attention siamese networks

    Xiankai Lu, Wenguan Wang, Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli. See more, know more: Unsupervised video object segmentation with co-attention siamese networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3623–3632...

    2019

  35. [43]

    Multi-subspace representation and discovery

    Dijun Luo, Feiping Nie, Chris Ding, and Heng Huang. Multi-subspace representation and discovery. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part II 22, pages 405–420. Springer...

    2011

  36. [44]

    A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation

    Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and ...

    2016

  37. [45]

    Unsupervised space-time network for temporally-consistent segmentation of multiple motions

    Etienne Meunier and Patrick Bouthemy. Unsupervised space-time network for temporally-consistent segmentation of multiple motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22139–22148, June 2023. 8

    2023

  38. [46]

    Unsupervised motion segmentation in one go: Smooth long-term model over a video

    Etienne Meunier and Patrick Bouthemy. Unsupervised motion segmentation in one go: Smooth long-term model over a video. arXiv preprint arXiv:2310.01040, 2023. 2

    2023 arXiv

  39. [47]

    Em-driven unsupervised learning for efficient motion segmentation

    Etienne Meunier, Anaïs Badoual, and Patrick Bouthemy. Em-driven unsupervised learning for efficient motion segmentation. CoRR, abs/2201.02074, 2022. 2

    2022 arXiv

  40. [48]

    Object segmentation in video: a hierarchical variational approach for turning point trajectories into dense regions

    Peter Ochs and Thomas Brox. Object segmentation in video: a hierarchical variational approach for turning point trajectories into dense regions. In 2011 international conference on computer vision, pages 1583–1590. IEEE, 2011. 2

    2011

  41. [49]

    Higher order motion models and spectral clustering

    Peter Ochs and Thomas Brox. Higher order motion models and spectral clustering. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 614–621. IEEE, 2012. 3

    2012

  42. [50]

    Segmentation of moving objects by long term video analysis

    Peter Ochs, Jitendra Malik, and Thomas Brox. Segmentation of moving objects by long term video analysis. IEEE transactions on pattern analysis and machine intelligence, 36(6):1187–1200, 2013. 3

    2013

  43. [51]

    Segmentation of moving objects by long term video analysis

    Peter Ochs, Jitendra Malik, and Thomas Brox. Segmentation of moving objects by long term video analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36:1187–1200, 2014. 2, 7

    2014

  44. [52]

    Fast object segmentation in unconstrained video

    Anestis Papazoglou and Vittorio Ferrari. Fast object segmentation in unconstrained video. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2013. 2, 8

    2013

  45. [53]

    Perazzi, J

    F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Computer Vision and Pattern Recognition, 2016. 2, 7

    2016

  46. [54]

    A simple and powerful global optimization for unsupervised video object segmentation

    Georgy Ponimatkin, Nermin Samet, Yang Xiao, Yuming Du, Renaud Marlet, and Vincent Lepetit. A simple and powerful global optimization for unsupervised video object segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5892–5903...

    2023

  47. [55]

    Motion segmentation via robust subspace separation in the presence of outlying, incomplete, or corrupted trajectories

    Shankar R Rao, Roberto Tron, René Vidal, and Yi Ma. Motion segmentation via robust subspace separation in the presence of outlying, incomplete, or corrupted trajectories. In 2008 IEEE conference on computer vision and pattern recognition, pages 1–8. IEEE, 2008. 3

    2008

  48. [56]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015. 7, 18

    2015

  49. [57]

    Multi-object discovery by low-dimensional object motion

    Sadra Safadoust and Fatma Güney. Multi-object discovery by low-dimensional object motion. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 734–744, 2023. 2

    2023

  50. [58]

    Simple unsupervised object-centric learning for complex and naturalistic videos

    Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. Advances in Neural Information Processing Systems, 35:18181–18196, 2022. 2

    2022

  51. [59]

    Locate: Self-supervised object discovery via flow-guided graph-cut and bootstrapped self-training

    Silky Singh, Shripad Deshmukh, Mausoom Sarkar, and Balaji Krishnamurthy. Locate: Self-supervised object discovery via flow-guided graph-cut and bootstrapped self-training. In British Machine Vision Conference (BMVC), 2023. 2, 8

    2023

  52. [60]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision (ECCV), 2020. 1, 7, 8, 17

    2020

  53. [61]

    Learning to segment moving objects

    Pavel Tokmakov, Cordelia Schmid, and Karteek Alahari. Learning to segment moving objects. Int. J. Comput. Vision, 127(3):282–301, mar 2019. ISSN 0920-5691. 2

    2019

  54. [62]

    Philip H. S. Torr. Geometric motion segmentation and model selection. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 356:1321 – 1340,

  55. [63]

    Differentiating the singular value decomposition

    James Townsend. Differentiating the singular value decomposition. 2016. URL https://j-towns. github.io/papers/svd-derivative.pdf. 9

    2016

  56. [64]

    Low rank subspace clustering (lrsc)

    René Vidal and Paolo Favaro. Low rank subspace clustering (lrsc). Pattern Recognition Letters, 43:47–61,

  57. [65]

    Videocutler: Surprisingly simple unsupervised video instance segmentation

    Xudong Wang, Ishan Misra, Ziyun Zeng, Rohit Girdhar, and Trevor Darrell. Videocutler: Surprisingly simple unsupervised video instance segmentation. arXiv preprint arXiv:2308.14710, 2023. 2, 3, 15

    2023 arXiv

  58. [66]

    Experimentelle studien uber das sehen von bewegung

    Max Wertheimer. Experimentelle studien uber das sehen von bewegung. Zeitschrift fur psychologie, 61: 161–165, 1912. 1

    1912

  59. [67]

    Segmenting moving objects via an object-centric layered representation

    Jun Xie, Weidi Xie, and Andrew Zisserman. Segmenting moving objects via an object-centric layered representation. ArXiv, abs/2207.02206, 2022. 2, 8

    2022 arXiv

  60. [68]

    A general framework for motion segmentation: Independent, articulated, rigid, non-rigid, degenerate and non-degenerate

    Jingyu Yan and Marc Pollefeys. A general framework for motion segmentation: Independent, articulated, rigid, non-rigid, degenerate and non-degenerate. In Computer Vision–ECCV 2006: 9th European Confer- ence on Computer Vision, Graz, Austria, May 7-13, 2006, Proceedings, Part I...

    2006

  61. [69]

    Self-supervised video object segmentation by motion grouping

    Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7177–7188, 2021. 2, 7, 17

    2021

  62. [70]

    Unsupervised moving object detection via contextual information separation

    Yanchao Yang, Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervised moving object detection via contextual information separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 2, 7

    2019

  63. [71]

    Dystab: Unsupervised object segmentation via dynamic- static bootstrapping

    Yanchao Yang, Brian Lai, and Stefano Soatto. Dystab: Unsupervised object segmentation via dynamic- static bootstrapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2826–2836, 2021. 2

    2021

  64. [72]

    Deformable sprites for unsupervised video decomposition

    Vickie Ye, Zhengqi Li, Richard Tucker, Angjoo Kanazawa, and Noah Snavely. Deformable sprites for unsupervised video decomposition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2657–2666, June 2022. 8

    2022

  65. [73]

    Object-centric learning for real-world videos by predicting temporal feature similarities

    Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-centric learning for real-world videos by predicting temporal feature similarities. Advances in Neural Information Processing Systems, 36, 2024. 2, 15

    2024

  66. [74]

    Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J

    Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J. Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In ICCV, 2023. 14 13 Table 5: Influence of r, the rank of the trajectory matrix used in loss function (4). r DA VIS(J ↑...

    2023

  67. [75]

    We also considered alternative formulations of the trajectories and found them to underperform

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We have confirmed the viability of our loss formulation in controlled simu- lated settings, per-sequence optimisation ...

  68. [76]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: See Section 5.3. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but tho...

  69. [77]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 19 Answer: [NA] Justification: the paper does not include theoretical results. Guidelines: • The answer NA means that the...

  70. [78]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  71. [79]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code 20 Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We do not include code at...

  72. [80]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We include brief summary of k...

  73. [81]

    We report±σ intervals in our feasibility study

    Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: We do not include confidence intervals when reporti...

  74. [82]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We give the computation...

  75. [83]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We make use of publicly available and open-source code and models, respect- ing...

  76. [84]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: See Appendix A. Guidelines: • The answer NA means that there is no societal impact of the work performed....

  77. [85]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: As our key prop...

  78. [86]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the mode...

  79. [87]

    Guidelines: • The answer NA means that the paper does not release new assets

    New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not release new assets. Guidelines: • The answer NA means that the paper does not release new assets. ...

  80. [88]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  81. [89]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.