Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read St4RTrack claims that a single feed-forward network can simultaneously reconstruct dynamic scene geometry and track every visible point in a consistent world coordinate frame from monocular RGB video.

desk verdict A genuinely new pointmap redefinition for joint 4D tracking and reconstruction, with strong results and some honest evaluation gaps that need closing. read the letter →

arxiv 2504.13152 v1 pith:WQLKEVI6 submitted 2025-04-17 cs.CV

Haiwen Feng , Junyi Zhang , Qianqian Wang , Yufei Ye , Pengcheng Yu , Michael J. Black , Trevor Darrell , Angjoo Kanazawa This is my paper · ORCID
classification cs.CV
keywords 4Dreconstructionlong-rangepointtrackingworldcoordinatepointmaprepresentationmonocularvideotest-timeadaptationreprojectionlossdynamicscenes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that dynamic 3D reconstruction and dense 3D point tracking, usually treated as separate tasks, are two sides of the same coin once scene motion is accounted for, and it backs that claim with one network: St4RTrack takes a pair of frames and, in a single feed-forward pass, outputs both the second frame's geometry and where every pixel of the first frame has moved, expressed in one shared world coordinate frame. Repeating this pairing against a fixed first frame chains the predictions into long-range world-frame 3D tracks and per-frame reconstruction simultaneously. Because the predicted geometry also yields camera poses through a differentiable PnP step, the same architecture can adapt to new videos using only 2D trajectory and monocular depth pseudo-labels, with no 4D ground truth. The paper also introduces WorldTrack, a benchmark for world-frame 3D tracking, and reports that St4RTrack outperforms the combined baselines on both tracking and reconstruction.

What carries the argument

The load-bearing object is the time-dependent pointmap, a per-pixel 3D map stamped with three factors: whose content it describes, which camera frame expresses it, and at which timestamp. St4RTrack realizes Eq. (1) with a DUSt3R-style siamese transformer whose two branches share weights but not roles: the tracking branch outputs $^{1}\mathbf{X}^{1}_{j}$ (frame 1's content moved to frame $j$'s moment, in frame 1's world) and the reconstruction branch outputs $^{1}\mathbf{X}^{j}_{j}$ (frame $j$'s geometry in the same world), with alternating self- and cross-attention letting the tracking branch borrow geometric cues from the reconstruction branch. Pairing the anchor frame with every later frame and chaining the outputs turns these pairwise predictions into a full sequence. The second mechanism is the adaptation loop: a differentiable Gauss-Newton PnP step recovers camera extrinsics from the reconstruction pointmap, then a scale-invariant reprojection loss against 2D tracks and monocular depth, plus a 3D self-consistency term between the two branches, fine-tunes the model on the target video without any 4D labels.

What would settle it

A decisive check is a WorldTrack-style sequence with large camera translation between the anchor frame and a distant later frame plus a complete occlusion of a tracked region: the paper's own Discussion flags this regime as one where scale misalignment and occlusions are not fully resolved, so if the predicted world-frame tracks of the occluded points jump or drift when they reappear, the pairwise chaining of Eq. (1) has failed to carry correspondence through the gap. On the quantitative side, dynamic-point APD3D on the Point Odyssey split should not fall faster than the reconstruction branch's own error as the frame gap grows from 8 to 64; a sudden drop in tracking accuracy with gap size would expose the pairwise observability assumption as the load-bearing premise.

Watch

Extended reading notes

Core claim

The central claim is that the representation in Eq. (1) suffices: a learned function $f_\theta(I_i, I_j) = (\,^{i}\mathbf{X}^{i}_{j},\,^{i}\mathbf{X}^{j}_{j})$ maps any pair of frames from a dynamic video into two pointmaps in one shared coordinate frame. Here the tracking pointmap $^{i}\mathbf{X}^{i}_{j}$ records where the content visible in frame $i$ is located at time $j$, expressed in frame $i$'s coordinates, while the reconstruction pointmap $^{i}\mathbf{X}^{j}_{j}$ records frame $j$'s own geometry in those same coordinates. Both pointmaps are thus at the same timestamp and in the same world, which is what makes geometry and correspondence dual again: once the network has accounted for motion, the dynamic problem falls back to the rigid case. Anchoring on the first frame, $\{f(I_1,I_1), f(I_1,I_2), \ldots, f(I_1,I_T)\}$ yields the world-frame tracks $\{^{1}\mathbf{X}^{1}_{1}, \ldots, ^{1}\mathbf{X}^{1}_{T}\}$ and the per-frame geometry $\{^{1}\mathbf{X}^{1}_{1}, ^{1}\mathbf{X}^{2}_{2}, \ldots, ^{1}\mathbf{X}^{T}_{T}\}$, and the paper reports that this unified prediction beats the best available combinations of separate trackers and reconstructors on WorldTrack, and improves dynamic reconstruction on Point Odyssey and TUM-Dynamics.

Load-bearing premise

From just the two images in a pair, the network must recover where every pixel of the first frame has moved by the second frame's moment, even when the camera has moved far, points are occluded, or the temporal gap is large, and the paper's own discussion concedes that scale misalignment, large camera movements, and occlusions are not fully resolved.

Editorial extensions

If this is right

  • Dense 3D reconstruction and long-range point tracking become a single prediction, so improvements to the representation or the backbone improve both tasks at once rather than two separate pipelines.
  • Any monocular video, including footage with no ground-truth 3D or motion data, becomes usable training material, since the reprojection losses supply supervision from 2D tracks and monocular depth.
  • A consumer RGB video can yield a consistent world coordinate frame with per-frame camera poses, scene geometry, and dense 3D trajectories in one feed-forward pass, which is the input a downstream dynamic-view-synthesis or robotics system would need.
  • The WorldTrack benchmark gives the field a way to measure world-frame 3D tracking accuracy, and the paper's reported numbers on Panoptic Studio show the method also holds up when the camera is static.
  • During test-time adaptation only the tracking branch is fine-tuned; the paper reports that this preserves the view-alignment capability while correcting drifting points, making per-sequence adaptation converge in minutes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the pairwise chaining is as robust as reported, adding temporal attention across frames, which the paper lists as future work, should push hardest on exactly the failure modes it names: scale misalignment, large camera movement, and occlusion, where the pairwise assumption is strained.
  • The scale-invariant reprojection losses inherit the accuracy of their pseudo-labels (CoTracker3 for 2D tracks, MoGe for depth); replacing those priors with stronger monocular estimators as they appear should improve the world-frame 3D tracks without any architectural change, a claim the paper does not make.
  • World-frame tracking separates camera motion from scene motion, so St4RTrack's output could seed optimization-based 4D reconstruction methods that currently optimize from scratch per video, potentially replacing their expensive initialization with a feed-forward prior.
  • Because the tracking pointmap is expressed in the anchor frame's coordinates by construction, the representation is naturally reversible: pairing the last frame as anchor would yield tracks backwards in time, which could serve video editing or re-timing applications that the paper does not discuss.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes St4RTrack, a feed-forward framework for simultaneous dense 3D reconstruction and point tracking in a single world coordinate frame from monocular RGB video. For a pair of frames (Ii, Ij), the network predicts two pointmaps in the coordinate system defined by frame i, both evaluated at time j: the tracking pointmap iXi_j, which gives the 3D positions at time j of the content visible in frame i, and the reconstruction pointmap iXj_j, which gives the geometry of frame j. Chaining the first frame with every later frame yields long-range 3D point tracks and a per-frame reconstruction aligned to the first frame's coordinates. The model is initialized from MASt3R, pretrained on synthetic 4D datasets (Point Odyssey, Dynamic Replica, Kubric) with dense supervision, and refined at test time through a reprojection loss that uses CoTracker3 2D trajectories and MoGe monocular depth, with camera poses obtained via differentiable PnP. The authors introduce a new benchmark, WorldTrack, covering ADT, Panoptic Studio, Point Odyssey, and Dynamic Replica (50 sequences of 64 frames each), and report state-of-the-art results on world-frame 3D tracking (Table 1) and dynamic 3D reconstruction (Table 2), together with ablations of the test-time adaptation components.

Significance. If the results hold, this is a valuable contribution. The representation is minimal and clean: a small modification of the DUSt3R pointmap formulation unifies two traditionally separate tasks, and the proposed reprojection-based test-time adaptation provides a practical path toward self-supervised 4D training without dense 4D labels. The empirical gains over combinational baselines are consistent and large across multiple datasets, and the reconstruction branch also improves over MonST3R and DUSt3R. The new WorldTrack benchmark addresses a real gap in the literature, since prior benchmarks only evaluate tracking in camera coordinates. The architecture is simple, the pairwise inference is efficient (30 FPS on an RTX 4090 is claimed), and the authors plan to release code, models, and the benchmark. The paper is generally clearly written and the self-assessment in Section 5 is candid.

major comments (4)
  1. [Section 4.2 / Eq. (12), Table 1] The APD3D metric as written in Eq. (12) is defined for a single threshold delta_3D and does not state whether it is evaluated over all query points or only those visible in each target frame. This is load-bearing because the two-frame mapping in Eq. (1) is underdetermined for points that are occluded or out of view at time j, and Section 5 explicitly concedes that occlusions and large camera movements are not fully resolved. If the reported scores include occluded or out-of-view points, the headline claim of long-range dense world-frame tracking is not established outside the visible/overlapping-frame regime; if the TAPVid-3D protocol does mask these points, that must be stated explicitly. I request that the authors specify the visibility handling in the evaluation and, ideally, report performance separately for visible and occluded/out-of-view points, along with a breakdown of the fraction of evaluated points that are occluded per dataset.
  2. [Section 4.2 / Table 1] MonST3R is listed as a feed-forward tracking baseline in Table 1, but MonST3R is a dense reconstruction method that does not by itself output temporal correspondences. The paper never describes how the pointmap predictions of MonST3R are converted into the point tracks used to compute APD3D and EPE. Without this explanation, the 'MonST3R' row in the tracking tables is not reproducible and the reader cannot assess whether this non-tracking baseline is evaluated in a fair manner. Please provide a precise description of the procedure used to generate MonST3R tracks, including any matching or nearest-neighbor steps.
  3. [Section 3.2 / Section 4.1] The synthetic 4D supervision for the tracking branch is underspecified. The text states that scene mesh vertices provide 'sparse, masked supervision' for the tracking pointmap iX1_j, but it does not explain how world-space mesh vertices at time j are associated with pixels of frame 1, nor how the mask is defined. This supervision is the only source that teaches the network the time-dependent pointmap representation, so the training loss construction must be described in enough detail to be reproduced. Please clarify the exact loss terms, the masking, and how correspondences between frame-1 pixels and mesh vertices at other timestamps are obtained.
  4. [Section 4.1 vs. Appendix C.1] The test-time adaptation protocol is described inconsistently. Section 4.1 states 'we run 500 optimization steps on a single sequence,' while Appendix C.1 specifies instance-level adaptation as training for 3 epochs on 300 sampled frames per epoch with batch size 4, which corresponds to a different number of gradient steps (75 steps per epoch, i.e., 225 total). The discrepancy between these descriptions must be resolved, and the main text should state the exact number of adaptation steps and the batch size used to produce the Table 1 results.
minor comments (6)
  1. [Eq. (12), Table 1 caption] The APD3D definition in Eq. (12) is a function of a single threshold delta_3D, but Table 1 reports one number per cell. Please state that the reported value is the average over the threshold set {0.1, 0.3, 0.5, 1.0} m and, if applicable, over time steps and query points; the caption phrase 'average points under distance' should be 'average percentage of points within distance.'
  2. [Section 4.2, baselines paragraph] MonST3R is described as a 'dynamic SLAM method' in the description of the second combinational baseline, but MonST3R is a feed-forward dense 3D reconstruction method, not an optimization-based SLAM system. Please correct this characterization to avoid confusion.
  3. [Section 3.1, notation] The shorthand Xi_j := iXi_j is introduced after Eq. (3), but later equations and paragraphs do not consistently restate the left superscript. A short 'notational conventions' note listing the meaning of the superscript, left subscript, and right subscript would improve readability.
  4. [Section 4.5 / Table 6] The ablation of the pretraining stage is only qualitative in the main text; Table 6 provides quantitative results but only on DR and PStudio and only for the test-time-adaptation variants. Consider reporting a quantitative feed-forward ablation on the full WorldTrack benchmark so the contribution of each component is measured on all four datasets.
  5. [Table 6 caption] The rows labeled 'w/o trajectory loss', 'w/o depth loss', etc. should specify that they refer to the St4RTrack + TTA configuration with that loss component removed. As written, the baseline for these ablations is ambiguous.
  6. [Abstract / Section 4.2] The abstract claims 'dense 3D point tracks,' but the WorldTrack evaluation downsamples query points to approximately 1,000 per sequence. Please clarify whether the reported metrics are computed on the sampled queries or on the full dense pointmaps, and adjust the wording if the evaluation is on a sparse subset.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: St4RTrack's pointmap prediction is a learned empirical mapping with external synthetic and off-the-shelf supervision; the Sec. 5 limitation is an identifiability caveat, not a circular reduction.

full rationale

The claimed result is an empirical feed-forward mapping fθ(Ii,Ij)=(iXi_j,iXj_j) learned from synthetic 4D supervision and reprojection losses; no predicted quantity is constructed from its own target. Eq. (1) defines the task and representation rather than deriving an output from a fitted constant: the tracking pointmap iXi_j is the 3D position of frame-i content at time j, and this is what the network is trained to regress, not a function of the benchmark labels. Pretraining uses ground-truth mesh, depth, and camera from external synthetic datasets (Point Odyssey, Dynamic Replica, Kubric). Test-time adaptation uses pseudo-labels from independent off-the-shelf models (CoTracker3 for 2D tracks, MoGe for monocular depth); the reconstruction branch is frozen, and camera pose is solved from the frozen reconstruction pointmap via differentiable PnP, so the tracking branch is not supervised with its own output. The reprojection and self-consistency losses (Eqs. 5-11) are training objectives, not reported predictions. Evaluation on WorldTrack is against external baselines under a fixed alignment protocol; the only fitted quantities are hand-chosen loss weights and eval-time global median/Sim(3) alignment, which are not presented as derived results. The paper's Sec. 5 statement that 'scale misalignment, large camera movements, and occlusions are not fully resolved' is an identifiability/robustness caveat about the two-frame observability assumption, not a circular reduction. Self-citations to DUSt3R, MonST3R, and MASt3R are architectural lineage and initialization sources; they are not invoked as uniqueness theorems and do not carry the argument.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method is an empirical neural system. The central claim rests on the validity of the pseudo-labels used during adaptation, the transferability of synthetic pretraining, and the observability of anchor-frame content in later frames. No invented physical entities are introduced.

free parameters (2)
  • Loss weights in TTA reprojection loss = lambda_traj=1, lambda_depth=10, lambda_align=5
    Chosen by hand in Eq. (11); the paper's ablations (Table 6) show each component helps, but no tuning analysis is given.
  • Per-sequence scale factors in TTA losses = s in Eq. (6), alpha* in Eq. (9)
    Computed per sample by closed-form least squares to align predictions with pseudo-labels; they are not learned model parameters but are fit at adaptation time and affect the loss landscape.
assumptions (4)
  • domain assumption CoTracker 3 and MoGe provide reliable pseudo-labels for test-time adaptation
    The reprojection losses L_traj and L_depth (Eq. 7, 9) treat these off-the-shelf outputs as ground truth when adapting to new videos; if their errors are large, the adapted model inherits them.
  • domain assumption Synthetic 4D datasets (Point Odyssey, Dynamic Replica, Kubric) transfer to real dynamic scenes
    Pretraining (Sec. 4.1) uses only synthetic meshes and renders; the paper's own Discussion (Sec. 5) acknowledges limited diversity and realism, motivating TTA.
  • domain assumption Anchor-frame pairing retains enough observability of frame 1 content in later frames
    Eq. (1) defines f on a single pair (I1,Ij) and tracking chains these pairs; the method assumes frame 1 content can be located at time j, which fails for large camera motion or occlusion (Sec. 5).
  • standard math Differentiable PnP (EPro-PnP) provides correct pose gradients
    Used in Sec. 3.3 and App. A to backpropagate through camera pose via one Gauss-Newton step from a detached RANSAC-PnP solution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World." pith.science (2026). https://pith.science/paper/WQLKEVI6

@misc{pith2026250413152,
  author       = {Pith},
  title        = {Pith review of: St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WQLKEVI6}},
  note         = {Machine review of arXiv:2504.13152}
}
read the original abstract

Dynamic 3D reconstruction and point tracking in videos are typically treated as separate tasks, despite their deep connection. We propose St4RTrack, a feed-forward framework that simultaneously reconstructs and tracks dynamic video content in a world coordinate frame from RGB inputs. This is achieved by predicting two appropriately defined pointmaps for a pair of frames captured at different moments. Specifically, we predict both pointmaps at the same moment, in the same world, capturing both static and dynamic scene geometry while maintaining 3D correspondences. Chaining these predictions through the video sequence with respect to a reference frame naturally computes long-range correspondences, effectively combining 3D reconstruction with 3D tracking. Unlike prior methods that rely heavily on 4D ground truth supervision, we employ a novel adaptation scheme based on a reprojection loss. We establish a new extensive benchmark for world-frame reconstruction and tracking, demonstrating the effectiveness and efficiency of our unified, data-driven framework. Our code, model, and benchmark will be released.

Figures

Figures reproduced from arXiv: 2504.13152 by the authors.

Figure 1
Figure 1. St4RTrack: Given an RGB video capturing dynamic scenes, St4RTrack simultaneously tracks the points from the initial frame (visualized in purple) and reconstructs the geometry of the subsequent frames (in orange) in a consistent world coordinate frame. St4RTrack is a feed-forward framework that takes a pair of images as input and outputs two pointmaps in the world frame, as the visualization shown in the middle. By i… view at source ↗
Figure 2
Figure 2. Pointmap Comparison of MonST3R and St4RTrack. Given two input frames, MonST3R handles dynamic scenes by reconstructing both pointmaps in their own timestamp. St4RTrack predicts where the points in the first frame move in the second frame, and reconstructs the geometry of the second frame. More details of the representation definition are introduced in Sec. 3.1. over extended time periods and overcoming these limita￾… view at source ↗
Figure 3
Figure 3. Overview of St4RTrack. Given frame 1 and frame j as input, the tracking branch outputs 1X1 j , the pointmap that corresponds to observed content of the first frame at timestep j in its own camera coordinate (i.e. world coordinate); the reconstruction branch outputs 1X j j , the pointmap of the content in frame j at its own timestamp in the world coordinate. To adapt to new videos without any 4D labels, the camera is… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Qualitative Results. From left to right, we show our results in feed-forward inference: 1) the input video, 2) two pointmaps at frame j overlayed together, 3) the accumulated reconstruction branch result, and 4) the accumulated tracking branch result. The ac￾cumulated …
Figure 5
Figure 5. Figure 5: Ablation Study. We show the qualitative comparison of our full method and variants that do not pretrain or do not adapt in test time. Predicted pointmaps from two heads are visualized together. Results. As shown in Tab. 1, we achieve state-of-the-art performance, with …
Figure 6
Figure 6. Figure 6: Reconstruction Results of St4RTrack on TUM-Dynamics Dataset. From left to right, we show 1) the sampled frames from the input sequence of 64 frames, 2) the subsampled ground truth pointmaps, 3) the predicted pointmaps of our method, and 4) the aligned results of the pr…
Figure 7
Figure 7. Figure 7: Reconstruction Results of St4RTrack on Point Odyssey Dataset. From left to right, we show 1) the sampled frames from the input sequence of 64 frames, 2) the subsampled ground truth pointmaps, 3) the predicted pointmaps of our method, and 4) the aligned results of the p…
Figure 8
Figure 8. Figure 8: Tracking Results of St4RTrack on WorldTrack Benchmark. We show the results of the predicted tracks aligned with the ground truth tracks, visualized in 2D and 3D. The corresponding datasets are Point Odyssey (top left), Dynamic Replica (top right), Arial Digital Twin (b…
Figure 9
Figure 9. Figure 9: Fully Feed-Forward Inference Results of St4RTrack. We show from left to right: 1) the input video, 2) the pairwise output for tracking (in blue) and reconstruction (in yellow) of the same frame, 3) the accumulated results of the reconstruction pointmaps, and 4) the acc…
Figure 10
Figure 10. Figure 10: Test-Time Adaptation Results of St4RTrack. From left to right, we show 1) the input video, 2) the pairwise output for tracking (in blue) and reconstruction (in yellow) of the same frame, 3) the accumulated results of the reconstruction pointmaps, and 4) the accumulate…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. 4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time

    cs.CV 2025-06 conditional novelty 7.0 of 10

    4D-LRM is a transformer that maps sparse posed frames scattered across time to a cloud of 4D Gaussians and renders any query view at any query time in under 1.5 seconds.

  2. UAV4D: Dynamic Neural Rendering of Human-Centric UAV Imagery using Gaussian Splatting

    cs.CV 2025-06 conditional novelty 5.0 of 10

    UAV4D reconstructs 4D scenes from monocular drone video by fitting a single global scale to align human meshes with the background mesh, then renders with separate Gaussian splats.

  3. Advances in 4D Representation: Geometry, Motion, and Interaction

    cs.CV 2025-10 conditional novelty 4.0 of 10

    A representation-centric survey of 4D generation and reconstruction, organized by geometry, motion, and interaction, with qualitative trade-off comparisons across seven representation families.

  4. Reconstructing 4D Spatial Intelligence: A Survey

    cs.CV 2025-07 accept novelty 4.0 of 10

    A review that classifies 4D scene reconstruction methods into five progressive levels: low-level cues, scene components, dynamic scenes, interactions, and physics.

Reference graph

Works this paper leans on

73 extracted references · 45 canonical work pages · cited by 4 Pith papers

  1. [1]

    Building rome in a day

    Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Si- mon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome in a day. Communications of the ACM , 54 (10):105–112, 2011. 2

    2011

  2. [2]

    Zoedepth: Zero-shot trans- fer by combining relative and metric depth

    Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias M ¨uller. Zoedepth: Zero-shot trans- fer by combining relative and metric depth. arXiv preprint arXiv:2302.12288, 2023. 2

    arXiv 2023

  3. [3]

    A framework for the robust estimation of optical flow

    Michael J Black and Padmanabhan Anandan. A framework for the robust estimation of optical flow. In 1993 (4th) In- ternational Conference on Computer Vision, pages 231–236. IEEE, 1993. 2

    work page 1993

  4. [4]

    Recovering non-rigid 3d shape from image streams

    Christoph Bregler, Aaron Hertzmann, and Henning Bier- mann. Recovering non-rigid 3d shape from image streams. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662), pages 690–696. IEEE, 2000. 3

    2000

  5. [5]

    EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation

    Hansheng Chen, Wei Tian, Pichao Wang, Fan Wang, Lu Xiong, and Hao Li. Epro-pnp: Generalized end-to-end prob- abilistic perspective-n-points for monocular object pose esti- mation. arXiv preprint arXiv:2303.12787, 2023. 5, 13

    work page Pith review arXiv 2023

  6. [6]

    A simple prior- free method for non-rigid structure-from-motion factoriza- tion

    Yuchao Dai, Hongdong Li, and Mingyi He. A simple prior- free method for non-rigid structure-from-motion factoriza- tion. International Journal of Computer Vision , 107:101– 122, 2014. 3

    work page 2014

  7. [7]

    Monoslam: Real-time single camera slam

    Andrew J Davison, Ian D Reid, Nicholas D Molton, and Olivier Stasse. Monoslam: Real-time single camera slam. IEEE transactions on pattern analysis and machine intelli- gence, 29(6):1052–1067, 2007. 2

    2007

  8. [8]

    TAP-vid: A benchmark for track- ing any point in a video

    Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Re- casens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. TAP-vid: A benchmark for track- ing any point in a video. Advances in Neural Information Processing Systems, 35:13610–13626, 2022. 2

    work page 2022

Show all 73 references
  1. [9]

    TAPIR: Tracking any point with per-frame initialization and temporal refinement

    Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. TAPIR: Tracking any point with per-frame initialization and temporal refinement. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pag...

    2023

  2. [10]

    Simultaneous local- ization and mapping: part i

    Hugh Durrant-Whyte and Tim Bailey. Simultaneous local- ization and mapping: part i. IEEE robotics & automation magazine, 13(2):99–110, 2006. 2

    2006

  3. [11]

    Martin A Fischler and Robert C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.Communications of the ACM, 24(6):381–395, 1981. 5, 7

    1981

  4. [12]

    Grouping-based low-rank trajectory comple- tion and 3d reconstruction

    Katerina Fragkiadaki, Marta Salas, Pablo Arbelaez, and Ji- tendra Malik. Grouping-based low-rank trajectory comple- tion and 3d reconstruction. advances in neural information processing systems, 27, 2014. 3

    2014

  5. [13]

    K-planes: Explicit radiance fields in space, time, and appearance

    Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 12479–12488, 2023. 3

    2023

  6. [14]

    Monocular dynamic view synthesis: A reality check

    Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dynamic view synthesis: A reality check. Advances in Neural Information Processing Systems, 35:33768–33780, 2022. 3

    2022

  7. [15]

    Kubric: A scalable dataset generator

    Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapra- gasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF conference on computer vision and pattern re...

    2022

  8. [16]

    Particle video revisited: Tracking through occlusions using point trajectories

    Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vi- sion, pages 59–75. Springer, 2022. 2

    2022

  9. [17]

    Multiple view ge- ometry in computer vision

    Richard Hartley and Andrew Zisserman. Multiple view ge- ometry in computer vision . Cambridge university press,

  10. [18]

    Determining opti- cal flow

    Berthold KP Horn and Brian G Schunck. Determining opti- cal flow. Artificial intelligence, 17(1-3):185–203, 1981. 2

    1981

  11. [19]

    Self-supervised monocular scene flow estimation

    Junhwa Hur and Stefan Roth. Self-supervised monocular scene flow estimation. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 7396–7405, 2020. 2

    2020

  12. [20]

    Stereo4d: Learning how things move in 3d from internet stereo videos.arXiv preprint arXiv:2412.09621, 2024

    Linyi Jin, Richard Tucker, Zhengqi Li, David Fouhey, Noah Snavely, and Aleksander Holynski. Stereo4d: Learning how things move in 3d from internet stereo videos.arXiv preprint arXiv:2412.09621, 2024. 3

    2024 arXiv

  13. [21]

    Panoptic studio: A massively multiview system for social interaction capture, 2016

    Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Godisart, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social interaction capture, 2016. 7

    2016

  14. [22]

    Dy- namicstereo: Consistent dynamic depth from stereo videos

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Dy- namicstereo: Consistent dynamic depth from stereo videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13229–13239, 2023...

    2023

  15. [23]

    Co- Tracker3: Simpler and better point tracking by pseudo- labelling real videos

    Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- Tracker3: Simpler and better point tracking by pseudo- labelling real videos. 2024. 2, 5, 6

    2024

  16. [24]

    Co- tracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker: It is better to track together. In Proc. ECCV, 2024. 1, 2, 5

    2024

  17. [25]

    Ro- bust consistent video depth estimation

    Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Ro- bust consistent video depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1611–1621, 2021. 2

    2021

  18. [26]

    Tapvid-3d: A benchmark for tracking any point in 3d, 2024

    Skanda Koppula, Ignacio Rocco, Yi Yang, Joe Heyward, Jo˜ao Carreira, Andrew Zisserman, Gabriel Brostow, and Carl Doersch. Tapvid-3d: A benchmark for tracking any point in 3d, 2024. 7, 13

    2024

  19. [27]

    Harley, Leonidas Guibas, and Kostas Daniilidis

    Jiahui Lei, Yijia Weng, Adam W. Harley, Leonidas Guibas, and Kostas Daniilidis. MoSca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds. arXiv preprint arXiv:2405.17421, 2024. 3, 6 10

    2024 arXiv

  20. [28]

    EPnP: An accurate O(n) solution to the PnP problem

    Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. EPnP: An accurate O(n) solution to the PnP problem. IJCV, 81:155–166, 2009. 5

    2009

  21. [29]

    Ground- ing image matching in 3d with mast3r

    Vincent Leroy, Yohann Cabon, and J´erˆome Revaud. Ground- ing image matching in 3d with mast3r. In European Confer- ence on Computer Vision, pages 71–91. Springer, 2024. 5

    2024

  22. [30]

    Neural 3d video synthesis from multi-view video

    Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF conference on computer vi- si...

    2022

  23. [31]

    Neural scene flow fields for space-time view synthesis of dy- namic scenes

    Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dy- namic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6498– 6508, 2021. 3

    2021

  24. [32]

    Dynibar: Neural dynamic image-based rendering

    Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4273– 4284, 2023. 3

    2023

  25. [33]

    MegaSaM: Accurate, fast, and robust structure and motion from casual dynamic videos

    Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holyn- ski, and Noah Snavely. MegaSaM: Accurate, fast, and robust structure and motion from casual dynamic videos. In Pro- ceedings of the IEEE/CVF Conference on Comput...

    2025

  26. [34]

    Modgs: Dy- namic gaussian splatting from casually-captured monocular videos, 2024

    Qingming Liu, Yuan Liu, Jiepeng Wang, Xianqiang Lyv, Peng Wang, Wenping Wang, and Junhui Hou. Modgs: Dy- namic gaussian splatting from casually-captured monocular videos, 2024. 3

    2024

  27. [35]

    An iterative image reg- istration technique with an application to stereo vision

    Bruce D Lucas and Takeo Kanade. An iterative image reg- istration technique with an application to stereo vision. In IJCAI’81: 7th international joint conference on Artificial in- telligence, pages 674–679, 1981. 2

    1981

  28. [36]

    Dynamic 3d gaussians: Tracking by per- sistent dynamic view synthesis, 2023

    Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by per- sistent dynamic view synthesis, 2023. 3

    2023

  29. [37]

    Orb-slam: A versatile and accurate monocular slam system

    Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: A versatile and accurate monocular slam system. IEEE transactions on robotics , 31(5):1147–1163,

  30. [38]

    DELTA: Dense efficient long-range 3d tracking for any video

    Tuan Duc Ngo, Peiye Zhuang, Chuang Gan, Evange- los Kalogerakis, Sergey Tulyakov, Hsin-Ying Lee, and Chaoyang Wang. DELTA: Dense efficient long-range 3d tracking for any video. In International Conference on Learning Representations (ICLR), 2025. 1, 3

    2025

  31. [39]

    C3dpo: Canonical 3d pose networks for non-rigid structure from motion

    David Novotny, Nikhila Ravi, Benjamin Graham, Natalia Neverova, and Andrea Vedaldi. C3dpo: Canonical 3d pose networks for non-rigid structure from motion. In Proceed- ings of the IEEE/CVF international conference on computer vision, pages 7688–7697, 2019. 3

    2019

  32. [40]

    Aria digital twin: A new benchmark dataset for egocentric 3d machine percep- tion, 2023

    Xiaqing Pan, Nicholas Charron, Yongqian Yang, Scott Pe- ters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Carl Yuheng Ren. Aria digital twin: A new benchmark dataset for egocentric 3d machine percep- tion, 2023. 7

    2023

  33. [41]

    Hypernerf: A higher- dimensional representation for topologically varying neural radiance fields

    Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin- Brualla, and Steven M Seitz. Hypernerf: A higher- dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.13228, 2021. 3

    2021 arXiv

  34. [42]

    Unidepth: Universal monocular metric depth estimation

    Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10106–10116, 2024. 2

    2024

  35. [43]

    D-nerf: Neural radiance fields for dynamic scenes

    Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 10318–10327, 2021. 3

    2021

  36. [44]

    Vi- sion transformers for dense prediction

    Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12179–12188, 2021. 2

    2021

  37. [45]

    To- wards longer long-range motion trajectories

    Michael Rubinstein, Ce Liu, and William T Freeman. To- wards longer long-range motion trajectories. 2012. 2

    2012

  38. [46]

    Sand and S

    P. Sand and S. Teller. Particle video: Long-range motion es- timation using point trajectories. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recog- nition (CVPR’06), pages 2195–2202, 2006. 2

    2006

  39. [47]

    Structure- from-motion revisited

    Johannes L Schonberger and Jan-Michael Frahm. Structure- from-motion revisited. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 4104–4113, 2016. 2

    2016

  40. [48]

    A comparison and evalua- tion of multi-view stereo reconstruction algorithms

    Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. A comparison and evalua- tion of multi-view stereo reconstruction algorithms. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06), pages 519–528. IEE...

    2006

  41. [49]

    A benchmark for the eval- uation of RGB-D SLAM systems

    J ¨urgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the eval- uation of RGB-D SLAM systems. pages 573–580, 2012. 9

    2012

  42. [50]

    Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume

    Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934–8943,

  43. [51]

    Raft: Recurrent all-pairs field transforms for optical flow, 2020

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow, 2020. 1, 2

    2020

  44. [52]

    Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras

    Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neu- ral information processing systems, 34:16558–16569, 2021. 2

    2021

  45. [53]

    Raft-3d: Scene flow using rigid- motion embeddings

    Zachary Teed and Jia Deng. Raft-3d: Scene flow using rigid- motion embeddings. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 8375–8384, 2021. 2

    2021

  46. [54]

    Least-squares estimation of transforma- tion parameters between two point patterns

    Shinji Umeyama. Least-squares estimation of transforma- tion parameters between two point patterns. IEEE Transac- tions on Pattern Analysis & Machine Intelligence , 13(04): 376–380, 1991. 7 11

    1991

  47. [55]

    Three-dimensional scene flow

    Sundar Vedula, Simon Baker, Peter Rander, Robert Collins, and Takeo Kanade. Three-dimensional scene flow. In Pro- ceedings of the Seventh IEEE International Conference on Computer Vision, pages 722–729. IEEE, 1999. 2

    1999

  48. [56]

    Vggsfm: Visual geometry grounded deep structure from motion

    Jianyuan Wang, Nikita Karaev, Christian Rupprecht, and David Novotny. Vggsfm: Visual geometry grounded deep structure from motion. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 21686–21697, 2024. 2

    2024

  49. [57]

    Shape of mo- tion: 4d reconstruction from a single video

    Qianqian Wang, Vickie Ye, Hang Gao, Weijia Zeng, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of mo- tion: 4d reconstruction from a single video. arXiv preprint arXiv:2407.13764, 2024. 3, 6

    2024

  50. [58]

    Efros, and Angjoo Kanazawa

    Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 1, 2, 9

    2025

  51. [59]

    Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision, 2024

    Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision, 2024. 2, 6

    2024

  52. [60]

    Deepvo: Towards end-to-end visual odometry with deep re- current convolutional neural networks

    Sen Wang, Ronald Clark, Hongkai Wen, and Niki Trigoni. Deepvo: Towards end-to-end visual odometry with deep re- current convolutional neural networks. In 2017 IEEE in- ternational conference on robotics and automation (ICRA) , pages 2043–2050. IEEE, 2017. 2

    2017

  53. [61]

    Dust3r: Geometric 3d vision made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and J ´erˆome Revaud. Dust3r: Geometric 3d vision made easy. 2024 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 20697– 20709, 2023. 2, 3, 5, 13

    2024

  54. [62]

    Sur le point pour lequel la somme des dis- tances de n points donn´es est minimum

    Endre Weiszfeld. Sur le point pour lequel la somme des dis- tances de n points donn´es est minimum. Tohoku Mathemati- cal Journal, First Series, 43:355–386, 1937. 5

    1937

  55. [63]

    4d gaussian splatting for real-time dynamic scene rendering

    Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 20310–2032...

    2024

  56. [64]

    Spatialtracker: Tracking any 2d pixels in 3d space

    Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2, 3, 7, 8

    2024

  57. [65]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371–10381, 2024. 2

    2024

  58. [66]

    Deformable 3d gaussians for high- fidelity monocular dynamic scene reconstruction

    Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high- fidelity monocular dynamic scene reconstruction. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20331–20341, 2024. 3

    2024

  59. [67]

    MonST3R: A simple approach for estimating geometry in the presence of motion

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jam- pani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming- Hsuan Yang. MonST3R: A simple approach for estimating geometry in the presence of motion. In International Con- ference on Learning Representations (ICLR), 2025. 1...

    2025

  60. [68]

    Zhoutong Zhang, Forrester Cole, Zhengqi Li, Michael Ru- binstein, Noah Snavely, and William T. Freeman. Structure and motion from casual videos. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), pages 20–37,

  61. [69]

    Pointodyssey: A large-scale synthetic dataset for long-term point tracking

    Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wet- zstein, and Leonidas J Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 19855–19865, 2023. 5, 6, 9

    2023

  62. [70]

    Sparseness meets deepness: 3d human pose estimation from monocular video

    Xiaowei Zhou, Menglong Zhu, Spyridon Leonardos, Kon- stantinos G Derpanis, and Kostas Daniilidis. Sparseness meets deepness: 3d human pose estimation from monocular video. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4966–4975, 2016. ...

    2016

  63. [71]

    Unified 4D Representation of St4RTrack

    Simultaneous Reconstruction and Tracking 3 3.1. Unified 4D Representation of St4RTrack . . 3 3.2. Joint Learning of Tracking and Reconstruction 5 3.3. Adapt to Any Video without 4D Label . . . . 5

  64. [72]

    Experimental Details

    Experiments 6 4.1. Experimental Details . . . . . . . . . . . . . 6 4.2. 3D Tracking in World Coordinates . . . . . . 7 4.3. Dynamic 3D Reconstruction . . . . . . . . . 9 4.4. Joint Tracking and Reconstruction in the World 9 4.5. Ablation Study . . . . . . . . . . . . . . . . 9

  65. [73]

    Differentiable Camera Pose Estimation 13 B

    Acknowledgements 9 A . Differentiable Camera Pose Estimation 13 B . Details on theWorldTrackBenchmark 13 B.1. Datasets . . . . . . . . . . . . . . . . . . . . 13 B.2. Additional Quantitative Evaluation . . . . . 13 B.3. Qualitative Evaluation . . . . . . . . . . . . 14 C . Det...

    1969

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.