Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Matrix3D: Large Photogrammetry Model All-in-One

T0 review · 2 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single diffusion transformer, trained by masking whole views and modalities, claims state-of-the-art pose estimation, depth prediction, and novel view synthesis with one set of weights.

desk verdict A genuinely unified photogrammetry model that deserves review, but the headline pose numbers rest on a train/eval split the paper never states, and the paper's own ablation weakens its central claim. read the letter →

arxiv 2502.07685 v2 pith:M4LATBDW submitted 2025-02-11 cs.CV

Yuanxun Lu , Jingyang Zhang , Tian Fang , Jean-Daniel Nahmias , Yanghai Tsin , Long Quan , Xun Cao , Yao Yao
show 1 more author
Shiwei Li
This is my paper · ORCID
classification cs.CV
keywords multi-modaldiffusiontransformerphotogrammetryposeestimationnovelviewsynthesisdepthpredictionmaskedlearningPlückerraymaps3DGaussianSplatting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Matrix3D is a single diffusion transformer that performs pose estimation, depth prediction, and novel view synthesis with the same weights. The paper's central claim is that mask learning, which randomly masks whole views and modalities during training, is what makes large-scale multi-modal training feasible, because it lets the model train on partially complete data such as image-pose pairs and image-depth pairs. Trained on a mixture of six datasets, the model reports state-of-the-art pose accuracy on CO3D and strong novel-view synthesis on GSO, and it can serve as a front end for 3D Gaussian Splatting reconstruction from one or a few unposed images. The pitch is that one model with flexible input/output configurations can replace several task-specific models and streamline the photogrammetry pipeline.

What carries the argument

The central object is a multi-view, multi-modal diffusion transformer built on the Hunyuan-DiT architecture, with a multi-view encoder that ingests conditioning modalities and a multi-view decoder that denoises the target maps. Cameras are represented as Plücker ray maps, making poses image-like 2D data; depth is converted to disparity and regularized. Positional encodings for views and modalities let one network distinguish which maps are conditions and which are targets. Masked learning, sampling task-specific masks for novel view synthesis, pose estimation, depth prediction, and fully random configurations in a 3:3:3:1 ratio, is what allows flexible input/output configurations and training on partially complete data.

What would settle it

Train the same model while withholding all CO3D sequences from the training mixture, or evaluate on a new unposed multi-view dataset such as a held-out slice of ARKitScenes, and check whether relative rotation accuracy at 15 degrees stays above roughly 90 percent for two-view inputs; a large drop would indicate that the reported state-of-the-art pose numbers depended on training-set overlap.

Watch

Extended reading notes

Core claim

The paper claims that a unified multi-modal diffusion transformer, with images, Plücker-ray camera poses, and depth maps all encoded as 2D maps, can learn a joint distribution over all three modalities, so that any subset can be used as conditioning and the rest generated. The key mechanism is masked learning: training randomly masks views and modalities, so the model learns to denoise missing images, ray maps, and depth maps, and this also allows training on datasets that supply only some modalities. On the CO3D pose benchmark the model reports relative rotation accuracy above 95 percent for two to eight views, beating prior dedicated pose networks, and it matches or exceeds task-specific multi-view diffusion models on novel view synthesis. The same model also produces metric depth competitive with monocular depth foundations, and the authors demonstrate unposed sparse-view 3D reconstruction by completing poses, depth, and new views before 3DGS optimization.

Load-bearing premise

The headline pose results treat CO3D as a fair external test of generalization, but the paper trains on CO3D-v2 and evaluates on CO3D without stating how evaluation scenes were excluded from the training mixture; if scenes overlap, the reported accuracy measures memorization rather than generalization.

Editorial extensions

If this is right

  • If the central claim holds, a single set of weights can replace separate SfM, MVS, and multi-view diffusion systems for sparse-view pipelines, eliminating error accumulation between stages.
  • Because the model accepts arbitrary input combinations, users can inject extra depth from active sensors to improve pose and novel-view quality without retraining.
  • Unposed sparse-view 3D reconstruction becomes practical: the same model estimates poses, densifies views, and predicts depth before a 3DGS optimization, removing the traditional SfM bottleneck.
  • Training data efficiency improves: datasets with only bi-modality annotations (image-pose or image-depth) can be used for full-modality training, enlarging the usable pool of internet-scale data.
  • The 8-view and 512-resolution model suggests a scaling path: more views and higher resolution should improve consistency and reconstruction quality as compute allows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors do not test: because masking operates on whole views, the same recipe should work for other modality triples, such as color, normal maps, and semantic labels, or for video frames where missing-view prediction is the same operation.
  • The pose results are reported on CO3D, which is also in the training mixture; until a held-out split is demonstrated, the generalization margin over baselines should be treated as an open question rather than a settled fact.
  • Since the ablation shows geometry outputs converge more slowly than RGB from a text-to-image initialization, pre-training on cheap geometry tasks before joint fine-tuning is a plausible speed-up that remains untested.
  • The model's joint distribution could also serve as a proposal generator that warm-starts classical bundle adjustment, rather than replacing it, which would give a direct way to measure whether generated poses actually improve downstream SfM accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Matrix3D, a unified multi-modal diffusion transformer that performs pose estimation, depth prediction, and novel view synthesis with a single set of weights. The model represents images, Plücker-ray camera poses, and depth maps in a common 2D token space, and is trained with a masked multi-task objective over partially complete data from six datasets. The authors report state-of-the-art pose accuracy on CO3D, strong novel-view synthesis on GSO, competitive monocular depth on DTU, and demonstrate an integrated pipeline that completes sparse unposed inputs for 3D Gaussian Splatting reconstruction.

Significance. If the central claims hold, Matrix3D is a meaningful step toward a single photogrammetry foundation model that can replace several task-specific model families. The paper's strengths include a clear architectural description, a sensible masked-training formulation that increases usable training data, extensive quantitative evaluation across tasks, a useful supplementary with training hyper-parameters and ablations, and an end-to-end demonstration on unposed sparse-view reconstruction. However, the headline pose result in Table 1 is not fully verifiable as a generalization measure because the evaluation protocol on CO3D is not specified relative to the CO3D-v2 training mixture, and the ablation in Table 10 shows that masked multi-task training underperforms task-specific training on pose and depth, which complicates the paper's central claim that mask learning is the key to large-scale multi-modal training.

major comments (2)
  1. [Section 4.1 / Table 1 vs. Section 3.3 / Table 8] The pose evaluation on CO3D is stated without a held-out split, while Section 3.3 and Table 8 list CO3D-v2 as a training dataset with 19K scenes and 1.5M images. This creates a potential train/eval overlap: if the CO3D scenes used for Table 1 are drawn from the same distribution as the CO3D-v2 training mixture, the reported margins over RayDiffusion (e.g., 95.6 vs. 90.4 RRA@15 for two images) could partly reflect memorization rather than generalization. The phrase 'following RayDiffusion' is not enough to establish that scenes were excluded from training. Please specify the exact version of CO3D used for evaluation, the number of scenes, and the procedure that prevents those scenes from appearing in the CO3D-v2 training mixture, or re-evaluate on a clearly held-out set.
  2. [Section 10.3 / Table 10] The ablation that is meant to support the masked multi-task training design shows that the multi-task model underperforms the task-specific model on pose estimation (e.g., RRA@15 for 2 images: 81.1 vs. 89.2) and depth estimation (rel: 10.76 vs. 9.07), while only outperforming on NVS. The paper attributes this to slower convergence and notes that the ablation models use halved batch size and steps, but as reported the ablation does not substantiate the claim that mask learning is 'the key' to the method's pose and depth performance. Please provide a matched-compute or fully converged ablation, or temper the claim about the key mechanism for these tasks.
minor comments (5)
  1. [Section 4.1] The evaluation description would benefit from stating whether 'CO3D' in this section refers to the original CO3D v1, CO3D-v2, or a subset, and from citing the exact evaluation protocol used by RayDiffusion rather than only saying 'following RayDiffusion'.
  2. [Tables 4 and 5] The column heading 'Pose Range' is not defined in the table captions or in the main text; please clarify what 'range' refers to and how it affects the comparison categories.
  3. [Section 8 / Table 8] The training data mixture is described as having dataset proportions '4:4:4:4:4:1', which sums to 21; it would help to state explicitly whether these are relative weights or percentages and how they are applied in each batch.
  4. [Figure 4 caption] The caption contains a typo, 'Qualitive evaluation results', which should be 'Qualitative evaluation results'.
  5. [Section 10.3] The text 'Quantitative reuslts are shown in Table 10' contains a typo ('reuslts' for 'results').

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are supported by external benchmarks and ablations, not by equations that reduce to their inputs; the CO3D train/eval split gap is a documentation concern, not a demonstrated circularity.

full rationale

Matrix3D is an empirical systems paper. Its load-bearing claim—that one masked multi-modal diffusion transformer can perform pose estimation, depth prediction, and novel view synthesis—is supported by ablations (Table 10) and by benchmark evaluations against external baselines (Tables 1–6), not by a derivation that reduces to its inputs. The training loss (Eq. 1) is a standard v-prediction diffusion objective; the Plücker-ray pose representation is adopted from RayDiffusion and is not used to define the evaluation metric in a circular way. No equation or protocol in the paper defines the target result in terms of an input parameter, and no uniqueness theorem is invoked to forbid alternatives. The few self-citations (e.g., Direct2.5, JointNet, Vis-MVSNet, MvSNet) are related-work or baseline citations, not load-bearing support for the central claim. The paper's own limitation statement (Sec. 10.5) concedes outdoor degradation due to training-data coverage, which is an honest external-validity statement rather than circularity. One documentation gap exists: Sec. 3.3 trains on CO3D-v2 while Sec. 4.1 evaluates pose 'on the CO3D dataset' without stating the held-out split, so the reader cannot verify that Table 1 is an out-of-distribution test. This is a benchmark-hygiene concern that could affect conclusion validity, but it is not an exhibited reduction of the prediction to training data; the hard rules require a quoted equation or defined equivalence for circularity, and none is present. Therefore no significant circularity is found.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central contribution is an empirical system, not a derivation. The ledger shows the hand-tuned knobs applied after the model was trained (dataset normalization, task mix, CFG scales, 3DGS loss weights) and the unproven premises: the canonical ray-frame encoding across heterogeneous datasets, the benefit of masked multi-task training (contradicted at matched compute by Table 10), the affine-depth to metric-depth conversion, and the CO3D train/test hygiene.

free parameters (5)
  • Task mix ratio 3:3:3:1 (NVS:pose:depth:random) = 3:3:3:1
    Hand-chosen split of training tasks in Sec 3.2; no experiment justifies this ratio.
  • Classifier-free guidance scale = 1.5 for RGB/poses, 1.0 for depth
    Empirically chosen in supplementary Sec 7; CFG=1.0 means no guidance for depth.
  • Per-dataset scene scale normalization = median depth of first view set to 1.0; max camera distance set to 1.0
    Supplementary Sec 8; this normalization defines the metric scale that later supports the metric-depth claim, but the rule differs per dataset.
  • 3DGS optimization loss weights = wL1=1.0/10/20, wSSIM=0.2, wLPIPS=10.0, wmask=5.0, waccum=5.0/0.5, wdepth=10.0, wrel-depth=20.0
    Hand-chosen in supplementary Sec 10.4; directly affect reported reconstruction metrics.
  • Fixed shift and scale for ray and depth maps = not specified numerically
    Sec 3.1 states a fixed shift and scale factor is applied so ray and depth distributions approximate standard Gaussian; the values are not given, so this data standardization cannot be reproduced.
assumptions (5)
  • domain assumption Pose can be faithfully encoded as Plücker ray maps in a canonical frame with the first view as identity camera.
    Sec 3.1 and supplementary Sec 8 camera normalization; if this canonicalization is not consistent across datasets, the learned joint distribution is not a valid camera geometry distribution.
  • domain assumption Masked multi-task training on partially complete data teaches the joint RGB-pose-depth distribution and improves task performance.
    Sec 3.2 core claim; Table 10 shows the masked multi-task ablation underperforms task-specific models on pose and depth at matched compute, so the assumption is not yet demonstrated.
  • domain assumption Affine-invariant (multi-view aligned) depth converted to disparity is sufficient to recover metric depth via normalization.
    Sec 3.1 and Table 3; the path from affine-invariant depth to the claimed monocular metric depth on DTU is not described, making the metric-depth claim unverified.
  • standard math Diffusion v-prediction objective and HunyuanDiT architecture provide a sound base for multi-modal generation.
    Sec 3.1, Eq. (1)-(2), following Salimans and Ho 2022 and Li et al. 2024; standard diffusion background.
  • domain assumption The CO3D evaluation scenes are disjoint from CO3D-v2 training data.
    Sec 3.3 includes CO3D-v2 in training; Sec 4.1 evaluates pose on CO3D without stating a held-out split. If overlapping, the pose results are inflated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Matrix3D: Large Photogrammetry Model All-in-One." pith.science (2026). https://pith.science/paper/M4LATBDW

@misc{pith2026250207685,
  author       = {Pith},
  title        = {Pith review of: Matrix3D: Large Photogrammetry Model All-in-One},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M4LATBDW}},
  note         = {Machine review of arXiv:2502.07685}
}
read the original abstract

We present Matrix3D, a unified model that performs several photogrammetry subtasks, including pose estimation, depth prediction, and novel view synthesis using just the same model. Matrix3D utilizes a multi-modal diffusion transformer (DiT) to integrate transformations across several modalities, such as images, camera parameters, and depth maps. The key to Matrix3D's large-scale multi-modal training lies in the incorporation of a mask learning strategy. This enables full-modality model training even with partially complete data, such as bi-modality data of image-pose and image-depth pairs, thus significantly increases the pool of available training data. Matrix3D demonstrates state-of-the-art performance in pose estimation and novel view synthesis tasks. Additionally, it offers fine-grained control through multi-round interactions, making it an innovative tool for 3D content creation. Project page: https://nju-3dv.github.io/projects/matrix3d.

Figures

Figures reproduced from arXiv: 2502.07685 by the authors.

Figure 1
Figure 1. Utilizing Matrix3D for single/few-shot reconstruction. Before 3DGS optimization, we complete the input set by pose estimation, depth estimation and novel view synthesis, all of which are done by the same model. Abstract We present Matrix3D, a unified model that performs several photogrammetry subtasks, including pose estimation, depth prediction, and novel view synthesis using just the same model. Matrix3D utilizes … view at source ↗
Figure 2
Figure 2. We train the Matrix3D by masked learning. Multi-modal data are randomly masked by noise corruption. Observations (green) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Sparse-view pose estimation results on CO3D dataset. The black axes are ground-truth and the colored ones are the estimation. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Qualitive evaluation results of novel view synthesis from single images on GSO and ARKitScenes dataset: a) random novel [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Monocular 3D reconstruction. Additional novel view [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Unposed sparse-view 3D reconstruction results. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Evaluation results of stage 1 model for a) pose estimation, b) NVS and c) Depth estimation with respect to training step. For pose [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: More unposed sparse-view 3D reconstruction results. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Visualization of multi-view depth prediction results. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    E3D-Bench compares 16 3D geometric foundation models on depth, reconstruction, pose, and view-synthesis tasks with a unified evaluation toolkit.

Reference graph

Works this paper leans on

146 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    Large-scale data for multiple-view stereopsis

    Henrik Aanæs, Rasmus Ramsbøl Jensen, George V ogiatzis, Engin Tola, and Anders Bjorholm Dahl. Large-scale data for multiple-view stereopsis. International Journal of Com- puter Vision, 120:153–168, 2016. 6

    2016

  2. [2]

    Multimae: Multi-modal multi-task masked autoencoders

    Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi-task masked autoencoders. In European Conference on Computer Vi- sion, pages 348–367. Springer, 2022. 3

    2022

  3. [3]

    Zip-nerf: Anti-aliased grid-based neural radiance fields

    Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19697–19705, 2023. 7

    2023

  4. [4]

    Arkitscenes: A di- verse real-world dataset for 3d indoor scene understanding using mobile rgb-d data

    Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A di- verse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. arXiv preprint arXiv:2111.08897,

    arXiv

  5. [5]

    Surf: Speeded up robust features

    Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. In Computer Vision–ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006. Proceedings, Part I 9, pages 404–

    2006

  6. [6]

    Auto-rectify network for unsupervised indoor depth estimation

    Jia-Wang Bian, Huangying Zhan, Naiyan Wang, Tat-Jun Chin, Chunhua Shen, and Ian Reid. Auto-rectify network for unsupervised indoor depth estimation. IEEE transac- tions on pattern analysis and machine intelligence, 44(12): 9802–9813, 2021. 3

    2021

  7. [7]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 3

    arXiv 2005

  8. [8]

    Using multiple hypotheses to improve depth-maps for multi-view stereo

    Neill DF Campbell, George V ogiatzis, Carlos Hern ´andez, and Roberto Cipolla. Using multiple hypotheses to improve depth-maps for multi-view stereo. In Computer Vision– ECCV 2008: 10th European Conference on Computer Vi- sion, Marseille, France, October 12-18, 2008, Proceedings, Part I 10, pages 766–779. Springer, 2008. 2, 7

    2008

Show all 146 references
  1. [9]

    Chan, Connor Z

    Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. In CVPR, 2022. 3

    2022

  2. [10]

    Generative novel view synthesis with 3d-aware diffusion models

    Eric R Chan, Koki Nagano, Matthew A Chan, Alexan- der W Bergman, Jeong Joon Park, Axel Levy, Miika Ait- tala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. In Proceedings of the IEEE/CVF International Confe...

    2023

  3. [11]

    Shapenet: An information-rich 3d model repository

    Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015. 3

    2015 arXiv

  4. [12]

    Aspanformer: Detector-free image matching with adaptive span transformer

    Hongkai Chen, Zixin Luo, Lei Zhou, Yurun Tian, Ming- min Zhen, Tian Fang, David Mckinnon, Yanghai Tsin, and Long Quan. Aspanformer: Detector-free image matching with adaptive span transformer. In European Conference on Computer Vision, pages 20–36. Springer, 2022. 2

    2022

  5. [13]

    Wide- baseline relative camera pose estimation with directional learning

    Kefan Chen, Noah Snavely, and Ameesh Makadia. Wide- baseline relative camera pose estimation with directional learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3258– 3268, 2021. 2

    2021

  6. [14]

    Point- based multi-view stereo network

    Rui Chen, Songfang Han, Jing Xu, and Hao Su. Point- based multi-view stereo network. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1538–1547, 2019. 2

    2019

  7. [15]

    Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation

    Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.13873, 2023. 3

    2023 arXiv

  8. [16]

    Deep stereo us- ing adaptive thin volume representation with uncertainty awareness

    Shuo Cheng, Zexiang Xu, Shilin Zhu, Zhuwen Li, Li Er- ran Li, Ravi Ramamoorthi, and Hao Su. Deep stereo us- ing adaptive thin volume representation with uncertainty awareness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2524– 253...

    2020

  9. [17]

    Sfm with mrfs: Discrete-continuous optimization for large-scale structure from motion

    David J Crandall, Andrew Owens, Noah Snavely, and Daniel P Huttenlocher. Sfm with mrfs: Discrete-continuous optimization for large-scale structure from motion. IEEE transactions on pattern analysis and machine intelligence , 35(12):2841–2853, 2012. 2

    2012

  10. [18]

    Hsfm: Hybrid structure-from-motion

    Hainan Cui, Xiang Gao, Shuhan Shen, and Zhanyi Hu. Hsfm: Hybrid structure-from-motion. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 1212–1221, 2017. 2

    2017

  11. [19]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2023

  12. [20]

    Objaverse-xl: A universe of 10m+ 3d objects

    Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Chris- tian Laforte, Vikram V oleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36, 2024. 6

    2024

  13. [21]

    Superpoint: Self-supervised interest point detection and description

    Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabi- novich. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 224–236, 2018. 2

    2018

  14. [22]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 4

    2010 arXiv

  15. [23]

    Google scanned objects: A high-quality dataset of 3d scanned household items

    Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automa- tion (ICRA), ...

    2022

  16. [24]

    D2- net: A trainable cnn for joint description and detection of local features

    Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Polle- feys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2- net: A trainable cnn for joint description and detection of local features. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, ...

  17. [25]

    Accurate, dense, and robust multiview stereopsis

    Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE transactions on pattern analysis and machine intelligence, 32(8):1362–1376, 2009. 2, 7

    2009

  18. [26]

    Multi-view stereo: A tutorial

    Yasutaka Furukawa, Carlos Hern ´andez, et al. Multi-view stereo: A tutorial. Foundations and Trends® in Computer Graphics and Vision, 9(1-2):1–148, 2015. 2

    2015

  19. [27]

    Massively parallel multiview stereopsis by surface normal diffusion

    Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Massively parallel multiview stereopsis by surface normal diffusion. In Proceedings of the IEEE international confer- ence on computer vision, pages 873–881, 2015. 2, 7

    2015

  20. [28]

    Relightable 3d gaussians: Re- alistic point cloud relighting with brdf decomposition and ray tracing

    Jian Gao, Chun Gu, Youtian Lin, Zhihao Li, Hao Zhu, Xun Cao, Li Zhang, and Yao Yao. Relightable 3d gaussians: Re- alistic point cloud relighting with brdf decomposition and ray tracing. In European Conference on Computer Vision, pages 73–89. Springer, 2024. 3

    2024

  21. [29]

    Cat3d: Create anything in 3d with multi-view diffusion models

    Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul Srinivasan, Jonathan T Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint arXiv:2405.10314, 2024. 2, 3, 7, 8

    2024 arXiv

  22. [30]

    Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion

    Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ra- mamoorthi. Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion. In Inter- national Conference on Machine Learning , pages 11808– 118...

    2023

  23. [31]

    Cascade cost volume for high- resolution multi-view stereo and stereo matching

    Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high- resolution multi-view stereo and stereo matching. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2495–2504, 2020. 2, 7

    2020

  24. [32]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 3

    2023 arXiv

  25. [33]

    Instruct-nerf2nerf: Edit- ing 3d scenes with instructions

    Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2nerf: Edit- ing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19740–19750, 2023. 3

    2023

  26. [34]

    A combined corner and edge detector

    Chris Harris, Mike Stephens, et al. A combined corner and edge detector. In Alvey vision conference, pages 10–5244. Citeseer, 1988. 2

    1988

  27. [35]

    Multiple view ge- ometry in computer vision

    Richard Hartley and Andrew Zisserman. Multiple view ge- ometry in computer vision . Cambridge university press,

  28. [36]

    Masked autoencoders are scal- able vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scal- able vision learners. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 16000–16009, 2022. 3

    2022

  29. [37]

    Stereo processing by semiglobal matching and mutual information

    Heiko Hirschmuller. Stereo processing by semiglobal matching and mutual information. IEEE Transactions on pattern analysis and machine intelligence , 30(2):328–341,

  30. [38]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 4

    2022 arXiv

  31. [39]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural informa- tion processing systems, 33:6840–6851, 2020. 4

    2020

  32. [40]

    Lrm: Large reconstruction model for single image to 3d

    Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400, 2023. 3

    2023 arXiv

  33. [41]

    Metric3d v2: A versatile monocular geomet- ric foundation model for zero-shot metric depth and surface normal estimation

    Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geomet- ric foundation model for zero-shot metric depth and surface normal estimation. arXiv preprint arXiv:2404.15506, 2024. 6

    2024 arXiv

  34. [42]

    Dreamtime: An improved optimization strategy for text-to-3d content creation

    Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi, Zheng-Jun Zha, and Lei Zhang. Dreamtime: An improved optimization strategy for text-to-3d content creation. arXiv preprint arXiv:2306.12422, 2023. 3

    2023 arXiv

  35. [43]

    Few-view object reconstruction with unknown cate- gories and camera poses

    Hanwen Jiang, Zhenyu Jiang, Kristen Grauman, and Yuke Zhu. Few-view object reconstruction with unknown cate- gories and camera poses. In 2024 International Conference on 3D Vision (3DV), pages 31–41. IEEE, 2024. 2

    2024

  36. [44]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of naacL-HLT, page 2. Minneapolis, Minnesota, 2019. 3

    2019

  37. [45]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1,

  38. [46]

    A theory of shape by space carving

    Kiriakos N Kutulakos and Steven M Seitz. A theory of shape by space carving. International journal of computer vision, 38:199–218, 2000. 2

    2000

  39. [47]

    Vivid-1-to-3: Novel view synthesis with video diffusion models

    Jeong-gi Kwak, Erqun Dong, Yuhe Jin, Hanseok Ko, Shweta Mahajan, and Kwang Moo Yi. Vivid-1-to-3: Novel view synthesis with video diffusion models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6775–6785, 2024. 3

    2024

  40. [48]

    A quasi-dense approach to surface reconstruction from uncalibrated images

    Maxime Lhuillier and Long Quan. A quasi-dense approach to surface reconstruction from uncalibrated images. IEEE transactions on pattern analysis and machine intelligence , 27(3):418–433, 2005. 2

    2005

  41. [49]

    Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model

    Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprint arXiv:2311.06214, 2023. 2, 3

    2023 arXiv

  42. [50]

    Mst: Masked self-supervised transformer 10 for visual representation

    Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. Mst: Masked self-supervised transformer 10 for visual representation. Advances in Neural Information Processing Systems, 34:13165–13176, 2021. 3

    2021

  43. [51]

    Hunyuan- dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan- dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint arXiv:2405.0874...

    2024 arXiv

  44. [52]

    Relpose++: Recovering 6d poses from sparse- view observations

    Amy Lin, Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose++: Recovering 6d poses from sparse- view observations. arXiv preprint arXiv:2305.04926, 2023. 2, 6

    2023 arXiv

  45. [53]

    Barf: Bundle-adjusting neural radiance fields

    Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Si- mon Lucey. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5741–5751, 2021. 2

    2021

  46. [54]

    Gaussian-flow: 4d reconstruction with dynamic 3d gaus- sian particle

    Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flow: 4d reconstruction with dynamic 3d gaus- sian particle. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21136– 21145, 2024. 3

    2024

  47. [55]

    Pixel-perfect structure-from- motion with featuremetric refinement

    Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Lars- son, and Marc Pollefeys. Pixel-perfect structure-from- motion with featuremetric refinement. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 5987–5997, 2021. 2

    2021

  48. [56]

    Lightglue: Local feature matching at light speed

    Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Pollefeys. Lightglue: Local feature matching at light speed. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17627–17638, 2023. 2

    2023

  49. [57]

    One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion

    Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Ji- ayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. In Proceedings of the IEEE/CVF Conference on Co...

    2024

  50. [58]

    One-2-3-45: Any sin- gle image to 3d mesh in 45 seconds without per-shape opti- mization

    Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any sin- gle image to 3d mesh in 45 seconds without per-shape opti- mization. Advances in Neural Information Processing Sys- tems, 36, 2024. 3

    2024

  51. [59]

    Zero-1- to-3: Zero-shot one image to 3d object

    Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tok- makov, Sergey Zakharov, and Carl V ondrick. Zero-1- to-3: Zero-shot one image to 3d object. arXiv preprint arXiv:2303.11328, 2023. 3, 6

    2023 arXiv

  52. [60]

    Syncdreamer: Generating multiview-consistent images from a single-view image

    Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453, 2023. 3, 6

    2023 arXiv

  53. [61]

    Wonder3d: Single image to 3d using cross-domain diffusion

    Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15008, 2023. 3, 6

    arXiv

  54. [62]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 2

    2017 arXiv

  55. [63]

    Distinctive image features from scale- invariant keypoints

    David G Lowe. Distinctive image features from scale- invariant keypoints. International journal of computer vi- sion, 60:91–110, 2004. 2

    2004

  56. [64]

    Direct2.5: Diverse text-to-3d generation via multi- view 2.5d diffusion

    Yuanxun Lu, Jingyang Zhang, Shiwei Li, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan, Xun Cao, and Yao Yao. Direct2.5: Diverse text-to-3d generation via multi- view 2.5d diffusion. Computer Vision and Pattern Recog- nition (CVPR), 2024. 3

    2024

  57. [65]

    Aslfeat: Learning local features of accurate shape and lo- calization

    Zixin Luo, Lei Zhou, Xuyang Bai, Hongkai Chen, Jiahui Zhang, Yao Yao, Shiwei Li, Tian Fang, and Long Quan. Aslfeat: Learning local features of accurate shape and lo- calization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6589–6598,

  58. [66]

    Multiview stereo with cascaded epipolar raft

    Zeyu Ma, Zachary Teed, and Jia Deng. Multiview stereo with cascaded epipolar raft. InProceedings of the European conference on computer vision (ECCV), 2022. 7

    2022

  59. [67]

    Im-3d: Iterative multiview diffusion and re- construction for high-quality 3d generation

    Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, Na- talia Neverova, Andrea Vedaldi, Oran Gafni, and Filippos Kokkinos. Im-3d: Iterative multiview diffusion and re- construction for high-quality 3d generation. arXiv preprint arXiv:2402.08682, 2024. 3, 7

    2024 arXiv

  60. [68]

    4m: Massively multimodal masked modeling.Advances in Neu- ral Information Processing Systems, 36, 2024

    David Mizrahi, Roman Bachmann, Oguzhan Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4m: Massively multimodal masked modeling.Advances in Neu- ral Information Processing Systems, 36, 2024. 4

    2024

  61. [69]

    Orb-slam: a versatile and accurate monocular slam system

    Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics , 31(5):1147–1163,

  62. [70]

    Dinov2: Learning robust visual features without supervi- sion

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervi- sion. arXiv preprint arXiv:2304.07193, 2023. 1

    2023 arXiv

  63. [71]

    Scalable diffusion mod- els with transformers

    William Peebles and Saining Xie. Scalable diffusion mod- els with transformers. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 4195– 4205, 2023. 3, 1

    2023

  64. [72]

    Sdxl: improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 4

    2023 arXiv

  65. [73]

    Dreamfusion: Text-to-3d using 2d diffusion

    Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Milden- hall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022. 3

    2022 arXiv

  66. [74]

    Improving language understanding by gen- erative pre-training

    Alec Radford. Improving language understanding by gen- erative pre-training. 2018. 3

    2018

  67. [75]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9,

  68. [76]

    Dreambooth3d: Subject-driven text-to-3d generation

    Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al. Dreambooth3d: Subject-driven text-to-3d generation. In Proceedings of the 11 IEEE/CVF international conference on compute...

    2023

  69. [77]

    Vi- sion transformers for dense prediction

    Rene Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 12179–12188, 2021. 3

    2021

  70. [78]

    Towards robust monocu- lar depth estimation: Mixing datasets for zero-shot cross- dataset transfer

    Ren ´e Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocu- lar depth estimation: Mixing datasets for zero-shot cross- dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3), 2022. 3

    2022

  71. [79]

    Com- mon objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction

    Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Com- mon objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In International Con- ference on Computer Vision, 2021. 4, 1, 2

    2021

  72. [80]

    R2d2: Reliable and repeatable detec- tor and descriptor

    Jerome Revaud, Cesar De Souza, Martin Humenberger, and Philippe Weinzaepfel. R2d2: Reliable and repeatable detec- tor and descriptor. Advances in neural information process- ing systems, 32, 2019. 2

    2019

  73. [81]

    Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding

    Mike Roberts, Jason Ramapuram, Anurag Ranjan, At- ulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference o...

    2021

  74. [82]

    High-resolution image synthesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models, 2021. 1

    2021

  75. [83]

    Machine learning for high-speed corner detection

    Edward Rosten and Tom Drummond. Machine learning for high-speed corner detection. In Computer Vision–ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006. Proceedings, Part I 9, pages 430–

    2006

  76. [84]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022. 4

    2022 arXiv

  77. [85]

    Zeronvs: Zero-shot 360-degree view synthesis from a single real im- age

    Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Her- rmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, et al. Zeronvs: Zero-shot 360-degree view synthesis from a single real im- age. arXiv preprint arXiv:2310.17994, 2023. 3, 7

    arXiv

  78. [86]

    Superglue: Learning feature matching with graph neural networks

    Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938–4947, 2020. 2

    2020

  79. [87]

    Structure-from-motion revisited

    Johannes Lutz Sch ¨onberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Com- puter Vision and Pattern Recognition (CVPR), 2016. 2, 6, 7

    2016

  80. [88]

    Pixelwise view selection for unstructured multi-view stereo

    Johannes Lutz Sch ¨onberger, Enliang Zheng, Marc Polle- feys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016. 2, 7

    2016

  81. [89]

    A benchmark and a baseline for robust multi-view depth estimation

    Philipp Schr ¨oppel, Jan Bechtold, Artemij Amiranashvili, and Thomas Brox. A benchmark and a baseline for robust multi-view depth estimation. In 2022 International Con- ference on 3D Vision (3DV) , pages 637–645. IEEE, 2022. 7

    2022

  82. [90]

    Photorealistic scene reconstruction by voxel coloring

    Steven M Seitz and Charles R Dyer. Photorealistic scene reconstruction by voxel coloring. International journal of computer vision, 35:151–173, 1999. 2

    1999

  83. [91]

    Zero123++: a single image to consis- tent multi-view diffusion base model

    Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consis- tent multi-view diffusion base model. arXiv preprint arXiv:2310.15110, 2023. 3

    arXiv

  84. [92]

    Mvdream: Multi-view diffusion for 3d generation

    Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308.16512, 2023. 2, 3

    2023 arXiv

  85. [93]

    Sparsepose: Sparse- view camera pose regression and refinement

    Samarth Sinha, Jason Y Zhang, Andrea Tagliasacchi, Igor Gilitschenski, and David B Lindell. Sparsepose: Sparse- view camera pose regression and refinement. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21349–21359, 2023. 2

    2023

  86. [94]

    Photo tourism: exploring photo collections in 3d

    Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. In ACM sig- graph 2006 papers, pages 835–846. 2006. 2

    2006

  87. [95]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024. 4

    2024

  88. [96]

    Loftr: Detector-free local feature matching with transformers

    Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 8922–8931, 2021. 2

    2021

  89. [97]

    Nerfstudio: A modular framework for neural radiance field development

    Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristof- fersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In ACM SI...

    2023

  90. [98]

    Dreamgaussian: Generative gaussian splat- ting for efficient 3d content creation

    Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splat- ting for efficient 3d content creation. arXiv preprint arXiv:2309.16653, 2023. 3

    2023 arXiv

  91. [99]

    Lgm: Large multi- view gaussian model for high-resolution 3d content cre- ation

    Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi- view gaussian model for high-resolution 3d content cre- ation. arXiv preprint arXiv:2402.05054, 2024. 3

    2024 arXiv

  92. [100]

    Mvdiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion

    Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion. arXiv preprint arXiv:2307.01097, 2023. 3

    2023 arXiv

  93. [101]

    Deepv2d: Video to depth with differentiable structure from motion

    Zachary Teed and Jia Deng. Deepv2d: Video to depth with differentiable structure from motion. arXiv preprint arXiv:1812.04605, 2018. 7

    2018 arXiv

  94. [102]

    Efficient large-scale multi-view stereo for ultra high-resolution im- age sets

    Engin Tola, Christoph Strecha, and Pascal Fua. Efficient large-scale multi-view stereo for ultra high-resolution im- age sets. Machine Vision and Applications , 23:903–920,

  95. [103]

    Demon: Depth and motion network for learning monocular stereo

    Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, and 12 Thomas Brox. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 5038–504...

    2017

  96. [104]

    Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion

    Vikram V oleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. arXiv preprint arXiv:2403.12008 ,

    arXiv

  97. [105]

    Patchmatchnet: Learned multi-view patchmatch stereo

    Fangjinhua Wang, Silvano Galliani, Christoph V ogel, Pablo Speciale, and Marc Pollefeys. Patchmatchnet: Learned multi-view patchmatch stereo. In CVPR, 2021. 7

    2021

  98. [106]

    Sparsenerf: Distilling depth ranking for few- shot novel view synthesis

    Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth ranking for few- shot novel view synthesis. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9065– 9076, 2023. 3

    2023

  99. [107]

    Posediffusion: Solving pose estimation via diffusion-aided bundle adjustment

    Jianyuan Wang, Christian Rupprecht, and David Novotny. Posediffusion: Solving pose estimation via diffusion-aided bundle adjustment. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 9773– 9783, 2023. 2, 6

    2023

  100. [108]

    Vggsfm: Visual geometry grounded deep structure from motion

    Jianyuan Wang, Nikita Karaev, Christian Rupprecht, and David Novotny. Vggsfm: Visual geometry grounded deep structure from motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21686–21697, 2024. 2

    2024

  101. [109]

    Imagedream: Image-prompt multi-view diffusion for 3d generation

    Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02201, 2023. 7

    2023 arXiv

  102. [110]

    Pf-lrm: Pose-free large reconstruction model for joint pose and shape prediction

    Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-lrm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint arXiv:2311.12024, 2023. 2, 3

    2023 arXiv

  103. [111]

    Dust3r: Geometric 3d vision made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 20697–20709, 2024. 2, 3, 6, 7, 1

    2024

  104. [112]

    Prolificdreamer: High- fidelity and diverse text-to-3d generation with variational score distillation

    Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongx- uan Li, Hang Su, and Jun Zhu. Prolificdreamer: High- fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305.16213, 2023. 3

    2023 arXiv

  105. [113]

    Croco: Self-supervised pre-training for 3d vision tasks by cross-view completion

    Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Ro- main Br´egier, Yohann Cabon, Vaibhav Arora, Leonid Ants- feld, Boris Chidlovskii, Gabriela Csurka, and J ´erˆome Re- vaud. Croco: Self-supervised pre-training for 3d vision tasks by cross-view completion. Advances in Neura...

    2022

  106. [114]

    Croco v2: Improved cross-view completion pre- training for stereo matching and optical flow

    Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Br ´egier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and J ´erˆome Revaud. Croco v2: Improved cross-view completion pre- training for stereo matching and optical flow. In Proceed-...

    2023

  107. [115]

    Reconfusion: 3d reconstruction with diffusion priors

    Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P Srinivasan, Dor Verbin, Jonathan T Barron, Ben Poole, et al. Reconfusion: 3d reconstruction with diffusion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  108. [116]

    Direct3d: Scalable image-to-3d generation via 3d latent diffusion transformer

    Shuang Wu, Youtian Lin, Feihu Zhang, Yifei Zeng, Jingxi Xu, Philip Torr, Xun Cao, and Yao Yao. Direct3d: Scalable image-to-3d generation via 3d latent diffusion transformer. arXiv preprint arXiv:2405.14832, 2024. 3

    2024 arXiv

  109. [117]

    Level- s2fm: Structure from motion on neural level set of implicit surfaces

    Yuxi Xiao, Nan Xue, Tianfu Wu, and Gui-Song Xia. Level- s2fm: Structure from motion on neural level set of implicit surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17205– 17214, 2023. 2

    2023

  110. [118]

    Physgaussian: Physics-integrated 3d gaussians for generative dynamics

    Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 4389–4398, 2024. 3

    2024

  111. [119]

    Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models

    Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07191 ,

    arXiv

  112. [120]

    Learning inverse depth regression for multi-view stereo with correlation cost vol- ume

    Qingshan Xu and Wenbing Tao. Learning inverse depth regression for multi-view stereo with correlation cost vol- ume. In Proceedings of the AAAI conference on artificial intelligence, 2020. 7

    2020

  113. [121]

    Dmv3d: Denoising multi-view diffu- sion using 3d large reconstruction model

    Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Ji- ahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffu- sion using 3d large reconstruction model. arXiv preprint arXiv:2311.09217, 2023. 3

    2023 arXiv

  114. [122]

    Alvarez, and Miaomiao Liu

    Jiayu Yang, Wei Mao, Jose M. Alvarez, and Miaomiao Liu. Cost volume pyramid based depth inference for multi-view stereo. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 7

    2020

  115. [123]

    Consistnet: Enforcing 3d consistency for multi- view images diffusion

    Jiayu Yang, Ziang Cheng, Yunfei Duan, Pan Ji, and Hong- dong Li. Consistnet: Enforcing 3d consistency for multi- view images diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7079–7088, 2024. 3

    2024

  116. [124]

    Depth anything: Un- leashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Ji- ashi Feng, and Hengshuang Zhao. Depth anything: Un- leashing the power of large-scale unlabeled data. In CVPR,

  117. [125]

    Depth any- thing v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. arXiv:2406.09414, 2024. 6

    2024 arXiv

  118. [126]

    Mvs2d: Efficient multi-view stereo via attention-driven 2d convolutions

    Zhenpei Yang, Zhile Ren, Qi Shan, and Qixing Huang. Mvs2d: Efficient multi-view stereo via attention-driven 2d convolutions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8574– 8584, 2022. 7 13

    2022

  119. [127]

    Mvsnet: Depth inference for unstructured multi- view stereo

    Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi- view stereo. In Proceedings of the European conference on computer vision (ECCV), pages 767–783, 2018. 2, 7, 1

    2018

  120. [128]

    Recurrent mvsnet for high-resolution multi-view stereo depth inference

    Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan. Recurrent mvsnet for high-resolution multi-view stereo depth inference. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5525–5534, 2019. 2

    2019

  121. [129]

    Multiview neu- ral surface reconstruction by disentangling geometry and appearance

    Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neu- ral surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33, 2020. 6, 1

    2020

  122. [130]

    Learning to recover 3d scene shape from a single image

    Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Learning to recover 3d scene shape from a single image. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 204–213, 2021. 3

    2021

  123. [131]

    Met- ric3d: Towards zero-shot metric 3d prediction from a sin- gle image

    Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Met- ric3d: Towards zero-shot metric 3d prediction from a sin- gle image. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9043–9053, 2023. 6

    2023

  124. [132]

    Mvimgnet: A large-scale dataset of multi-view images

    Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recogn...

    2023

  125. [133]

    Stag4d: Spatial-temporal anchored generative 4d gaus- sians

    Yifei Zeng, Yanqin Jiang, Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, and Yao Yao. Stag4d: Spatial-temporal anchored generative 4d gaus- sians. In European Conference on Computer Vision, pages 163–179. Springer, 2024. 3

    2024

  126. [134]

    Vis-mvsnet: Visibility-aware multi-view stereo net- work

    Jingyang Zhang, Shiwei Li, Zixin Luo, Tian Fang, and Yao Yao. Vis-mvsnet: Visibility-aware multi-view stereo net- work. International Journal of Computer Vision , 131(1): 199–214, 2023. 2, 7

    2023

  127. [135]

    Joint- net: Extending text-to-image diffusion for dense distribu- tion modeling

    Jingyang Zhang, Shiwei Li, Yuanxun Lu, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan, and Yao Yao. Joint- net: Extending text-to-image diffusion for dense distribu- tion modeling. International Conference on Learning Rep- resentations (ICLR), 2024. 3

    2024

  128. [136]

    Relpose: Predicting probabilistic relative rotation for single objects in the wild

    Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose: Predicting probabilistic relative rotation for single objects in the wild. In European Conference on Computer Vision, pages 592–611. Springer, 2022. 2

    2022

  129. [137]

    Cameras as rays: Pose estimation via ray diffusion

    Jason Y Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, and Shubham Tulsiani. Cameras as rays: Pose estimation via ray diffusion. arXiv preprint arXiv:2402.14817, 2024. 2, 4, 6, 1

    2024 arXiv

  130. [138]

    Gs-lrm: Large reconstruction model for 3d gaussian splatting

    Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-lrm: Large reconstruction model for 3d gaussian splatting. arXiv preprint arXiv:2404.19702, 2024. 3

    2024 arXiv

  131. [139]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 3

    2018

  132. [140]

    Ge- omvsnet: Learning multi-view stereo with geometry per- ception

    Zhe Zhang, Rui Peng, Yuxi Hu, and Ronggang Wang. Ge- omvsnet: Learning multi-view stereo with geometry per- ception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21508– 21518, 2023. 7

    2023

  133. [141]

    Stereo magnification: Learning view synthesis using multiplane images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817, 2018. 4, 1, 2 14 Matrix3D: Large Photogrammetry Model All-in-One Supplementary Material Here, we pr...

    2018 arXiv

  134. [142]

    The modality-specific encoders are composed of stacked convo- lution and linear layers following [71] to patchify image- like 2D data into 1D tokens

    Model Architecture For RGB data, we use DINOv2 [70] and Stable Diffu- sion [82] V AE to extract deep features from pixels be- fore sending them into the modality-specific encoders. The modality-specific encoders are composed of stacked convo- lution and linear layers following...

  135. [143]

    In each training batch, the datasets have a proportion of 4:4:4:4:4:1

    Dataset Pre-processing As illustrated in the main paper, we train Matrix3D on a mixture of six datasets, including Objaverse [19], MVImgNet [132], CO3D-v2 [79], RealEstate10k [141], Hypersim [81], and ARKitScenes [4]. In each training batch, the datasets have a proportion of 4...

  136. [144]

    We didn’t apply any data augmenta- tion techniques and center-cropped the input images into a square

    Training Details Table 7 reports the detailed training hyper-parameter set- tings of three stages. We didn’t apply any data augmenta- tion techniques and center-cropped the input images into a square

  137. [145]

    DTU Dataset Split for Depth Evaluation In Sec

    Experiments 10.1. DTU Dataset Split for Depth Evaluation In Sec. 4.3, we use different evaluation set for monodepth and multi-view depth evaluation. Specifically, we use the IDR [129] subset for monodepth because perfect fore- ground masks are provided, and follow previous wor...

  138. [146]

    The evaluation met- rics for each task is the same as the main paper

    initialization with halved batch size and total steps due to limited time and compute resources. The evaluation met- rics for each task is the same as the main paper. Quantitative reuslts are shown in Table 10. The model with masked learning strategy ( Multi-task) surpasses th...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.