Pith. sign in

REVIEW 4 major objections 7 minor 54 references

LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusion

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A generative video-diffusion framework can build open-vocabulary 3D language-embedded scenes from as few as two images, outperforming per-scene optimization baselines.

desk verdict Novel pipeline that plausibly builds language-embedded 3D scenes from sparse views, but the headline margins over baselines are not interpretable until the view-count protocol is disclosed; worth reviewing, needs heavy revision. read the letter →

arxiv 2507.02813 v1 pith:RNVICPCD submitted 2025-07-03 cs.CV

Fangfu Liu , Hao Li , Jiawei Chi , Hanyang Wang , Minghui Yang , Fudong Wang , Yueqi Duan This is my paper
classification cs.CV
keywords 3Dlanguage-embeddedscenessparse-viewreconstructionvideodiffusionmodelsopen-vocabularysceneunderstandinglanguagequantizedcompressorGaussiansplattingsemanticsegmentationgenerative
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LangScene-X claims that a single video-diffusion model can generate the missing appearance, geometry, and semantics that a 3D language-embedded scene needs, so that only two input views are required instead of the twenty-plus calibrated views that current language-field methods demand. A sympathetic reader should care because this changes the practical bottleneck for open-vocabulary 3D querying from data capture to prompt engineering: robots, AR/VR, and navigation systems could build queryable 3D memories from a quick photo burst. The paper also argues that its language quantized compressor, trained once on a large image dataset, replaces the per-scene autoencoders used by prior work, giving cross-scene generalization and faster rendering. On LERF-OVS and ScanNet, the reported numbers put LangScene-X ahead of both per-scene optimized language fields and a generalizable baseline, with a reported +10.58 mIoU and +31.18 mAcc on LERF-OVS.

What carries the argument

The load-bearing object is the TriMap video diffusion model, a transformer-based video diffusion adapted for key-frame interpolation: it takes two views, encodes them with a causal VAE, concatenates the latent with noise, and denoises to RGB, normal, and semantic videos. Its four-stage progressive training — web-scale interpolation, then 3D-consistent video finetuning, then normal annotation, then semantic annotation — is what lets one model carry appearance, geometry, and semantics simultaneously, and the paper reports that this schedule yields better geometric feature matching than a vanilla video diffusion. The second named mechanism is the Language Quantized Compressor (LQC), a vector-quantized encoder-decoder with a learnable codebook of 2048 embeddings of dimension 3; it compresses high-dimensional dense language features to discrete 3-channel indices while preserving text-query activation, trained with reconstruction, embedding, and mask-alignment losses on a large image dataset. These two mechanisms feed the language-embedded surface fields: Gaussian splatting initialized by a geometric prior from the input views, regularized by generated normals with an angle-based uncertainty filter, and supervised by 2D/3D clustering losses on the generated segmentation masks. The argument's force is that the generated frames, not the original sparse inputs, are what supply the dense multimodal supervision.

What would settle it

Take a ScanNet scene with ground-truth depth and semantic labels, run LangScene-X with only two input views, and compare the generated normal maps and semantic masks against the ground truth per frame using depth error and semantic mIoU. If frames with large geometric error or hallucinated object masks still yield high 3D query accuracy, the pipeline's robustness is not explained by frame fidelity; if the accuracy collapses on such frames, the central claim that generated frames are trustworthy supervision is falsified.

Watch

Extended reading notes

Core claim

The paper's core discovery, stated on its own terms, is that reconstruction and understanding can be unified in a generative prior: instead of recovering 3D language fields directly from sparse views, LangScene-X first uses a TriMap video diffusion model to synthesize a dense, 3D-consistent video of RGB frames, normal maps, and hierarchical semantic masks from as few as two key-frames, then trains language-embedded 3D Gaussians on those generated frames. The architecture uses progressive multi-task training — web-scale key-frame interpolation, then 3D-consistent video finetuning, then normal annotation, then semantic annotation — to make one DiT-based model produce consistent outputs across all three modalities. The paper's second claim is that a Language Quantized Compressor trained on a large image dataset maps dense language features to a 3-channel discrete codebook with vector quantization, and that this discrete representation preserves language alignment well enough to replace scene-specific autoencoders. Finally, the language surface fields are trained by aligning the compressed semantics onto the geometry from generated normal maps, with clustering losses in 2D and 3D. The result is a pipeline that claims to beat per-scene optimized and feed-forward baselines on open-vocabulary localization and segmentation.

Load-bearing premise

The load-bearing premise is that frames generated by the video diffusion model from two input views are 3D-consistent and semantically faithful enough to supervise the language surface fields; if the generated RGB, normal, or segmentation maps contain hallucinations or misalignments, the final 3D language queries inherit those errors.

Editorial extensions

If this is right

  • A device with two overlapping photos of an unseen scene could construct a 3D field that answers open-vocabulary text queries about object locations, without dense video capture or per-scene retraining.
  • The large-dataset-trained language quantized compressor should transfer to new scenes as-is, reducing per-scene work to the diffusion generation step plus a short Gaussian optimization.
  • The reported gains on the two benchmarks — +10.58 mIoU and +31.18 mAcc on LERF-OVS — imply that sparse-view generative reconstruction can outperform methods that see all training views on indoor tabletop and kitchen scenes.
  • Because semantics are aligned to generated normals, text queries can return sharp object boundaries rather than fuzzy radiance-cloud activations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the pipeline's success is only as trustworthy as the generated frames, so a direct measure of semantic and geometric fidelity of those frames — not just feature matching — would be the decisive next evaluation.
  • Editorial inference: since each domain enters through a channel-preserving mapper, the same progressive integration scheme could in principle generate depth, material, affordance, or other maps, turning TriMap into a general multimodal 3D prior.
  • Editorial inference: the reported experiments use two input views; sweeping the input count from one to several would show whether the generative prior or the downstream Gaussian fitting is the binding constraint.
  • Editorial inference: a discrete language codebook could support text-driven 3D editing — moving, copying, or deleting an object identified by its code — although the paper does not explore that use.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. LangScene-X proposes a generative framework for building generalizable 3D language-embedded Gaussian fields from as few as two sparse calibrated views. The method trains a TriMap video diffusion model to generate a sequence of 49 frames comprising RGB, normal maps, and semantic segmentation masks, then uses a Language Quantized Compressor (LQC) to encode dense CLIP features into 3-dimensional discrete latents, and finally reconstructs language surface fields with RGB, normal, and semantic losses. Experiments on LERF-OVS and ScanNet report large improvements over LSeg, LangSplat, LangSurf, and LSM in mIoU and mAcc, supporting the central claim that a generative video prior can replace dense calibrated views and per-scene language autoencoders for open-vocabulary 3D scene understanding.

Significance. If the reported results hold under controlled comparison, this is a significant step toward practical sparse-view 3D scene understanding: it replaces per-scene optimization and dense capture with a generative video prior, and it introduces a generalizable quantized language compressor that avoids scene-specific retraining. The paper's strengths include the progressive multi-task training scheme for the video diffusion model, the explicit design of a 3-channel discrete language representation, and the breadth of real-scene evaluations spanning two benchmarks. The main caveat is that the evaluation protocol is under-specified, which currently prevents full confidence in the headline quantitative margins.

major comments (4)
  1. [§4.1–4.2, Tables 1–2] The comparison against LangSplat, LangSurf, and LSM is not controlled. For each baseline, the manuscript does not state the number of input views (dense calibrated for per-scene optimization methods, sparse unposed for LSM), nor whether the baseline numbers were recomputed on the same prompts and ground-truth masks or copied from prior papers. Because Sec. 3.2 characterizes LangSplat/LangSurf as requiring very dense views (typically more than 20) while LangScene-X uses as few as two input images, the reported margins (+10.58 mIoU and +31.18 mAcc on LERF-OVS; +14.92 mIoU on ScanNet) cannot be attributed to the proposed method without specifying matched input regimes. Please provide per-method view counts, recompute or cite the exact source of each baseline score, and report per-prompt breakdowns and variance across scenes.
  2. [§4.3, Table 3] The ablation study is carried out on one scene per dataset (ScanNet scene0085 and LERF 'Teaime' [sic]), and the table legend is illegible (rows contain a mixture of '!' and '%' symbols with no key). A single scene cannot support the conclusion that progressive training and LQC are necessary components; the observed differences (e.g., 44.25→51.68 on ScanNet) are within plausible scene-to-scene variation. Moreover, the LQC comparison in Fig. 7 is a training curve on COCO, not a downstream 3D language-field metric; please run the ablations on all scenes and report the same mIoU/mAcc metrics as the main tables.
  3. [§3.2, §4.3, Fig. 6] The core enabling assumption that the TriMap video diffusion produces 3D-consistent and semantically accurate frames is only supported by a feature-matching inlier count on an unspecified number of videos. There is no quantitative measurement of generated RGB fidelity, normal-map angular error, or semantic-mask agreement with ground truth. Since the language surface fields are trained on these generated frames, any hallucination or misalignment propagates into the final language queries. Please add quantitative fidelity evaluations of the generated modalities on held-out scenes, and ideally analyze the effect of generated-frame errors on downstream localization/segmentation accuracy.
  4. [§4.1, §3.1] The number of input views actually used in the main experiments is never specified. The abstract and Sec. 3.1 say 'as few as two images,' but Sec. 4.1 does not state whether every LERF/ScanNet scene was reconstructed from exactly two views, how those views were selected (baseline, overlap), or whether additional views were used. This is central to the sparse-view claim and to reproducing the experiments. Please report the exact input configuration per scene, including the number of views, the view-selection criterion, and the camera poses.
minor comments (7)
  1. [§3.2, Eq. (7)] Equation (7) is malformed; the piecewise definition does not display the norm on the second branch, and the threshold variable is denoted θn in the text but θp in Eq. (8). Please clarify the notation and the exact formulation.
  2. [§3.3, Eq. (5)] The 'pseudo-mask supervision' in Eq. (5) is not defined; please specify how the text-guided activation maps and pseudo masks are generated during LQC training on COCO.
  3. [§4.3, Fig. 6] The number of video clips used for the feature-matching comparison is not reported; without this information, the figure is anecdotal.
  4. [References] Reference [32] is the SIFT mutation-prediction paper, not the image feature matcher; the correct citation for SIFT keypoints should be added.
  5. [Tables 1–2] The LSeg baseline numbers appear inconsistent with previously published LSeg results; please state the exact evaluation protocol and the source of each baseline number.
  6. [Throughout] Please correct typographical errors: 'Language-Embeded' in the Section 3.4 heading, 'Teaime' in Table 3, 'pratice' in Sec. 3.4, and 'Dingewn Zhang' in reference [20].
  7. [§3.4, Eq. (9)] The KL term f_sem_k log(f_sem_k / f_sem_j) has mismatched summation indices; please define f_sem_j clearly and verify the clustering objective.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pipeline is supervised by external models and held-out evaluation.

full rationale

The derivation chain is self-contained. TriMap video diffusion (Sec. 3.2) is trained with a standard diffusion objective (Eq. 1) on external web data, RealEstate-10K, and ACID, using external domain mappers StableNormal and SAM2; the generated RGB/normal/semantic frames are then used as pseudo-ground-truth for the downstream Gaussian field, not as the evaluation target. LQC (Sec. 3.3) is a VQ-VAE-style compressor trained on COCO with reconstruction, codebook, and text-activation-preservation losses (Eqs. 3-6); though it shares the CLIP feature space used later, that is the intended function of a compressor rather than circularity, and it is not per-scene fitted. The language surface fields (Sec. 3.4) are optimized against the generated frames and CLIP/LQC features, while the quantitative claims (Sec. 4.2) are computed against held-out real-world ground-truth annotations on LERF-OVS and ScanNet. Self-citations (ReconX, DimensionX, etc.) appear only in related work and are not load-bearing for the central claim. The concern about uncontrolled baseline view counts is an experimental-validity issue, not a circularity reduction, and cannot be scored as circularity under the evidence rule.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The central claim rests on standard pretrained models (CogVideoX, StableNormal, SAM2, CLIP) and on the paper's new TriMap diffusion and LQC components. The main modeling assumption is that CLIP features can be safely quantized to a 3-channel code, and that generated multi-modal frames are reliable enough to supervise the final 3D field. Several hyperparameters that affect the normal supervision are not specified.

free parameters (4)
  • lambda1, lambda2, lambda3 (LQC loss weights) = 1, 0.2, 0.5
    Chosen by hand; balance reconstruction, embedding, and mask-alignment losses in Eq. (6).
  • tau_thr (normal angle threshold)
    Filters uncertain normal regions in Eq. (7); value not reported, making the geometry supervision incompletely specified.
  • Tn (normal loss step threshold)
    Switches between two normal loss forms in Eq. (7); value not reported.
  • Codebook size K and latent channels D = K=2048, D=3
    Design choices for LQC; not swept or justified beyond the paper's experiments.
assumptions (4)
  • domain assumption CogVideoX backbone provides a strong pretrained prior for novel-view video synthesis
    The TriMap video diffusion builds on CogVideoX and expects it to generalize to unseen scenes; no evidence beyond the reported experiments.
  • domain assumption StableNormal and SAM2 produce accurate normal maps and segmentation masks for training and inference
    Used to annotate training videos and to extract semantics at inference (Section 4.1); errors in these modules propagate into the generated frames.
  • ad hoc to paper CLIP language features are inherently discrete and can be compressed to 3 channels via vector quantization without losing discriminative information
    This is the paper's key motivation for LQC (Section 3.3, 'language features are inherently discrete'); it is a modeling assumption, not proven, and the 3-channel choice is not justified beyond experiments.
  • standard math Standard autoencoder losses and straight-through gradient estimator are valid for training LQC
    Standard practice in discrete representation learning; no need to prove.
invented entities (2)
  • TriMap video diffusion model
    purpose: Jointly generates RGB, normal, and semantic videos from sparse input views to provide dense supervision for 3D language fields.
    New component introduced in this paper; evaluated only through the paper's own experiments, with no external reproduction yet.
  • Language Quantized Compressor (LQC)
    purpose: Compresses dense CLIP features into 3-channel discrete codes for efficient Gaussian language fields.
    New module trained on COCO; its only evaluation is in this paper's ablation and downstream tasks, with no external benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusion." pith.science (2026). https://pith.science/paper/RNVICPCD

@misc{pith2026250702813,
  author       = {Pith},
  title        = {Pith review of: LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RNVICPCD}},
  note         = {Machine review of arXiv:2507.02813}
}
read the original abstract

Recovering 3D structures with open-vocabulary scene understanding from 2D images is a fundamental but daunting task. Recent developments have achieved this by performing per-scene optimization with embedded language information. However, they heavily rely on the calibrated dense-view reconstruction paradigm, thereby suffering from severe rendering artifacts and implausible semantic synthesis when limited views are available. In this paper, we introduce a novel generative framework, coined LangScene-X, to unify and generate 3D consistent multi-modality information for reconstruction and understanding. Powered by the generative capability of creating more consistent novel observations, we can build generalizable 3D language-embedded scenes from only sparse views. Specifically, we first train a TriMap video diffusion model that can generate appearance (RGBs), geometry (normals), and semantics (segmentation maps) from sparse inputs through progressive knowledge integration. Furthermore, we propose a Language Quantized Compressor (LQC), trained on large-scale image datasets, to efficiently encode language embeddings, enabling cross-scene generalization without per-scene retraining. Finally, we reconstruct the language surface fields by aligning language information onto the surface of 3D scenes, enabling open-ended language queries. Extensive experiments on real-world data demonstrate the superiority of our LangScene-X over state-of-the-art methods in terms of quality and generalizability. Project Page: https://liuff19.github.io/LangScene-X.

Figures

Figures reproduced from arXiv: 2507.02813 by the authors.

Figure 1
Figure 1. LangScene-X: Given sparse views as input (e.g., as few as two images), we design a generative paradigm to build the 3D generalizable language-embedded surface fields with TriMap video diffusion and language quantized compressor (LQC), which supports open-ended language queries in any 3D scenes. For example, given a prompt like “stuffed bear” in the Teatime scene, a relevancy map can be rendered focusing on the locat… view at source ↗
Figure 2
Figure 2. Pipeline of LangScene-X. Given two sparse-view images as input, we first generate a sequence of 3D consistent RGB images, normal maps, and segmentation maps from TriMap video diffusion model, which provides the dense frames for later 3D scene reconstruc￾tion and understanding. Then we project high-dimensional semantic features into low-dimensional discrete space through a generalizable Language Quantized Compressor … view at source ↗
Figure 3
Figure 3. The illustration of Language Quantized Compressor [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: 2D Segmentation Results on LERF-OVS [17] Dataset. Here, we showcase two cases (i.e., Teatime, Kitchen) with multiple segmentation masks with text query. On the top, we display the rendered results of our method and other methods, along with the corresponding ground tru…
Figure 5
Figure 5. Figure 5: 2D Segmentation Results on Scannet [7] Dataset. Here, we showcase two cases (i.e., 0085 00, 0114 00) with multiple segmentation masks with text query. The masks predicted by ours contain more comprehensive regions and sharper boundaries than other methods, such as the …
Figure 6
Figure 6. Figure 6: Feature Matching comparison between our method and vanilla video diffusion mdoel . L2 Loss (log) Step Method AE (channel=3) LQC (channel=3) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Training Curve comparison between our LQC and reg￾ular autoencoder technique. curacy in both mIoU and mAcc metrics with a large mar￾gin, i.e., a 10.58% in terms of mIoU and a 31.18% in terms of mAcc on LERF-OVS dataset. On the ScanNet dataset, the improvement upon the …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 22 canonical work pages

  1. [1]

    Segdiff: Image segmentation with diffusion proba- bilistic models

    Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. Segdiff: Image segmentation with diffusion proba- bilistic models. arXiv preprint arXiv:2112.00390, 2021. 2

    arXiv 2021

  2. [2]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 19129– 19139, 2022. 2

    work page 2022

  3. [3]

    Vd3d: Taming large video diffu- sion transformers for 3d camera control

    Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siaro- hin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffu- sion transformers for 3d camera control. arXiv preprint arXiv:2407.12781, 2024. 3

    arXiv 2024

  4. [4]

    pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction

    David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19457–19467, 2024. 3

    2024

  5. [5]

    Pgsr: Planar-based gaussian splatting for efficient and high-fidelity surface reconstruction

    Danpeng Chen, Hai Li, Weicai Ye, Yifan Wang, Weijian Xie, Shangjin Zhai, Nan Wang, Haomin Liu, Hujun Bao, and Guofeng Zhang. Pgsr: Planar-based gaussian splatting for efficient and high-fidelity surface reconstruction. arXiv preprint arXiv:2406.06521, 2024. 2

    arXiv 2024

  6. [6]

    Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images

    Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. In European Conference on Computer Vision, pages 370–386. Springer, 2024. 3

    work page 2024

  7. [7]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 2, 6, 7, 8

    work page 2017

  8. [8]

    Large spatial model: End-to-end unposed images to semantic 3d

    Zhiwen Fan, Jian Zhang, Wenyan Cong, Peihao Wang, Renjie Li, Kairun Wen, Shijie Zhou, Achuta Kadambi, Zhangyang Wang, Danfei Xu, et al. Large spatial model: End-to-end unposed images to semantic 3d. Advances in Neural Information Processing Systems , 37:40212–40229,

    work page

Show all 54 references
  1. [9]

    Random sample consensus: a paradigm for model fitting with applications to image analy- sis and automated cartography

    MA FISCHLER AND. Random sample consensus: a paradigm for model fitting with applications to image analy- sis and automated cartography. Commun. ACM, 24(6):381– 395, 1981. 8

    1981

  2. [10]

    Iqa: Visual question answering in interactive environments

    Daniel Gordon, Aniruddha Kembhavi, Mohammad Raste- gari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4089–4098, 2018. 2

    2018

  3. [11]

    Sugar: Surface- aligned gaussian splatting for efficient 3d mesh reconstruc- tion and high-quality mesh rendering

    Antoine Gu ´edon and Vincent Lepetit. Sugar: Surface- aligned gaussian splatting for efficient 3d mesh reconstruc- tion and high-quality mesh rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5354–5363, 2024. 2

    2024

  4. [12]

    Cameractrl: Enabling camera control for text-to-video generation

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024. 3

    2024 arXiv

  5. [13]

    Sparselgs: Sparse view language embedded gaussian splat- ting

    Jun Hu, Zhang Chen, Zhong Li, Yi Xu, and Juyong Zhang. Sparselgs: Sparse view language embedded gaussian splat- ting. arXiv preprint arXiv:2412.02245, 2024. 2

    2024 arXiv

  6. [14]

    Depthcrafter: Generating consistent long depth sequences for open-world videos

    Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos. arXiv preprint arXiv:2409.02095, 2024. 3

    2024 arXiv

  7. [15]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics ,

  8. [17]

    Lerf: Language embedded radiance fields

    Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 19729–19739,

  9. [18]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023. 3

    2023 arXiv

  10. [19]

    Language-driven semantic seg- mentation

    Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic seg- mentation. In International Conference on Learning Rep- resentations, 2022. 5, 6

    2022

  11. [20]

    Langsurf: Language-embedded surface gaussians for 3d scene under- standing

    Hao Li, Roy Qin, Zhengyu Zou, Diqi He, Bohan Li, Bingquan Dai, Dingewn Zhang, and Junwei Han. Langsurf: Language-embedded surface gaussians for 3d scene under- standing. arXiv preprint arXiv:2412.17635, 2024. 2, 5, 6

    2024

  12. [21]

    Gp- nerf: Generalized perception nerf for context-aware 3d scene understanding

    Hao Li, Dingwen Zhang, Yalun Dai, Nian Liu, Lechao Cheng, Jingfeng Li, Jingdong Wang, and Junwei Han. Gp- nerf: Generalized perception nerf for context-aware 3d scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 217...

    2024

  13. [22]

    Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normaliza- tion

    Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normaliza- tion. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 2...

  14. [23]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceeding...

    2014

  15. [24]

    Infinite na- ture: Perpetual view generation of natural scenes from a sin- gle image

    Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite na- ture: Perpetual view generation of natural scenes from a sin- gle image. In Proceedings of the IEEE/CVF International 9 Conference on Computer Vision, pages 14458–14467, 2021. 6

    2021

  16. [25]

    Semantic ray: Learning a generalizable semantic field with cross-reprojection attention

    Fangfu Liu, Chubin Zhang, Yu Zheng, and Yueqi Duan. Semantic ray: Learning a generalizable semantic field with cross-reprojection attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17386–17396, 2023. 2

    2023

  17. [26]

    Re- conx: Reconstruct any scene from sparse views with video diffusion model

    Fangfu Liu, Wenqiang Sun, Hanyang Wang, Yikai Wang, Haowen Sun, Junliang Ye, Jun Zhang, and Yueqi Duan. Re- conx: Reconstruct any scene from sparse views with video diffusion model. arXiv preprint arXiv:2408.16767, 2024. 3, 4

    2024 arXiv

  18. [27]

    Make-your-3d: Fast and consistent subject- driven 3d content generation

    Fangfu Liu, Hanyang Wang, Weiliang Chen, Haowen Sun, and Yueqi Duan. Make-your-3d: Fast and consistent subject- driven 3d content generation. In European Conference on Computer Vision, pages 389–406. Springer, 2024. 2

    2024

  19. [28]

    Physics3d: Learning physical properties of 3d gaussians via video diffusion.arXiv preprint arXiv:2406.04338, 2024

    Fangfu Liu, Hanyang Wang, Shunyu Yao, Shengjun Zhang, Jie Zhou, and Yueqi Duan. Physics3d: Learning physical properties of 3d gaussians via video diffusion.arXiv preprint arXiv:2406.04338, 2024. 2

    2024 arXiv

  20. [29]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6

    2017 arXiv

  21. [30]

    Gaussian splatting slam

    Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and An- drew J Davison. Gaussian splatting slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 18039–18048, 2024. 2

    2024

  22. [31]

    Nerf: Representing scenes as neural radiance fields for view syn- thesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. Communications of the ACM, 65(1):99–106, 2021. 2, 3

    2021

  23. [32]

    Sift: Predicting amino acid changes that affect protein function

    Pauline C Ng and Steven Henikoff. Sift: Predicting amino acid changes that affect protein function. Nucleic acids re- search, 31(13):3812–3814, 2003. 8

    2003

  24. [33]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 4195–4205,

  25. [34]

    Langsplat: 3d language gaussian splatting

    Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20051–20060, 2024. 2, 3, 5, 6

    2024

  26. [35]

    Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever

    Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from nat- ural language supervision. In ICML, 2021. 2, 3

    2021

  27. [36]

    Language embedded radiance fields for zero-shot task-oriented grasping

    Adam Rashid, Satvik Sharma, Chung Min Kim, Justin Kerr, Lawrence Yunliang Chen, Angjoo Kanazawa, and Ken Gold- berg. Language embedded radiance fields for zero-shot task-oriented grasping. In 7th Annual Conference on Robot Learning, 2023. 2

    2023

  28. [37]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 6

    2024 arXiv

  29. [38]

    Scaling properties of diffusion models for perceptual tasks

    Rahul Ravishankar, Zeeshan Patel, Jathushan Rajasegaran, and Jitendra Malik. Scaling properties of diffusion models for perceptual tasks. arXiv preprint arXiv:2411.08034, 2024. 2, 4

    2024 arXiv

  30. [39]

    Robustness of segment anything model (sam) for autonomous driving in adverse weather conditions

    Xinru Shan and Chaoning Zhang. Robustness of segment anything model (sam) for autonomous driving in adverse weather conditions. arXiv preprint arXiv:2306.13290, 2023. 2

    2023 arXiv

  31. [40]

    Learning temporally consistent video depth from video diffusion priors

    Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Vitor Guizilini, Yue Wang, Matteo Poggi, and Yiyi Liao. Learning temporally consistent video depth from video diffusion priors. arXiv preprint arXiv:2406.01493 ,

    arXiv

  32. [41]

    Dimensionx: Create any 3d and 4d scenes from a single image with controllable video diffusion

    Wenqiang Sun, Shuo Chen, Fangfu Liu, Zilong Chen, Yueqi Duan, Jun Zhang, and Yikai Wang. Dimensionx: Create any 3d and 4d scenes from a single image with controllable video diffusion. arXiv preprint arXiv:2411.04928, 2024. 3

    2024 arXiv

  33. [42]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 5

    2017

  34. [43]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 3

    2017

  35. [44]

    Sv3d: Novel multi-view syn- thesis and 3d generation from a single image using latent video diffusion

    Vikram V oleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view syn- thesis and 3d generation from a single image using latent video diffusion. In European Conference on Computer...

    2024

  36. [45]

    Dust3r: Geometric 3d vi- sion made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vi- sion made easy. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20697– 20709, 2024. 3, 5

    2024

  37. [46]

    Opengaussian: Towards point-level 3d gaussian-based open vocabulary understanding

    Yanmin Wu, Jiarui Meng, Haijie Li, Chenming Wu, Yahao Shi, Xinhua Cheng, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, et al. Opengaussian: Towards point-level 3d gaussian-based open vocabulary understanding. arXiv preprint arXiv:2406.02058, 2024. 3

    2024 arXiv

  38. [47]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In Eu- ropean Conference on Computer Vision , pages 399–417. Springer, 2024. 3

    2024

  39. [48]

    Thinking in space: How mul- timodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How mul- timodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14171, 2024. 2

    2024 arXiv

  40. [49]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 3, 4, 6

    2024 arXiv

  41. [50]

    Stablenormal: Reducing diffusion variance for stable and sharp normal

    Chongjie Ye, Lingteng Qiu, Xiaodong Gu, Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. ACM Transactions on Graphics (TOG), 43(6):1–18, 2024. 2, 6 10

    2024

  42. [51]

    Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chieh Chen. Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36:32215–32234,

  43. [52]

    Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis.arXiv preprint arXiv:2409.02048, 2024

    Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien-Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis.arXiv preprint arXiv:2409.02048, 2024. 3, 4

    2024 arXiv

  44. [53]

    Sam-e: leveraging visual foundation model with sequence imitation for embod- ied manipulation

    Junjie Zhang, Chenjia Bai, Haoran He, Wenke Xia, Zhigang Wang, Bin Zhao, Xiu Li, and Xuelong Li. Sam-e: leveraging visual foundation model with sequence imitation for embod- ied manipulation. arXiv preprint arXiv:2405.19586 , 2024. 2

    2024 arXiv

  45. [54]

    Motiondirector: Motion customization of text-to-video diffusion models

    Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Jun- hao Zhang, Jia-Wei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. In European Conference on Computer Vision, pages 273–290. Springer, 2024. 3

    2024

  46. [55]

    Stereo magnification: Learning view synthesis using multiplane images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817, 2018. 6 11

    2018 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.