REVIEW 3 major objections 5 minor 2 cited by
UniForward: Unified 3D Scene and Semantic Field Reconstruction via Feed-Forward Gaussian Splatting from Only Sparse-View Images
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read UniForward reconstructs a 3D scene and its semantic field from two unposed, uncalibrated images in a single feed-forward pass that needs no depth labels during training.
desk verdict Useful integrated feed-forward splatting with strong results, but the SOTA claim against LSM rests on a comparison the paper itself contradicts. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a 3D Gaussian with an extra semantic feature vector: each Gaussian carries position, covariance, opacity, color, and an $N$-dimensional semantic embedding, so the same splatting operation renders images and semantic feature maps. The argument runs through a dual-branch decoupled decoder with separately supervised heads for geometry, appearance, and semantics, plus a loss-guided view sampler that starts training on close views with small baselines and gradually increases view angle once pose loss stabilizes. This sampler is what removes the need for ground-truth depth or masks, and the decoupled design is what prevents joint regression of geometry and semantics from collapsing into the suboptimal results the paper attributes to earlier single-branch feed-forward models.
What would settle it
Feed UniForward two images of a room whose camera baseline and metric depth are known, then compare the reconstructed Gaussian centers to a metric laser scan without any scale alignment; if the cloud matches only after applying a constant scale factor, the scale-ambiguity premise is confirmed, and if it matches metrically without alignment, the paper's normalization claim is wrong.
Extended reading notes
Core claim
The central claim is that a feed-forward network can jointly predict, from only two unposed and uncalibrated images, a set of 3D Gaussians whose per-Gaussian semantic features constitute a 3D semantic field. Rendering those features from arbitrary views and decoding them with a pre-trained open-vocabulary decoder produces dense, view-consistent segmentation masks. The paper shows that separating the prediction into a geometry branch and an attribute branch, supervised by photometric and semantic-distillation losses, lets the model train without depth labels or masks and outperform prior feed-forward and per-scene baselines on both novel view synthesis and novel view segmentation.
Load-bearing premise
The method assumes that reconstructing a scene up to an unknown scale from two images, after training with ground-truth poses and normalized baseline distances, is enough for the downstream use case, because at inference the model never sees metric distances.
Editorial extensions
If this is right
- A single forward pass (about 0.1 seconds on one GPU) yields both a renderable 3D scene and a 3D semantic field, so novel-view segmentation becomes available at interactive rates without per-scene optimization.
- Because the training objective needs no depth or masks, the same pipeline can in principle be trained on large collections of posed video frames rather than only depth-annotated datasets.
- The reconstructed semantic field is view-consistent and open-vocabulary: a text query can segment the same object consistently from different novel viewpoints.
- Reconstruction is limited to regions visible in the two input views, and the fidelity of the semantic field is capped by the pre-trained 2D semantic model used for distillation.
- All geometry is recovered up to an unknown scale, so metric distances are not available from the model output alone.
Reading between the lines
- If the scale ambiguity were resolved by injecting any known metric length (an object of known size, a stereo baseline, or a monocular metric-depth head), the same architecture could slot directly into navigation or AR pipelines without retraining.
- Since the semantic head already accounts for a large segmentation gain in the paper's ablations, swapping in a stronger dense open-vocabulary feature extractor should directly raise novel-view segmentation quality.
- The loss-guided view sampler is effectively a curriculum on angular baseline; a natural extension is to replace its angle threshold with a learned overlap predictor so that it handles irregular video framerates from handheld or egocentric capture.
- Reporting metric-scale errors after aligning predicted and ground-truth scenes would make explicit how much of the performance depends on the scale normalization.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. UniForward is a feed-forward 3D Gaussian Splatting model that takes two unposed, uncalibrated sparse-view images and, in a single forward pass, reconstructs a 3D scene together with a semantic field. The Gaussian centers are regressed directly in a shared canonical space, while appearance and semantic features are predicted by a separate attribute branch. Training uses a photometric loss, a pose loss as an auxiliary geometry signal, and distillation from a pretrained 2D semantic model (LSeg), without ground-truth depth. To stabilize training, the paper introduces a loss-guided view sampler that progressively increases the angular separation between context views. Experiments on ScanNet++ and ScanNet report improved novel-view synthesis and novel-view segmentation over feed-forward baselines (pixelSplat, MVSplat, LSM, Splatt3R, NoPoSplat) and per-scene optimized methods (Feature 3DGS, DFFs).
Significance. If the empirical claims hold, UniForward is a meaningful contribution: it is the first feed-forward 3DGS model I am aware of that jointly predicts geometry, appearance, and open-vocabulary semantic features from two unposed, uncalibrated images without depth supervision, while running in about 0.1 s at inference. The ablations in Table 3 support the loss-guided sampler, the pose head, the semantic head, and the shortcut connection, and the novel-view segmentation comparison against 2D LSeg is a genuine test of 3D-consistent semantics rather than a self-supervised circularity. The scale ambiguity is disclosed in Section 3.3 and is common to pose-free reconstruction, though it is not fully addressed in the downstream-motivation claims. The main risk to the headline state-of-the-art claim is the inconsistency in the LSM baseline protocol, which must be resolved before the quantitative ranking can be accepted.
major comments (3)
- [Section 4.1 and Supplementary A.3, Tables 1 and 4] Section 4.1 states 'For a fair comparison, we retrained all the comparison methods under the same setting,' but Supplementary A.3 says that for LSM the authors 'directly use the official pre-trained weights, as the model is trained on ScanNet and ScanNet++, which include our datasets.' These two statements conflict. If LSM's official training included the ScanNet++ validation scenes used here for testing, or if LSM was trained at a different resolution, with different data, or with more compute, then the reported LSM numbers (16.005 PSNR, 0.320 mIoU in Table 1; 19.083 PSNR, 0.341 mIoU in Table 4) are not a valid same-setting baseline. Because LSM is the only prior feed-forward method that also performs unified scene and semantic reconstruction, the central claim of state-of-the-art unified reconstruction rests on this comparison. Please retrain LSM under the same training split and protocol, or provide rigorous evidence that the official weights were not trained on the evaluation scenes and that evaluation conditions match.
- [Section 4.3, Table 2] Table 2 compares UniForward with per-scene optimized methods (Feature 3DGS, DFFs) only on scenes where SfM succeeded for those baselines, while Table 1 reports results on the full 49-scene ScanNet++ validation set. The 28.097 PSNR reported for Ours in Table 2 is therefore not directly comparable to the 26.147 PSNR in Table 1, and the claim of outperforming per-scene optimized methods is established only on a favorable subset. Please report the number of scenes used in Table 2, results of all methods on that same subset, and, where feasible, results on the full test set for the per-scene baselines.
- [Section 3.3 and Section 5] The paper correctly notes in Section 3.3 that the distance between input views is normalized to resolve scale ambiguity and that the model reconstructs scaled scenes, but at inference the model receives only images and cannot recover metric scale. This means all reconstructed geometry and the semantic field live in a scale-ambiguous normalized frame. The introduction and conclusion motivate the method for robot navigation and AR/VR, which require metric or at least consistently scaled geometry, yet the Limitations paragraph in Section 5 does not mention this limitation and no experiment measures scale error. Please add an explicit limitation statement and, if possible, an evaluation of scale recovery or an external alignment step.
minor comments (5)
- [Tables 1-4] All quantitative results are reported from a single run without error bars or multiple seeds. Given the modest margins over NoPoSplat in Table 1 (PSNR 26.147 vs 24.755), reporting variance or repeated-run statistics would strengthen the comparison.
- [Table 1 and Figure 4] The LSeg baseline setup should be clarified: in qualitative figures LSeg is applied to ground-truth novel views (labeled 'w GT'), while in Table 1 the LSeg row lists N/A for synthesis and only segmentation numbers. Please state explicitly whether the Table 1 LSeg segmentation is evaluated on ground-truth novel views, rendered images, or input views.
- [Supplementary A.3] The sentence 'as the model is trained on ScanNet and ScanNet++, which include our datasets' is ambiguous: it could mean LSM was trained on the same training splits or on the same entire datasets including the validation scenes used for testing. Please clarify the exact overlap with the evaluation split.
- [Section 3.2.2 and Equation (3)] The semantic head predicts an 'N-dimensional' feature, but N is never specified. Please state the feature dimension used in the experiments.
- [Algorithm 1] The loss-guided view sampler depends on a sliding window over the pose loss list L and a schedule function Schedule(i), but neither the window size nor the schedule is defined in the text. Please provide these details for reproducibility.
Circularity Check
No significant circularity: UniForward's claims are validated against external ground-truth benchmarks; the LSM baseline inconsistency is a fairness issue, not a circular derivation.
full rationale
UniForward's central claims are not circular. The scene branch is trained with a photometric loss against ground-truth novel-view images and the pose head against ground-truth extrinsics, and it is tested on held-out ScanNet++ and ScanNet scenes with standard PSNR/SSIM/LPIPS and mAcc/mIoU metrics, so the reported numbers are externally grounded rather than forced by construction. The semantic head distills features from the pre-trained 2D LSeg model, and the same LSeg decoder is used during evaluation, but the evaluated quantity is whether rendered 3D semantic fields, when decoded, match ground-truth class labels at novel views. This is a genuine 3D-consistency test; the improvement over 2D LSeg applied directly to ground-truth views is an empirical result, not an identity. The normalized-scale limitation is disclosed in Section 3.3, affects all pose-free methods equally, and does not encode the outcome. The only notable inconsistency is in the baseline protocol: Section 4.1 states 'For a fair comparison, we retrained all the comparison methods under the same setting,' while Supplementary A.3 says 'For LSM, we directly use the official pre-trained weights.' This could undermine the SOTA comparison with LSM, but it is an experimental-fairness or correctness concern, not circular reasoning. Reference [22] is a self-citation, but it appears only as one related-work example and is not load-bearing; no fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work. Overall, the derivation chain is self-contained against external benchmarks.
Assumptions & free parameters
free parameters (5)
- lambda_pose =
0.1
- lambda_sem =
0.1
- eta =
0.15
- initial_angle_threshold_theta =
not reported
- angle_increment_Delta_theta =
not reported
assumptions (6)
- domain assumption MASt3R pretrained weights provide a good feature space for joint pose-free point regression.
- domain assumption Ground truth camera extrinsics are available during training for both context and target views.
- ad hoc to paper The distance between input views is normalized to resolve scale ambiguity, and target poses are scaled accordingly.
- domain assumption LSeg 2D semantic features are dense and view-consistent enough to be lifted into a 3D semantic field via Gaussian splatting.
- ad hoc to paper Pose-loss stability is a valid signal that the model can handle larger view angles.
- domain assumption The dual-branch decoupled decoder architecture improves over joint regression of geometry and attributes.
Cite this review
Pith. "Pith review of UniForward: Unified 3D Scene and Semantic Field Reconstruction via Feed-Forward Gaussian Splatting from Only Sparse-View Images." pith.science (2026). https://pith.science/paper/URCTKSSN
@misc{pith2026250609378,
author = {Pith},
title = {Pith review of: UniForward: Unified 3D Scene and Semantic Field Reconstruction via Feed-Forward Gaussian Splatting from Only Sparse-View Images},
year = {2026},
howpublished = {\url{https://pith.science/paper/URCTKSSN}},
note = {Machine review of arXiv:2506.09378}
}
read the original abstract
We propose a feed-forward Gaussian Splatting model that unifies 3D scene and semantic field reconstruction. Combining 3D scenes with semantic fields facilitates the perception and understanding of the surrounding environment. However, key challenges include embedding semantics into 3D representations, achieving generalizable real-time reconstruction, and ensuring practical applicability by using only images as input without camera parameters or ground truth depth. To this end, we propose UniForward, a feed-forward model to predict 3D Gaussians with anisotropic semantic features from only uncalibrated and unposed sparse-view images. To enable the unified representation of the 3D scene and semantic field, we embed semantic features into 3D Gaussians and predict them through a dual-branch decoupled decoder. During training, we propose a loss-guided view sampler to sample views from easy to hard, eliminating the need for ground truth depth or masks required by previous methods and stabilizing the training process. The whole model can be trained end-to-end using a photometric loss and a distillation loss that leverages semantic features from a pre-trained 2D semantic model. At the inference stage, our UniForward can reconstruct 3D scenes and the corresponding semantic fields in real time from only sparse-view images. The reconstructed 3D scenes achieve high-quality rendering, and the reconstructed 3D semantic field enables the rendering of view-consistent semantic features from arbitrary views, which can be further decoded into dense segmentation masks in an open-vocabulary manner. Experiments on novel view synthesis and novel view segmentation demonstrate that our method achieves state-of-the-art performances for unifying 3D scene and semantic field reconstruction.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 2 Pith papers
-
Visual Execution and Validation of Finite-State Machines and Pushdown Automata
Two new visualization tools for the FSM language step through all computations of nondeterministic finite-state machines and pushdown automata and let users check state properties during transitions.
-
Sparse-View 3D Reconstruction: Recent Advances and Open Challenges
A comprehensive survey that organizes sparse-view 3D reconstruction methods into geometry-based, NeRF, 3DGS, and diffusion-based categories, with benchmarks and open challenges.
Reference graph
Works this paper leans on
-
[1]
D. Charatan, S. L. Li, A. Tagliasacchi, and V . Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19457–19467, 2024
work page 2024
-
[2]
Y . Chen, H. Xu, C. Zheng, B. Zhuang, M. Pollefeys, A. Geiger, T.-J. Cham, and J. Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images.arXiv preprint arXiv:2403.14627, 2024
arXiv 2024
-
[3]
A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richly- annotated 3d reconstructions of indoor scenes. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017
work page 2017
-
[4]
J. Duan, S. Yu, H. L. Tan, H. Zhu, and C. Tan. A survey of embodied ai: From simulators to research tasks.IEEE Transactions on Emerging Topics in Computational Intelligence (TETCI), 6(2):230–244, 2022
work page 2022
-
[5]
Z. Fan, P. Wang, Y . Jiang, X. Gong, D. Xu, and Z. Wang. Nerf-sos: Any-view self-supervised object segmentation on complex scenes.arXiv preprint arXiv:2209.08776, 2022
arXiv 2022
-
[6]
Z. Fan, J. Zhang, W. Cong, P. Wang, R. Li, K. Wen, S. Zhou, A. Kadambi, Z. Wang, D. Xu, et al. Large spatial model: End-to-end unposed images to semantic 3d.Advances in Neural Information Processing Systems (NeurIPS), 37:40212–40229, 2025. 10
work page 2025
-
[7]
Y . Ji, H. Zhu, J. Tang, W. Liu, Z. Zhang, X. Tan, and Y . Xie. Fastlgs: Speeding up language embedded gaussians with feature grid mapping. InProceedings of the AAAI Conference on Artificial Intelligence, 2025
work page 2025
- [8]
Show all 34 references
-
[9]
J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded radiance fields. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 19729–19739, 2023
2023
-
[10]
Kobayashi, E
S. Kobayashi, E. Matsumoto, and V . Sitzmann. Decomposing nerf for editing via feature field distillation.Advances in Neural Information Processing Systems (NeurIPS), 35:23311–23330, 2022
2022
-
[11]
Leroy, Y
V . Leroy, Y . Cabon, and J. Revaud. Grounding image matching in 3d with mast3r. InEuropean Conference on Computer Vision (ECCV), pages 71–91. Springer, 2024
2024
-
[12]
B. Li, K. Q. Weinberger, S. Belongie, V . Koltun, and R. Ranftl. Language-driven semantic segmentation.arXiv preprint arXiv:2201.03546, 2022
2022 arXiv
-
[13]
F. Liu, C. Zhang, Y . Zheng, and Y . Duan. Semantic ray: Learning a generalizable semantic field with cross-reprojection attention. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17386–17396, 2023
2023
-
[14]
Mildenhall, P
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis.Communications of the ACM, 65(1):99–106, 2021
2021
-
[15]
M. Qin, W. Li, J. Zhou, H. Wang, and H. Pfister. Langsplat: 3d language gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20051–20060, 2024
2024
-
[16]
Ranftl, A
R. Ranftl, A. Bochkovskiy, and V . Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12179–12188, 2021
2021
-
[17]
Z. Ren, A. Agarwala, B. Russell, A. G. Schwing, and O. Wang. Neural volumetric object selection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6133–6142, 2022
2022
-
[18]
J.-C. Shi, M. Wang, H.-B. Duan, and S.-H. Guan. Language embedded 3d gaussians for open- vocabulary scene understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5333–5343, 2024
2024
-
[19]
Siddiqui, L
Y . Siddiqui, L. Porzi, S. R. Buló, N. Müller, M. Nießner, A. Dai, and P. Kontschieder. Panop- tic lifting for 3d scene understanding with neural fields. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9043–9052, 2023
2023
-
[20]
Smart, C
B. Smart, C. Zheng, I. Laina, and V . A. Prisacariu. Splatt3r: Zero-shot gaussian splatting from uncalibrated image pairs.arXiv preprint arXiv:2408.13912, 2024
2024 arXiv
-
[21]
Snavely, S
N. Snavely, S. M. Seitz, and R. Szeliski. Photo tourism: Exploring photo collections in 3d. In ACM SIGGRAPH 2006 Papers, pages 835–846, 2006
2006
-
[22]
Q. Tian, X. Tan, Y . Xie, and L. Ma. Drivingforward: Feed-forward 3d gaussian splatting for driving scene reconstruction from flexible surround-view input. InProceedings of the AAAI Conference on Artificial Intelligence, 2025
2025
-
[23]
Tschernezki, I
V . Tschernezki, I. Laina, D. Larlus, and A. Vedaldi. Neural feature fusion fields: 3d distillation of self-supervised 2d image representations. In2022 International Conference on 3D Vision (3DV), pages 443–453. IEEE, 2022. 11
2022
-
[24]
S. Wang, V . Leroy, Y . Cabon, B. Chidlovskii, and J. Revaud. Dust3r: Geometric 3d vision made easy. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20697–20709, 2024
2024
-
[25]
T. Wang, X. Mao, C. Zhu, R. Xu, R. Lyu, P. Li, X. Chen, W. Zhang, K. Chen, T. Xue, et al. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19757–19767, 2024
2024
-
[26]
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity.IEEE Transactions on Image Processing (TIP), 13(4): 600–612, 2004
2004
-
[27]
B. Ye, S. Liu, H. Xu, X. Li, M. Pollefeys, M.-H. Yang, and S. Peng. No pose, no prob- lem: Surprisingly simple 3d gaussian splats from sparse unposed images.arXiv preprint arXiv:2410.24207, 2024
2024 arXiv
-
[28]
Yeshwanth, Y .-C
C. Yeshwanth, Y .-C. Liu, M. Nießner, and A. Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12–22, 2023
2023
-
[29]
A. Yu, V . Ye, M. Tancik, and A. Kanazawa. pixelnerf: Neural radiance fields from one or few images. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4578–4587, 2021
2021
-
[30]
Zhang, P
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 586–595, 2018
2018
-
[31]
S. Zhi, T. Laidlow, S. Leutenegger, and A. J. Davison. In-place scene labelling and understanding with implicit scene representation. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15838–15847, 2021
2021
-
[32]
S. Zhou, H. Chang, S. Jiang, Z. Fan, Z. Zhu, D. Xu, P. Chari, S. You, Z. Wang, and A. Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pa...
2024
-
[33]
X. Zhou, Z. Lin, X. Shan, Y . Wang, D. Sun, and M.-H. Yang. Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21634– 21643, 2024
2024
-
[34]
wall”, “floor
S. Zhu, G. Wang, D. Kong, and H. Wang. 3d gaussian splatting in robotics: A survey.arXiv preprint arXiv:2410.12262, 2024. 12 Supplementary Material A Additional Implementation Details A.1 Datasets ScanNet++[ 28] is a large-scale dataset of indoor scenes containing sub-millimet...
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.