REVIEW 4 major objections 5 minor 35 references
Endo-NeRF++: Uncertainty-Aware Neural Rendering with Multi-Resolution Hash Encoding for Dynamic Surgical Scene Reconstruction
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Endo-NeRF++ claims that uncertainty-guided adaptive sampling, multi-resolution hash encoding, and temporal feature blending improve dynamic surgical scene reconstruction, with the adaptive variant beating EndoNeRF by up to 1.22 dB PSNR.
desk verdict A plausible engineering extension of EndoNeRF with real but thin evidence; the adaptive-sampling ablation is internally inconsistent and needs fixing before the central claim can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is the per-sample uncertainty score $S_{u,j}^{(sample)} = \tilde{\sigma}^2_{\sigma,j} + \lambda \tilde{\sigma}^2_{c,j}$, where the two variances are min–max normalized along each ray and combine aleatoric variance (predicted log-variances) with epistemic variance (variance across $K$ Monte Carlo dropout passes). After a 5,000-iteration warm-up, 30% of rays are selected by a ray-level opacity-weighted aggregate of this score and re-sampled so that uncertain regions receive more points; multi-resolution hash grids $\{H_i\}$ with temporal blending weights $\beta_t^{(i)}$ supply the features that two MLP heads turn into predicted means and variances for color and density.
What would settle it
Compute per-pixel Pearson and Spearman correlations between predicted uncertainty and absolute color error on held-out frames of the pulling and cutting sequences; if the correlations are near zero or negative, or if reallocating samples with shuffled uncertainty scores produces the same PSNR/SSIM/LPIPS gains, the claim that uncertainty guidance drives the improvement fails.
Extended reading notes
Core claim
The paper's central claim is that the combination of three modifications to EndoNeRF—multi-resolution hash-grid encoding, temporal feature blending, and uncertainty-guided adaptive sampling—improves reconstruction of deformable surgical scenes. The authors show that each modification contributes: hash-grid encoding with temporal blending raises SSIM from 0.912 to 0.949 on the pulling sequence, adding uncertainty estimation preserves the gain while producing error-correlated confidence maps, and the full adaptive version reaches the best overall numbers on both tissue types. The reported margins against the EndoNeRF baseline are up to 1.22 dB PSNR, 5.3% SSIM, and 55.1% LPIPS, with the largest gains on the pulling sequence.
Load-bearing premise
The adaptive-sampling gain rests on the assumption that the per-sample uncertainty score computed from Monte Carlo dropout variances is a reliable guide to where reconstruction error is large; the paper's own correlation tables show only weak-to-moderate agreement (Pearson around 0.20–0.38, Spearman around 0.28), so if that signal does not track error, the reported gains would not be explained by uncertainty guidance.
Editorial extensions
If this is right
- On the two EndoNeRF sequences, the adaptive variant improves PSNR by up to 1.22 dB, SSIM by up to 5.3%, and LPIPS by up to 55.1% relative to EndoNeRF.
- Hash-grid encoding plus temporal blending alone accounts for part of the gain, e.g., pulling-sequence SSIM rises from 0.912 to 0.949 before adaptive sampling is added.
- The uncertainty maps align with surgical tools, tissue borders, and strongly deformed areas, giving a confidence signal for downstream use in surgical scene understanding.
- The same hyperparameters transfer to the StereoMIS dataset, where the adaptive variant beats EndoNeRF on both tested sequences.
- The method keeps EndoNeRF's tool-aware canonicalization, depth cueing, and bidirectional mapping intact, so the reported gains come from the three new components rather than a different overall architecture.
Reading between the lines
- A direct consequence the authors do not draw: the same uncertainty-guided sampling rule could be applied to other deformable neural rendering pipelines, concentrating computation on moving boundaries in any dynamic scene, not just endoscopy.
- Because the paper's uncertainty-error correlations are weak, an alternative explanation for the adaptive-sampling gains is that non-uniform sample allocation alone, rather than the accuracy of the uncertainty estimate, is what helps; comparing against random or shuffled reallocation would separate these.
- The evaluation is limited to four short sequences, so the reported margins should be read as evidence about those clips; testing on longer multi-view surgical recordings would show whether the gains generalize.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Endo-NeRF++, an extension of the EndoNeRF dynamic surgical scene reconstruction method, with three claimed contributions: multi-resolution hash-grid encoding, temporal feature blending, and uncertainty-guided adaptive sampling based on Monte-Carlo dropout variances. Experiments on two endoscopic sequences (pulling and cutting) report that the adaptive variant improves PSNR by up to 1.22 dB, SSIM by up to 5.3%, and LPIPS by up to 55.1% over EndoNeRF, with additional cross-dataset results on StereoMIS. The paper also reports uncertainty calibration metrics (Pearson/Spearman correlation, AUSE) and qualitative comparisons.
Significance. If the reported results are reliable, the proposed combination of efficient hash encoding and uncertainty-aware sampling would be a useful step for deformable surgical scene reconstruction, an application area where existing NeRF-based methods are computationally heavy and rarely quantify uncertainty. The paper is clearly written in terms of the overall architecture, and it makes a reasonable attempt to evaluate uncertainty quality rather than only photometric metrics. However, the experimental support is weakened by a very small proprietary dataset (two sequences, roughly 250 frames, single viewpoint), by apparent inconsistencies between Tables 2, 3, and 4, and by selection of the hash configuration on the test sequences. These issues directly affect the central claim that uncertainty-guided adaptive sampling produces the reported gains.
major comments (4)
- [Section 5.5, Tables 2-4] The evidence for the central contribution, uncertainty-guided adaptive sampling, is internally inconsistent. Table 3, titled "Performance of uncertainty-guided adaptive sampling compared with the baseline," reports Pulling PSNR 29.256, SSIM 0.943, LPIPS 0.055 and Cutting PSNR 27.356, SSIM 0.931, LPIPS 0.074, which are identical to the Table 2 results for EndoNeRF++ with uncertainty estimation but without any stated adaptive sampling. In contrast, Table 4's "Unc. EndoNeRF++" row reports Cutting SSIM 0.922 and LPIPS 0.087, values that differ from both Table 2 and Table 3, while its Pulling row matches Tables 2 and 3 exactly. Table 3 also reports Pulling Spearman correlation 0.605 and AUSE 0.317, whereas Table 2 reports Spearman 0.280 and AUSE 0.698 for the same sequence. Because the adaptive-sampling ablation table duplicates or contradicts the other tables, the comparison between uniform and uncertainty-guided sampling is not verifiable from the reported data, and the claimed improvements cannot currently be attributed to the proposed mechanism.
- [Section 4.1 and Section 5.2] There is a mismatch between the method description and the implementation. Equations (13)-(14) and the surrounding text define a per-sample uncertainty score that reallocates additional points along each ray proportionally to normalized variance ("Samples with higher uncertainty are allocated additional points proportionally to their normalized uncertainty score"). However, Section 5.2 states that "adaptive ray sampling selects 30% uncertainty-guided rays and 70% uniformly sampled rays," which describes selecting entire rays rather than reallocating samples within rays. The paper therefore does not specify whether the reported results come from the per-sample reallocation derived in Section 4.1 or from a ray-selection heuristic. This ambiguity must be resolved before the adaptive-sampling results can be interpreted.
- [Table 2 and Section 5.5] The uncertainty signal used to drive adaptive sampling is weakly correlated with reconstruction error: Table 2 reports Pearson correlations of 0.201 and 0.380 and Spearman correlations of 0.280 and 0.284 for the Pulling and Cutting sequences, respectively. Even the best correlations over trials are modest (Pearson 0.302-0.511, Spearman 0.465-0.475). With such weak correlations, the claimed substantial gains from uncertainty-guided sampling (up to 1.22 dB PSNR, 5.3% SSIM, 55.1% LPIPS) are not mechanistically explained. The paper should report whether the sampling reallocation actually concentrates samples in regions with high error, and should include an ablation that directly compares uniform sampling with uncertainty-guided sampling under otherwise identical settings, rather than relying on the inconsistent tables.
- [Section 5.7, Table 5] The multi-resolution hash configuration is selected on the test sequences. Table 5 evaluates several hash configurations on the same Pulling and Cutting sequences, and the authors then state that the configuration "is thus used in all subsequent studies." This is a selection-on-the-test-set procedure: the reported results for the chosen configuration are optimistic, and the generalization claim to new surgical scenes is not supported by the experimental design. The paper should either use a validation split for configuration selection or report results across configurations without presenting the best one as the final model.
minor comments (5)
- [General] There are numerous typos and inconsistent naming throughout: "T able" in table captions, "EndoNerf" versus "EndoNeRF", "E-DSSR" versus "DSSR", and "Qualtative Results" in Section 5.8. These should be corrected.
- [Equation (18)] The negative log-likelihood loss in Eq. (18) sums over rays and over samples j, but the ground-truth color C_gt is defined per ray. The notation should clarify how the per-sample color loss relates to the final rendered pixel color, since the predictive mean color for each sample is compared with the ray's ground truth.
- [Section 5.2] The implementation paragraph lists "K = 4000" in the same sentence as the depth loss weight and exponential moving average coefficient, but K is defined earlier as the number of stochastic forward passes for uncertainty estimation (Eq. 7). Please clarify what K=4000 refers to, and if it is the depth refinement interval, use a distinct symbol.
- [References] Reference [31] is listed as "Unknown" and reference [26] appears to duplicate reference [27]; several related works discussed in the text (e.g., EndoSurf, NeRFscopy) are not included in the quantitative comparisons. Please complete the reference list and clearly distinguish the baselines used in experiments from those only discussed.
- [Section 6] The limitations paragraph appropriately mentions dataset scarcity and computational cost, but the paper does not discuss the consequences of using a single viewpoint and a proprietary dataset for the validity of the reported improvements. A sentence acknowledging this would be helpful.
Circularity Check
No significant circularity: rendering, uncertainty estimation, and adaptive sampling do not reduce to their own inputs by construction, though the reported tables are internally inconsistent.
full rationale
The paper's derivation chain is self-contained and does not reduce to its inputs by construction. The rendered color and depth (Eqs. 15–16) depend only on the predictive means (Eq. 8), while the uncertainty scores (Eqs. 13–14, 17) are separate outputs used for sampling allocation; no predicted quantity is defined in terms of the target it is supposed to explain. The uncertainty branch is trained with an NLL loss (Eq. 18) that encourages predicted variances to match the model's own residuals, so a positive uncertainty-error correlation on training data would be partly loss-induced; however, Table 2 reports correlations on test frames, and the weak values (Pearson 0.20–0.38, Spearman about 0.28) show the signal is not forced. The central comparisons are against the external EndoNeRF and E-DSSR baselines (Tables 1 and 4), and the cited EndoNeRF works have no author overlap with the present paper, so no self-citation chain carries the argument. The serious problem in the manuscript is internal inconsistency rather than circularity: Table 3's reconstruction numbers duplicate Table 2 and disagree with the Adap. EndoNeRF++ row of Table 4, so the adaptive-sampling improvement is not verifiable from the reported data, and Section 5.2's implementation note describes ray selection rather than the per-sample reallocation in Eqs. (13)–(14). These are data-integrity and reporting issues, not circular reductions, and per the instructions they do not raise the circularity score.
Assumptions & free parameters
free parameters (4)
- Hash grid configuration =
resolutions {8,16,32}, scales {1.4,1.7,1.9}, 16 levels, 8 channels
- Uncertainty loss weight lambda_unc =
0.02 (cutting), 0.03 (pulling)
- Adaptive ray sampling fraction =
30% uncertainty-guided, 70% uniform
- Warm-up iterations =
5k
assumptions (4)
- domain assumption The deformation network G_phi can map deformed-space points to a consistent canonical space using tool motion encoding w_t from a single viewpoint.
- domain assumption Monte Carlo dropout variances approximate epistemic uncertainty in this dynamic setting.
- ad hoc to paper Min-max normalization of variances along each ray preserves the relative uncertainty ranking needed for sampling.
- domain assumption The STTR-light stereo depth maps and manually annotated tool masks are accurate enough to guide training.
Cite this review
Pith. "Pith review of Endo-NeRF++: Uncertainty-Aware Neural Rendering with Multi-Resolution Hash Encoding for Dynamic Surgical Scene Reconstruction." pith.science (2026). https://pith.science/paper/ADIER2TZ
@misc{pith2026260727825,
author = {Pith},
title = {Pith review of: Endo-NeRF++: Uncertainty-Aware Neural Rendering with Multi-Resolution Hash Encoding for Dynamic Surgical Scene Reconstruction},
year = {2026},
howpublished = {\url{https://pith.science/paper/ADIER2TZ}},
note = {Machine review of arXiv:2607.27825}
}
read the original abstract
Reconstructing dynamic surgical scenes is crucial for robot-assisted minimally invasive surgery; however, it continues to be difficult because of tissue deformation, occlusions, specular reflections, and restricted viewpoints. In this study, we introduce Endo-NeRF++, a neural rendering framework that accounts for uncertainty in the reconstruction of dynamic surgical scenes. Expanding on EndoNeRF, the suggested approach incorporates multi-resolution hash-grid encoding, temporal feature merging, and uncertainty-informed adaptive sampling to enhance reconstruction accuracy and temporal coherence in deformable endoscopic scenes.The multi-resolution hash-grid representation within the framework effectively captures both coarse and fine anatomical details, while temporal feature blending ensures stable reconstruction during tissue deformation and surgical tool occlusions. Additionally, uncertainty-driven adaptive sampling assigns more samples to uncertain areas to enhance rendering quality and geometric coherence. Experiments on robotic surgical video sequences demonstrate that the proposed uncertainty-guided adaptive sampling improves PSNR by up to 1.22dB (4.3%), increases SSIM by up to 5.3%, and reduces LPIPS by up to 55.1% compared with the EndoNeRF baseline.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
In: Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Chang, J.R., Chen, Y.S.: Pyramid stereo matching network. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
work page 2018
-
[3]
arXiv preprint arXiv:2602.23574 (2026)
Duan, R., Wong, A.: Evidential neural radiance fields. arXiv preprint arXiv:2602.23574 (2026)
arXiv 2026
-
[4]
IEEE Transactions on Pattern Analysis and Machine Intelligence (2018)
Engel, J., Koltun, V., Cremers, D.: Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence (2018)
work page 2018
-
[5]
Goli, L., et al.: Bayes’ rays: Uncertainty quantification for neural radiance fields. In: CVPR (2024)
work page 2024
-
[6]
International Journal of Computer Assisted Radiology and Surgery (2023)
Hayoz, M., Hahne, C., Gallardo, M., Candinas, D., Kurmann, T., Allan, M., Sznitman, R.: Learning how to robustly estimate camera pose in endoscopic videos. International Journal of Computer Assisted Radiology and Surgery (2023). https://doi.org/10.1007/s11548-023-02919-w
-
[7]
IEEE Transactions on Pattern Analysis and Machine Intelligence30(2), 328– 341 (2008)
Hirschmuller, H.: Stereo processing by semiglobal matching and mutual informa- tion. IEEE Transactions on Pattern Analysis and Machine Intelligence30(2), 328– 341 (2008)
work page 2008
-
[8]
IEEE Robotics and Automation Letters (2024)
Kim, J.S., Kim, M., Kim, G., Oh, T.H., Kim, J.H.: Factorized multi-resolution hashgrid for efficient neural radiance fields: Execution on edge-devices. IEEE Robotics and Automation Letters (2024)
work page 2024
-
[9]
arXiv preprint arXiv:2305.03027 (2023) Endo-NeRF++19
Kirschstein, T., Qian, S., Giebenhain, S., Walter, T., Nießner, M.: Nersem- ble: Multi-view radiance field reconstruction of human heads. arXiv preprint arXiv:2305.03027 (2023) Endo-NeRF++19
arXiv 2023
Show all 35 references
-
[10]
In: International Conference on 3D Vision (3DV) (2021)
Lipson, L., Teed, Z., Deng, J.: Raft-stereo: Multilevel recurrent field transforms for stereo matching. In: International Conference on 3D Vision (3DV) (2021)
2021
-
[11]
In: Medical Image Computing and Computer-Assisted Interven- tion (MICCAI)
Long, Y., Li, Z., Yee, C.H., Ng, C.F., Taylor, R.H., Unberath, M., Dou, Q.: E-dssr: Efficientdynamicsurgicalscenereconstructionwithtransformer-basedstereoscopic depth perception. In: Medical Image Computing and Computer-Assisted Interven- tion (MICCAI). pp. 415–425. Springer (2021)
2021
-
[12]
arXiv preprint arXiv:2505.03042 (2025)
Luo, S.T.S.: A new perspective to understanding multi-resolution hash encoding for neural fields. arXiv preprint arXiv:2505.03042 (2025)
2025 arXiv
-
[13]
Na- ture Biomedical Engineering1, 691–696 (2017)
Maier-Hein, L., et al.: Surgical data science for next-generation interventions. Na- ture Biomedical Engineering1, 691–696 (2017)
2017
-
[14]
In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Martin-Brualla, R., Radwan, N., Sajjadi, M.S.M., Barron, J.T., Dosovitskiy, A., Duckworth, D.: Nerf in the wild: Neural radiance fields for unconstrained photo collections. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
2021
-
[15]
In: ECCV (2020)
Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: ECCV (2020)
2020
-
[16]
IEEE Transactions on Medical Imaging29(11), 1876–1888 (2010)
Mountney, P., Stoyanov, D., Davison, A.J., Yang, G.Z.: Three-dimensional tissue deformation recovery and tracking. IEEE Transactions on Medical Imaging29(11), 1876–1888 (2010)
2010
-
[17]
ACM Transactions on Graphics (SIGGRAPH) (2022)
Müller,T.,Evans,A.,Schied,C.,Keller,A.:Instantneuralgraphicsprimitiveswith a multiresolution hash encoding. ACM Transactions on Graphics (SIGGRAPH) (2022)
2022
-
[18]
In: ISMAR (2011)
Newcombe, R.A., et al.: Kinectfusion: Real-time dense surface mapping and track- ing. In: ISMAR (2011)
2011
-
[19]
Surgical Oncology20(3), 189–201 (2011)
Nicolau, S., Soler, L., Mutter, D., Marescaux, J.: Augmented reality in laparoscopic surgical oncology. Surgical Oncology20(3), 189–201 (2011)
2011
-
[20]
Cureus17(3) (2025)
Osman, E.I.A., Ismail, M.M.E.M., Mukhtar, M.A.H., Ahmed, A.U.B., Mohamed, N.A.A.E., Ibrahim, A.A.A., Mohamed, N.A.A., Ibrahim, A.A.: Artificial intelli- gence and robotics in minimally invasive and complex surgical procedures: a sys- tematic review. Cureus17(3) (2025)
2025
-
[21]
Park, K., Sinha, U., Barron, J.T., Bouaziz, S., Goldman, D.B., Seitz, S.M., Martin- Brualla,R.:Nerfies:Deformableneuralradiancefields.In:InternationalConference on Computer Vision (ICCV) (2021)
2021
-
[22]
Journal of Plastic, Reconstructive & Aesthetic Surgery71(3), 456–462 (2018)
Pratt, P., Ives, M., Lawton, G., Simmons, J., Radev, N., Spyropoulou, L., Amiras, D.: Through the hololens looking glass: augmented reality for extremity recon- struction surgery. Journal of Plastic, Reconstructive & Aesthetic Surgery71(3), 456–462 (2018)
2018
-
[23]
In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Pumarola, A., Corona, E., Pons-Moll, G., Moreno-Noguer, F.: D-nerf: Neural ra- diance fields for dynamic scenes. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
2021
-
[24]
Journal of Robotic Surgery20(1), 186 (2026)
Raptis,S.P.,Theocharopoulos,A.,Theocharopoulos,C.,Papadakos,S.P.,Levantis, G., Kontis, E., Vrahatis, A.G.: Artificial intelligence analysis of minimally invasive surgery data. Journal of Robotic Surgery20(1), 186 (2026)
2026
-
[25]
In: WACV (2026)
Safadoust, S., et al.: Warprf: Multi-view consistency for training-free uncertainty quantification. In: WACV (2026)
2026
-
[27]
arXiv preprint arXiv:2602.15775 (2026) 20 Habib et al
Salort-Benejam, L., Agudo, A.: Nerfscopy: Neural radiance fields for in-vivo time- varying tissues from endoscopy. arXiv preprint arXiv:2602.15775 (2026) 20 Habib et al
2026
-
[28]
International Journal of Computer Vision47(1), 7–42 (2002)
Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense stereo correspon- dence algorithms. International Journal of Computer Vision47(1), 7–42 (2002)
2002
-
[29]
In: Proceedings of CVPR (2016)
Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of CVPR (2016)
2016
-
[30]
arXiv preprint arXiv:2209.08718 (2022)
Sünderhauf, N., Abou-Chakra, J., Miller, D.: Density-aware nerf ensembles: Quantifying predictive uncertainty in neural radiance fields. arXiv preprint arXiv:2209.08718 (2022)
2022 arXiv
-
[31]
arXiv preprint (2025)
Unknown: Endo2dgs: Endoscopic scene reconstruction with high-fidelity geometry. arXiv preprint (2025)
2025
-
[32]
In: Medical Image Computing and Computer-Assisted Intervention (MICCAI)
Wang, Y., Long, Y., Fan, S., Dou, Q.: Neural rendering for stereo 3d reconstruc- tion of deformable tissues in robotic surgery. In: Medical Image Computing and Computer-Assisted Intervention (MICCAI). pp. 431–441. Springer (2022)
2022
-
[33]
arXiv preprint arXiv:2404.15339 (2024)
Wang, Y., Gong, B., Long, Y., Fan, S.H., Dou, Q.: Efficient endonerf recon- struction and its application for data-driven surgical simulation. arXiv preprint arXiv:2404.15339 (2024)
2024 arXiv
-
[34]
In: RSS (2015)
Whelan, T., et al.: Elasticfusion: Dense slam without a pose graph. In: RSS (2015)
2015
-
[35]
In: Conference on Com- puter Vision and Pattern Recognition (CVPR) (2023)
Yu, A., et al.: Uncertainty-aware neural radiance fields. In: Conference on Com- puter Vision and Pattern Recognition (CVPR) (2023)
2023
- [36]
-
[37]
In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
Zhang, F., Prisacariu, V., Yang, R., Torr, P.H.: Ga-net: Guided aggregation net for end-to-end stereo matching. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
2019
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.