REVIEW 4 major objections 5 minor 34 references
DNRSelect: Active Best View Selection for Deferred Neural Rendering
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A reinforcement-learning view selector trained on cheap rasterized images can pick the few camera views worth ray tracing, letting deferred neural rendering reach near-full-data quality with far fewer ray-traced images.
desk verdict Genuinely new combination of RL view selection and DNR, but the central rasterized-to-ray-traced transfer is untested and the headline claim is undefined. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two components carry the argument. The first is the reinforcement-learning view selector: the state at step $t$ is a camera embedding of previously selected views plus a CNN feature of the current spatial neural texture; Q-learning with temporal-difference targets (Eqs. 2--4) scores candidate next views, and the next action is chosen greedily with $\epsilon$-greedy exploration. Its reward is the negative coarse DNR loss, so the selector and the renderer are optimized toward the same objective. The second is the 3D texture aggregator: hierarchical sampling $\pi(\cdot)$ separately extracts neural textures $\{T^{ex}_D, T^{ex}_N, T^{ex}_U\}$ from depth, normal, and UV maps; these are concatenated and processed by a CNN into the spatial neural texture $T$ that feeds the U-Net renderer and also serves as the selector's observation. This separate-per-modality fusion is what the paper credits for preserving geometric consistency that early channel-wise concatenation loses.
What would settle it
Train the identical two-step pipeline but compute the Step-1 reward on ray-traced images instead of rasterized images for the same candidate views. If the ray-traced selector picks a different set of $M$ views and the resulting fine-tuned renderer scores measurably higher PSNR at the same $M$, then rasterized images are not a sufficient proxy for choosing views, and the paper's cost-saving argument weakens.
Extended reading notes
Core claim
The paper's central claim is that optimal views for deferred neural rendering can be discovered without ray tracing. DNRSelect trains a Q-learning view selector in Step 1 on rasterized images, rewarding it with the negative current DNR loss, so the selector outputs a sequence of $M<N$ camera views that maximize expected future rendering quality. Step 2 then fine-tunes the coarse DNR model on ray-traced images only for those $M$ views, using a six-loss supervision (photometric, SSIM, perceptual, frequency, total variation, and texture regularization). The authors further claim that a 3D texture aggregator, which processes depth, normal, and UV maps separately and then fuses their neural textures, reduces the artifacts that sparse views cause, and that the complete system on NeRF-Synthetic outperforms uncertainty-based NeRF view-selection methods and nearly reaches, sometimes exceeds, full-data quality.
Load-bearing premise
The load-bearing premise is that cheap rasterized images reveal which viewpoints are most valuable for final ray-traced rendering quality; if the two rankings diverge, the selected views are not the informative ones.
Editorial extensions
If this is right
- At a fixed number of training views, DNRSelect reports higher PSNR and SSIM than ActiveNeRF, density-aware NeRF ensembles, and other uncertainty-based information-gain view selectors on NeRF-Synthetic.
- Because the selector is trained on rasterized images, the expensive ray-traced acquisition can be deferred until after the useful viewpoints are known, so the cost of data collection is concentrated on the views that matter.
- Each of the three additions---data augmentation, multi-loss supervision, and the RL selector plus aggregator---improves the ablation results, with the full system reaching 28.12 PSNR versus 26.64 without the proposed components at 25 views.
- Fusing depth, normal, and UV maps with the separate-texture aggregator beats using UV maps alone, UV plus depth, or UV plus normal, and it beats concatenating all three into a single 9-channel input.
- The method's compatibility with DNR suggests deferred rendering can be made active, letting an agent choose views rather than passively consuming a fixed dataset.
Reading between the lines
- The paper does not compare rasterized-trained versus ray-traced-trained selectors, so a direct test of the proxy assumption remains open; if the two rankings diverge, view selection could be improved further by using a small amount of ray-traced data during training.
- Exceeding the 100-view upper limit with fewer selected views implies the upper limit is not a hard ceiling; it may reflect that random full training includes redundant views, so a well-chosen subset can be a better training set, not just a cheaper one.
- The same two-step pattern---cheap proxy renders for selection, expensive renders for fine-tuning---could transfer to other neural rendering stacks, such as Gaussian-splatting pipelines that need high-quality shading only at chosen viewpoints.
- For robotics, the selector could be run online: an exploring agent could use rasterized proxies to decide where to take the next high-fidelity measurement, extending the next-best-view idea from reconstruction to deferred rendering.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DNRSelect, a two-stage deferred neural rendering framework. In Step 1, a Q-learning-based view selector (Sec. III-C) is trained jointly with DNR on cheap rasterized images, using the negative DNR photometric loss as the reward in Eq. (5). In Step 2, the coarse DNR model is fine-tuned on ray-traced images only at the M views selected in Step 1, under a six-term loss (Eq. (9)). A 3D texture aggregator (Sec. III-D) fuses depth, normal, and UV maps into neural textures to improve geometric consistency under sparse views. Experiments on NeRF-Synthetic compare DNRSelect with NeRF-based view-selection baselines (ActiveNeRF, Density-aware Ensembles, uncertainty sampling, InstantNGP) across view budgets from 5 to 100. The paper claims that DNRSelect outperforms these baselines, 'nearly reaches the theoretical upper limit,' and 'can sometimes exceed this limit' with fewer views (Sec. IV-B, paragraph after Fig. 6).
Significance. If the central claims hold, DNRSelect would be a practically useful way to cut DNR's dependence on expensive ray-traced training images: it uses rasterized images for view selection, acquires ray-traced data only at selected views, and reports quality gains over NeRF-based competitors. The work is timely given DNR's data cost and the growing interest in active view selection for neural rendering. Credit should be given for testing several design choices (aggregator strategy, reward strategies, data modalities) and for identifying a real gap: no prior work specifically tackles active view selection for deferred neural rendering. However, the supporting evidence is currently thin: no error bars or repeated seeds, no vanilla DNR baseline under identical view budgets, and no experiment validating the rasterized-image proxy that underpins the view selector. The 'theoretical upper limit' claim is also undefined, so the headline quantitative result is not evaluable as written.
major comments (4)
- [Sec. IV-B, paragraph after Fig. 6] The paper states that DNRSelect 'nearly reaches the theoretical upper limit' and 'with fewer selected views, it can sometimes exceed this limit,' but no oracle, all-views baseline, or mathematical definition of this limit is given anywhere in the manuscript. Without a specified upper-bound curve, this load-bearing quantitative claim cannot be checked. The authors should define the limit explicitly (e.g., DNR trained on all 100 training views with the same architecture and losses) and plot the corresponding curve in Fig. 6.
- [Sec. III-C/E, Eq. (5), Fig. 8c] The view selector is trained in Step 1 solely on rasterized images with reward -L_DNR^c, but the selected views are then used to acquire ray-traced images for Step 2 fine-tuning. The paper never tests whether the ranking of views induced by the rasterized-image loss correlates with the ranking induced by the ray-traced-image loss after fine-tuning. Since ray-traced images contain global illumination, shadows, and reflections absent from rasterized images, the selected views could be systematically suboptimal for the final rendering task. A direct experiment comparing (i) a selector trained on rasterized loss and (ii) a selector trained on ray-traced loss at the same view budgets, or reporting the correlation between the two loss rankings, is needed to support the central cost-saving claim.
- [Sec. IV-B, Fig. 6] The main quantitative comparison is against NeRF-based methods (ActiveNeRF, Density-aware Ensembles, uncertainty sampling variants, InstantNGP), not against vanilla DNR or DNR with simple view-selection baselines such as random or uniform sampling. Because DNRSelect changes both the rendering architecture and the view-selection strategy, the curves in Fig. 6 do not isolate the contribution of the RL selector. The authors should add vanilla DNR trained on the same number of randomly or uniformly selected views, vanilla DNR trained on all views, and DNR with the 3D texture aggregator but without RL selection. In addition, no error bars or repeated-seed results are reported, so 'consistently outperforms' is not substantiated.
- [Sec. IV-C, Table I] The ablation table reports results at a single view count (25 views) and the checkmark layout is visually ambiguous: it is difficult to tell which components are enabled in each row. Moreover, the table reports only point estimates without variance. Since the RL selector and texture aggregator are the two main contributions, their ablations should be shown across several view budgets (e.g., 10, 25, 50 views) and with multiple evaluation seeds to demonstrate that the gains are consistent and not specific to one configuration.
minor comments (5)
- [Sec. III-C] The notation 'scam t' and 'sobs t' appears to be a typesetting error for s_t^cam and s_t^obs; please fix the superscript/subscript formatting throughout.
- [Eq. (4)] The symbol T is used both for the neural texture in Sec. III-A and for the trajectory length in Eq. (4); please use separate symbols (e.g., T_tex and T_horizon) to avoid ambiguity.
- [Sec. IV-A] Implementation details list many hyperparameters but omit the RL discount factor γ, the epsilon-greedy exploration schedule, and the number of action/state dimensions; these are needed for reproducibility.
- [Fig. 8c] The reward-strategy comparison would be easier to interpret if the text described which bars correspond to which reward and whether 'Raster' and 'Ray Tracing' refer to the data modality used for training or for evaluation; the current caption is ambiguous.
- [Abstract and Sec. I] The phrase 'the code will be released' is not a substitute for an available implementation; providing a link or supplementary material would help reviewers and readers reproduce the reported numbers.
Circularity Check
No significant circularity: view selection optimizes a legitimate active-learning objective; self-citations are not load-bearing.
full rationale
The central claim is that an RL view selector trained on rasterized images can pick a small set of ray-traced views for DNR fine-tuning. The reward is the negative DNR loss (Eqs. 2, 3, 5), which is also used to train the DNR model in Step 1. This is a standard active-learning/RL objective: the selector is trained to pick views that minimize the current renderer's loss, and the renderer is trained to minimize the same loss. The evaluation, however, measures PSNR/SSIM/LPIPS on held-out ray-traced test views, which are independent of the selector's training objective. There is no fitted parameter disguised as a prediction; the selector's output is a set of views, and the final model is trained on the ray-traced versions of those views. The paper's 'theoretical upper limit' claim is undefined and unsupported, but that is an empirical/correctness concern, not a circularity. The two self-citations ([13], [18], both by author Xiaobao Wei) appear only in lists of related neural-rendering works and do not justify any premise of DNRSelect; they are not load-bearing. No equation reduces to another by construction, and no uniqueness theorem is imported from the authors' own prior work. The rasterized-to-ray-traced proxy gap is a potential weakness but not circularity.
Assumptions & free parameters
free parameters (4)
- Step 1 loss weights λ_RL and λ_DNRc =
0.1 and 1.0
- Step 2 loss weights =
λ_DNRf=1.0, λ_SSIM=0.1, λ_p=0.1, λ_FR=0.01, λ_TV=0.001, λ_reg=0.1
- Training schedule (epochs and learning rate) =
50 epochs in Step 1, 100 in Step 2; LR 0.001 then 0.0001
- RL discount factor and exploration schedule =
not reported
assumptions (4)
- domain assumption Ground-truth depth maps, normal maps, and UV maps are available for every view
- domain assumption Rasterized images are a valid proxy for ray-traced images for view selection
- domain assumption The DNR renderer and neural texture framework from Thies et al. [5] works as described
- domain assumption Q-learning with epsilon-greedy exploration converges to a useful policy in this setting
Cite this review
Pith. "Pith review of DNRSelect: Active Best View Selection for Deferred Neural Rendering." pith.science (2026). https://pith.science/paper/OPG6AUPU
@misc{pith2026250112150,
author = {Pith},
title = {Pith review of: DNRSelect: Active Best View Selection for Deferred Neural Rendering},
year = {2026},
howpublished = {\url{https://pith.science/paper/OPG6AUPU}},
note = {Machine review of arXiv:2501.12150}
}
read the original abstract
Deferred neural rendering (DNR) is an emerging computer graphics pipeline designed for high-fidelity rendering and robotic perception. However, DNR heavily relies on datasets composed of numerous ray-traced images and demands substantial computational resources. It remains under-explored how to reduce the reliance on high-quality ray-traced images while maintaining the rendering fidelity. In this paper, we propose DNRSelect, which integrates a reinforcement learning-based view selector and a 3D texture aggregator for deferred neural rendering. We first propose a novel view selector for deferred neural rendering based on reinforcement learning, which is trained on easily obtained rasterized images to identify the optimal views. By acquiring only a few ray-traced images for these selected views, the selector enables DNR to achieve high-quality rendering. To further enhance spatial awareness and geometric consistency in DNR, we introduce a 3D texture aggregator that fuses pyramid features from depth maps and normal maps with UV maps. Given that acquiring ray-traced images is more time-consuming than generating rasterized images, DNRSelect minimizes the need for ray-traced data by using only a few selected views while still achieving high-fidelity rendering results. We conduct detailed experiments and ablation studies on the NeRF-Synthetic dataset to demonstrate the effectiveness of DNRSelect. The code will be released.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[28]
Learning to Select Camera Views: Efficient Multiview Understanding at Few Glances
Y . Hou, S. Gould, and L. Zheng, “Learning to select camera views: Efficient multiview understanding at few glances,” arXiv preprint arXiv:2303.06145, 2023
work page Pith review arXiv 2023
-
[1]
Nerf: Representing scenes as neural radiance fields for view synthesis,
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoor- thi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM , vol. 65, no. 1, pp. 99– 106, 2021
work page 2021
-
[2]
3d gaussian splatting for real-time radiance field rendering,
B. Kerbl, G. Kopanas, T. Leimk ¨uhler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” ACM Transactions on Graphics, vol. 42, July 2023
work page 2023
-
[3]
Activenerf: Learning where to see with uncertainty estimation,
X. Pan, Z. Lai, S. Song, and G. Huang, “Activenerf: Learning where to see with uncertainty estimation,” in European Conference on Computer Vision, pp. 230–246, Springer, 2022
work page 2022
-
[4]
Cg-slam: Efficient dense rgb-d slam in a consistent uncertainty-aware 3d gaussian field,
J. Hu, X. Chen, B. Feng, G. Li, L. Yang, H. Bao, G. Zhang, and Z. Cui, “Cg-slam: Efficient dense rgb-d slam in a consistent uncertainty-aware 3d gaussian field,” arXiv preprint arXiv:2403.16095 , 2024
arXiv 2024
-
[5]
Deferred neural rendering: image synthesis using neural textures,
J. Thies, M. Zollh ¨ofer, and M. Nießner, “Deferred neural rendering: image synthesis using neural textures,” ACM Trans. Graph. , vol. 38, jul 2019
work page 2019
-
[6]
De- ferred neural lighting: free-viewpoint relighting from unstructured photographs,
D. Gao, G. Chen, Y . Dong, P. Peers, K. Xu, and X. Tong, “De- ferred neural lighting: free-viewpoint relighting from unstructured photographs,” ACM Transactions on Graphics (TOG) , vol. 39, no. 6, pp. 1–15, 2020
work page 2020
-
[7]
Anr: Articulated neural rendering for virtual avatars,
A. Raj, J. Tanke, J. Hays, M. V o, C. Stoll, and C. Lassner, “Anr: Articulated neural rendering for virtual avatars,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 3722–3731, 2021
work page 2021
Show all 34 references
-
[8]
Multi-view mesh reconstruction with neural deferred shading,
M. Worchel, R. Diaz, W. Hu, O. Schreer, I. Feldmann, and P. Eisert, “Multi-view mesh reconstruction with neural deferred shading,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6187–6197, 2022
2022
-
[9]
Deferredgs: Decoupled and editable gaussian splatting with deferred shading,
T. Wu, J.-M. Sun, Y .-K. Lai, Y . Ma, L. Kobbelt, and L. Gao, “Deferredgs: Decoupled and editable gaussian splatting with deferred shading,” arXiv preprint arXiv:2404.09412 , 2024
2024 arXiv
-
[10]
Advances in neural rendering,
A. Tewari, J. Thies, B. Mildenhall, P. Srinivasan, E. Tretschk, W. Yifan, C. Lassner, V . Sitzmann, R. Martin-Brualla, S. Lombardi, et al. , “Advances in neural rendering,” inComputer Graphics Forum, vol. 41, pp. 703–735, Wiley Online Library, 2022
2022
-
[11]
State of the art on neural rendering,
A. Tewari, O. Fried, J. Thies, V . Sitzmann, S. Lombardi, K. Sunkavalli, R. Martin-Brualla, T. Simon, J. Saragih, M. Nießner, et al., “State of the art on neural rendering,” in Computer Graphics Forum , vol. 39, pp. 701–727, Wiley Online Library, 2020
2020
-
[12]
Neural rendering and its hardware acceleration: A review,
X. Yan, J. Xu, Y . Huo, and H. Bao, “Neural rendering and its hardware acceleration: A review,” arXiv preprint arXiv:2402.00028 , 2024
2024 arXiv
-
[13]
I-medsam: Implicit medical image segmentation with segment anything,
X. Wei, J. Cao, Y . Jin, M. Lu, G. Wang, and S. Zhang, “I-medsam: Implicit medical image segmentation with segment anything,” arXiv preprint arXiv:2311.17081, 2023
2023 arXiv
-
[14]
Instant neural graphics primitives with a multiresolution hash encoding,
T. M ¨uller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM transactions on graphics (TOG) , vol. 41, no. 4, pp. 1–15, 2022
2022
-
[15]
Plenoxels: Radiance fields without neural networks,
S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa, “Plenoxels: Radiance fields without neural networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5501–5510, 2022
2022
-
[16]
Tensorf: Tensorial radiance fields,
A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su, “Tensorf: Tensorial radiance fields,” in European conference on computer vision, pp. 333– 350, Springer, 2022
2022
-
[17]
Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,
P. Wang, L. Liu, Y . Liu, C. Theobalt, T. Komura, and W. Wang, “Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,” arXiv preprint arXiv:2106.10689 , 2021
2021 arXiv
-
[18]
Nto3d: Neural target object 3d reconstruction with segment anything,
X. Wei, R. Zhang, J. Wu, J. Liu, M. Lu, Y . Guo, and S. Zhang, “Nto3d: Neural target object 3d reconstruction with segment anything,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20352–20362, 2024
2024
-
[19]
Gaussianpro: 3d gaussian splatting with progressive propa- gation,
K. Cheng, X. Long, K. Yang, Y . Yao, W. Yin, Y . Ma, W. Wang, and X. Chen, “Gaussianpro: 3d gaussian splatting with progressive propa- gation,” in Forty-first International Conference on Machine Learning , 2024
2024
-
[20]
S3gaussian: Self-supervised street gaus- sians for autonomous driving,
N. Huang, X. Wei, W. Zheng, P. An, M. Lu, W. Zhan, M. Tomizuka, K. Keutzer, and S. Zhang, “S3gaussian: Self-supervised street gaus- sians for autonomous driving,”arXiv preprint arXiv:2405.20323, 2024
2024 arXiv
-
[21]
Scaffold-gs: Structured 3d gaussians for view-adaptive rendering,
T. Lu, M. Yu, L. Xu, Y . Xiangli, L. Wang, D. Lin, and B. Dai, “Scaffold-gs: Structured 3d gaussians for view-adaptive rendering,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20654–20664, 2024
2024
-
[22]
Gradient-based local next-best-view planning for improved perception of targeted plant nodes,
A. K. Burusa, E. J. van Henten, and G. Kootstra, “Gradient-based local next-best-view planning for improved perception of targeted plant nodes,” in 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 15854–15860, 2024
2024
-
[23]
Active vision in robotic systems: A survey of recent developments,
S. Chen, Y . Li, and N. M. Kwok, “Active vision in robotic systems: A survey of recent developments,” The International Journal of Robotics Research, vol. 30, no. 11, pp. 1343–1377, 2011
2011
-
[24]
Humanoid robot next best view planning under occlusions using body movement primitives,
R. Monica, J. Aleotti, and D. Piccinini, “Humanoid robot next best view planning under occlusions using body movement primitives,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2493–2500, IEEE, 2019
2019
-
[25]
Neu-nbv: Next best view planning using uncertainty estimation in image-based neural rendering,
L. Jin, X. Chen, J. R ¨uckin, and M. Popovi ´c, “Neu-nbv: Next best view planning using uncertainty estimation in image-based neural rendering,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 11305–11312, IEEE, 2023
2023
-
[26]
Multi-view picking: Next-best- view reaching for improved grasping in clutter,
D. Morrison, P. Corke, and J. Leitner, “Multi-view picking: Next-best- view reaching for improved grasping in clutter,” in 2019 International Conference on Robotics and Automation (ICRA) , pp. 8762–8768, IEEE, 2019
2019
-
[27]
Online next-best-view planner for 3d-exploration and inspection with a mobile manipulator robot,
M. Naazare, F. G. Rosas, and D. Schulz, “Online next-best-view planner for 3d-exploration and inspection with a mobile manipulator robot,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3779– 3786, 2022
2022
-
[29]
Density-aware nerf ensembles: Quantifying predictive uncertainty in neural radiance fields,
N. S ¨underhauf, J. Abou-Chakra, and D. Miller, “Density-aware nerf ensembles: Quantifying predictive uncertainty in neural radiance fields,” in 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9370–9376, 2023
2023
-
[30]
Bayes’ rays: Uncertainty quantification for neural radiance fields,
L. Goli, C. Reading, S. Sell ´an, A. Jacobson, and A. Tagliasacchi, “Bayes’ rays: Uncertainty quantification for neural radiance fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20061–20070, 2024
2024
-
[31]
Nerf director: Revisiting view selection in neural volume rendering,
W. Xiao, R. S. Cruz, D. Ahmedt-Aristizabal, O. Salvado, C. Fookes, and L. Lebrat, “Nerf director: Revisiting view selection in neural volume rendering,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 20742–20751, 2024
2024
-
[32]
Continuous control with deep reinforcement learning,
T. Lillicrap, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971 , 2015
2015 arXiv
-
[33]
Improved adam optimizer for deep neural networks,
Z. Zhang, “Improved adam optimizer for deep neural networks,” in 2018 IEEE/ACM 26th international symposium on quality of service (IWQoS), pp. 1–2, Ieee, 2018
2018
-
[34]
Instant neural graphics primitives with a multiresolution hash encoding,
T. M ¨uller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Trans. Graph., vol. 41, pp. 102:1–102:15, July 2022
2022
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.