REVIEW 5 major objections 7 minor 42 references
Virtual Memory for 3D Gaussian Splatting
T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper proposes rendering page IDs on a proxy mesh to a visibility buffer so that only visible Gaussian pages are streamed to the GPU, bounding memory use and accelerating rendering.
desk verdict Solid new application of virtual texturing to 3DGS, with an honest but incomplete evaluation that omits transparent scenes. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the page-ID visibility buffer: a low-resolution render of the proxy mesh in which each fragment outputs the ID of the Gaussian page covering that mesh face. This buffer converts occlusion into an explicit selection step, because the pixels it contains name exactly the pages that can contribute to the final image. Around it sits a page table that maps virtual pages to physical GPU pages, uses least-recently-used eviction, and stores per-page LOD information; page links, found by random sampling inside ellipsoids and checking the nearest proxy face, ensure that overlapping Gaussians on neighboring pages are requested together. The proxy mesh itself is built from plane-ellipsoid intersections cleaned with morphological operations and reconstructed with Marching Cubes, and the LOD levels are created by k-means clustering of Gaussian attributes followed by averaging.
What would settle it
Take a scene with a known Gaussian whose mean lies on an occluded page but whose visible tail extends into a neighboring page, render it with page links disabled and a deliberately small visibility buffer, and check whether the final image shows a hole; the paper's own ablation, in which disabling links drops PSNR from 42.89 dB to 35.02 dB, is already a strong version of this test.
Extended reading notes
Core claim
The central claim is that a 3D Gaussian Splatting scene can be rendered within a strict GPU memory budget by selecting Gaussians through a visibility-buffer page table instead of loading the whole scene. Preprocessing converts the scene into a simplified proxy mesh, groups Gaussians into pages by position, links pages whose ellipsoids overlap, and generates several LOD levels per page by clustering and averaging Gaussian attributes. Each frame, the proxy mesh is rendered with page IDs into a low-resolution visibility buffer; a compute shader reduces that buffer to the list of required pages, the page table uploads missing pages and evicts least-recently-used ones, and the LOD level for each page is chosen from its closest pixel depth. The paper reports that this selection reduces memory use and rendering time on the test scenes, with the largest gains when occlusion is abundant, and that page links are required to avoid visible artifacts.
Load-bearing premise
The method assumes that a coarse proxy mesh captures the scene's occlusion structure accurately enough, and that the random-sampling page-link test catches every overlap between Gaussian pages, so that every Gaussian that contributes to the image is on a page the visibility buffer requests.
Editorial extensions
If this is right
- A scene larger than GPU memory can be rendered with a fixed-size GPU buffer, because only pages visible from the current camera position are resident at any time.
- Occluded environments benefit most: the visibility buffer removes Gaussians hidden behind other structure, which frustum culling alone cannot do.
- When the GPU buffer is under pressure, the adaptive LOD mechanism pushes level transitions closer to the camera, trading distant detail for a hole-free frame.
- The method accepts standard 3D Gaussian Splatting scenes without retraining, so existing reconstructions can be preprocessed and rendered at larger scale.
- The page table and visibility buffer add only a small fraction of frame time compared with the depth sorting and splatting they save.
Reading between the lines
- Because the visibility buffer sees only what the proxy mesh exposes, scenes with heavy transparency would need pinned pages on transparent surfaces; the paper lists this as future work, and a coarse depth channel in the buffer would make the fix straightforward.
- Replacing the random-sampling page-link heuristic with a bounding-volume hierarchy over Gaussian extents would likely make links both cheaper and more complete, directly attacking the main source of missing pages.
- The same page-table structure could serve any scene representation that can provide an occlusion proxy, so the idea extends beyond Gaussian splats to hybrid mesh-radiance-field renderers.
- Pushing page assignment and proxy-mesh construction into the training stage, instead of running them as an offline post-process, would remove the hour-scale preprocessing the largest test scene requires and could optimize pages for few links from the start.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a virtual-memory system for rendering 3D Gaussian Splatting scenes that exceed GPU memory. An offline stage extracts a proxy mesh from the Gaussian ellipsoids, assigns Gaussians to pages, links overlapping pages, and builds LOD levels by k-means merging. At runtime, the proxy mesh is rendered to a low-resolution visibility buffer of page IDs; the resulting page list, augmented by links, drives a page table that streams required pages into a bounded GPU buffer, with LRU eviction and distance-based LOD selection. The system is implemented in Vulkan and ported to iOS via MoltenVK, and evaluated on four scenes (SciArt, Residence, Alameda, Berlin) on a GTX 1070 desktop and an iPad Pro M1. The paper reports that the method reduces GPU memory and frame times relative to rendering the full scene without virtual memory.
Significance. If the claims hold, the paper offers a practical mechanism for rendering 3DGS scenes larger than GPU memory, borrowing mature virtual-texturing technology. The authors provide a full implementation, a mobile port, and ablation studies that isolate page linking, LOD, buffer size, and page size; they also clearly identify several limitations (low-resolution visibility buffer, lack of LOD blending, transparency). The contribution is incremental but useful: it is one of the first to apply occlusion-based page streaming rather than frustum-only streaming to 3DGS. However, the central assertion that the method 'efficiently identifies visible Gaussians' is only demonstrated for the tested opaque scenes; the transparency limitation and the approximate page-linking heuristic are load-bearing and need stronger validation before the contribution can be fully accepted.
major comments (5)
- [Sec. 4.1, Sec. 4.3] The page-determination stage is an opaque-surface test: page IDs are rendered on the proxy mesh with ordinary depth testing, so any face behind another face is treated as occluded. 3DGS, however, composites Gaussians with alpha blending, so semi-transparent structures do not fully hide the Gaussians behind them; those background pages will be marked invisible and never streamed, creating holes. The manuscript acknowledges this in Sec. 4.3 ('Transparency challenges in page determination echo traditional virtual texturing issues'), but the abstract and Sec. 4.1 still claim the method 'efficiently identifies visible Gaussians.' This is a load-bearing gap: either restrict the claim to opaque scenes, evaluate a scene with significant transparency, or implement the suggested pinning of transparent pages.
- [Sec. 3.3, Table 2] Page linking is the mechanism that repairs missed visibility, but its correctness is not established. Links are found by generating random points inside each ellipsoid and checking only the nearest proxy-mesh face; the paper gives no sampling density, no convergence criterion, and no analysis of failure modes. Table 2 shows how much depends on this heuristic: disabling page links drops PSNR from 42.89/47.19 to 35.02, and Figure 16 shows visible artifacts. Because the central claim is that the visibility buffer plus links returns the exact set of pages needed for artifact-free rendering, please provide an empirical sensitivity analysis (number of samples per ellipsoid, number of nearest faces) and, on a small scene, compare against exhaustive ellipsoid-intersection testing.
- [Table 2] The rows 'Without page links, with LOD' and 'Without page links, without LOD' report identical PSNR (35.02) and SSIM (0.97) values. Either this is a copy-paste error or LOD has no effect when page links are disabled; in either case the table must be corrected and the result explained, since Table 2 is the main evidence for the contribution of page linking.
- [Sec. 5] The quantitative evaluation compares only ablations of the proposed method and does not include any existing large-scene 3DGS system as a baseline (e.g., CityGaussian [20], Hierarchical 3DGS [15], or a full-scene non-streaming renderer on the same hardware). All timings are medians with no error bars or repeated-run variance, despite the acknowledged nondeterminism of the adaptive LOD thresholds (Sec. 5.2). To support the claims of reduced memory and accelerated rendering, please report mean/median with variance over multiple runs and, where feasible, compare against a published system or a clear non-streaming baseline.
- [Sec. 4.3, Table 4] The paper acknowledges that rendering the visibility buffer at low resolution 'can cause distant pages to be missed' (Sec. 4.3). This is not peripheral: the streaming decision is based entirely on that buffer, so a missed distant page translates directly into holes in the final image. Page links only cover geometric overlap, not visibility through distant gaps. Please add a quantitative study of visibility-buffer resolution versus the 'Missing Pages' metric introduced in Table 4, and either bound the resolution error or relax the claim of exact visibility identification.
minor comments (7)
- [Sec. 5.1.3] The heading 'Synchronsitaion' is a typo for 'Synchronization'.
- [Table 2] The table caption uses 'ArtSci' while Figure 13 and the text call the scene 'SciArt'; please unify the naming.
- [Table 4] The 'Missing Pages' metric is not defined; with a 250-page buffer, a value of 549 suggests it counts repeated requests or page faults rather than unique missing pages, so please define it in the text.
- [Table 1] The text around Table 1 is inconsistent about storage overhead: it says the overhead is negligible, but the final file size for SciArt (1505.5 MiB) is nearly double its initial size (804.9 MiB) due to padding and LOD levels; please clarify that 'overhead' refers only to the mesh/links file and state the LOD storage cost explicitly.
- [Sec. 3.1] In Equation (1), the symbols v_z, s, and n are not defined at first use; please define them in the text before or immediately after the equation.
- [Figures 20 and 21] The stacked time-segment bars are not accompanied by an explicit legend or color key; please add a legend so the reader can map each segment to the stages described in the text.
- [Sec. 5.3.2] The claim that frame times decrease as the number of pages decreases is based on unsmoothed FPS; please report a correlation coefficient or a smoothed plot to support the relationship.
Circularity Check
No significant circularity: visible-page determination is an implemented heuristic validated by image-quality ablations, not a claim true by construction.
full rationale
The paper's central claim is that rendering page IDs of a proxy mesh into a visibility buffer, together with page links, identifies the Gaussian pages required for the final image. This is not circular, because the claim is not true by definition: the correctness of the identified page set is checked empirically against full-scene renders using PSNR/SSIM, and the paper's own ablation (Table 2) shows that disabling page links degrades PSNR from 42.89/47.19 to 35.02. Thus the page-selection mechanism is validated by its downstream image quality rather than being defined to match the output. The 50-80% memory-usage target in Section 5.1.1 is an explicit adaptive control goal, not a fitted parameter later repackaged as a prediction; memory and frame-time results are direct measurements. The page-linking heuristic using random points inside ellipsoids and nearest proxy-mesh faces is approximate, and the paper explicitly acknowledges its failure modes in Section 4.3 (low-resolution visibility buffer can miss distant pages) and in the Conclusion (transparency challenges in page determination). These are legitimate correctness risks, not hidden circularity. The only self-citation, Haberl [10], is referenced in Section 5.1 as a source of more in-depth implementation description and is not load-bearing for the method's correctness or for any derived result. The paper does not claim a first-principles derivation; it is an empirical systems contribution, and no step in its derivation chain reduces by construction to its own inputs.
Assumptions & free parameters
free parameters (6)
- Page size =
2048 Gaussians
- Staging buffer size =
~18.4 MiB (40 pages)
- LOD fixed scale-up factor
- k-means attribute weights
- Adaptive LOD threshold step size =
1% adjustment
- Visibility buffer resolution
assumptions (5)
- standard math Hartmann's ellipsoid-plane intersection method and Marching Cubes produce a valid proxy mesh.
- domain assumption The proxy mesh generated from axis-aligned slice intersections is a faithful visibility proxy for occluded Gaussians.
- ad hoc to paper Random point sampling within each ellipsoid followed by nearest-face lookup discovers all page overlaps that matter for visual quality.
- ad hoc to paper Merging Gaussians by averaging attributes and scaling up by a fixed factor produces acceptable LOD levels.
- domain assumption The global sort plus hardware rasterizer pipeline is a fair baseline for comparing virtual memory on and off.
Cite this review
Pith. "Pith review of Virtual Memory for 3D Gaussian Splatting." pith.science (2026). https://pith.science/paper/G4YTEFU5
@misc{pith2026250619415,
author = {Pith},
title = {Pith review of: Virtual Memory for 3D Gaussian Splatting},
year = {2026},
howpublished = {\url{https://pith.science/paper/G4YTEFU5}},
note = {Machine review of arXiv:2506.19415}
}
read the original abstract
3D Gaussian Splatting represents a breakthrough in the field of novel view synthesis. It establishes Gaussians as core rendering primitives for highly accurate real-world environment reconstruction. Recent advances have drastically increased the size of scenes that can be created. In this work, we present a method for rendering large and complex 3D Gaussian Splatting scenes using virtual memory. By leveraging well-established virtual memory and virtual texturing techniques, our approach efficiently identifies visible Gaussians and dynamically streams them to the GPU just in time for real-time rendering. Selecting only the necessary Gaussians for both storage and rendering results in reduced memory usage and effectively accelerates rendering, especially for highly complex scenes. Furthermore, we demonstrate how level of detail can be integrated into our proposed method to further enhance rendering speed for large-scale scenes. With an optimized implementation, we highlight key practical considerations and thoroughly evaluate the proposed technique and its impact on desktop and mobile devices.
Figures
Figures from the paper (15 more)
Reference graph
Works this paper leans on
-
[20]
Y . Liu, H. Guan, C. Luo, L. Fan, N. Wang, J. Peng, and Z. Zhang. Citygaus- sian: Real-time high-quality large-scale scene rendering with gaussians. (arXiv:2404.01133), July 2024. arXiv:2404.01133 [cs]. doi: 10.48550/ arXiv.2404.01133 2
- [15]
-
[1]
S. Barrett. Sparse virtual textures. https://silverspaceship.com/ src/svt/, 2008. Accessed: 2024-06-04. 3, 5
work page 2008
-
[2]
J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields.ICCV, 2023. 8
work page 2023
-
[3]
H. Chen, C. Li, and G. H. Lee. Neusg: Neural implicit surface reconstruc- tion with 3d gaussian splatting guidance. (arXiv:2312.00846), Dec. 2023. arXiv:2312.00846 [cs]. 2
arXiv 2023
-
[4]
Z. Fan, K. Wang, K. Wen, Z. Zhu, D. Xu, and Z. Wang. Lightgaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps. (arXiv:2311.17245), Feb. 2024. arXiv:2311.17245 [cs]. 2
arXiv 2024
- [5]
-
[6]
L. Gao, J. Yang, B.-T. Zhang, J.-M. Sun, Y .-J. Yuan, H. Fu, and Y .-K. Lai. Mesh-based gaussian splatting for real-time large-scale deformation. (arXiv:2402.04796), Feb. 2024. arXiv:2402.04796 [cs]. 2
arXiv 2024
Show all 42 references
-
[7]
Girish, K
S. Girish, K. Gupta, and A. Shrivastava. Eagles: Efficient accelerated 3d gaussians with lightweight encodings. (arXiv:2312.04564), Dec. 2023. arXiv:2312.04564 [cs]. 2
2023 arXiv
-
[8]
T. K. Group. Khronos vulkan portability initiative. https://www. khronos.org/vulkan/portability-initiative, 2024. Accessed: 2024-10-15. 7
2024
-
[9]
Guédon and V
A. Guédon and V . Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. (arXiv:2311.12775), Dec. 2023. arXiv:2311.12775 [cs]. 2
2023 arXiv
-
[10]
Haberl.Virtual memory for 3D Gaussian Splatting
J. Haberl.Virtual memory for 3D Gaussian Splatting. Master’s Thesis, Institute of Visual Computing, Graz University of Technology, Inffeldgasse 16/2, A-8010 Graz, Austria, Dec. 2024. doi: 10.3217/xstz7-q3q02 6
2024 doi
-
[11]
J. Hable. Variable rate shading with visibility buffer render- ing. https://advances.realtimerendering.com/s2024/index. html#hable, 2024. Accessed: 2024-08-07. 6
2024
-
[12]
Hartmann
E. Hartmann. Computerunterstützte darstellende und konstruktive geome- trie. https://www2.mathematik.tu-darmstadt.de/~ehartmann/ cdg-skript-1998.pdf, 1997. Accessed: 2024-09-09. 3
1998
-
[14]
Kerbl, G
B. Kerbl, G. Kopanas, T. Leimkuehler, and G. Drettakis. 3D Gaussian Splatting for Real-Time Radiance Field Rendering.ACM Transactions on Graphics, 42(4):1–14, Aug. 2023. doi: 10.1145/3592433 1, 2, 5, 6, 11
2023 doi
-
[16]
J. C. Lee, D. Rho, X. Sun, J. H. Ko, and E. Park. Compact 3d gaus- sian representation for radiance field. (arXiv:2311.13681), Nov. 2023. arXiv:2311.13681 [cs]. 2
2023 arXiv
-
[17]
doi: 10.1145/3658160 2, 8, 11
- [18]
-
[19]
Z. Li, Z. Chen, Z. Li, and Y . Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. (arXiv:2312.16812), Dec. 2023. arXiv:2312.16812 [cs]. 2
2023 arXiv
-
[21]
L. Lin, Y . Liu, Y . Hu, X. Yan, K. Xie, and H. Huang. Capturing, recon- structing, and simulating: the urbanscene3d dataset. (arXiv:2107.04286), July 2022. arXiv:2107.04286 [cs]. 8
2022 arXiv
-
[22]
T. Lu, M. Yu, L. Xu, Y . Xiangli, L. Wang, D. Lin, and B. Dai. Scaffold-gs: Structured 3d gaussians for view-adaptive rendering. (arXiv:2312.00109), Nov. 2023. arXiv:2312.00109 [cs]. 2
2023 arXiv
-
[23]
W. E. Lorensen and H. E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. InProceedings of the 14th Annual Confer- ence on Computer Graphics and Interactive Techniques, SIGGRAPH ’87, 7 pages, p. 163–169. Association for Computing Machinery, New York,...
1987
-
[24]
Mittring and C
M. Mittring and C. GmbH. Advanced virtual texture topics. InACM SIGGRAPH 2008 Games, SIGGRAPH ’08, 29 pages, p. 23–51. Asso- ciation for Computing Machinery, New York, NY , USA, 2008. doi: 10. 1145/1404435.1404438 4
2008
-
[25]
A. J. Mayer.Virtual Texturing. Master’s Thesis, Institute of Computer Graphics and Algorithms, Vienna University of Technology, Favoriten- strasse 9-11/E193-02, A-1040 Vienna, Austria, Oct. 2010. 3, 6, 11
2010
-
[26]
Apple ipad pro 11" (3rd gen, 2021)
NanoReview.net. Apple ipad pro 11" (3rd gen, 2021). https://nanoreview.net/en/tablet/ apple-ipad-pro-11-3rd-gen-2021?m=r.1 , 2021. Accessed: 2024-10-18. 10
2021
-
[27]
Morgenstern, F
W. Morgenstern, F. Barthel, A. Hilsmann, and P. Eisert. Compact 3d scene representation via self-organizing gaussian grids. (arXiv:2312.13299), Dec. 2023. arXiv:2312.13299 [cs]. 2
2023 arXiv
-
[28]
Niedermayr, J
S. Niedermayr, J. Stumpfegger, and R. Westermann. Compressed 3d gaus- sian splatting for accelerated novel view synthesis. (arXiv:2401.02436), Jan. 2024. arXiv:2401.02436 [cs]. 2
2024 arXiv
-
[29]
Navaneet, K
K. Navaneet, K. Pourahmadi Meibodi, S. Koohpayegani, and H. Pirsiavash. Compact3D: Compressing Gaussian Splat Radiance Field Models with Vector Quantization. Jan. 2024. 2
2024
-
[30]
Pranckeviˇcius
A. Pranckeviˇcius. Making gaussian splats smaller. https://aras-p. info/blog/2023/09/13/Making-Gaussian-Splats-smaller/ , Sep 2023. Accessed: 2024-06-04. 2
2023
-
[31]
Pranckevi ˇcius
A. Pranckevi ˇcius. Making gaussian splats more smaller. https://aras-p.info/blog/2023/09/27/ Making-Gaussian-Splats-more-smaller/ , Sep 2023. Accessed: 2024-04-06. 2
2023
-
[32]
Richermoz
A. Richermoz. Sparse texture binding is painfully slow. https://forums.developer.nvidia.com/t/ sparse-texture-binding-is-painfully-slow , 2023. Accessed: 2024-09-01. 5
2023
-
[33]
K. Ren, L. Jiang, T. Lu, M. Yu, L. Xu, Z. Ni, and B. Dai. Octree-gs: Towards consistent real-time rendering with lod-structured 3d gaussians. (arXiv:2403.17898), Mar. 2024. arXiv:2403.17898 [cs]. 2
2024 arXiv
-
[34]
Tatarchuk
N. Tatarchuk. Welcome and introduction – trends in games and render- ing. https://advances.realtimerendering.com/s2024/index. html#intro, 2024. Accessed: 2024-08-20. 7
2024
-
[35]
Tancik, E
M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, J. Kerr, T. Wang, A. Kristof- fersen, J. Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa. Nerfstudio: A modular framework for neural radiance field development. InACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH ’23,
2023
-
[36]
Moltenvk
The Brenwill Workshop Ltd. Moltenvk. https://github.com/ KhronosGroup/MoltenVK, 2015. Accessed: 2024-09-24. 5
2015
-
[37]
Nvidia geforce gtx 1070
techpowerup.com. Nvidia geforce gtx 1070. https://www. techpowerup.com/gpu-specs/geforce-gtx-1070.c2840 , 2024. Accessed: 2024-10-18. 10
2024
-
[38]
S. Willems. Hardware capability database for vulkan. http://vulkan. gpuinfo.org, 2016. Accessed: 2024-09-24. 5
2016
-
[39]
Waczy´nska, P
J. Waczy´nska, P. Borycki, S. Tadeja, J. Tabor, and P. Spurek. Games: Mesh- based adapting and modification of gaussian splatting. (arXiv:2402.01459), Feb. 2024. arXiv:2402.01459 [cs]. 2
2024 arXiv
-
[40]
Y . Yuan, X. Li, Y . Huang, S. De Mello, K. Nagano, J. Kautz, and U. Iqbal. Gavatar: Animatable 3d gaussian avatars with implicit mesh learning. (arXiv:2312.11461), Dec. 2023. arXiv:2312.11461 [cs]. 2
2023 arXiv
-
[41]
Z. Yan, W. F. Low, Y . Chen, and G. H. Lee. Multi-scale 3d gaussian splatting for anti-aliased rendering. (arXiv:2311.17089), Nov. 2023. arXiv:2311.17089 [cs]. 2, 4, 11
2023 arXiv
-
[43]
H. Zhao, H. Weng, D. Lu, A. Li, J. Li, A. Panda, and S. Xie. On scal- ing up 3d gaussian splatting training. (arXiv:2406.18533), June 2024. arXiv:2406.18533 [cs]. 2
2024
-
[2024]
arXiv:2403.11367 [cs]. 2
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.