pith. sign in

arxiv: 2605.15852 · v2 · pith:EUVLVGO5new · submitted 2026-05-15 · 💻 cs.CV

GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction

Leyang Chen , Junyi Wu , Zhiteng Li , Yulun Zhang This is my paper

Pith reviewed 2026-06-30 19:17 UTC · model grok-4.3

classification 💻 cs.CV
keywords streaming 3D reconstructionKV cache evictionmonocular videotoken managementtraining-free methodgeometry-guided scoringefficient inference
0
0 comments X

The pith

GHOST uses a model's 3D geometry outputs to evict redundant tokens from the KV cache in streaming reconstruction.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Streaming 3D reconstruction from long monocular video sequences builds a KV cache that grows linearly and creates a memory bottleneck. GHOST tackles this with a training-free framework that evicts tokens online by consulting the model's own geometry predictions instead of attention-score heuristics or fixed anchor frames. It combines hierarchical dual-level importance scoring, a privilege mechanism that shields special tokens, and cosine-similarity-guided layer-wise budget allocation. The result keeps reconstruction quality intact while shrinking the cache by nearly half and accelerating inference. A reader would care because the approach makes extended video sequences feasible on limited hardware without retraining or quality trade-offs.

Core claim

GHOST is a training-free KV cache management framework that exploits the model's own 3D geometry outputs to evict redundant tokens online during streaming 3D reconstruction. It rests on three components: a hierarchical dual-level importance scoring scheme, a privilege mechanism that protects special tokens from eviction, and a cosine-similarity-guided layer-wise budget allocation.

What carries the argument

Geometry-hierarchical online streaming token eviction that scores token importance from the model's 3D outputs and allocates eviction budgets layer by layer.

If this is right

  • KV cache size drops by nearly half relative to full-cache baselines.
  • Inference runs 1.75 times faster than prior state-of-the-art eviction methods.
  • Reconstruction quality remains comparable across standard benchmarks.
  • Eviction decisions occur online without any additional model training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same geometry-driven eviction logic could be tested on other streaming transformer tasks such as long video generation.
  • Extending the method to sequences far longer than the evaluated benchmarks might expose edge cases in the privilege mechanism.
  • If geometry predictions become unstable under domain shift, the eviction policy could inadvertently remove useful tokens.

Load-bearing premise

The model's own 3D geometry outputs stay accurate enough to mark which tokens are truly redundant without hurting final reconstruction quality.

What would settle it

Compare reconstruction quality on a long video sequence when geometry predictions are deliberately perturbed versus when they are left untouched; a measurable drop would falsify the claim.

Figures

Figures reproduced from arXiv: 2605.15852 by Junyi Wu, Leyang Chen, Yulun Zhang, Zhiteng Li.

Figure 1
Figure 1. Figure 1: Radar comparison across 7-Scenes, NRGBD and Bonn (averaged over all input lengths; outer = better). GHOST consistently dominates all baselines on every axis. Transformer models [17] have achieved remarkable results in 3D reconstruction from monocular im￾ages [21, 11, 18], learning to predict dense depth, point maps, and camera poses in a single forward pass. VGGT [18] extends this to multi-view sequences b… view at source ↗
Figure 2
Figure 2. Figure 2: Correlation between Key-sim score and two frame attributes: Left: Negligible linear correla￾tion between Key-sim score and camera pose change (ρ = −0.07); right: Moderate positive linear correla￾tion between Key-sim score and depth gradient variance (ρ = +0.31). Dashed lines denote linear fitting trends. As shown in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Top row: Raw RGB inputs from Long3D Lecture Hall, 7-Scenes Heads, 7-Scenes Chess, [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: GHOST inference pipeline. Offline: Cosine-similarity profiling allocates per-layer budgets. Online: An eviction mode that prunes KV cache to layer-wise budget computed offline with Geometry-Hierarchical Importance scoring and Special token boost . GHOST assigns per-patch importance ϕ(t, p) = wf sframe(t) + wkstoken(t, p), where sframe combines camera motion scam, depth variance sgeo, and recency stemp, whi… view at source ↗
Figure 5
Figure 5. Figure 5: Layer-wise budget allocation guided by cosine similarity. Larger input–output colour discrepancy and larger arrows indicate lower ρ¯ℓ; the cylinder shows how Btotal is dis￾tributed, with such layers receiving larger Bℓ. Camera tokens ct and register tokens {r i t} encode global scene geometry state and structural priors. Evicting these tokens can corrupt pose estimation and scene globalisation, yet standar… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative reconstruction comparison on 7-Scenes (Chess, Fire, Heads, Office, Kitchen [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Per-layer cosine similarity ρ¯ℓ (blue) and GHOST budget Bℓ (orange, τ=0.5). Lower similarity layers receive larger budgets [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Accuracy Mean (↓) versus sequence length on the Long3D benchmark. The shaded region highlights the gap between GHOST and InfiniteVGGT. GHOST’s advantage over key-similarity eviction (InfiniteVGGT) grows with sequence length, confirming that geometry-aware eviction scales more gracefully to very long sequences. Limitations. GHOST is not directly applicable to architectures that lack any causal structure (e.… view at source ↗
read the original abstract

Streaming 3D reconstruction from long monocular video sequences requires maintaining a key-value (KV) cache that grows linearly with sequence length, creating a severe memory bottleneck. Existing approaches either truncate the cache to a fixed set of anchor frames, leading to reconstruction quality degradation, or rely on attention-score heuristics that are agnostic to 3D scene structure, failing to preserve geometrically valuable tokens. To address these problems, we present GHOST (Geometry-Hierarchical Online Streaming Token Eviction), a training-free KV cache management framework that exploits the model's own 3D geometry outputs to evict redundant tokens online. GHOST introduces three mutually reinforcing innovations: a hierarchical dual-level importance scoring scheme, a privilege mechanism that protects special tokens from eviction, and a cosine-similarity-guided layer-wise budget allocation. Experiments on various benchmarks show that GHOST preserves excellent reconstruction quality while cutting the KV cache by nearly half and delivering 1.75x faster inference compared to state-of-the-art methods. Our code is available at https://github.com/lokiniuniu/GHOST.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces GHOST, a training-free KV cache management framework for streaming 3D reconstruction from monocular video. It exploits the model's own 3D geometry outputs via a hierarchical dual-level importance scoring scheme, a privilege mechanism to protect special tokens, and cosine-similarity-guided layer-wise budget allocation to evict redundant tokens online. The central empirical claim is that this preserves excellent reconstruction quality while reducing KV cache size by nearly half and delivering 1.75x faster inference versus state-of-the-art methods on various benchmarks.

Significance. If the results hold, the work addresses a key memory bottleneck in long-sequence transformer-based 3D reconstruction by incorporating geometric structure into eviction decisions rather than relying on generic attention heuristics or fixed anchor frames. The training-free design and public code release support reproducibility and practical deployment for extended video sequences.

major comments (2)
  1. [Method description of hierarchical dual-level importance scoring and privilege mechanism] The core eviction mechanism relies on importance scores derived from the model's own evolving 3D geometry outputs (hierarchical dual-level scheme plus cosine-similarity layer allocation). This creates a self-referential dependency where early-frame geometry errors can trigger eviction of tokens that later become critical, violating the quality-preservation guarantee in the streaming setting. The privilege mechanism and budget allocation do not address this feedback risk, which is load-bearing for the central claim.
  2. [Abstract and Experiments] Abstract and experimental claims: quantitative assertions (cache reduction by nearly half, 1.75x speedup, preserved quality on various benchmarks) are presented without reference to specific tables, error bars, dataset details, or ablation studies that would allow verification of robustness under the self-referential scoring.
minor comments (1)
  1. [Abstract] The phrase 'nearly half' for KV cache reduction is imprecise; reporting exact ratios or percentages with standard deviations would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the thoughtful and detailed review. We address each major comment below and outline targeted revisions to improve clarity and robustness.

read point-by-point responses

  1. Referee: The core eviction mechanism relies on importance scores derived from the model's own evolving 3D geometry outputs (hierarchical dual-level scheme plus cosine-similarity layer allocation). This creates a self-referential dependency where early-frame geometry errors can trigger eviction of tokens that later become critical, violating the quality-preservation guarantee in the streaming setting. The privilege mechanism and budget allocation do not address this feedback risk, which is load-bearing for the central claim.

    Authors: We appreciate the referee's identification of this potential feedback loop in the streaming regime. The hierarchical dual-level scoring explicitly combines per-token geometric saliency with frame-level consistency checks to reduce sensitivity to isolated early errors, while the privilege mechanism reserves a fixed budget for tokens whose geometry scores exceed a threshold in the initial frames. Nevertheless, we agree that an explicit treatment of this risk is warranted. We will add a new subsection in the method discussion analyzing the self-referential dependency and include additional ablation experiments that inject controlled early-frame geometry noise to quantify its effect on later eviction decisions and final reconstruction quality. revision: partial

  2. Referee: Abstract and experimental claims: quantitative assertions (cache reduction by nearly half, 1.75x speedup, preserved quality on various benchmarks) are presented without reference to specific tables, error bars, dataset details, or ablation studies that would allow verification of robustness under the self-referential scoring.

    Authors: The supporting evidence appears in Section 4: Table 1 reports cache-size reduction and wall-clock speedup, Table 2 presents PSNR/SSIM/LPIPS on the listed benchmarks with per-sequence statistics, and Figure 5 together with Table 3 contain the ablation studies. Error bars are shown for multi-run metrics in the supplementary figures. Dataset specifications and evaluation protocols are detailed in Section 4.1. We will revise the abstract to include inline citations to these tables and figures, expand the main-text discussion of robustness under geometry estimation variance, and ensure all quantitative claims are explicitly linked to the corresponding results. revision: yes

Circularity Check

0 steps flagged

No significant circularity; method is algorithmic and self-contained

full rationale

The paper presents a training-free algorithmic framework (GHOST) that uses the model's existing 3D geometry outputs for online token eviction scoring via hierarchical dual-level importance, privilege mechanism, and cosine-similarity budget allocation. No equations, derivations, or first-principles results are shown that reduce claimed performance to quantities defined by fitted parameters or self-citations. The approach is evaluated on external benchmarks without any load-bearing self-citation chains or self-definitional reductions. This is the normal case of a self-contained empirical method with no circularity in its derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract supplies no explicit free parameters, mathematical axioms, or newly postulated entities; the framework is described at the level of high-level design choices.

pith-pipeline@v0.9.1-grok · 5726 in / 1115 out tokens · 48096 ms · 2026-06-30T19:17:10.405191+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 8 canonical work pages · 3 internal anchors

  1. [1]

    Neural rgb-d surface reconstruction

    Dejan Azinovi´c, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. InICCV, 2022

    2022

  2. [2]

    Token Merging: Your ViT But Faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  3. [3]

    PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. PyramidKV: Dynamic KV cache compression based on pyramidal information funneling.arXiv:2406.02069, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    Diffrate: Differentiable compression rate for efficient vision transformers

    Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate: Differentiable compression rate for efficient vision transformers. InICCV, 2023

    2023

  5. [5]

    Ttt3r: 3d reconstruc- tion as test-time training

    Xingyu Chen, Yue Chen, Yuliang Xiu, Andreas Geiger, and Anpei Chen. Ttt3r: 3d reconstruc- tion as test-time training. InICLR, 2026

    2026

  6. [6]

    Evit: Privacy-preserving image retrieval via encrypted vision transformer in cloud computing.TCSVT, 2024

    Qihua Feng, Peiya Li, Zhixun Lu, Chaozhuo Li, Zefan Wang, Zhiquan Liu, Chunhui Duan, Feiran Huang, Jian Weng, and Philip S Yu. Evit: Privacy-preserving image retrieval via encrypted vision transformer in cloud computing.TCSVT, 2024

    2024

  7. [7]

    Exploiting uncertainty in regression forests for accurate camera relocalization

    Abner Guzman-Rivera, Pushmeet Kohli, Ben Glocker, Jamie Shotton, Toby Sharp, Andrew Fitzgibbon, and Shahram Izadi. Exploiting uncertainty in regression forests for accurate camera relocalization. InCVPR, 2014

    2014

  8. [8]

    Gradpruner: Gradient-guided layer pruning enabling efficient fine-tuning and inference for llms.arXiv preprint arXiv:2601.19503, 2026

    Wei Huang, Anda Cheng, and Yinggui Wang. Gradpruner: Gradient-guided layer pruning enabling efficient fine-tuning and inference for llms.arXiv preprint arXiv:2601.19503, 2026

    work page arXiv 2026

  9. [9]

    3d gaussian splatting for real-time radiance field rendering.ACM TOG, 2023

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis, et al. 3d gaussian splatting for real-time radiance field rendering.ACM TOG, 2023

    2023

  10. [10]

    A fast post-training pruning framework for transformers

    Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers. InNeurIPS, 2022

    2022

  11. [11]

    MASt3R: Grounding image matching in 3D with MASt3R

    Vincent Leroy, Yohann Cabon, and Jerome Revaud. MASt3R: Grounding image matching in 3D with MASt3R. InECCV, 2024

    2024

  12. [12]

    Analyzing the mechanism of attention collapse in vggt from a dynamics perspective.arXiv preprint arXiv:2512.21691, 2025

    Huan Li, Longjun Luo, Yuling Shi, and Xiaodong Gu. Analyzing the mechanism of attention collapse in vggt from a dynamics perspective.arXiv preprint arXiv:2512.21691, 2025

    work page arXiv 2025

  13. [13]

    SnapKV: LLM knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. InNeurIPS, 2024

    2024

  14. [14]

    Evict3r: Training-free token eviction for memory-bounded streaming visual geometry trans- formers.arXiv preprint arXiv:2509.17650, 2025

    Soroush Mahdi, Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada, and Mahdi Javanmardi. Evict3r: Training-free token eviction for memory-bounded streaming visual geometry trans- formers.arXiv preprint arXiv:2509.17650, 2025

    work page arXiv 2025

  15. [15]

    Palazzolo, J

    E. Palazzolo, J. Behley, P. Lottes, P. Giguère, and C. Stachniss. ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residuals. InIROS, 2019

    2019

  16. [16]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. InNeurIPS, 2021

    2021

  17. [17]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS, 2017

    2017

  18. [18]

    VGGT: Visual geometry grounded deep structured feature transformer

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. VGGT: Visual geometry grounded deep structured feature transformer. InCVPR, 2025

    2025

  19. [19]

    Efficient video transformers with spatial-temporal token selection

    Junke Wang, Xitong Yang, Hengduo Li, Li Liu, Zuxuan Wu, and Yu-Gang Jiang. Efficient video transformers with spatial-temporal token selection. InECCV, 2022. 10

    2022

  20. [20]

    Continuous 3d perception model with persistent state

    Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. InCVPR, 2025

    2025

  21. [21]

    DUSt3R: Geometric 3D vision made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. DUSt3R: Geometric 3D vision made easy. InCVPR, 2024

    2024

  22. [22]

    Quantcache: Adaptive importance-guided quantization with hierarchical latent and layer caching for video generation

    Junyi Wu, Zhiteng Li, Zheng Hui, Yulun Zhang, Linghe Kong, and Xiaokang Yang. Quantcache: Adaptive importance-guided quantization with hierarchical latent and layer caching for video generation. InICCV, 2025

    2025

  23. [23]

    FlashEdit: Decoupling Speed, Structure, and Semantics for Precise Image Editing

    Junyi Wu, Zhiteng Li, Haotong Qin, Xiaohong Liu, Linghe Kong, Yulun Zhang, and Xiaokang Yang. Flashedit: Decoupling speed, structure, and semantics for precise image editing.arXiv preprint arXiv:2509.22244, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  24. [24]

    Balancegs: Algorithm-system co-design for efficient 3d gaussian splatting training on gpu

    Junyi Wu, Jiaming Xu, Jinhao Li, Yongkang Zhou, Jiayi Pan, Xingyang Li, and Guohao Dai. Balancegs: Algorithm-system co-design for efficient 3d gaussian splatting training on gpu. In Asia and South Pacific Design Automation Conference, 2026

    2026

  25. [25]

    Point3r: Streaming 3d reconstruction with explicit spatial pointer memory

    Yuqi Wu, Wenzhao Zheng, Jie Zhou, and Jiwen Lu. Point3r: Streaming 3d reconstruction with explicit spatial pointer memory. InNeurIPS, 2025

    2025

  26. [26]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InICLR, 2024

    2024

  27. [27]

    Specontext: Enabling efficient long-context reasoning with speculative context sparsity in llms

    Jiaming Xu, Jiayi Pan, Hanzhen Wang, Yongkang Zhou, Jiancai Ye, Yu Wang, and Guohao Dai. Specontext: Enabling efficient long-context reasoning with speculative context sparsity in llms. InACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2026

    2026

  28. [28]

    Streamingvlm: Real-time understanding for infinite video streams.ICLR, 2026

    Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Kelly Peng, Yao Lu, and Song Han. Streamingvlm: Real-time understanding for infinite video streams.ICLR, 2026

    2026

  29. [29]

    Evo-vit: Slow-fast token evolution for dynamic vision trans- former

    Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision trans- former. InAAAI, 2022

    2022

  30. [30]

    Global vision transformer pruning with hessian-aware saliency

    Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision transformer pruning with hessian-aware saliency. InCVPR, 2023

    2023

  31. [31]

    2410.18517 , archivePrefix =

    Yifei Yang, Zouying Cao, Qiguang Chen, Libo Qin, Dongjie Yang, Hai Zhao, and Zhi Chen. Kvsharer: Efficient inference via layer-wise dissimilar kv cache sharing.arXiv preprint arXiv:2410.18517, 2024

    work page arXiv 2024

  32. [32]

    A-vit: Adaptive tokens for efficient vision transformer

    Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. InICCV, 2022

    2022

  33. [33]

    arXiv preprint arXiv:2601.02281 (2026)

    Shuai Yuan, Yantai Yang, Xiaotian Yang, Xupeng Zhang, Zhonghao Zhao, Lingming Zhang, and Zhipeng Zhang. Infinitevggt: Visual geometry grounded transformer for endless streams. arXiv preprint arXiv:2601.02281, 2026

    work page arXiv 2026

  34. [34]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. InNeurIPS, 2020

    2020

  35. [35]

    Magnitude pruning of large pretrained trans- former models with a mixture gaussian prior.Journal of Data Science: JDS, 2024

    Mingxuan Zhang, Yan Sun, and Faming Liang. Magnitude pruning of large pretrained trans- former models with a mixture gaussian prior.Journal of Data Science: JDS, 2024

    2024

  36. [36]

    H2O: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2O: Heavy-hitter oracle for efficient generative inference of large language models. InNeurIPS, 2023

    2023

  37. [37]

    Streaming 4d visual geometry transformer

    Dong Zhuo, Wenzhao Zheng, Jiahe Guo, Yuqi Wu, Jie Zhou, and Jiwen Lu. Streaming 4d visual geometry transformer. InICLR, 2026. 11 A GHOST Eviction Algorithm A.1 Online Incremental Computation Computing full importance from scratch at every eviction step would require O(T 2) operations. GHOST maintains animportance cachethat stores per-frame raw scores: • O...

    2026