REVIEW 4 major objections 6 minor 41 references
Hierarchical Neural Collapse Detection Transformer for Class Incremental Object Detection
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper proposes Hier-DETR, claiming that a frozen hierarchical tree of class prototypes guided by Proxy-NCA on each decoder layer prevents catastrophic forgetting in incremental object detection without replay or teacher distillation…
desk verdict A genuinely new hierarchical GOF prototype tree for incremental DETR, with large reported gains but the continual-learning claim is unmeasured and the 'knowledge distillation' label is misleading. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Hierarchical Neural Collapse (HNC) tree, a multi-level collection of General Orthogonal Frame (GOF) prototypes. A GOF is a set of mutually orthogonal vectors whose lengths are proportional to the number of samples in each class, which makes it suitable for imbalanced detection data. The tree's first level holds one frozen orthogonal vector per class; higher levels hold averages of child vectors for super-classes. The Proxy-NCA loss is the mechanism that uses the tree: after Hungarian matching assigns each query a label, the query's output at each decoder layer is pushed toward the positive prototype and away from all negative prototypes, so early layers learn super-class separation and later layers commit to the fine class. The tree is what carries the claim that forgetting is avoided without replay or distillation.
What would settle it
Run the 70+10 COCO experiment and compute AP only on the 70 classes trained in the first phase; if that old-class AP drops substantially compared with a model trained only on the first phase, the frozen prototypes are not preventing catastrophic forgetting.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that a fixed, hierarchy-aware set of class prototypes can serve as both the classifier and the memory in class-incremental object detection. Instead of learning a new classifier head for each task, Hier-DETR initializes a tree of General Orthogonal Frame vectors—orthogonal class directions whose lengths follow class frequencies—and never updates the first level. New classes are added as new orthogonal vectors, and each decoder layer's query output is pulled by Proxy-NCA toward the appropriate coarse or fine prototype. The paper argues that this arrangement preserves old-task knowledge, keeps Hungarian matching (the assignment algorithm that pairs predicted boxes with ground-truth objects) stable, and makes the model converge faster, and it reports accuracy gains of roughly 8 AP on COCO 70+10 and 7.5 AP on MTSD 150+71 over the prior best method.
Load-bearing premise
The load-bearing assumption is that freezing the first-level class prototypes acts as knowledge distillation, so old classes stay accurate even though the model never sees old images and has no teacher-student loss; the paper reports only overall AP, not old-class AP, so forgetting could be hidden.
Editorial extensions
If this is right
- If the reported accuracy holds, incremental detection can be made practical: no exemplar buffer, no generative replay, and no teacher-student distillation, so memory and inference costs stay low.
- Any DETR-style detector could adopt the HNC guidance, since the method only changes how decoder query outputs are supervised, not the backbone or attention machinery.
- The coarse-to-fine alignment should let models converge faster, because early decoder layers learn super-class separation first and only the last layer commits to fine classes; the paper's instability plot supports this.
- Hier-DETR improves even the join-training setting on MTSD (from 38.4 to 39.8 AP), which suggests the hierarchical prototype tree helps imbalanced detection in general, not only continual learning.
Reading between the lines
- Beyond the paper, the reported gains do not yet separate old-class from new-class accuracy; a direct measurement of old-class AP would show whether the frozen prototypes truly prevent forgetting or simply improve new-class learning.
- A testable extension is to apply the HNC tree to a single-phase long-tailed detection benchmark, since the GOF vectors already encode class-imbalance information through their lengths.
- Another open direction is growing the tree from data rather than a fixed taxonomy; if the hierarchy must be known in advance, the method's applicability outside labeled taxonomies is limited.
- The layer-wise coarse-to-fine supervision suggests a general recipe for stabilizing DETR-style matching that could be ported to other transformer detectors, but the paper only demonstrates it on RT-DETR.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Hier-DETR, a class-incremental object detection framework built on RT-DETR. The central idea is to replace conventional replay or teacher-student distillation with a fixed hierarchical structure of class prototypes: a Hierarchical Neural Collapse (HNC) tree whose first layer contains class-level General Orthogonal Frame (GOF) prototypes, with coarser super-class prototypes in higher layers. After Hungarian matching assigns queries to ground-truth objects, a Proxy-NCA loss (Eq. 4-6) guides each query's intermediate decoder outputs toward the prototypes of its assigned class and super-classes, while the first-layer prototypes are kept fixed across tasks and described as a form of knowledge distillation. The authors report state-of-the-art results on COCO 2017 and MTSD in two-phase incremental settings (e.g., 48.4 AP on COCO 70+10 and 45.46 AP on MTSD 150+71), plus a small joint-training comparison and an ablation-style instability plot.
Significance. If verified, the proposal is an interesting and potentially practical direction: replacing expensive replay buffers and distillation losses with a fixed, hierarchy-aware prototype geometry could make incremental object detection more efficient and better behaved. The construction is clearly presented, and Theorem 3.1's orthogonality argument is simple and internally consistent. The experimental claim of large gains over prior work is striking. However, the empirical case is currently under-supported: there are no error bars or code, no old-class forgetting metric, no full ablation table, and the central 'knowledge distillation' claim lacks a direct measure of catastrophic forgetting. The approach is not internally inconsistent, and the missing evidence is in principle obtainable, so the manuscript merits revision rather than rejection.
major comments (4)
- [Abstract and Section 3.2] The claim that fixing the first-layer GOF prototypes acts as 'knowledge distillation' and preserves upstream-task knowledge is not supported by any measured forgetting metric. Under the strict two-phase protocol of Section 3.1, task-1 annotations contain only new-class objects, so in Eq. (4)-(6) old-class prototypes appear only in the negative denominator; there is no distillation loss, no feature-regularization term, and no replay buffer. Tables 1 and 2 report only combined AP after all phases, which can remain high if new-class gains mask old-class losses. The authors should report old-class AP, per-task AP, and a forgetting metric (e.g., average forgetting) for every setting before the continual-learning benefit can be accepted.
- [Section 4.2, Tables 1 and 2] The state-of-the-art comparison is weakened by missing experimental rigor: CL-RTDETR is a new baseline reproduced by the authors, but no configuration details or code are given, and no error bars or multiple-seed standard deviations are reported for any entry in Tables 1 and 2. The claimed improvements of +8 AP and +7.5 AP need variance information to be credible, particularly because the comparison baselines are partly taken from other papers or reproduced without a released implementation.
- [Section 3.2, Theorem 3.1] Theorem 3.1 establishes only pairwise orthogonality of parent-layer vectors, assuming orthogonality of child-layer GOF columns. It does not establish that the recursively constructed parent vectors form a General Orthogonal Frame with norms proportional to parent-class sample counts, as required by Definition 2.2 and condition 2 of the HNC construction. Because GOF is defined by both orthogonality and a specific norm scaling, the proof is incomplete for the claim that every HNC layer is a GOF; please provide the norm computation for parent vectors after the described normalization and scaling.
- [Section 4.3, Ablations] The ablation study is not a full ablation. Figure 3 shows a training-curve comparison on a '150+70' setting that is not used in the main tables (which use 129+92 and 150+71), and the instability metric IS is referenced to DN-DETR but never defined or quantified in a table. There is no ablation isolating the hierarchical structure, the choice of decoder layers receiving the Proxy-NCA loss, the loss weight, or the distance function d(q_i,w) in Eq. (4)-(6). Without these ablations, the contribution of each component to the final AP is unknown.
minor comments (6)
- [Abstract and Table 2] The dataset is called MTSD throughout, but 'MSTD' appears in the Abstract and in the Table 2 caption; please correct the typo.
- [Section 3.2] There is a typo in 'querry' (should be 'query') and in the heading 'HNC tree Contruction' (should be 'Construction').
- [Figure 3] The caption says '150+70' while the main experiments use '150+71'; please reconcile the setting name.
- [Section 3.3, Eq. (4)] The distance function d(q_i,w) is never specified; please state whether it is cosine distance, negative squared L2, or another metric, and how query outputs are normalized or projected before computing it.
- [Section 3.3] Please specify how many decoder layers l are used and which of them receive the Proxy-NCA loss; the current text says only that 'several decoder layers' are used.
- [References] Reference [35] appears to share authors with the present paper; please clarify the relationship and state what new theoretical content Theorem 3.1 adds beyond that prior result.
Circularity Check
No significant circularity: HNC prototypes are constructed from class hierarchy and sample counts, Proxy-NCA is a training loss, and reported APs are measured against external benchmarks; the forgetting concern is an empirical limitation, not a circular step.
full rationale
The paper's derivation chain is self-contained relative to its reported metrics. The Hierarchical Neural Collapse (HNC) prototypes are constructed from class hierarchy and per-class sample counts (Section 3.2), not from the reported AP values. The Proxy-NCA loss (Eq. 4-6) is a training objective that aligns decoder query outputs to those fixed prototypes; it contains no term fitted to the evaluation metric. The GOF convergence result is cited from prior work [35], and although that reference shares authors with the present paper, it is a stated mathematical theorem about convergence of class means under imbalanced data, with assumptions that do not include the target detection AP; it is therefore independent evidence rather than a fitted input. Theorem 3.1 is a direct algebraic consequence of defining parent GOF vectors as means of orthogonal child vectors, so it is not circular. The paper's claim that fixing first-layer prototypes acts as 'knowledge distillation' is not derived formally, and the absence of old-class AP or a forgetting metric is a legitimate correctness/empirical limitation (also acknowledged in the Limitations section), but that does not make any prediction reduce to its inputs by construction. The SOTA comparisons are against external benchmarks and controlled baselines such as CL-RTDETR. No self-definitional, fitted-input, or renaming pattern is present, so the appropriate score is 0.
Assumptions & free parameters
free parameters (4)
- Per-class sample counts a_i for GOF prototype norms =
Per-class counts in training splits
- Proxy-NCA loss weight =
not stated
- Distance function d(q_i, w) in Proxy-NCA =
not stated
- Set of decoder layers receiving HNC loss =
not stated ('several decoder layers')
assumptions (5)
- domain assumption Deep linear network GOF convergence theorem from [35] transfers to transformer decoder query features in imbalanced detection data.
- domain assumption A valid semantic hierarchy T is available for all classes in both COCO and MTSD before training.
- ad hoc to paper Frozen first-layer GOF prototypes preserve old-class knowledge without replay or distillation.
- domain assumption Decoder query outputs can be treated as last-layer features eligible for neural-collapse alignment.
- standard math Sums of vectors from disjoint sets of mutually orthogonal GOF columns remain mutually orthogonal.
Cite this review
Pith. "Pith review of Hierarchical Neural Collapse Detection Transformer for Class Incremental Object Detection." pith.science (2026). https://pith.science/paper/VYRRV4GI
@misc{pith2026250608562,
author = {Pith},
title = {Pith review of: Hierarchical Neural Collapse Detection Transformer for Class Incremental Object Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/VYRRV4GI}},
note = {Machine review of arXiv:2506.08562}
}
read the original abstract
Recently, object detection models have witnessed notable performance improvements, particularly with transformer-based models. However, new objects frequently appear in the real world, requiring detection models to continually learn without suffering from catastrophic forgetting. Although Incremental Object Detection (IOD) has emerged to address this challenge, these existing models are still not practical due to their limited performance and prolonged inference time. In this paper, we introduce a novel framework for IOD, called Hier-DETR: Hierarchical Neural Collapse Detection Transformer, ensuring both efficiency and competitive performance by leveraging Neural Collapse for imbalance dataset and Hierarchical relation of classes' labels.
Reference graph
Works this paper leans on
-
[1]
RT-DETRv3: Real-time End-to-End Object Detection with Hierarchical Dense Positive Supervision
Wang, S., Xia, C., Lv, F., Shi, Y.: RT-DETRv3: Real-time End-to-End Object Detection with Hierarchical Dense Positive Supervision (2024). https://arxiv.org/ abs/2409.08475
work page Pith review arXiv 2024
-
[2]
https://arxiv.org/abs/2211.12860
Zong, Z., Song, G., Liu, Y.: DETRs with Collaborative Hybrid Assignments Training (2023). https://arxiv.org/abs/2211.12860
arXiv 2023
-
[3]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp
Liu, Y., Schiele, B., Vedaldi, A., Rupprecht, C.: Continual detection transformer for incremental object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23799–23808 (2023)
work page 2023
-
[4]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp
Kim, J., Cho, H., Kim, J., Tiruneh, Y.Y., Baek, S.: Sddgr: Stable diffusion-based deep generative replay for class incremental object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28772–28781 (2024)
work page 2024
-
[5]
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. ArXivabs/2005.12872(2020)
arXiv 2020
-
[6]
arXiv preprint arXiv:2010.04159 (2020)
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 (2020)
arXiv 2020
-
[7]
In: Proceedings of the IEEE/CVF 14 Conference on Computer Vision and Pattern Recognition, pp
Li, F., Zhang, H., Liu, S., Guo, J., Ni, L.M., Zhang, L.: Dn-detr: Accelerate detr training by introducing query denoising. In: Proceedings of the IEEE/CVF 14 Conference on Computer Vision and Pattern Recognition, pp. 13619–13627 (2022)
work page 2022
-
[8]
arXiv preprint arXiv:2203.03605 (2022)
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L.M., Shum, H.-Y.: Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605 (2022)
arXiv 2022
Show all 41 references
-
[9]
arXiv preprint arXiv:2201.12329 (2022)
Liu, S., Li, F., Zhang, H., Yang, X., Qi, X., Su, H., Zhu, J., Zhang, L.: Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201.12329 (2022)
2022 arXiv
-
[10]
TPAMI40(12), 2935–2947 (2018)
Li, Z., Hoiem, D.: Learning without forgetting. TPAMI40(12), 2935–2947 (2018)
2018
-
[11]
In: ECCV, pp
Douillard, A., Cord, M., Ollion, C., Robert, T., Valle, E.: Podnet: Pooled outputs distillation for small-tasks incremental learning. In: ECCV, pp. 86–102 (2020)
2020
-
[12]
In: CVPR, pp
Zhao, B., Xiao, X., Gan, G., Zhang, B., Xia, S.-T.: Maintaining discrimination and fairness in class incremental learning. In: CVPR, pp. 13208–13217 (2020)
2020
-
[13]
In: CVPR, pp
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: Incremental classifier and representation learning. In: CVPR, pp. 5533–5542 (2017)
2017
-
[14]
In: CVPR, pp
Liu, Y., Su, Y., Liu, A., Schiele, B., Sun, Q.: Mnemonics training: Multi-class incremental learning without forgetting. In: CVPR, pp. 12245–12254 (2020)
2020
-
[15]
In: ICLR (2022)
Wang, L., Zhang, X., Yang, K., Yu, L., Li, C., Hong, L., Zhang, S., Li, Z., Zhong, Y., Zhu, J.: Memory replay with data compression for continual learning. In: ICLR (2022)
2022
-
[16]
In: ECCV, pp
Castro, F.M., Mar ´ ın-Jim´ enez, M.J., Guil, N., Schmid, C., Alahari, K.: End-to-end incremental learning. In: ECCV, pp. 241–257 (2018)
2018
-
[17]
https://arxiv.org/abs/2302.03004
Yang, Y., Yuan, H., Li, X., Lin, Z., Torr, P., Tao, D.: Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning (2023). https://arxiv.org/abs/2302.03004
2023 arXiv
-
[18]
https://arxiv.org/abs/2203.09081
Yang, Y., Chen, S., Li, X., Xie, L., Lin, Z., Tao, D.: Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network? (2022). https://arxiv.org/abs/2203.09081
2022 arXiv
-
[20]
In: European Conference on Computer Vision, pp
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll´ ar, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European Conference on Computer Vision, pp. 740–755 (2014). Springer 15
2014
-
[21]
ArXivabs/1706.03762 (2017)
Vaswani, A., Shazeer, N.M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. ArXivabs/1706.03762 (2017)
2017 arXiv
-
[22]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp
Meng, D., Chen, X., Fan, Z., Zeng, G., Li, H., Yuan, Y., Sun, L., Wang, J.: Conditional detr for fast training convergence. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3651–3660 (2021)
2021
-
[23]
Advances in neural information processing systems28(2015)
Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detec- tion with region proposal networks. Advances in neural information processing systems28(2015)
2015
-
[24]
Advances in Neural Information Processing Systems33, 21002–21012 (2020)
Li, X., Wang, W., Wu, L., Chen, S., Hu, X., Li, J., Tang, J., Yang, J.: Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. Advances in Neural Information Processing Systems33, 21002–21012 (2020)
2020
-
[25]
In: ICCV, pp
Shmelkov, K., Schmid, C., Alahari, K.: Incremental learning of object detectors without catastrophic forgetting. In: ICCV, pp. 3420–3429 (2017)
2017
-
[26]
Pattern Recognition131, 108863 (2022)
Yang, D., Zhou, Y., Zhang, A., Sun, X., Wu, D., Wang, W., Ye, Q.: Multi-view correlation distillation for incremental object detection. Pattern Recognition131, 108863 (2022)
2022
-
[27]
arXiv preprint arXiv:2009.01129 (2020)
Zhou, W., Chang, S., Sosa, N., Hamann, H., Cox, D.: Lifelong object detection. arXiv preprint arXiv:2009.01129 (2020)
2020 arXiv
-
[28]
In: CVPR, pp
Feng, T., Wang, M., Yuan, H.: Overcoming catastrophic forgetting in incremental object detection via elastic response distillation. In: CVPR, pp. 9427–9436 (2022)
2022
-
[29]
In: IJCNN, pp
Chen, L., Yu, C., Chen, L.: A new knowledge distillation for incremental object detection. In: IJCNN, pp. 1–7 (2019)
2019
-
[30]
In: ICME, pp
Hao, Y., Fu, Y., Jiang, Y.-G., Tian, Q.: An end-to-end architecture for class- incremental object detection with knowledge distillation. In: ICME, pp. 1–6 (2019)
2019
-
[31]
Pattern Recognition Letter140, 109–115 (2020)
Peng, C., Zhao, K., Lovell, B.C.: Faster ILOD: incremental learning for object detectors based on faster RCNN. Pattern Recognition Letter140, 109–115 (2020)
2020
-
[32]
In: CVPR, pp
Joseph, K.J., Khan, S.H., Khan, F.S., Balasubramanian, V.N.: Towards open world object detection. In: CVPR, pp. 5830–5840 (2021)
2021
-
[33]
In: CVPR, pp
Dai, Z., Cai, B., Lin, Y., Chen, J.: UP-DETR: Unsupervised pre-training for object detection with transformers. In: CVPR, pp. 1601–1610 (2021)
2021
-
[34]
Proceedings of the National Academy of 16 Sciences117(40), 24652–24663 (2020) https://doi.org/10.1073/pnas.2015509117
Papyan, V., Han, X.Y., Donoho, D.L.: Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of 16 Sciences117(40), 24652–24663 (2020) https://doi.org/10.1073/pnas.2015509117
2020 doi
-
[35]
arXiv preprint arXiv:2301.00437 (2023)
Dang, H., Tran, T., Osher, S., Tran-The, H., Ho, N., Nguyen, T.: Neural col- lapse in deep linear networks: from balanced to imbalanced data. arXiv preprint arXiv:2301.00437 (2023)
2023 arXiv
-
[36]
In: Proceedings of the IEEE International Conference on Computer Vision, pp
Movshovitz-Attias, Y., Toshev, A., Leung, T.K., Ioffe, S., Singh, S.: No fuss dis- tance metric learning using proxies. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 360–368 (2017)
2017
-
[37]
Naval research logistics quarterly2(1-2), 83–97 (1955)
Kuhn, H.W.: The hungarian method for the assignment problem. Naval research logistics quarterly2(1-2), 83–97 (1955)
1955
-
[38]
In: Chen, S., Onishi, R., Ananthanarayanan, G., Li, Q
Li, D., Tasci, S., Ghosh, S., Zhu, J., Zhang, J., Heck, L.P.: RILOD: near real-time incremental learning for object detection at the edge. In: Chen, S., Onishi, R., Ananthanarayanan, G., Li, Q. (eds.) SEC, pp. 113–126 (2019)
2019
-
[39]
CVIU 210, 103229 (2021)
Peng, C., Zhao, K., Maksoud, S., Li, M., Lovell, B.C.: SID: incremental learning for anchor-free object detection via selective and inter-related distillation. CVIU 210, 103229 (2021)
2021
-
[40]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp
Feng, T., Wang, M., Yuan, H.: Overcoming catastrophic forgetting in incre- mental object detection via elastic response distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9427–9436 (2022)
2022
-
[41]
In: European Conference on Computer Vision, pp
Ertler, C., Mislej, J., Ollmann, T., Porzi, L., Neuhold, G., Kuang, Y.: The map- illary traffic sign dataset for detection and classification on a global scale. In: European Conference on Computer Vision, pp. 68–84 (2020). Springer
2020
-
[42]
Sensors22(12), 4389 (2022) 17
Ruiz, I., Serrat, J.: Hierarchical novelty detection for traffic sign recognition. Sensors22(12), 4389 (2022) 17
2022
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.