REVIEW 3 major objections 5 minor 1 cited by
Cross-Hierarchical Bidirectional Consistency Learning for Fine-Grained Visual Classification
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The central claim is that using a label tree's hierarchy as a bidirectional consistency constraint improves fine-grained classification accuracy and consistency, with reported finest-level gains of 3.1%, 3.0%, and 2.1% over a multi-label…
desk verdict A clean, well-ablated combination of known hierarchical-consistency ideas, but the unexplained dataset-pretrained initialization of the predict submodule makes the reported gains unproven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two modules carry the argument. MGE generates a CAM attention mask and a level-specific feature map for each hierarchy level; the matrix orthogonal decomposition $M_{\mathrm{orth}} = M_{\mathrm{fine}} - \frac{\sum_{m,n} M_{\mathrm{fine}}^{(m,n)} M_{\mathrm{coarse}}^{(m,n)}}{\sum_{m,n} M_{\mathrm{coarse}}^{(m,n)} M_{\mathrm{coarse}}^{(m,n)}} M_{\mathrm{coarse}}$ removes the coarse component from the fine matrix and adds the residual scaled by $\alpha$ back, applied to attention from the previous level and features from the coarsest level. CBC uses the tree adjacency matrix $D_{i,j}$ to expand coarse distributions to the fine dimension and to aggregate fine distributions to the coarse dimension, then defines the combined distribution $\hat{s}_l$ from all other levels and minimizes the Jensen-Shannon divergence $JS(s_l, \hat{s}_l)$ for every level $l$, including an all-level concatenated header. This all-to-all bidirectional loss is what makes the hierarchy constrain every prediction at once.
What would settle it
Retrain CHBC and baseline-multi with the predict branches initialized only from ImageNet weights, removing all target-dataset pretraining; if the reported 3.1% CUB species-level gain over baseline-multi shrinks to near zero, the improvement comes from the hidden in-domain initialization rather than from MGE and CBC.
Extended reading notes
Core claim
The paper proposes CHBC and claims that decomposing attention masks and features across hierarchies, then enforcing bidirectional consistency among all hierarchy levels, improves both accuracy and label consistency. The key reported result is that finest-level accuracy improves by 3.1% on CUB-200-2011, 3.0% on FGVC-Aircraft, and 2.1% on Stanford Cars over the multi-label baseline, while weighted average accuracy and the proposed Tree-based Consistency Rate are the best among the compared methods on all three datasets. The authors attribute this to the Multi Granularity Enhancement (MGE) module, which uses matrix orthogonal decomposition to pull each level's discriminative content away from coarser levels, and to the Cross-hierarchical Bidirectional Consistency (CBC) module, which projects coarse predictions down to fine levels and fine predictions up to coarse levels and minimizes their JS divergence in an all-to-all interaction scheme.
Load-bearing premise
The per-level predict branches are initialized with parameters pre-trained on the target datasets themselves, and the paper does not show that the baseline and comparison models received the same in-domain pretraining, so if that pretraining drives the gains, the proposed MGE and CBC modules may not be responsible.
Editorial extensions
If this is right
- If the reported gains hold, hierarchical label trees improve even finest-level accuracy, so FGVC models should use existing label hierarchies instead of discarding them.
- The ablations claim that all-to-all bidirectional consistency beats both neighbor-only and all-to-finest interaction, implying consistency should be enforced globally across the tree rather than only along adjacent edges.
- The Tree-based Consistency Rate gains (85.0% on CUB, 92.5% on Aircraft, 94.3% on Cars) imply that when the model errs, the wrong fine label is more likely to stay inside the correct superclass, making errors less misleading in practice.
- At the finest level CHBC reports the best accuracy on Aircraft and Cars among compared single-label methods and remains competitive on CUB, so using hierarchy does not appear to trade away fine-grained performance.
Reading between the lines
- The CBC loss can likely be detached from MGE and applied as a standalone regularizer to any hierarchical classifier; a clean test would add only the all-to-all JS term to baseline-multi and measure the gain.
- Because CBC raises the probabilities of sibling subclasses under a confident superclass, it should systematically improve Top-3 and Top-5 accuracy, and the gains could be evaluated as a calibration improvement rather than only Top-1 accuracy.
- The all-to-all scheme computes a consistency term for every level pair, so on deeper trees the cost grows; a sampled or neighbor-pruned consistency graph might recover most of the benefit at lower cost.
- The reported dependence on target-dataset pretraining of the predict branches means the architecture's contribution should be re-measured under a shared initialization protocol before attributing the accuracy gains to MGE and CBC.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CHBC, a framework for fine-grained visual classification that exploits tree-structured label hierarchies without extra annotations. The method has two main components: a Multi-Granularity Enhancement (MGE) module that decomposes and enhances attention masks and features across hierarchy levels via matrix orthogonalization, and a Cross-hierarchical Bidirectional Consistency (CBC) module that enforces consistency between predictions at different granularities using JS divergence in an all-to-all interaction scheme. Experiments are reported on CUB-200-2011, FGVC-Aircraft, and Stanford Cars, with comparisons to hierarchical multi-label methods such as HSE, FGN, HRN, and HCSL, plus ablations of the proposed components. The central claims are that CHBC improves both weighted-average accuracy and the proposed Tree-based Consistency Rate (TCR) over baselines and prior multi-level methods.
Significance. If the empirical results are reliable, the paper makes a modest but useful contribution: it addresses hierarchical fine-grained classification without additional part or bounding-box annotations, and it introduces a bidirectional consistency mechanism plus a TCR metric that directly measures whether predicted labels respect the hierarchy. The ablations in Tables 5-7 are internally consistent and show that each component contributes. The method also does not fit parameters to the reported metrics, and the consistency loss is derived from the externally given tree hierarchy. However, the main empirical claim depends on an undocumented dataset-pretraining step for the predict submodule and on single-run results with small margins over strong baselines, so the significance of the reported gains is currently not fully established.
major comments (3)
- [Section 4.2 (Implementation details)] The paper states that the predict submodule is initialized with parameters pre-trained on the specific datasets in Section 4.1, but it does not specify the pre-training split (training-only versus full dataset), the task, or the duration. This is load-bearing because Tables 1-3 and the ablations in Tables 5-7 compare CHBC against baseline-multi and reproduced SOTA methods, and the reader cannot tell whether those baselines received the same in-domain initialization. If the pre-training uses the test split, the reported numbers would be inflated by leakage; if it merely gives CHBC a better starting point, the gains in Tables 1-3 (e.g., 3.1% on CUB species over baseline-multi) would not be attributable to the MGE and CBC modules. The authors must report the exact pre-training protocol, apply it identically to all compared baselines, or remove this dataset-level pre-training and re-run the experiments.
- [Section 4.3 (Results and analysis), Tables 1-3] All accuracy and TCR numbers are reported from single runs with no error bars or repeated-seed statistics, while several improvements over the strongest prior method are very small: Air wa_acc is 95.3 versus 95.1 for HCSL in Table 2, CUB species accuracy is 87.8 versus 87.7 for HCSL in Table 1, and Cars maker accuracy is 97.8, which is actually below HCSL's 97.9. With such margins, the claim that CHBC consistently outperforms prior methods is not supported without variance estimates or significance testing. Please provide mean and standard deviation over at least three seeds, or otherwise demonstrate that the differences are not within run-to-run noise.
- [Section 4.4 (Ablation study), Tables 5-7] The ablation rows labeled 'Base' are described as baseline-multi, but it is not stated whether baseline-multi uses the same predict-submodule initialization on the target datasets as CHBC. Since the full model and all ablated variants presumably share this initialization, the ablations isolate the effect of MGE and CBC only under the assumption that the initialization is constant across rows. This assumption is not documented, and if baseline-multi is initialized purely from ImageNet while CHBC is initialized from target-dataset weights, the 1.5% MGE-only and 2.3% CBC-only gains in Table 7 could reflect the initialization rather than the modules. Please clarify the initialization for every row in the ablation tables.
minor comments (5)
- [Section 3.2, Eq. (5)] The denominator in the MOD formula sums squared entries of the coarse matrix; the behavior when a coarse matrix is entirely zero at some spatial locations is not discussed. Please specify whether any numerical stabilization is used.
- [Section 3.3, Eq. (13)] The notation "s_i × D_i,j" is ambiguous: it should be a matrix-vector product, not entrywise multiplication. Please clarify the operation and specify the dimensions of the resulting vector.
- [Section 4.2] The sentence "According to HSE [16], this initialization accelerates the convergence of models" cites a prior work for the initialization practice, but HSE [16] is not cited with a page or section number, and the current paper does not describe how the pre-training is performed in practice. Please expand this description.
- [Figure 7] The plots for hyperparameters α and T lack axis labels and numerical tick values, making it difficult to read the actual sensitivities. Please add proper axis labels and legend entries.
- [References] Reference [11] has a typo in the title: "Fine-grained, ornot" should be "Fine-grained, or not". Please also check for consistent capitalization of "Tree Hierarchy" throughout the text.
Circularity Check
No significant circularity: accuracy and TCR are measured against external ground-truth labels, and no load-bearing argument reduces to an input by construction; the §4.2 in-domain initialization of the predict submodule is a comparability confound, not a circular step.
full rationale
The central derivation is self-contained: the MGE modules (Eqs. 1–11) transform features and attention masks, and the CBC loss (Eqs. 12–18) is computed from the ground-truth Tree Hierarchy adjacency matrix, which is external to the model. The reported accuracy numbers in Tables 1–4 are evaluated against held-out labels, so no fitted parameter is renamed as a prediction. The TCR metric (Eq. 22) is not defined in terms of the training loss: it requires the finest-level prediction to equal the ground-truth label, an external correctness condition. The only mild self-referential aspect is that minimizing the JS consistency loss (Eq. 18) will naturally tend to raise the consistency component of TCR; this is a designed alignment between objective and metric rather than an equation-level reduction, and it does not account for the accuracy gains. There are no load-bearing self-citations: the initialization heuristic is attributed to HSE [16], an external prior work, and no uniqueness or ansatz argument is imported from the authors' own prior papers. The §4.2 sentence 'the predict submodule is initialized with parameters pre-trained on the specific datasets in Section 4.1' is a potential confound for the baseline comparison, because the paper does not state the pretraining split or whether baseline-multi received the same initialization; but without evidence that the pretraining used the test split or that the baselines were disadvantaged, this is an attribution/comparability issue, not circularity. Overall, the paper's claims rest on independent empirical evaluation, so the circularity burden is low.
Assumptions & free parameters
free parameters (2)
- alpha (enhancement factor in MGE) =
0.4
- T (temperature coefficient in CBC) =
2
assumptions (2)
- domain assumption The Tree Hierarchy adjacency matrices D_{i,j} accurately reflect the containment relations among labels.
- domain assumption Removing the projection of a fine-grained matrix onto a coarser matrix yields more discriminative fine-grained information.
Cite this review
Pith. "Pith review of Cross-Hierarchical Bidirectional Consistency Learning for Fine-Grained Visual Classification." pith.science (2026). https://pith.science/paper/FVCPE7Y7
@misc{pith2026250413608,
author = {Pith},
title = {Pith review of: Cross-Hierarchical Bidirectional Consistency Learning for Fine-Grained Visual Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/FVCPE7Y7}},
note = {Machine review of arXiv:2504.13608}
}
read the original abstract
Fine-Grained Visual Classification (FGVC) aims to categorize closely related subclasses, a task complicated by minimal inter-class differences and significant intra-class variance. Existing methods often rely on additional annotations for image classification, overlooking the valuable information embedded in Tree Hierarchies that depict hierarchical label relationships. To leverage this knowledge to improve classification accuracy and consistency, we propose a novel Cross-Hierarchical Bidirectional Consistency Learning (CHBC) framework. The CHBC framework extracts discriminative features across various hierarchies using a specially designed module to decompose and enhance attention masks and features. We employ bidirectional consistency loss to regulate the classification outcomes across different hierarchies, ensuring label prediction consistency and reducing misclassification. Experiments on three widely used FGVC datasets validate the effectiveness of the CHBC framework. Ablation studies further investigate the application strategies of feature enhancement and consistency constraints, underscoring the significant contributions of the proposed modules.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
FlexiGrad: Adaptive Gradient Modulation for Hierarchical Fine-Grained Classification
FlexiGrad selectively removes conflicting and reinforces agreeing gradient components between hierarchy levels, improving multi-granularity accuracy on CUB, FGVC-Aircraft and Stanford Cars.
Reference graph
Works this paper leans on
-
[1]
J. Wang, Q. Xu, B. Jiang, B. Luo, J. Tang, Multi-granularity part sampling attention for fine-grained visual classification, IEEE Transactions on Image Processing 33 (2024) 4529–4542.doi:10.1109/TIP.2024.3441813
arXiv 2024
-
[2]
X. Yang, Y. Wang, K. Chen, Y. Xu, Y. Tian, Fine-grained object classification via self-supervised pose alignment, in: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 7389–7398.doi:10.1109/CVPR52688.2022.00725
arXiv 2022
- [3]
- [5]
-
[7]
4682–4692.doi:10.1109/CVPR52688
H.Zhu,W.Ke,D.Li,J.Liu,L.Tian,Y.Shan,Dualcross-attentionlearningforfine-grainedvisualcategorizationandobjectre-identification, in: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 4682–4692.doi:10.1109/CVPR52688. 2022.00465
-
[8]
M.Liu,C.Zhang,H.Bai,R.Zhang,Y.Zhao,Cross-partlearningforfine-grainedimageclassification,IEEETransactionsonImageProcessing 31 (2022) 748–758.doi:10.1109/TIP.2021.3135477
arXiv 2022
-
[9]
M. Wang, P. Zhao, X. Lu, F. Min, X. Wang, Fine-grained visual categorization: A spatial–frequency feature fusion perspective, IEEE Transactions on Circuits and Systems for Video Technology 33 (6) (2023) 2798–2812.doi:10.1109/TCSVT.2022.3227737
arXiv 2023
-
[10]
Q.Diao,Y.Jiang,B.Wen,J.Sun,Z.Yuan,Metaformer:Aunifiedmetaframeworkforfine-grainedrecognition(2022). arXiv:2203.02751
arXiv 2022
Show all 36 references
-
[11]
flamingo
D.Chang,K.Pang,Y.Zheng,Z.Ma,Y.-Z.Song,J.Guo,Your“flamingo”ismy“bird”:Fine-grained,ornot,in:2021IEEE/CVFConference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 11471–11480.doi:10.1109/CVPR46437.2021.01131
2021
-
[12]
R. Wang, C. Zou, W. Zhang, Z. Zhu, L. Jing, Consistency-aware feature learning for hierarchical fine-grained visual classification, in: Proceedings of the 31st ACM International Conference on Multimedia, MM ’23, Association for Computing Machinery, New York, NY, USA, 2023, p. ...
2023
-
[13]
C. N. Silla, A. A. Freitas, A survey of hierarchical classification across different application domains, Springer US (1) (2011)
2011
-
[14]
4848–4857.doi:10.1109/CVPR52688
J.Chen,P.Wang,J.Liu,Y.Qian,Labelrelationgraphsenhancedhierarchicalresidualnetworkforhierarchicalmulti-granularityclassification, in: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 4848–4857.doi:10.1109/CVPR52688. 2022.00481
2022
-
[15]
Y. Liu, L. Zhou, P. Zhang, X. Bai, L. Gu, X. Yu, J. Zhou, E. R. Hancock, Where to focus: Investigating hierarchical attention relationship for fine-grained visual classification, in: Computer Vision – ECCV 2022, Springer Nature Switzerland, Cham, 2022, pp. 57–73
2022
-
[16]
T. Chen, W. Wu, Y. Gao, L. Dong, X. Luo, L. Lin, Fine-grained representation learning and recognition by exploiting hierarchical semantic embedding,in:Proceedingsofthe26thACMInternationalConferenceonMultimedia,MM’18,AssociationforComputingMachinery,New York, NY, USA, 2018, p. ...
2018
-
[17]
K. T. Noor, A. Robles-Kelly, L. Y. Zhang, M. R. Bouadjenek, W. Luo, A consistency-aware deep capsule network for hierarchical multi-label image classification, Neurocomputing 604 (2024) 128376.doi:https://doi.org/10.1016/j.neucom.2024.128376
2024
-
[18]
T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, S. Belongie, Feature pyramid networks for object detection, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 936–944.doi:10.1109/CVPR.2017.106. P. Gao et al.:Preprint submitted to Else...
2017 doi
-
[19]
Y.Ding,Z.Ma,S.Wen,J.Xie,D.Chang,Z.Si,M.Wu,H.Ling,Ap-cnn:Weaklysupervisedattentionpyramidconvolutionalneuralnetwork for fine-grained visual classification, IEEE Transactions on Image Processing 30 (2021) 2826–2836.doi:10.1109/TIP.2021.3055617
2021
-
[20]
W.Luo,X.Yang,X.Mo,Y.Lu,L.S.Davis,J.Li,J.Yang,S.-N.Lim,Cross-xlearningforfine-grainedvisualcategorization,2019IEEE/CVF International Conference on Computer Vision (ICCV) (2019) 8241–8250
2019
-
[21]
R.Du,D.Chang,A.K.Bhunia,J.Xie,Z.Ma,Y.-Z.Song,J.Guo,Fine-grainedvisualclassificationviaprogressivemulti-granularitytraining of jigsaw patches, in: Computer Vision – ECCV 2020, Springer International Publishing, Cham, 2020, pp. 153–168
2020
-
[22]
H.Zheng,J.Fu,Z.-J.Zha,J.Luo,T.Mei,Learningrichparthierarchieswithprogressiveattentionnetworksforfine-grainedimagerecognition, IEEE Transactions on Image Processing 29 (2020) 476–488.doi:10.1109/TIP.2019.2921876
2020
-
[23]
Zhuang, Y
P. Zhuang, Y. Wang, Y. Qiao, Learning attentive pairwise interaction for fine-grained classification, in: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020, pp. 13130–13137.doi:10.1609/aaai.v34i07.7016
2020 doi
-
[24]
Q. Xu, S. Li, J. Wang, B. Jiang, J. Tang, Context-semantic quality awareness network for fine-grained visual categorization, ArXiv abs/2403.10298 (2024)
2024 arXiv
-
[25]
597–602.doi:10.1109/CAI59869.2024.00117
T.Do,H.Tran,E.Tjiputra,Q.D.Tran,A.Nguyen,Fine-grainedvisualclassificationusingselfassessmentclassifier,in:2024IEEEConference on Artificial Intelligence (CAI), 2024, pp. 597–602.doi:10.1109/CAI59869.2024.00117
2024
-
[26]
Y. Liu, L. Yang, Y. Wang, Hierarchical fine-grained visual classification leveraging consistent hierarchical knowledge, in: Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2024, Vilnius, Lithuania, September 9–13, 2024, Pro...
2024 doi
-
[27]
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba, Learning deep features for discriminative localization, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 2921–2929.doi:10.1109/CVPR.2016.319
2016 doi
-
[28]
C. Wah, S. Branson, P. Welinder, P. Perona, S. Belongie, The caltech-ucsd birds-200-2011 dataset, Tech. Rep. CNS-TR-2011-001, California Institute of Technology (2011)
2011
-
[29]
S. Maji, E. Rahtu, J. Kannala, M. Blaschko, A. Vedaldi, Fine-grained visual classification of aircraft (2013).arXiv:1306.5151
2013 arXiv
-
[30]
Krause, M
J. Krause, M. Stark, J. Deng, L. Fei-Fei, 3d object representations for fine-grained categorization, in: 2013 IEEE International Conference on Computer Vision Workshops, 2013, pp. 554–561.doi:10.1109/ICCVW.2013.77
2013 doi
-
[31]
K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.doi:10.1109/CVPR.2016.90
2016 doi
-
[32]
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255.doi:10.1109/CVPR.2009.5206848
2009
-
[33]
E. D. Cubuk, B. Zoph, D. Mané, V. Vasudevan, Q. V. Le, Autoaugment: Learning augmentation strategies from data, in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 113–123.doi:10.1109/CVPR.2019.00020
2019
-
[34]
Wehrmann, R
J. Wehrmann, R. Cerri, R. Barros, Hierarchical multi-label classification networks, in: J. Dy, A. Krause (Eds.), Proceedings of the 35th International Conference on Machine Learning, Vol. 80 of Proceedings of Machine Learning Research, PMLR, 2018, pp. 5075–5084
2018
-
[35]
Giunchiglia, T
E. Giunchiglia, T. Lukasiewicz, Coherent hierarchical multi-label classification networks, in: 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada, 2020
2020
-
[36]
Chang, Y
D. Chang, Y. Ding, J. Xie, A. K. Bhunia, X. Li, Z. Ma, M. Wu, J. Guo, Y.-Z. Song, The devil is in the channels: Mutual-channel loss for fine-grained image classification, IEEE Transactions on Image Processing 29 (2020) 4683–4695.doi:10.1109/TIP.2020.2973812
2020
-
[37]
Y. Gao, X. Han, X. Wang, W. Huang, M. Scott, Channel interaction networks for fine-grained image categorization, in: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020, pp. 10818–10825.doi:10.1609/aaai.v34i07.6712
2020 doi
-
[38]
Y. Hu, Y. Yang, J. Zhang, X. Cao, X. Zhen, Attentional kernel encoding networks for fine-grained visual categorization, IEEE Transactions on Circuits and Systems for Video Technology 31 (1) (2021) 301–314.doi:10.1109/TCSVT.2020.2978115. P. Gao et al.:Preprint submitted to Else...
2021
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.