REVIEW 3 major objections 5 minor 33 references
Towards Privacy-Preserving Fine-Grained Visual Classification via Hierarchical Learning from Label Proportions
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that fine-grained visual classification can be done with no instance labels at all, using only bag-level label proportions plus a known class hierarchy, and reports 76–77% accuracy on three benchmarks.
desk verdict A real empirical gain for LLP-based FGVC, but the paper's own ablations show the hierarchy adds only ~0.4 points; the dictionary learning module is what matters. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is the Unrolled Hierarchical Fine-Grained Sparse Dictionary Learning module: the iterative sparse-coding objective $\min_{D,Z} \frac{1}{2}\|F-DZ\|_2^2 + \lambda\|Z\|_1$ is unrolled into $L$ learnable network layers, with the dictionary reorganized into category blocks and masked by coarse- and medium-grained Sparsemax classifiers that suppress irrelevant categories. The companion Hierarchical Proportion Loss sums the standard proportion loss at each level, so bag supervision matches the granularity of the masks and drives progressive feature refinement.
What would settle it
One falsifying experiment: repeat the CUB, Aircraft, and Cars bag evaluations with randomly permuted coarse and medium class groupings while keeping all other settings identical; if accuracy stays near the reported 76–77% levels, the hierarchical refinement is not what carries the claim.
Extended reading notes
Core claim
The central claim, stated the way a sympathetic reader would state it, is that hierarchical structure is what lets bag-level proportion supervision do fine-grained work: a category-organized sparse dictionary, unrolled into learnable layers and masked at coarse and medium granularities, forces progressive refinement from coarse to fine features. LHFGLP couples this dictionary module with a proportion loss applied at every level of the hierarchy, and the paper reports that the full framework consistently outperforms LLP baselines on all three datasets while staying within a few points of instance-supervised FGVC models that do see every label.
Load-bearing premise
The load-bearing premise is that a correct coarse-to-medium-to-fine class hierarchy is known in advance for the target dataset; if that taxonomy is unavailable, mismatched, or not the true source of the reported gains, the central contribution loses its footing.
Editorial extensions
If this is right
- Accurate fine-grained recognition can be trained with no instance labels at all, so data owners can publish classifiers without releasing per-image annotations.
- Adding a coarse-to-medium-to-fine hierarchy to LLP improves accuracy over flat LLP baselines on CUB, Aircraft, and Cars.
- The unrolled sparse dictionary supplies most of the gain over plain LLP, with hierarchical masks contributing a smaller further improvement on the reported ablations.
- Because the module is plug-and-play, it can be attached to existing fine-grained feature extractors without changing their bag-level training setup.
Reading between the lines
- A direct testable extension is to run LHFGLP on a privacy-sensitive domain with a natural taxonomy, such as medical lesion subtypes; success there would show the hierarchy, not benchmark-specific tuning, is carrying the result.
- The paper's aircraft ablation shows the full coarse-plus-medium masks add only 0.37 points over no masks, so the marginal contribution of the hierarchy itself is not cleanly isolated; a shuffled-hierarchy control experiment would settle whether the claimed mechanism is responsible.
- Because the framework is backbone-agnostic, pairing it with larger pretrained visual encoders could shrink the remaining gap to instance-supervised FGVC, an avenue the paper lists as future work.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes LHFGLP, a framework for fine-grained visual classification (FGVC) under the Learning from Label Proportions (LLP) paradigm, where training uses only bag-level label proportions and no instance-level labels. The method combines an unrolled sparse dictionary learning module with hierarchical category-aware masking and a hierarchical proportion loss, supported by a ResNet-50 backbone. Experiments on CUB-200-2011, FGVC Aircraft, and Stanford Cars report mean accuracies of 76.05, 76.68, and 77.22, respectively, against best LLP baselines of 73.59, 71.94, and 73.76, with small standard deviations across three trials. Ablation studies attribute large gains to the unrolled dictionary learning module, while the hierarchical masks contribute a small increment on Aircraft (76.31 to 76.68). The paper concludes that bag-level proportions plus a hierarchy can yield competitive fine-grained recognition while protecting instance labels.
Significance. If the empirical claims hold, the paper makes a useful contribution by demonstrating that fine-grained classification is feasible under bag-level supervision, and by introducing an unrolled dictionary learning component that appears to be the main source of improvement. The evaluation is generally sound: three independent trials are reported with standard deviations, and evaluation uses held-out instance-level labels, so the reported gains are not an artifact of fitting the test labels. The main weakness is that the central 'hierarchical' contribution, emphasized in the title and abstract, is not cleanly isolated: the hierarchy-free ablation is reported only on Aircraft and shows a 0.37-point gain, while the much larger gains come from the dictionary module. This attribution issue determines what the paper has actually demonstrated.
major comments (3)
- [§4.3, Table 4] The claim that hierarchical learning is the source of the consistent outperformance is not supported by the ablations. On Aircraft, removing all hierarchical masks and using only the fine-grained proportion loss (LHFGLPnoMASK) yields 76.31%, versus 76.68% for the full model, so the entire coarse- and medium-granularity hierarchy adds only 0.37 points. No hierarchy-free ablation is reported for CUB or Cars, so the 'consistently' hierarchical benefit is unverified on two of the three datasets. Please provide the no-mask, coarse-only, and medium-only ablations on all three datasets, or reframe the contribution to emphasize unrolled dictionary learning under bag-level supervision rather than hierarchical learning.
- [§4.3, Table 3] The 'without Dictionary Learning' row reports exactly the same numbers as the LLP baseline on all three datasets (73.31, 71.38, 73.40). If the hierarchical proportion loss and hierarchical classifiers are still used in this ablation, one would not expect exact equality with plain LLP. If they are removed as well, then the table only compares the full method against a plain LLP pipeline and does not isolate the dictionary module while keeping hierarchical supervision constant. Please state precisely which components are kept in this ablation and report the corresponding configuration; alternatively, provide an ablation that removes only the unrolled dictionary while retaining the hierarchical proportion loss.
- [§3.3, Eq. (2)-(4) and §4.1] Several hyperparameters and implementation details that are necessary for reproducibility are omitted: the number of unrolled layers L, the number of dictionary atoms per category (n_atoms), the initialization and size of the dictionary D, the initialization of the stepsize mu, and the schedule for the sparsity weight lambda are not given. These details matter because the main empirical gain is attributed to the unrolled dictionary module; without them, a reader cannot assess or reproduce the reported improvements.
minor comments (5)
- [References and §4.2] The related work cites the LLP PI baseline as [14] (arXiv preprint), while the experiments cite [15] (ICLR version). Please unify the citation to the published venue and avoid duplicate references to the same work.
- [Eq. (7)] The symbol H is used for the number of hierarchy levels but is not explicitly defined in the text; please add a short definition when the loss is introduced.
- [Figures 2 and 3] The architectural diagrams are dense and the text is very small; please enlarge the figures or split them so that the unrolled layers and masking operations are legible.
- [§4.1] The hierarchical structures are taken from [5] without describing their construction or giving any statistics about the number of coarse and medium categories. Since the hierarchy is central to the proposed method, please provide these details or a reference with a public hierarchy definition.
- [§5 and data availability] The paper states that code and datasets will be released, but no link or repository identifier is provided. Please include a public repository or specify the release mechanism in the final version.
Circularity Check
No significant circularity: the empirical claims are evaluated on held-out instance labels, and the only self-citations (FGoN hierarchy, PMG baseline) are external published results that do not encode the paper's output.
full rationale
The paper's derivation chain is self-contained with respect to its main claim. LLP training uses only bag-level proportion labels, and all reported accuracies are measured on held-out instance-level labels, so no prediction is fitted to the evaluation labels by construction. The proportion loss (Eq. 5) and hierarchical proportion loss (Eq. 7) are standard LLP losses applied at coarser granularities; the coarse/medium proportions are deterministic sums of the fine-grained bag proportions, which is a legitimate use of the given bag labels rather than a circular reduction. The unrolled sparse dictionary learning is adapted from the cited SC-MIL method and is a learnable network module, not a renamed version of the target result. The main self-citation is in Sec. 4.1: 'The hierarchical structures for FGVC datasets follow the work in [5].' This supplies the category hierarchy used by the masking and hierarchical loss, but [5] (FGoN) is a published, parameter-free taxonomy that does not contain or depend on the LHFGLP results, so it is not a load-bearing self-citation in the circularity sense. The ablation in Tab. 4 shows the hierarchical masks add only 0.37 points on Aircraft, which weakens the paper's emphasis on the hierarchical component, but that is a support/attribution issue, not circularity: the no-mask variant is still an LLP method whose gains come from the dictionary-learning module, and no fitted parameter is renamed as a prediction. Therefore the central claim does not reduce to its inputs by construction, and only a minor, non-load-bearing self-citation is present.
Assumptions & free parameters
free parameters (5)
- sparsity regularization lambda =
not reported
- stepsize mu =
not reported
- unrolled layers L =
not reported
- dictionary atoms per category n_atoms =
not reported
- bag size =
10
assumptions (4)
- standard math The unrolled ISTA update (Eq. 2-3) preserves the behavior of the sparse dictionary learning objective in Eq. 1.
- domain assumption A predefined coarse-to-medium-to-fine hierarchy is available and useful for each FGVC dataset.
- domain assumption Randomly packed bags of size 10 with exact proportions are a valid privacy-preserving training setup.
- domain assumption Features extracted by ResNet-50 can be sparsely represented by an overcomplete category dictionary.
Cite this review
Pith. "Pith review of Towards Privacy-Preserving Fine-Grained Visual Classification via Hierarchical Learning from Label Proportions." pith.science (2026). https://pith.science/paper/PLAAADVO
@misc{pith2026250523031,
author = {Pith},
title = {Pith review of: Towards Privacy-Preserving Fine-Grained Visual Classification via Hierarchical Learning from Label Proportions},
year = {2026},
howpublished = {\url{https://pith.science/paper/PLAAADVO}},
note = {Machine review of arXiv:2505.23031}
}
read the original abstract
In recent years, Fine-Grained Visual Classification (FGVC) has achieved impressive recognition accuracy, despite minimal inter-class variations. However, existing methods heavily rely on instance-level labels, making them impractical in privacy-sensitive scenarios such as medical image analysis. This paper aims to enable accurate fine-grained recognition without direct access to instance labels. To achieve this, we leverage the Learning from Label Proportions (LLP) paradigm, which requires only bag-level labels for efficient training. Unlike existing LLP-based methods, our framework explicitly exploits the hierarchical nature of fine-grained datasets, enabling progressive feature granularity refinement and improving classification accuracy. We propose Learning from Hierarchical Fine-Grained Label Proportions (LHFGLP), a framework that incorporates Unrolled Hierarchical Fine-Grained Sparse Dictionary Learning, transforming handcrafted iterative approximation into learnable network optimization. Additionally, our proposed Hierarchical Proportion Loss provides hierarchical supervision, further enhancing classification performance. Experiments on three widely-used fine-grained datasets, structured in a bag-based manner, demonstrate that our framework consistently outperforms existing LLP-based methods. We will release our code and datasets to foster further research in privacy-preserving fine-grained classification.
Figures
Reference graph
Works this paper leans on
-
[5]
Your” flamingo” is my” bird”: Fine-grained, or not
Dongliang Chang, Kaiyue Pang, Yixiao Zheng, Zhanyu Ma, Yi-Zhe Song, and Jun Guo. Your” flamingo” is my” bird”: Fine-grained, or not. In CVPR, 2021. 2, 6
work page 2021
-
[1]
Co-training for demographic classification using deep learning from la- bel proportions
Ehsan Mohammady Ardehaly and Aron Culotta. Co-training for demographic classification using deep learning from la- bel proportions. In ICDMW, 2017. 3, 5, 6, 7
work page 2017
-
[2]
Mixbag: Bag-level data augmentation for learning from label proportions
Takanori Asanomi, Shinnosuke Matsuo, Daiki Suehiro, and Ryoma Bise. Mixbag: Bag-level data augmentation for learning from label proportions. In ICCV, 2023. 2, 3, 6
work page 2023
-
[3]
A fast iterative shrinkage- thresholding algorithm for linear inverse problems
Amir Beck and Marc Teboulle. A fast iterative shrinkage- thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences, 2009. 4
work page 2009
-
[4]
Easy learning from label proportions
Robert Busa-Fekete, Heejin Choi, Travis Dick, Claudio Gen- tile, and Andres Munoz Medina. Easy learning from label proportions. In NIPS, 2023. 3
work page 2023
-
[6]
Fet-fgvc: Feature-enhanced transformer for fine-grained visual classification
Huazhen Chen, Haimiao Zhang, Chang Liu, Jianpeng An, Zhongke Gao, and Jun Qiu. Fet-fgvc: Feature-enhanced transformer for fine-grained visual classification. Pattern Recognition, 2024. 1, 2
work page 2024
-
[7]
An iterative thresholding algorithm for linear inverse prob- lems with a sparsity constraint
Ingrid Daubechies, Michel Defrise, and Christine De Mol. An iterative thresholding algorithm for linear inverse prob- lems with a sparsity constraint. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 2004. 4
work page 2004
-
[8]
An image is worth 16x16 words: Trans- formers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. ICLR, 2020. 6
work page 2020
Show all 33 references
-
[9]
Fine-grained visual classification via progressive multi-granularity train- ing of jigsaw patches
Ruoyi Du, Dongliang Chang, Ayan Kumar Bhunia, Jiyang Xie, Zhanyu Ma, Yi-Zhe Song, and Jun Guo. Fine-grained visual classification via progressive multi-granularity train- ing of jigsaw patches. In ECCV, 2020. 1, 2, 6
2020
-
[10]
Fair comparison: Quantifying variance in re- sults for fine-grained visual categorization
Matthew Gwilliam, Adam Teuscher, Connor Anderson, and Ryan Farrell. Fair comparison: Quantifying variance in re- sults for fine-grained visual categorization. In WACV, 2021
2021
-
[11]
Transfg: A trans- former architecture for fine-grained recognition
Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, and Changhu Wang. Transfg: A trans- former architecture for fine-grained recognition. In AAAI,
-
[12]
Granularity-aware distillation and structure modeling region proposal network for fine-grained image classifica- tion
Xiao Ke, Yuhang Cai, Baitao Chen, Hao Liu, and Wenzhong Guo. Granularity-aware distillation and structure modeling region proposal network for fine-grained image classifica- tion. Pattern Recognition, 2023. 1, 3
2023
-
[13]
3d object representations for fine-grained categorization
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCVW, 2013. 5
2013
-
[14]
Temporal ensembling for semi- supervised learning
Samuli Laine and Timo Aila. Temporal ensembling for semi- supervised learning. arXiv preprint arXiv:1610.02242, 2016. 3, 6
2016 arXiv
-
[15]
Temporal ensembling for semi- supervised learning
Samuli Laine and Timo Aila. Temporal ensembling for semi- supervised learning. In ICLR, 2017. 6
2017
-
[16]
Llp-gan: a gan-based algorithm for learning from label proportions
Jiabin Liu, Bo Wang, Hanyuan Hang, Huadong Wang, Zhi- quan Qi, Yingjie Tian, and Yong Shi. Llp-gan: a gan-based algorithm for learning from label proportions. IEEE trans- actions on neural networks and learning systems, 2022. 3
2022
-
[17]
Sgdr: Stochas- tic gradient descent with warm restarts
Ilya Loshchilov and Frank Hutter. Sgdr: Stochas- tic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. 6
2016 arXiv
-
[18]
Fine-grained visual classi- fication of aircraft
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classi- fication of aircraft. arXiv preprint arXiv:1306.5151 , 2013. 5
2013 arXiv
-
[19]
From softmax to sparsemax: A sparse model of attention and multi-label clas- sification
Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label clas- sification. In ICML, 2016. 5
2016
-
[20]
Ssfe-net: Self-supervised feature enhancement for ultra-fine-grained few-shot class incremental learning
Zicheng Pan, Xiaohan Yu, Miaohua Zhang, and Yongsheng Gao. Ssfe-net: Self-supervised feature enhancement for ultra-fine-grained few-shot class incremental learning. In WACV, 2023. 2
2023
-
[21]
Sc-mil: Sparsely coded multiple instance learning for whole slide image classification
Peijie Qiu, Pan Xiao, Wenhui Zhu, Yalin Wang, and Aris- teidis Sotiras. Sc-mil: Sparsely coded multiple instance learning for whole slide image classification. arXiv preprint arXiv:2311.00048, 2023. 4
2023 arXiv
-
[22]
Learn- ing transferable visual models from natural language super- vision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 6
2021
-
[23]
Fine-grained recognition: Multi-granularity labels and category similarity matrix
Xin Shu, Lei Zhang, Zizhou Wang, Lituan Wang, and Zhang Yi. Fine-grained recognition: Multi-granularity labels and category similarity matrix. Knowledge-Based Systems, 2023. 3, 6
2023
-
[24]
Interweaving insights: high-order feature interaction for fine-grained vi- sual recognition
Arindam Sikdar, Yonghuai Liu, Siddhardha Kedarisetty, Yi- tian Zhao, Amr Ahmed, and Ardhendu Behera. Interweaving insights: high-order feature interaction for fine-grained vi- sual recognition. International Journal of Computer Vision,
-
[25]
Learning from label proportions with consistency regularization
Kuen-Han Tsai and Hsuan-Tien Lin. Learning from label proportions with consistency regularization. InACML, 2020. 2, 3, 6
2020
-
[26]
Visualiz- ing data using t-sne
Laurens Van der Maaten and Geoffrey Hinton. Visualiz- ing data using t-sne. Journal of machine learning research,
-
[27]
The caltech-ucsd birds-200-2011 dataset
Catherine Wah, Steve Branson, Peter Welinder, Pietro Per- ona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011. 5
2011
-
[28]
Label hierarchy transition: Delving into class hierarchies to enhance deep classifiers
Renzhen Wang, Kaiwen Xiao, Xixi Jia, Xiao Han, Deyu Meng, et al. Label hierarchy transition: Delving into class hierarchies to enhance deep classifiers. arXiv preprint arXiv:2112.02353, 2021. 2, 6
2021 arXiv
-
[29]
Mask-cnn: Localizing parts and selecting descriptors for fine-grained bird species categorization
Xiu-Shen Wei, Chen-Wei Xie, Jianxin Wu, and Chunhua Shen. Mask-cnn: Localizing parts and selecting descriptors for fine-grained bird species categorization. Pattern Recog- nition, 2018. 2
2018
-
[30]
Context-semantic quality awareness network for fine-grained visual categorization
Qin Xu, Sitong Li, Jiahui Wang, Bo Jiang, and Jinhui Tang. Context-semantic quality awareness network for fine-grained visual categorization. arXiv preprint arXiv:2403.10298 ,
-
[31]
\proptosvm for learning with label proportions
Felix Yu, Dong Liu, Sanjiv Kumar, Jebara Tony, and Shih-Fu Chang. \proptosvm for learning with label proportions. In ICML, 2013. 3 9
2013
-
[32]
On learning from label propor- tions
Felix X Yu, Krzysztof Choromanski, Sanjiv Kumar, Tony Jebara, and Shih-Fu Chang. On learning from label propor- tions. arXiv preprint arXiv:1402.5902, 2014. 3
2014 arXiv
-
[33]
Learning from label proportions by learning with label noise
Jianxin Zhang, Yutong Wang, and Clay Scott. Learning from label proportions by learning with label noise. NIPS, 2022. 2, 3 10
2022
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.