Pith. sign in

REVIEW 4 major objections 4 minor 34 references

Learning from Label Proportions with Generative Adversarial Networks

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A GAN whose optimal discriminator directly solves learning from label proportions.

desk verdict A useful LLP-GAN extension with solid empirical gains, but the global-optimality theory applies to a Jensen lower-bound surrogate, not the actual loss, and the disjoint-bag assumption clashes with the random-bag experiments. read the letter →

arxiv 1909.02180 v4 pith:EGKKXN2Y submitted 2019-09-05 cs.LG stat.ML

Jiabin Liu , Bo Wang , Zhiquan Qi , Yingjie Tian , Yong Shi This is my paper · ORCID
classification cs.LGstat.ML
keywords learningfromlabelproportionsgenerativeadversarialnetworksweaklysupervisedinstance-levelclassificationproportioncross-entropymulti-classdeepfeaturematching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Learning from label proportions asks for an instance-level classifier when training data comes only in bags, with each bag labeled by the fraction of its items in each class and no individual labels. This paper proposes LLP-GAN, an adversarial network in which a generator produces fake examples and a discriminator both separates real from fake and predicts the $K$ real classes. The paper claims that this game has a well-defined equilibrium: for a fixed generator the optimal discriminator has a closed form, and at the global optimum the generator density equals the average of the bag densities. If true, the discriminator itself is the classifier and LLP becomes a standard end-to-end deep learning problem, avoiding the distributional assumptions and scalability limits of earlier shallow LLP methods.

What carries the argument

The load-bearing object is the $(K{+}1)$-way discriminator, which treats generated samples as a separate fake class and real samples as one of $K$ classes. The argument works by replacing the bag-level proportion cross-entropy with its Jensen lower bound (Eq. 6), which swaps the logarithm and the bag average; maximizing that lower bound is equivalent to minimizing the KL divergence between each bag's known proportion vector and the discriminator's mean posterior over that bag. This yields the closed-form optimal discriminator of Theorem 1 and, through Jensen-Shannon divergence, the global optimality condition of Theorem 2. In practice the generator is trained with feature matching rather than the pure adversarial loss, to avoid the known instability of a perfect generator.

What would settle it

On a small two- or three-class dataset with known bag proportions, train LLP-GAN and then compute both the lower-bound objective and the true bag-level proportion cross-entropy for the trained discriminator on held-out bags. If the discriminator does not approximately maximize the true cross-entropy while maximizing the surrogate, the global optimality theorem does not carry over to the training objective. A more direct check is to verify whether the trained discriminator's bag-averaged posteriors match the true proportions on held-out bags; systematic mismatch would contradict Lemma 1's claim that the optimal posterior equals the bag prior at equilibrium.

Watch

Extended reading notes

Core claim

The central claim is that label-proportion supervision can be injected into a GAN through a proportion cross-entropy term, and the resulting adversarial objective has a solvable optimum. The paper proves that maximizing a Jensen lower bound of that term makes the optimal discriminator take the explicit form $P_D^*(y=k|x) = (\sum_{i=1}^n p_i(k) p_i^d(x))/(\sum_{i=1}^n p_i^d(x)+p_g(x))$ for the $K$ real classes; the normalized posterior $\tilde{p}_{D^*}(y|x)$ is then a weighted average of the bag priors $p_i(y)$ with weights proportional to the bag densities at $x$. Theorem 2 states that the global minimum of the generator loss is reached if and only if $p_g = \frac{1}{n}\sum_i p_i^d$, recovering the usual GAN equilibrium in the one-bag case. Consequently the discriminator can be used directly as the instance-level classifier, and the final classification does not depend on the generator density, so the result also covers generators whose density is degenerate.

Load-bearing premise

The proof of global optimality applies to the Jensen lower-bound surrogate of the proportion cross-entropy, not to the true proportion loss actually minimized during training, and the paper never bounds the gap between the two.

Editorial extensions

If this is right

  • The discriminator's output after training is the instance-level classifier, so no separate classifier or label-recovery postprocessing is needed.
  • LLP-GAN scales to large datasets and large bag sizes where the DLLP baseline diverges; for example, on CIFAR-10 at bag size 128 the reported error is 27.39% versus DLLP's 82.89%.
  • The global optimum pins down the generator's target density as the average of the bag densities, giving a clear training target and recovering the standard GAN result when there is one bag.
  • Because the optimal classifier aggregates bag priors with density weights, the method does not require bags to be i.i.d. samples from a single distribution; each bag can have its own marginal density.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable consequence not explored in the paper: if the Jensen lower-bound gap is large, training on the surrogate could converge to a discriminator that is optimal for the surrogate but suboptimal for the true proportion cross-entropy; comparing the two objectives on a small dataset would quantify the gap.
  • The density-weighted aggregation formula implies that bags concentrating their mass at $x$ dominate the prediction there; for non-random bags built on sensitive features, this could introduce bias, a limitation the paper lists as future work.
  • The independence of the final classifier from $p_g$ suggests a two-stage design where the generator is only used to estimate the bag densities, after which the classifier could be solved as a weighted mixture estimation problem, potentially making the method robust to generator collapse.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes LLP-GAN, a GAN-based method for learning from label proportions (LLP). The discriminator is trained with an adversarial real/fake loss plus a bag-level proportion cross-entropy term, which the authors replace by a Jensen lower bound to make the objective amenable to SGD. The generator is trained with feature matching. The main theoretical claims are an explicit optimal discriminator (Theorem 1, Eq. 9) and a global optimality result for the generator (Theorem 2) under a non-parametric, disjoint-bag idealization. Experiments on MNIST, SVHN, CIFAR-10, and CIFAR-100 report substantially lower test error than DLLP, especially for large bag sizes, together with comparisons to InvCal and alter-SVM in a binary subset.

Significance. If the theoretical claims were correct, the paper would be a useful contribution: it gives an end-to-end deep LLP algorithm with a direct instance-level classifier obtained from the discriminator, avoids parametric distributional assumptions, and shows consistent empirical gains over DLLP at large bag sizes. The authors also ship code and report results across four datasets and four bag sizes, including random-bag stability checks. However, the central theoretical results are not sound: the optimality theorems concern an unquantified surrogate objective, and the closed-form discriminator in Theorem 1 is not even a stationary point of that surrogate for the value of lambda used in the analysis. The empirical results may still indicate a useful heuristic, but the paper's stated theoretical foundation does not support its main claims.

major comments (4)
  1. [§3.1, Remark 2 and Eq. (6)] Remark 2 replaces the actual bag-proportion cross-entropy -CEL(p, p_bar) by its Jensen lower bound, and all subsequent theorems are stated for the surrogate objective ~V, not for the LLP objective V in Eq. (4). Since Algorithm 1 trains the discriminator on ~V, and no bound on the Jensen gap is provided, the global optimality proven in Theorems 1 and 2 does not transfer to the loss that motivates the method. The paper needs either a quantitative gap bound or an explicit argument that the surrogate and the true objective share the same optima; neither is supplied.
  2. [§3.2, Theorem 1 and Eq. (9)] The proposed optimal discriminator is not a maximizer of ~V. For fixed G, writing S = sum_k P_D(k|x), A_k = sum_i p_i(k) p_i^d(x), and p_d = sum_i p_i^d(x), the integrand in Eq. (10) is F = p_d log S + sum_k A_k log P_k + (p_g - p_d) log(1-S). At the claimed values P_k = A_k/(p_d+p_g) and S = p_d/(p_d+p_g), the derivative with respect to P_k evaluates to (p_d+p_g)^2 / p_g, not zero. The correct stationary condition gives S = 2p_d/(p_d+p_g) when lambda = 1, which is not a valid probability whenever p_d > p_g; in that regime ~V is unbounded above because (p_g - p_d) log(1-S) tends to +infinity as S tends to 1. Thus Eq. (9) and the results built on it are not valid as stated.
  3. [§3.2, Definition 1 and Lemma 1 vs. §4.1] The theoretical analysis assumes a fixed partition of the data space into disjoint sections, with each bag distribution p_i^d supported on its own section. The experiments instead form bags by randomly partitioning the finite training set, so the bag distributions all have the same (global) support. Under the surrogate objective, when the bag supports coincide, Eq. (11) gives a normalized posterior approximately equal to (1/n) sum_i p_i(k), i.e., the global class proportions independent of x. This contradicts the instance-level accuracy reported in Table 1 and shows that the experimental regime is not the regime covered by the optimality theorems.
  4. [§3.3, Eq. (13) and Eq. (14)] The derivation of C(G) is also internally inconsistent. In Eq. (13) the JSD term is written as JSD(p_d || p_g), but p_d = sum_i p_i^d is unnormalized (integral n) while p_g is normalized, so the standard Jensen-Shannon divergence is not defined for this pair. Evaluating Eq. (13) at the claimed optimum p_g = p_d/n gives an additional (n-1) log 2 term relative to the constant n log n - (n+1) log(n+1) stated in Eq. (14). The generator optimality calculation needs to be redone with normalized densities.
minor comments (4)
  1. [§3.1, Eq. (6)] The inequality in Eq. (6) is written as '>' but Jensen's inequality gives '>='; the strict sign is not justified in general.
  2. [Appendix §6] The sentence 'Following a standard setting in the previous work [? 28]' contains an unresolved placeholder that must be fixed.
  3. [§4 and Appendix §7.1] The naming of the SVM baselines is inconsistent: 'alter-SVM' appears in the abstract and Figure 5, 'alter-pSVM' in Table 4, and 'alter-proportional-SVM' in the text; the notation should be unified.
  4. [Appendix §7.4, Table 5] The table caption and column headers for the random bag-generation experiment are unclear: the number of random repetitions is mixed with the number of errors and the reported accuracy, which makes the stability claim hard to evaluate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical claims are derived from the paper's own explicitly stated surrogate objective, and the self-citation is background only.

full rationale

The paper's central results (Lemma 1, Theorem 1, Theorem 2) are derived from the explicit approximate objective ~V defined in Remark 2, not assumed or imported from a fitting step. Theorem 1's optimal discriminator (Eq. 9) is obtained by pointwise optimization of the integrand in Eq. 10, so it is a mathematical consequence of the stated objective rather than a parameter fitted to a subset and renamed as a prediction. The final classifier (Eq. 11) is a direct algebraic consequence of Eq. 9; its dependence on the input bag proportions is the intended goal of LLP, not a circular reduction. Theorem 2 follows the standard GAN global-optimality argument with the average bag density 1/n * sum_i p_i^d as the real distribution. The only self-citation ([22]) appears in related-work background on privacy constraints and SVM-based LLP methods and is not load-bearing for the paper's claims. The replacement of the true bag-level cross-entropy by its Jensen lower bound is explicitly disclosed in Remark 2 as an approximation; this may be a correctness limitation (no gap bound is provided), but it is not circularity because the theorems are stated for the surrogate objective. Empirical results are measured against external baselines and fully supervised CNNs on held-out test labels, so the performance claims are not forced by construction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two domain assumptions (infinite capacity, bag-specific distributions), one ad hoc approximation (the Jensen lower bound), and standard convexity arguments. No new physical or mathematical entities are invented. The free parameter λ is tuned in experiments and the theory fixes it to 1, so the theoretical guarantees do not cover the tuned regime.

free parameters (1)
  • λ (loss weight) = tuned per dataset; λ>1 preferred, fixed λ=1 in theory
    Balances the supervised proportion cross-entropy against the unsupervised adversarial loss; Section 4.4 shows CIFAR-10 performance is sensitive to λ.
assumptions (4)
  • domain assumption Discriminator and generator have infinite capacity (non-parametric setting)
    Assumed before Remark 2 to justify the pointwise optimal discriminator derivation; not satisfied by finite networks in practice.
  • domain assumption Bags are disjoint and each bag's data is sampled from a bag-specific marginal distribution p_i^d
    Definition 1; the optimal discriminator in Eq. (9) is expressed in terms of these p_i^d, which must be learnable or representable.
  • ad hoc to paper The Jensen lower bound in Eq. (6) is a faithful surrogate for the cross-entropy objective
    Remark 2 substitutes the lower bound for computational convenience; all subsequent optimality results apply to this approximate objective.
  • standard math Standard GAN non-convergence and convexity arguments from Goodfellow et al. and Boyd & Vandenberghe
    Invoked in Section 3.4 to assert convergence of alternating gradient descent; relies on convexity of max_D V(G,D) in G.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning from Label Proportions with Generative Adversarial Networks." pith.science (2026). https://pith.science/paper/EGKKXN2Y

@misc{pith2026190902180,
  author       = {Pith},
  title        = {Pith review of: Learning from Label Proportions with Generative Adversarial Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EGKKXN2Y}},
  note         = {Machine review of arXiv:1909.02180}
}
read the original abstract

In this paper, we leverage generative adversarial networks (GANs) to derive an effective algorithm LLP-GAN for learning from label proportions (LLP), where only the bag-level proportional information in labels is available. Endowed with end-to-end structure, LLP-GAN performs approximation in the light of an adversarial learning mechanism, without imposing restricted assumptions on distribution. Accordingly, we can directly induce the final instance-level classifier upon the discriminator. Under mild assumptions, we give the explicit generative representation and prove the global optimality for LLP-GAN. Additionally, compared with existing methods, our work empowers LLP solver with capable scalability inheriting from deep models. Several experiments on benchmark datasets demonstrate vivid advantages of the proposed approach.

Figures

Figures reproduced from arXiv: 1909.02180 by the authors.

Figure 1
Figure 1. An illustration of multi-class learning from label proportions. In detail, the data belongs to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An illustration of our LLP-GAN framework. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The convergence curves on CIFAR-10 w/ different bag sizes. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Generated samples on CIFAR-10. FM in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The average error rates w/ different bag sizes. Secondly, DLLP and LLP-GAN are carried out on four bench￾mark datasets with different bag sizes in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Analysis on hyperparameter and complexity. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Multi-class test error rates (%) on benchmark datasets with different bag sizes. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Sum of instance-level entropy on MNIST. 7.4 The Randomness of Bag Assignment The distribution of proportions has an huge impact on LLP algorithm performance. Hence, fixing bag size, we randomly construct bags for multiple times and present the accuracy performance in …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 23 canonical work pages

  1. [10]

    Deep multi-class learning from label proportions

    Gabriel Dulac-Arnold, Neil Zeghidour, Marco Cuturi, Lucas Beyer, and Jean-Philippe Vert. Deep multi-class learning from label proportions. arXiv preprint arXiv:1905.12909, 2019

    arXiv 1905

  2. [1]

    Ardehaly and Aron Culotta

    Ehsan M. Ardehaly and Aron Culotta. Co-training for demographic classification using deep learning from label proportions. In International Conference on Data Mining Workshops, pages 1017–1024. IEEE, 2017

    work page 2017

  3. [2]

    Towards principled methods for training generative adver- sarial networks

    Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adver- sarial networks. In International Conference on Learning Representations, 2016

    work page 2016

  4. [3]

    Wasserstein generative adversarial networks

    Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pages 214–223, 2017

    2017

  5. [4]

    Relational inductive biases, deep learning, and graph networks

    Peter W Battaglia, Jessica B Hamrick, Victor Bapst, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018

    arXiv 2018

  6. [5]

    Representation learning: A review and new perspectives

    Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798– 1828, 2013

    2013

  7. [6]

    Pattern Recognition and Machine Learning

    Christopher Bishop. Pattern Recognition and Machine Learning. Springer, January 2006

    work page 2006

  8. [7]

    Convex optimization

    Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004

    2004

Show all 34 references
  1. [8]

    Good semi-supervised learning that requires a bad gan

    Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Ruslan R Salakhutdinov. Good semi-supervised learning that requires a bad gan. In Advances in neural information processing systems, pages 6510–6520, 2017

    2017

  2. [9]

    Dietterich, Richard H

    Thomas G. Dietterich, Richard H. Lathrop, and Tomás Lozano-Pérez. Solving the multiple instance problem with axis-parallel rectangles. Artificial Intelligence, 89(1-2):31–71, 1997

    1997

  3. [11]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, et al. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672–2680, 2014

    2014

  4. [12]

    Semi-supervised learning by entropy minimization

    Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in neural information processing systems, pages 529–536, 2005

    2005

  5. [13]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, et al. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition, pages 770–778, 2016

    2016

  6. [14]

    Deep neural networks for acoustic modeling in speech recognition

    Geoffrey Hinton, Li Deng, Dong Yu, et al. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6):82–97, 2012

    2012

  7. [15]

    Perceptual losses for real-time style transfer and super-resolution

    Justin Johnson, Alexandre Alahi, and Fei-Fei Li. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, pages 694–711. Springer, 2016

    2016

  8. [16]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 13

    2013 arXiv

  9. [17]

    Semi-supervised learning with deep generative models

    Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pages 3581–3589, 2014

    2014

  10. [18]

    Network in network.arXiv preprint arXiv:1312.4400, 2013

    Min Lin, Qiang Chen, and Shuicheng Yan. Network in network.arXiv preprint arXiv:1312.4400, 2013

    2013 arXiv

  11. [19]

    A framework for multiple-instance learning

    Oded Maron and Tomás Lozano-Pérez. A framework for multiple-instance learning. In Advances in Neural Information Processing Systems, pages 570–576, 1998

    1998

  12. [20]

    The expectation-maximization algorithm

    Todd K Moon. The expectation-maximization algorithm. IEEE Signal processing magazine, 13(6):47–60, 1996

    1996

  13. [21]

    (Almost) no label no cry

    Giorgio Patrini, Richard Nock, Paul Rivera, and Tiberio Caetano. (Almost) no label no cry. In Advances in Neural Information Processing Systems, pages 190–198, 2014

    2014

  14. [22]

    Learning with label proportions via NPSVM

    Zhiquan Qi, Bo Wang, Fan Meng, et al. Learning with label proportions via NPSVM. IEEE Transactions on Cybernetics, 47(10):3293–3305, 2017

    2017

  15. [23]

    Smola, Tiberio S

    Novi Quadrianto, Alex J. Smola, Tiberio S. Caetano, et al. Estimating labels from label proportions. Journal of Machine Learning Research, 10(Oct):2349–2374, 2009

    2009

  16. [24]

    Unsupervised representation learning with deep convolutional generative adversarial networks

    Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015

    2015 arXiv

  17. [25]

    Semi- supervised learning with ladder networks

    Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. InAdvances in neural information processing systems, pages 3546–3554, 2015

    2015

  18. [26]

    You only look once: Unified, real-time object detection

    Joseph Redmon, Santosh Divvala, Ross Girshick, et al. You only look once: Unified, real-time object detection. In Computer Vision and Pattern Recognition, pages 779–788, 2016

    2016

  19. [27]

    SVM classifier estimation from group probabilities

    Stefan Rueping. SVM classifier estimation from group probabilities. InInternational Conference on Machine Learning, pages 911–918, 2010

    2010

  20. [28]

    Improved techniques for training GANs

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, et al. Improved techniques for training GANs. In Advances in Neural Information Processing Systems, pages 2234–2242, 2016

    2016

  21. [29]

    Springenberg

    Jost T. Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv preprint arXiv:1511.06390, 2015

    2015 arXiv

  22. [30]

    Rethinking the inception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, et al. Rethinking the inception architecture for computer vision. In Computer Vision and Pattern Recognition, pages 2818–2826, 2016

    2016

  23. [31]

    Multi-class learning from class proportions

    Zilei Wang and Jiashi Feng. Multi-class learning from class proportions. Neurocomputing, 119(16):273–280, 2013

    2013

  24. [32]

    Adversarial perturbations of deep neural networks

    David Warde-Farley and Ian Goodfellow. Adversarial perturbations of deep neural networks. In Perturbations, Optimization, and Statistics, page 311. MIT Press, 2016

    2016

  25. [33]

    Yu, Liangliang Cao, Michele Merler, et al

    Felix X. Yu, Liangliang Cao, Michele Merler, et al. Modeling attributes from category-attribute proportions. In International Conference on Multimedia, pages 977–980. ACM, 2014

    2014

  26. [34]

    Yu, Dong Liu, Sanjiv Kumar, et al.∝-SVM for learning with label proportions

    Felix X. Yu, Dong Liu, Sanjiv Kumar, et al.∝-SVM for learning with label proportions. In International Conference on Machine Learning, pages 504–512, 2013. 14

    2013

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.