Structured IB: Improving Information Bottleneck with Structured Feature Learning

Dingzhu Wen; Hanzhe Yang; Yong Zhou; Youlong Wu; Yuanming Shi

REVIEW 4 major objections 6 minor 30 references

Structured IB: Improving Information Bottleneck with Structured Feature Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding small auxiliary encoders to the IB Lagrangian yields higher accuracy and more retained label information at equal or lower compression.

desk verdict A plausible empirical trick for IB Lagrangian training, but Theorem 1 is broken and should not be cited as theory. read the letter →

arxiv 2412.08222 v2 pith:ILJPN67I submitted 2024-12-11 cs.IT cs.LGmath.IT

Hanzhe Yang , Youlong Wu , Dingzhu Wen , Yong Zhou , Yuanming Shi This is my paper · ORCID

classification cs.ITcs.LGmath.IT MSC 94A1768T07

keywords informationbottleneckstructuredfeaturesauxiliaryencodersmutualrepresentationlearningIBLagrangianvariationalfeaturecompression

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the information bottleneck trade-off is not a fixed wall: a representation learned by a single IB-Lagrangian encoder leaves task-relevant information behind, and that information can be recovered by training a few small auxiliary encoders alongside the main one. The proposed Structured IB pipeline trains the main encoder with the usual IB Lagrangian, adds auxiliary encoders that each minimize the same Lagrangian plus a penalty on mutual information with previously collected features, and then tunes scalar weights on the summed features. On MNIST and CIFAR-10, the structured versions of VIB, square-VIB, and NIB report higher test accuracy and higher $I(Z,Y)$ at equal or lower $I(X,Z)$, even when per-encoder hidden widths are halved. The theoretical anchor is a Gaussian theorem asserting that adding an independent feature $Z'$ cannot decrease label information, provided the combined decoder output determines the label and the auxiliary covariance is not too small.

What carries the argument

The mechanism is additive structured features: the final representation is $\hat{Z}=w_0Z+\sum_{i=1}^K w_i Z_i$, where $Z$ comes from the main encoder and each $Z_i$ from an auxiliary encoder trained sequentially. Auxiliary training minimizes the IB Lagrangian plus a density-ratio estimate of $I(Z_i, Z+\sum_{j<i}Z_j)$, using a discriminator to encourage independence from the existing features. Theorem 1 is the load-bearing identity: for independent Gaussian $Z,Z'$, $H(Z'+Z,Z)-H(Z)=\frac{D}{2}\log(2\pi e)+\frac{1}{2}\log\det(\Sigma')$, so the added feature enlarges the entropy of the joint representation when $\det(\Sigma')\ge(2\pi e)^{-D}$; combined with the perfect-prediction condition on the decoder output, this yields $I(Z'+Z,Y)\ge I(Z,Y)$. The paper invokes variational encoders to justify Gaussianity and a fixed one-layer linear decoder to justify the one-hot condition.

What would settle it

Compute $H(Y \mid h(W(Z+Z')))$ on the test set for a trained Structured IB model. If it is far from zero while $I(Z'+Z,Y)\ge I(Z,Y)$ is still observed, the theorem's stated premise is not what drives the gain. Separately, check the proof's step (16): for independent Gaussian $Z,Z'$, $I(Z,Z+Z')$ should equal $H(Z+Z')-H(Z'\mid Z)$; if it is not $H(Z)$, the displayed derivation needs repair.

Watch

Extended reading notes

Core claim

The central claim is that the representation itself can be structured instead of monolithic. Theorem 1 states that for independent Gaussian features $Z,Z'$ with a full-rank linear decoder and one-hot outputs, $I(Z'+Z, Y) \ge I(Z,Y)$ holds when $I(h(Y'),Y)=H(Y)$ and $\det(\Sigma') \ge (2\pi e)^{-D}$. The paper reports IB-plane comparisons where structured variants sit above the corresponding single-encoder curves, with reduced parameter counts, and encoder-dropout experiments show accuracy and $I(Z,Y)$ fall as auxiliary branches are removed while $I(X,Z)$ rises.

Load-bearing premise

The load-bearing premise is condition (13), that the one-hot output of the fixed linear decoder on the combined feature determines the label $Y$; without that perfect-prediction state, the theorem does not say the auxiliary feature helps.

Editorial extensions

If this is right

Single-encoder IB curves are not an upper limit for a fixed architecture budget; the same Lagrangian can be wrapped with auxiliary branches to move up in the IB plane.
Because encoder widths can be halved without losing to the full-width baseline, structured training is a parameter-efficiency strategy, not just an accuracy strategy.
The encoder-dropout trend implies auxiliary branches contribute to compression: removing branches raises $I(X,Z)$.
Any IB-style objective satisfying the theorem's two conditions can in principle be run in structured mode; the paper demonstrates this for VIB, square-VIB, NIB, MINE, and KNIFE.
The main encoder naturally receives the largest combination weight without explicit constraints, suggesting the auxiliary branches act as targeted corrections rather than replacements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension is to replace the one-hot classification condition with a regression counterpart, but the entropy comparison would then need a continuous distortion measure; the paper does not make this move.
The paper does not explain why $I(X,Z)$ tends to fall as $K$ grows; if that pattern is real, auxiliary branches may act as a compression regularizer, and the mechanism could be studied by tracking per-encoder gradients.
Condition (13) alone may already imply the inequality through data processing, which would make the Gaussian entropy calculation an illustration of the core requirement rather than the essential driver.
The natural next experiment is to vary auxiliary encoder capacity and measure the marginal gain per branch, which would locate the diminishing-returns point the paper observes.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this for the empirical idea, not the theory. The SIB architecture—main encoder trained with the IB Lagrangian, auxiliary encoders trained sequentially to capture complementary information while being pushed toward independence, a fixed decoder, and a learned weighted sum—is a reasonable engineering contribution. The experiments on MNIST and CIFAR-10 show small but consistent gains in accuracy and I(Z,Y) over VIB/sqVIB/NIB, sometimes with fewer parameters, and the code is linked. The encoder dropout results are a nice sanity check that the auxiliary encoders matter.

The theory, though, does not support the weight. Theorem 1's assumption (13), I(h(Y'),Y)=H(Y), already forces the combined feature to be a perfect predictor of Y, so I(Z+Z',Y) ≥ I(Z,Y) follows immediately from the data processing inequality; the Gaussian entropy machinery is unnecessary. Condition (14) is irrelevant once (13) holds. Worse, the proof of (16) is algebraically wrong: for independent Z,Z', I(Z, Z+Z') = H(Z+Z') - H(Z'), not H(Z). With the corrected identity, the difference term becomes H(Z') for an invertible decoder, which again makes (14) decoration. The paper also never shows that the trained decoder actually satisfies (13); the reported test accuracies leave a small error floor. And the compression side, I(X,Z_hat), is explicitly deferred, so the central claim 'more I(Z,Y) at the same or lower I(X,Z)' has no theoretical backing.

Are the experiments enough? They are suggestive, not conclusive. No error bars or multiple seeds, the gains are a few tenths of a percent in accuracy and a few hundredths of a bit in MI, and the baselines are not always matched in architecture scale per encoder (they divide hidden dims by two, which is fair but should be stated more carefully). Still, the pattern is consistent across two datasets and three IB variants, and the encoder dropout experiment reinforces the mechanism.

My take: treat this as an empirical paper with a flawed theoretical wrapper. A serious referee should look at it because the idea is real and the comparison is useful, but the theorem should be either fixed or removed and the empirical claims need error bars. If the authors reframe as 'a heuristic that works in practice,' it could be a decent workshop paper; with a corrected theory or a larger empirical study, it could be a solid conference paper. Don't let the current Theorem 1 gatekeep the contribution—but don't cite it as proof either.

Referee Report

4 major / 6 minor

Summary. The paper proposes Structured IB (SIB), an extension of IB Lagrangian methods in which a main encoder is trained with the standard IB objective and K auxiliary encoders are trained sequentially to maximize I(Z_i,Y) - beta I(X,Z_i) while penalizing dependence on previous features. Features are combined by weighted summation and the weights are tuned with the IB Lagrangian. The authors claim that SIB achieves higher accuracy and I(Z,Y) at the same or lower I(X,Z) than VIB, sqVIB, and NIB, even with fewer parameters, and they provide Theorem 1 as a theoretical justification for weighted summation under Gaussianity and an independence/determinant condition. Experiments on MNIST and CIFAR-10 compare SVIB, sqSVIB, SNIB, and structured versions of MINE/KNIFE against their base algorithms. The paper includes code and an appendix with implementation details.

Significance. If the reported empirical gains are robust, SIB could be a simple drop-in enhancement for IB-Lagrangian representation learning; the code release and the breadth of comparisons (VIB, sqVIB, NIB, MINE, KNIFE) are strengths. However, the theoretical support as written does not hold: the displayed proof of Theorem 1 contains an algebraic error, and the main assumption (13) already makes the conclusion a direct consequence of the data processing inequality, so the Gaussian calculation is superfluous. The experiments also lack repeated runs or error bars, which matters because the reported I(Z,Y) gains are small. The paper's contribution would be substantially weakened if the theorem is reduced to a trivial DPI observation, but the empirical direction remains worth evaluating in a revision.

major comments (4)

[Appendix, Proof of Theorem 1, Eqs. (15)-(16)] The displayed equality (16) is algebraically incorrect. From (15) and the identity I(Z;Z+Z')=H(Z)+H(Z+Z')-H(Z,Z+Z'), the correct expression is H(Z+Z',Z)-H(Z), not H(Z+Z')-H(Z). As printed, the proof replaces I(Z;Z+Z') with H(Z), which is false in general. The subsequent block-determinant calculation actually evaluates H(Z+Z',Z)-H(Z), so the proof can be repaired locally, but the theorem statement and the role of condition (14) need to be revisited.
[Theorem 1, condition (13)] Condition (13) alone already implies I(Z+Z',Y) >= I(Z,Y), because Y is a deterministic function of h(Y'), hence of Y'=W(Z+Z'), hence of Z+Z'; the data processing inequality applied to the chain Z -> Z+Z' -> Y gives the result. Gaussianity, independence of Z and Z', and condition (14) play no role. Moreover, the paper does not verify (13): the claim that achieving global optimality ensures I(h(Y'),Y)=H(Y) is unsupported, and the reported test accuracies (about 98-99% on MNIST) leave nonzero error, so the theorem does not apply to the evaluated models.
[Justification of f(·) and compression claims] The abstract and introduction claim that SIB achieves the same or lower compression I(X,Z), but the manuscript explicitly defers the theoretical analysis of I(X,\hat Z) with the sentence 'More effort on the theoretical analysis of I(X, ˆZ) will be put on.' Figures 3 and 4 show only empirical estimates without error bars, so the claim of a favorable IB trade-off is not established beyond the specific runs shown.
[Experiments, Figures 3 and 4] The main empirical claims rest on single-run comparisons. The reported gains in I(Z,Y) are small (e.g., roughly 2.16 to 2.24 nats on MNIST), and the I(X,Z) differences are also small; without multiple seeds, confidence intervals, or significance tests, it is not possible to tell whether the structured variants genuinely dominate the baselines on the IB plane.

minor comments (6)

[Title and headings] The heading 'Relative Work' should be 'Related Work', and the appendix title contains the typo 'Supplimentary'.
[Equation (9)] Equation (9) has a malformed expression 'Z+i−1Xj=1Zj' caused by missing braces or delimiters in the LaTeX; please fix the rendering.
[Algorithm 1, lines 10 and 12] The discriminator objective is written with opposite signs from the one in Eq. (10); clarify the convention for d's output so the minimax game matches the stated KL objective.
[Figure 3] The caption's description of left and right figures is ambiguous because the figure contains eight panels; please label the panels explicitly as MNIST and CIFAR-10.
[References] Belghazi et al. 2018a and 2018b are the same paper and should be merged or clearly distinguished; the MINE citation is duplicated.
[Justification of f(·), NIB covariance] The sentence stating that the covariance matrix Σ can be treated as a hyperparameter when using NIB is unclear; specify how the covariance is set in the experiments.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 1's condition (13) already forces I(Z'+Z,Y)=H(Y), making the claimed inequality a corollary of the data processing inequality; the Gaussian/determinant machinery is superfluous and the proof's Eq. (16) is algebraically wrong.

self definitional [Justification of f(·), Theorem 1, Eqs. (12)–(14)]
"Then, given data X and its target Y , we have I(Z ′ + Z, Y) ≥ I(Z, Y), (12) when the following conditions are satisfied: I(h(Y ′), Y) = H(Y ), (13) det(Σ′) ≥ 1/(2πe)^D . (14)"

Under (13), h(Y')—the one-hot code of the decoder output on the combined feature—has full mutual information with Y, so h(Y') determines Y. Since Y'=d(Z+Z') is deterministic in Z+Z', the Markov chain Z+Z'→Y'→h(Y')→Y gives, by DPI, I(Z+Z',Y) ≥ I(h(Y'),Y)=H(Y); the upper bound H(Y) forces equality, so I(Z+Z',Y)=H(Y) ≥ I(Z,Y). The conclusion (12) is therefore a corollary of (13) alone; Gaussianity, independence and condition (14) are not used. The theorem's premise already contains the claimed improvement, so SIB's theoretical support reduces to assuming the combined feature is a perfect predictor.

full rationale

The central claim that SIB achieves higher task-relevant information I(Z,Y) at similar compression rests on Theorem 1. The theorem's load-bearing condition (13), I(h(Y'),Y)=H(Y), directly states that the aggregated feature's decoder output determines the label; the desired inequality (12) follows immediately from the data processing inequality, with no need for the Gaussian model, the independence assumption, or condition (14). The paper justifies (13) only by asserting that global optimality of training ensures it, citing an external theorem, and gives no argument that the trained networks (with test accuracies near 0.97–0.99) satisfy exact zero-error prediction. Independently of circularity, the proof of Theorem 1 contains an algebraic error: Eq. (16) replaces I(Z,Z+Z') with H(Z), but substituting X=Z, Y=Z+Z' into the cited identity yields H(Z)+H(Z+Z')−H(Z,Z+Z'), and for independent Z,Z' this equals H(Z+Z')−H(Z'), not H(Z); the subsequent determinant computation actually computes H(Z+Z',Z)−H(Z)=H(Z'). The compression side of the main claim, I(X,Ẑ)≤I(X,Z), is explicitly deferred ('More effort on the theoretical analysis of I(X, ˆZ) will be put on') and supported only empirically. Because the theoretical derivation founders and the principal prediction is essentially assumed, the paper is partially circular; the experiments remain independent empirical evidence, so the score is 6 rather than higher.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on Theorem 1, whose key conditions (13) and (14) are assumptions about the learned features that the training procedure does not guarantee. The empirical results depend on hand-set hyperparameters β and K and on a hidden-dimension scaling choice. No new physical entities are introduced, but the auxiliary encoders are new architectural components whose benefit is not isolated from ensemble effects.

free parameters (3)

β (Lagrange multiplier) = 1 for SVIB and sqVIB, 0.01 for SNIB, 1e-7 for MINE/KNIFE.
Chosen per algorithm and used throughout; the IB trade-off curves in Figure 4 are generated by sweeping β, so reported operating points depend on this hand-set parameter.
K (number of auxiliary encoders) = 2 to 6 in main experiments (1 to 3 for MINE/KNIFE).
Architecture hyperparameter; the paper notes performance plateaus for large K, so the choice affects the magnitude of reported gains.
Encoder hidden-dimension scaling = 0.5 (each encoder's hidden dimension divided by 2).
Design choice to keep total parameter counts comparable; underpins the 'reduced network size' claim but also changes the comparison baseline.

assumptions (5)

domain assumption The features Z and Z' are independent D-dimensional Gaussian random vectors with diagonal positive definite covariance matrices.
Stated in Theorem 1; the paper relies on variational encoders for Gaussianity and on adversarial training for independence, but neither guarantees the assumption. The proof of Theorem 1 uses Gaussian entropy formulas.
ad hoc to paper Y' = W(Z+Z') is a bijective deterministic function and Y is a bijective deterministic function of Y', expressed as I(h(Y'),Y)=H(Y).
Condition (13) in Theorem 1. It implies the combined feature perfectly determines Y, making the conclusion a data-processing consequence; the paper does not prove the training procedure achieves it.
ad hoc to paper The auxiliary covariance satisfies det(Σ') ≥ 1/(2πe)^D.
Condition (14) in Theorem 1, used to ensure the entropy gain is nonnegative. The paper claims it 'typically holds' without evidence.
ad hoc to paper Achieving global optimality during training ensures I(h(Y'),Y)=H(Y).
Stated in the justification of f(·), citing Pan et al. (2021); no proof is given that the proposed staged training reaches global optimality.
standard math Standard Gaussian differential entropy and block-determinant formulas.
Used in the proof of Theorem 1; these are unproblematic background results.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Structured IB: Improving Information Bottleneck with Structured Feature Learning." pith.science (2026). https://pith.science/paper/ILJPN67I

@misc{pith2026241208222,
  author       = {Pith},
  title        = {Pith review of: Structured IB: Improving Information Bottleneck with Structured Feature Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ILJPN67I}},
  note         = {Machine review of arXiv:2412.08222}
}

read the original abstract

The Information Bottleneck (IB) principle has emerged as a promising approach for enhancing the generalization, robustness, and interpretability of deep neural networks, demonstrating efficacy across image segmentation, document clustering, and semantic communication. Among IB implementations, the IB Lagrangian method, employing Lagrangian multipliers, is widely adopted. While numerous methods for the optimizations of IB Lagrangian based on variational bounds and neural estimators are feasible, their performance is highly dependent on the quality of their design, which is inherently prone to errors. To address this limitation, we introduce Structured IB, a framework for investigating potential structured features. By incorporating auxiliary encoders to extract missing informative features, we generate more informative representations. Our experiments demonstrate superior prediction accuracy and task-relevant information preservation compared to the original IB Lagrangian method, even with reduced network size.

Figures

Figures reproduced from arXiv: 2412.08222 by the authors.

**Figure 2.** Illustration of the feature space. Left: The entire feature space (large circle) containing the feature subspace (ellipse) generated by a single feature vector using the IB Lagrangian method. Middle: The feature subspace (small circle) spanned by two vectors from an untrained SIB within the overall feature space (large circle). Due to substantial overlap, this subspace is limited in its coverage. Right: The featur… view at source ↗

**Figure 3.** illustrates the accuracy, I(Z, Y ), I(X, Z), and model parameter count in relation to the number of encoders. The Lagrange multiplier, β, is set to 1 for SVIB and sqVIB, and 0.01 for SNIB. Importantly, the algorithms reduce to the standard IB Lagrangian when K = 1. MI I(X, Z) is estimated using Monte Carlo sampling (Goldfeld et al. 2018), while I(Z, Y ) = H(Y ) − H(Y |Z) is derived from the conditional entropy H(Y … view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Behavior of Algorithms on the IB Plane. The origi [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: The performance after encoder dropout. The upper and lower rows correspond to the MNIST and CIFAR10 datasets, [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 15 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

A.; Fischer, I.; Dillon, J

Alemi, A. A.; Fischer, I.; Dillon, J. V.; and Murphy, K. 2016. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410

arXiv 2016
[4]

Bardera, A.; Rigau, J.; Boada, I.; Feixas, M.; and Sbert, M. 2009. Image segmentation using information bottleneck method. IEEE Transactions on Image Processing, 18(7): 1601--1612

work page 2009
[5]

I.; Baratin, A.; Rajeshwar, S.; Ozair, S.; Bengio, Y.; Courville, A.; and Hjelm, D

Belghazi, M. I.; Baratin, A.; Rajeshwar, S.; Ozair, S.; Bengio, Y.; Courville, A.; and Hjelm, D. 2018 a . Mutual information neural estimation. In International conference on machine learning, 531--540. PMLR

work page 2018
[6]

I.; Baratin, A.; Rajeswar, S.; Ozair, S.; Bengio, Y.; Courville, A.; and Hjelm, R

Belghazi, M. I.; Baratin, A.; Rajeswar, S.; Ozair, S.; Bengio, Y.; Courville, A.; and Hjelm, R. D. 2018 b . Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062

arXiv 2018
[7]

Gilad-Bachrach, R.; Navot, A.; and Tishby, N. 2003. An information theoretic tradeoff between complexity and accuracy. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings, 595--609. Springer

work page 2003
[8]

Goldfeld, Z.; Berg, E. v. d.; Greenewald, K.; Melnyk, I.; Nguyen, N.; Kingsbury, B.; and Polyanskiy, Y. 2018. Estimating information flow in deep neural networks. arXiv preprint arXiv:1810.05728

arXiv 2018

Show all 30 references

[9]

Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014. Generative adversarial nets. Advances in neural information processing systems, 27

2014
[10]

Kim, H.; and Mnih, A. 2018. Disentangling by factorising. In International conference on machine learning, 2649--2658. PMLR

2018
[11]

Kolchinsky, A.; and Tracey, B. D. 2017. Estimating mixture entropy with pairwise distances. Entropy, 19(7): 361

2017
[12]

D.; and Wolpert, D

Kolchinsky, A.; Tracey, B. D.; and Wolpert, D. H. 2019. Nonlinear information bottleneck. Entropy, 21(12): 1181

2019
[13]

Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images

2009
[14]

LeCun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11): 2278--2324

1998
[15]

S.; Igel, C.; and Nielsen, M

Lorenzen, S. S.; Igel, C.; and Nielsen, M. 2021. Information bottleneck: Exact analysis of (quantized) neural networks. arXiv preprint arXiv:2106.12912

2021 arXiv
[16]

J.; and Jordan, M

Nguyen, X.; Wainwright, M. J.; and Jordan, M. 2007. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. Advances in neural information processing systems, 20

2007
[17]

Pan, Z.; Niu, L.; Zhang, J.; and Zhang, L. 2021. Disentangled information bottleneck. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 9285--9293

2021
[18]

Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32

2019
[19]

Pichler, G.; Colombo, P.; Boudiaf, M.; Koliander, G.; and Piantanida, P. 2022. KNIFE: Kernelized-Neural Differential Entropy Estimation

2022
[20]

Rodriguez Galvez, B. 2019. The Information Bottleneck: Connections to Other Problems, Learning and Exploration of the IB Curve

2019
[21]

Rodr \' guez G \'a lvez, B.; Thobaben, R.; and Skoglund, M. 2020. The convex information bottleneck lagrangian. Entropy, 22(1): 98

2020
[22]

Shamir, O.; Sabato, S.; and Tishby, N. 2010. Learning and generalization with the information bottleneck. Theoretical Computer Science, 411(29-30): 2696--2711

2010
[23]

Shwartz-Ziv, R.; and Tishby, N. 2017. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810

2017 arXiv
[24]

Slonim, N.; and Tishby, N. 2000. Document clustering using word clusters via the information bottleneck method. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, 208--215

2000
[25]

C.; and Bialek, W

Tishby, N.; Pereira, F. C.; and Bialek, W. 2000. The information bottleneck method. arXiv preprint physics/0004057

2000 arXiv
[26]

E.; Sun, J.; Chen, X.; and Zheng, Y

Wang, Z.; Huang, S.-L.; Kuruoglu, E. E.; Sun, J.; Chen, X.; and Zheng, Y. 2021. Pac-bayes information bottleneck. arXiv preprint arXiv:2109.14509

2021 arXiv
[27]

Witsenhausen, H.; and Wyner, A. 1975. A conditional entropy bound for a pair of discrete random variables. IEEE Transactions on Information Theory, 21(5): 493--501

1975
[28]

Xie, S.; Ma, S.; Ding, M.; Shi, Y.; Tang, M.; and Wu, Y. 2023. Robust information bottleneck for task-oriented communication with digital modulation. IEEE Journal on Selected Areas in Communications

2023
[29]

Yang, Y.; Wu, Y.; Ma, S.; and Shi, Y. 2023. Multi-Task-Oriented Broadcast for Edge AI Inference via Information Bottleneck. In GLOBECOM 2023-2023 IEEE Global Communications Conference, 3142--3147. IEEE

2023
[30]

Yu, S.; Yu, X.; L kse, S.; Jenssen, R.; and Principe, J. C. 2024. Cauchy-Schwarz Divergence Information Bottleneck for Regression. arXiv preprint arXiv:2404.17951

2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

A.; Fischer, I.; Dillon, J

Alemi, A. A.; Fischer, I.; Dillon, J. V.; and Murphy, K. 2016. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410

arXiv 2016

[4] [4]

Bardera, A.; Rigau, J.; Boada, I.; Feixas, M.; and Sbert, M. 2009. Image segmentation using information bottleneck method. IEEE Transactions on Image Processing, 18(7): 1601--1612

work page 2009

[5] [5]

I.; Baratin, A.; Rajeshwar, S.; Ozair, S.; Bengio, Y.; Courville, A.; and Hjelm, D

Belghazi, M. I.; Baratin, A.; Rajeshwar, S.; Ozair, S.; Bengio, Y.; Courville, A.; and Hjelm, D. 2018 a . Mutual information neural estimation. In International conference on machine learning, 531--540. PMLR

work page 2018

[6] [6]

I.; Baratin, A.; Rajeswar, S.; Ozair, S.; Bengio, Y.; Courville, A.; and Hjelm, R

Belghazi, M. I.; Baratin, A.; Rajeswar, S.; Ozair, S.; Bengio, Y.; Courville, A.; and Hjelm, R. D. 2018 b . Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062

arXiv 2018

[7] [7]

Gilad-Bachrach, R.; Navot, A.; and Tishby, N. 2003. An information theoretic tradeoff between complexity and accuracy. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings, 595--609. Springer

work page 2003

[8] [8]

Goldfeld, Z.; Berg, E. v. d.; Greenewald, K.; Melnyk, I.; Nguyen, N.; Kingsbury, B.; and Polyanskiy, Y. 2018. Estimating information flow in deep neural networks. arXiv preprint arXiv:1810.05728

arXiv 2018