Information Plane

10 papers with code • 0 benchmarks • 0 datasets

To obtain the Information Plane (IP) of deep neural networks, which shows the trajectories of the hidden layers during training in a 2D plane using as coordinate axes the mutual information between the input and the hidden layer, and the mutual information between the output and the hidden layer.

Most implemented papers

Opening the Black Box of Deep Neural Networks via Information

gtegner/mine-pytorch 2 Mar 2017

Previous work proposed to analyze DNNs in the \textit{Information Plane}; i. e., the plane of the Mutual Information values that each layer preserves on the input and output variables.

On the Information Bottleneck Theory of Deep Learning

artemyk/ibsgd ICLR 2018

The practical successes of deep neural networks have not been matched by theoretical progress that satisfyingly explains their behavior.

Scalable Mutual Information Estimation using Dependence Graphs

mrtnoshad/EDGE 27 Jan 2018

To the best of our knowledge EDGE is the first non-parametric MI estimator that can achieve parametric MSE rates with linear time complexity.

On the Information Plane of Autoencoders

nicolasigor/entropy 15 May 2020

Recently, the Information Plane (IP) was proposed to analyze them, which is based on the information-theoretic concept of mutual information (MI).

The Dual Information Bottleneck

ravidziv/dual_IB 8 Jun 2020

The Information Bottleneck (IB) framework is a general characterization of optimal representations obtained using a principled approach for balancing accuracy and complexity.

Malicious Network Traffic Detection via Deep Learning: An Information Theoretic View

erickgalinkin/jhu_masters 16 Sep 2020

Applying our results can serve to guide analysis methods for machine learning engineers and suggests that neural networks that can exploit the convolution theorem are equally accurate as standard convolutional neural networks, and can be more computationally efficient.

A Provably Convergent Information Bottleneck Solution via ADMM

hui811116/ib-admm 9 Feb 2021

Conventionally, it resorts to characterizing the information plane, that is, plotting $I(Y;Z)$ versus $I(X;Z)$ for all solutions obtained from different initial points.

Information flows of diverse autoencoders

Sungyeop/IPRL 15 Feb 2021

Thus, we conclude that the compression phase is not necessary for generalization in representation learning.

HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels

sarvanichinthapalli/hrel 22 Feb 2022

Even after pruning the filters from convolutional layers of LeNet-5 drastically (i. e. from 20, 50 to 2, 3, respectively), only a small accuracy drop of 0. 52\% is observed.

End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training

keitaroskmt/e2e-info 14 Feb 2024

End-to-end (E2E) training, optimizing the entire model through error backpropagation, fundamentally supports the advancements of deep learning.