Mutual Information Estimation
37 papers with code • 0 benchmarks • 0 datasets
To estimate mutual information from samples, specially for high-dimensional variables.
Benchmarks
These leaderboards are used to track progress in Mutual Information Estimation
Latest papers
Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models
This research provides an initial exploration of trustworthiness modeling during LLM pre-training, seeking to unveil new insights and spur further developments in the field.
Large Language Models are Efficient Learners of Noise-Robust Speech Recognition
To this end, we propose to extract a language-space noise embedding from the N-best list to represent the noise conditions of source speech, which can promote the denoising process in GER.
The Mixtures and the Neural Critics: On the Pointwise Mutual Information Profiles of Fine Distributions
Mutual information quantifies the dependence between two random variables and remains invariant under diffeomorphisms.
Rethinking Negative Pairs in Code Search
In our proposed loss function, we apply three methods to estimate the weights of negative pairs and show that the vanilla InfoNCE loss is a special case of Soft-InfoNCE.
Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization
Specifically, 1) we maximize the MI between visual and semantic space for distribution alignment; 2) we leverage the temporal information for estimating the MI by encouraging MI to increase as more frames are observed.
Variational $f$-Divergence and Derangements for Discriminative Mutual Information Estimation
We propose a novel class of discriminative mutual information estimators based on the variational representation of the $f$-divergence.
Improving Mutual Information Estimation with Annealed and Energy-Based Bounds
Since accurate estimation of MI without density information requires a sample size exponential in the true MI, we assume either a single marginal or the full joint density information is known.
FrankenSplit: Efficient Neural Feature Compression with Shallow Variational Bottleneck Injection for Mobile Edge Computing
The rise of mobile AI accelerators allows latency-sensitive applications to execute lightweight Deep Neural Networks (DNNs) on the client side.
Mutual Wasserstein Discrepancy Minimization for Sequential Recommendation
Wasserstein Discrepancy Measurement builds upon the 2-Wasserstein distance, which is more robust, more efficient in small batch sizes, and able to model the uncertainty of stochastic augmentation processes.
DiME: Maximizing Mutual Information by a Difference of Matrix-Based Entropies
We introduce an information-theoretic quantity with similar properties to mutual information that can be estimated from data without making explicit assumptions on the underlying distribution.