Mutual Information Estimation

37 papers with code • 0 benchmarks • 0 datasets

To estimate mutual information from samples, specially for high-dimensional variables.

Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models

chnq/tracingllm 29 Feb 2024

This research provides an initial exploration of trustworthiness modeling during LLM pre-training, seeking to unveil new insights and spur further developments in the field.

13
29 Feb 2024

Large Language Models are Efficient Learners of Noise-Robust Speech Recognition

yuchen005/robustger 19 Jan 2024

To this end, we propose to extract a language-space noise embedding from the N-best list to represent the noise conditions of source speech, which can promote the denoising process in GER.

50
19 Jan 2024

The Mixtures and the Neural Critics: On the Pointwise Mutual Information Profiles of Fine Distributions

cbg-ethz/bmi 16 Oct 2023

Mutual information quantifies the dependence between two random variables and remains invariant under diffeomorphisms.

20
16 Oct 2023

Rethinking Negative Pairs in Code Search

Alex-HaochenLi/Soft-InfoNCE 12 Oct 2023

In our proposed loss function, we apply three methods to estimate the weights of negative pairs and show that the vanilla InfoNCE loss is a special case of Soft-InfoNCE.

8
12 Oct 2023

Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization

yujieouo/smie 7 Aug 2023

Specifically, 1) we maximize the MI between visual and semantic space for distribution alignment; 2) we leverage the temporal information for estimating the MI by encouraging MI to increase as more frames are observed.

15
07 Aug 2023

Variational $f$-Divergence and Derangements for Discriminative Mutual Information Estimation

tonellolab/fdime 31 May 2023

We propose a novel class of discriminative mutual information estimators based on the variational representation of the $f$-divergence.

0
31 May 2023

Improving Mutual Information Estimation with Annealed and Energy-Based Bounds

huangsicong/ais_mi_estimation ICLR 2022

Since accurate estimation of MI without density information requires a sample size exponential in the true MI, we assume either a single marginal or the full joint density information is known.

0
13 Mar 2023

FrankenSplit: Efficient Neural Feature Compression with Shallow Variational Bottleneck Injection for Mobile Edge Computing

rezafuru/frankensplit 21 Feb 2023

The rise of mobile AI accelerators allows latency-sensitive applications to execute lightweight Deep Neural Networks (DNNs) on the client side.

10
21 Feb 2023

Mutual Wasserstein Discrepancy Minimization for Sequential Recommendation

zfan20/mstein 28 Jan 2023

Wasserstein Discrepancy Measurement builds upon the 2-Wasserstein distance, which is more robust, more efficient in small batch sizes, and able to model the uncertainty of stochastic augmentation processes.

9
28 Jan 2023

DiME: Maximizing Mutual Information by a Difference of Matrix-Based Entropies

uk-cliplab/DiME 19 Jan 2023

We introduce an information-theoretic quantity with similar properties to mutual information that can be estimated from data without making explicit assumptions on the underlying distribution.

2
19 Jan 2023