Entropy and Mutual Information Analysis

Entropy and mutual information analysis quantifies uncertainty with Shannon entropy and measures how much knowing one variable reduces uncertainty about another, capturing any form of dependence between categorical or binned variables.

상호정보량은 한 변수를 알면 다른 변수의 불확실성(섀넌 엔트로피)이 얼마나 줄어드는지를 재는 지표로, 범주형이나 구간화한 변수 사이의 모든 형태의 의존성을 잡아냅니다.

How much information does a customer's segment carry about which product they buy?

고객 세그먼트가 어떤 상품을 사는지에 대해 얼마나 많은 정보를 담고 있나요?

Entropy H quantifies the uncertainty of a variable, and mutual information H(X) + H(Y) − H(X,Y) is the reduction in uncertainty about one variable from knowing the other, equal to zero under independence. Normalised mutual information rescales it to 0–1, and a permutation test judges whether the observed value exceeds what chance and estimation bias produce.

When to use Entropy and Mutual Information Analysis

• Variables are categorical or can be meaningfully binned
• Dependence may be nonlinear or non-monotonic
• Ranking candidate predictors by information shared with an outcome

When to use something else

• Both variables are continuous with a linear relationship, where Pearson correlation is more efficient
• Samples are small relative to the number of cells, where Cramér's V with an exact test is more stable

Assumptions and requirements

• Independent observations
• Enough observations per cell for plug-in entropy estimates
• A sensible binning of continuous variables

What the analysis reports

• Entropies H(X), H(Y) and joint entropy H(X,Y)
• Mutual information and normalised mutual information
• Conditional entropy H(Y|X)
• Permutation and G-test p values

How to report it

The mutual information between [X] and [Y] was [value] bits (normalized MI = [value]; H(X) = [value], H(Y) = [value] bits), permutation p = [value] (N = [value]).

Related methods

• Distance Correlation
• Cramér's V and Phi Association
• Chi Square Test

Run Entropy and Mutual Information Analysis on Quorant