k-Nearest-Neighbour Mutual Information (KSG Estimator)

KSG mutual information estimation measures the total linear and nonlinear dependence between continuous variables from k-nearest-neighbour distances, without binning the data.

KSG 상호정보량 추정량은 구간을 나누지 않고 k-최근접 이웃 거리로 연속 변수 사이의 선형·비선형 의존 전체를 재는 방법입니다.

How can I quantify a nonlinear association between two continuous variables that a correlation coefficient misses?

상관계수로는 잡히지 않는 두 연속 변수의 비선형 관계를 어떻게 수치로 나타낼 수 있나요?

Mutual information is zero only when two variables are independent, so it captures curved and non-monotonic relationships that Pearson or Spearman correlations can miss. The Kraskov–Stögbauer–Grassberger estimator computes it from nearest-neighbour distances, and a permutation test indicates whether the dependence is larger than chance.

When to use k-Nearest-Neighbour Mutual Information (KSG Estimator)

• The relationship between an outcome and a continuous predictor may be nonlinear or non-monotonic.
• Several continuous predictors are screened for any kind of dependence with an outcome.
• A dependence measure in nats or bits is needed for an information-theoretic analysis.

When to use something else

• The variables are categorical or have many tied values; use binned or discrete mutual information.
• Only a linear association is of interest; use Pearson correlation, which has a simpler interpretation.

Assumptions and requirements

• Variables are continuous with few ties (tiny jitter is added to break ties).
• Observations are independent; time-series autocorrelation invalidates the permutation p-value.
• The sample is large enough for the chosen k (estimates are biased in small samples).

What the analysis reports

• Mutual information per feature and jointly for all features (nats and bits)
• Permutation p-value for each estimate
• Linfoot informational coefficient of correlation
• Mean of the permutation null distribution

How to report it

The KSG estimate (k = [value]) of the mutual information between [X] and [Y] was [value] nats, permutation p = [value].

Related methods

• Entropy and Mutual Information Analysis
• Spearman Rank Correlation
• Distance Correlation

Run k-Nearest-Neighbour Mutual Information (KSG Estimator) on Quorant