Permutation Test for Two Groups

The two-group permutation test assesses a difference in group means by comparing the observed difference with the differences obtained under all, or many random, reassignments of group labels.

두 집단 순열 검정은 관측된 평균 차이를 집단 이름표를 모든 방법(또는 많은 무작위 방법)으로 다시 붙였을 때의 평균 차이 분포와 비교하는 검정이다.

How do I compare two group means without assuming normality, especially with small samples?

표본이 작고 정규성을 가정하기 어려울 때 두 집단 평균을 어떻게 비교하나요?

A permutation test recomputes the mean difference for every way of relabeling the pooled observations into groups of the original sizes; the p-value is the share of relabelings at least as extreme as the observed difference. With few observations all relabelings are counted exactly, and otherwise a large seeded random sample of relabelings gives a Monte Carlo p-value.

When to use Permutation Test for Two Groups

• Samples are small or clearly non-normal.
• Units were randomly assigned to two conditions.
• You want an exact p-value for the observed data.

When to use something else

• The data are normal with adequate sample sizes; a Welch t test gives the same answer more simply.
• Groups have very different variances and the interest is only in means; use the Welch t test or a studentized permutation test instead.

Assumptions and requirements

• Observations are independent.
• Under the null hypothesis the group labels are exchangeable (same distribution in both groups).

What the analysis reports

• Observed mean difference
• Exact or Monte Carlo p-value
• Number of permutations and seed
• Cohen's d

How to report it

An exact permutation test showed that the mean of [group 1] (M = [value]) [differed/did not differ] from that of [group 2] (M = [value]), difference = [value], p = [value].

Related methods

• Welch's t-test
• Mann-Whitney U Test
• Bootstrap Confidence Intervals (BCa)

Run Permutation Test for Two Groups on Quorant