Bayesian A/B Test (Beta-Binomial)

The Bayesian A/B test gives each variant's success rate a beta posterior from a beta prior and binomial data, yielding the probability that one variant beats the other and each choice's expected loss.

베이즈 A/B 검정은 베타 사전분포와 이항 자료로 각 안의 전환율 사후분포를 구해, 한 안이 더 나을 확률과 각 선택의 기대 손실을 제시하는 방법이다.

What is the probability that version B converts better than version A, and how much do I risk by choosing it?

B안의 전환율이 A안보다 높을 확률은 얼마이고, B를 골랐을 때 감수하는 손실은 얼마인가요?

With a Beta prior and binomial data, each variant's success rate has a Beta posterior, and the probability that B exceeds A is the integral of B's density times A's distribution function. The expected loss of choosing B is the average amount by which A would exceed B under the posterior, which supports a decision rule.

When to use Bayesian A/B Test (Beta-Binomial)

• You compare conversion or success rates between two variants
• Stakeholders need a direct probability that one variant is better
• You want a decision based on expected loss rather than significance

When to use something else

• The outcome is continuous, such as revenue; use a Bayesian t test or bootstrap instead
• Users can appear in both arms; analyse at the user level with a paired or hierarchical model instead

Assumptions and requirements

• Each outcome is binary
• Observations are independent within and between arms
• The prior reflects belief before the test (Beta(1, 1) is uniform)
• The larger outcome value counts as success, and the groups in sorted order are taken as A and B

What the analysis reports

• Posterior mean and credible interval of each rate
• Probability that B exceeds A
• Expected loss of choosing each variant
• Credible interval of the difference and relative lift

How to report it

The posterior mean conversion rate was [value] for A and [value] for B; P(B > A) = [value], difference = [value], 95% CrI [[value], [value]].

Related methods

• Two Proportion z Test
• Bayesian t Test (Bayes Factor)

Run Bayesian A/B Test (Beta-Binomial) on Quorant