A/B Test Lecture 60

 

A/B Test

Lecture 60

The A/B or A/B/.../K test, also known as a bucket or split-run test, is a statistical method used to compare two (A, B) or more versions of the same variable to determine which version is more effective on a specific metric. Version A is called 'control', and version B is called 'experimental'. The businesses and industry widely used the A/B test to utilise their resources in a better way. The specific metric is

Average revenue per user per unit of time.

A/B testing is a handy technique for verifying optimisation when comparing a variation to the conventional. For instance, altering a movie poster might boost attendance, or altering a product's packing can boost sales.

Examples:

i. A company is creating two versions of an advertisement: one is version A with a folk song in the background, and the second is version B with fast music in the background. Both versions of advertisements are viral;check which one is more popular or effective in terms of revenue.

ii. A dress designer develops two brands of a school uniform and wants to check which brand is more attractive and share with kids’s parents.

Procedure to Perform A/B or A/B/…/K Test

Define your objectives. 

Define your variable or variables and metrics.

State your null and alternative hypotheses.

Procedure to Collect Data

i. Suppose you want to analyse two variants, A and B. The observations for both variants are collected randomly and independently.

ii. The observations for both variants are collected at the same time period to control extraneous sources of variations.

iii. The duration to collect observations for both variants will be the same and conducted in the same circumstances to measure the real change.

 Test Statistic

The test statistic is dependent; you define the hypothesis and metric. The following commonly employed test statistics are:

z-statistic, t-statistic, chi-square statistic, F statistic and ANOVA.

Assume the following forms of hypotheses along with a valid test statistic:

$$

H0: P1 = P2 

H0: P1 = P2 = ... = Pk

H0: μ1 = μ2

The test statistic for large samples:

H0: μ1 = μ2

The test statistic for small samples:

H0: μ1 = μ2 = ... = μk

The ANOVA technique will be used.