What is A/B Testing? -2

In the second article of the series, I will do more exercises. This topic covers the statistics section. So I will not go into detail. I will only talk about the methods I used in my own project.

A little reminder, A/B tests consist of a randomized experiment with two variants, A and B. Split testing, refers to an experiment technique to determine whether a new design brings improvement, according to a chosen metric.

What path should be followed and what should be considered?

1. Correct experimental design and correct data collection is required.

2. The correct measurement tool (test method) must be selected.

3. Correct statistical hypotheses need to be established and tested.

4. The optimum answer to the question of how long should the measurement period be?

5. Hypotheses should be interpreted and relevant action recommendations should be submitted to business units.

Here are the things we will do;

1. Loading Necessary Libraries
2. Sampling
3. Descriptive Statistics
4. Confidence Interval
5. Assumption Check
6. Testing the Normality Assumption
7. Testing the Homogeneity Assumption
8. Application of the Hypothesis (Independent Two Samples T-Test)

I used them;

2. Sampling

Probability sampling involves random selection, allowing you to make strong statistical inferences about the whole group. The methodology used to sample from a larger population depends on the type of analysis being performed, but it is may include simple random sampling or systematic sampling. It is an important concept because it is gives information about the total. The larger the sample, the closer the results are to the data.

3. Descriptive Statistics

Data is analyzed at this stage. Numerical values ​​are taken into statistical evaluation.

Observations and missing values ​​are processed according to the situation. Data should be made available.

4. Confidence Interval

This proposes a range of plausible values for an unknown parameter (for example, the mean.) General terms, a confidence interval for an unknown parameter is based on sampling the distribution of a corresponding estimator. Most commonly, a 95% confidence level is used. I also used this confidence interval in the Project.

And results;

95 out of 100 people entering the site make between 508 and 593 purchases.

95 out of 100 people entering the site make between 530 and 633 purchases.

5. Assumption Check

But first of all what is a hypothesis?

“hypothesis is a claim that can be tested through experiment or observation.”

“Hypothesis is described as a recommended solution for an undefinable incident which doesn’t into current theory”

There are two types of hypotheses; Null and Alternative Hypothesis Testing.

The null hypothesis testing is tested for the assumption that it is true. We assume that the empty hypothesis is correct until there is statistically sufficient evidence. The alternative hypothesis complements the null hypothesis. When the null and alternative hypothesis come together, the whole possibility occurs.

Null hypothesis=H0

Alternative hypothesis=H1

The statements are more correct as follows;

H0 is rejected or H0 cannot be rejected (there is not enough evidence to reject it) comments can be made.

P-value (Probability value)

The p-value is the smallest level of significance at which a null hypothesis can be rejected.

P-value is determined by assuming the null hypothesis is true, it is the probability of observing a test statistic of a value or more extreme.

If P-value < 0,05-> Reject H0

P-value ≥ 0,05 — -> Do not reject H0

6. Testing the Normality Assumption

There are many tests of assumption of normality in the literature. Here we will use Shapiro test(Shapiro-Wilk).

H0: There is no statistically significant different between sample distribution and the normal distribution.

H1: There is a statistically significant different between sample distribution and the normal distribution.

So how do we decide which one to use? -with P-value.

If P-value < 0,05-> Reject H0

P-value ≥ 0,05 — -> Do not reject H0

Let’s examine both data groups separately.

H0 cannot be rejected as the p-value is greater than 0.05 in both examinations.

7. Testing the Homogeneity Assumption

Note: This assumption is only relevant in the case of normal distribution.

We put both data into Levene test. In statistics, Levene test is an inferential statistic used to assess the equality of variances for a variable calculated for two or more groups. Here we examine two variables together.

Ho: The variances are equal across two groups.

Hı: The variances are not equal across two groups.

So how do we decide which one to use? -with P-value.

If P-value < 0,05-> Reject H0

P-value ≥ 0,05 — -> Do not reject H0

P-value = 0.1, H0 cannot be rejected. So the variances are equal across two groups.

8.Application of the Hypothesis (Independent Two Samples T-Test)

The assumptions are met. If assumptions are met, independent two sample t test (parametric test) is applied.

H0: There is no statistically significant difference between the means of the two groups.

H1: There is a statistically significant difference between the means of the two groups.

So how do we decide which one to use? -with P-value.

If P-value < 0,05-> Reject H0

P-value ≥ 0,05 — -> Do not reject H0

P-value = 0.3, H0 cannot be rejected. There is no statistically significant difference between the means of the two groups.

Why did we use t test?

Used to test whether there is a statistically significant difference between numerical (continuous) variables (or groups).

t-tests are a statistical way of testing a hypothesis when:

• We do not know the population variance
• Our sample size is small, n < 30

Result, the control group has higher reliability. The test group is statistically better. There is no statistically significant difference between the means of the two groups. The number of observations should be increased by continuing the test.

Visit my GitHub account to see the codes in more detail!

nihandincer/A-B-Testing

nihandincer/Pricing-A-B-Testing-

References:

1. https://www.veribilimiokulu.com/blog/istatistiksel-a-b-testleri-nasil-yapilir/

2. https://mathchi.medium.com/what-is-ab-test-how-are-statistical-a-b-tests-done-2-164b55fa1c30

3. https://tezverianaliz.com/biyoistatistik-dershanesi/t-test-students-t-test/

4.https://www.analyticsvidhya.com/blog/2021/01/an-introduction-to-hypothesis-testing/