Course Content
The Art of A/B Testing
The Art of A/B Testing
Levene's Test
Our next step after visualization will be a statistical check of the equality of variances for the control and test groups.
For this, we need the Levene's test. It is used to test the null hypothesis that the variances are equal. By comparing the variances in both groups, the test uses a statistic based on the standard deviation.
It's called the F-statistic. If the obtained value of the F-statistic exceeds the critical value, this indicates significant differences in the variances of the groups.
Let's do a Levene's test for the 'Impression'speakers. Let's start with hypotheses:
H₀: The variances in both groups are the same;
Hₐ: Variances differ between groups.
Let's take a look at the code. The syntax is very simple:
# Import libraries import pandas as pd from scipy.stats import levene # Read files df_control = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/c3b98ad3-420d-403f-908d-6ab8facc3e28/ab_control.csv', delimiter=';') df_test = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/c3b98ad3-420d-403f-908d-6ab8facc3e28/ab_test.csv', delimiter=';') # Do Levene's test statistic, p_value = levene(df_control['Impression'], df_test['Impression']) # Print result of Levene's test print('Statistic:', statistic) print('p-value:', p_value) # Determine whether the variances are similar if p_value > 0.05: print('The variances of the two groups are NOT statistically different') else: print('The variances of the two groups are statistically different')
As you can see, the p-value is much higher than 0.05. This means that we do not have enough statistical evidence to reject the null hypothesis of equality of variances. There is no statistically significant difference between the variances of the two samples.
Now let's draw a violin plot and a swarm plot for the 'Click' columns of both datasets:
# Import libraries import matplotlib.pyplot as plt import pandas as pd import seaborn as sns # Read .csv files df_control = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/c3b98ad3-420d-403f-908d-6ab8facc3e28/ab_control.csv', delimiter=';') df_test = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/c3b98ad3-420d-403f-908d-6ab8facc3e28/ab_test.csv', delimiter=';') # Define colors for graphs colors_list = ['#ff8a00', '#33435c'] # Add to the dataframes columns-labels, which mean belonging to either the control or the test group df_control['group'] = 'Contol group' df_test['group'] = 'Test group' # Concat the dataframes df_combined = pd.concat([df_control, df_test]) # Plotting violin plots sns.violinplot(data=df_combined, x='group', y='Click', palette=colors_list) # Plotting swarm plots sns.swarmplot(data=df_combined, x='group', y='Click', color="r", alpha=0.8) # Sign the axes plt.xlabel('') plt.ylabel('Clicks') plt.title('Comparison of Click') # Show the results plt.show()
Are the variances equal? Let's do a Levene test for the 'Click' columns of both datasets:
# Import libraries import pandas as pd from scipy.stats import levene # Read .csv files df_control = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/c3b98ad3-420d-403f-908d-6ab8facc3e28/ab_control.csv', delimiter=';') df_test = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/c3b98ad3-420d-403f-908d-6ab8facc3e28/ab_test.csv', delimiter=';') # Levene's test statistic, p_value = levene(df_control['Click'], df_test['Click']) # Results of the Levene's test print('Statistic:', statistic) print('p-value:', p_value) # Determine whether the variances are similar if p_value > 0.05: print('The variances of the two groups are NOT statistically different') else: print('The variances of the two groups are statistically different')
We need an understanding of whether the variances are equal to conduct a T-test. In the case of inequality of variances, we will use a modification of the T-test. Therefore, the Levene test is an important part of A/B testing. As you can see, the variances for these two columns are statistically different. Now it's your turn!
Everything was clear?
