The real-world data is not always distributed the way we want, that is Normal-Distribution It is always distributed in some distribution which we have no idea about some time it is skewed towards the right other time it has a long tail this leads us to miss normal distribution, Why we miss normal distribution you ask?
The normal distribution is the most important probability distribution in statistics because it fits many natural phenomena. It is symmetric distribution where most of the observations cluster around the central peak and the probabilities for values further away from the mean taper off equally in both directions.
The Box-Cox transformation is a particularly useful family of transformations. It is defined as:
where y^λ is the response variable and λ is the transformation parameter, For λ = 0, the natural log of the data is taken instead of using the above formula, here λ is a hyperparameter which has to be tuned according to the datasetLet’s see box-cox in action
Import the necessary libraries
from scipy import stats import pandas as pd import numpy as np import pylab import matplotlib.pyplot as plt import seaborn as sns from scipy import stats %matplotlib inline
Let’s create a skewed distribution
skewed_dist = stats.loggamma.rvs(5, size=10000) + 5
Now let’s create a normal distribution
normal_dist = np.random.normal(0, 1, 10000)
Now let’s calculate the skewness of the normal distribution and look at the plot of the distributions
sns.distplot(s) print("Skewness for the normal distribution:",skew(normal_dist)) sns.distplot(x) print("Skewness for the skewed distribution:",skew(skewed_dist))
Here you can see how the second distribution is left-skewed
skewness = 0 : normally distributed.
skewness > 0 : more weight in the left tail of the distribution.
skewness < 0 : more weight in the right tail of the distribution.
Pearson’s Coefficient of Skewness
skewness = 3(X-Me) / σ
where X = mean of the distribution
Me = median of the distribution and
σ = standard deviation
The probability plot is a graphical technique for assessing whether or not a data set follows a normal distribution
stats.probplot(normal_dist, dist=”norm”, plot=pylab) pylab.show() stats.probplot(skewed_dist, dist=”norm”, plot=pylab) pylab.show()
In the above diagram for the skewed plot, we can see that the data is not normally distributed since the point don’t align with the red line the plot is not normally distributed
Now we apply our box-cox Transformation and plot it
skewed_box_cox, lmda = stats.boxcox(skwed_dist) sns.distplot(skewed_box_cox)
Let’s check the Probability-Plot and see whether the data is normally distributed or not and get the appropriate lambda value.
stats.probplot(skewed_box_cox, dist=”norm”, plot=pylab) pylab.show() print ("lambda parameter for Box-Cox Transformation is:",lmda)
In Conclusion, Box-cox transformation attempts to transform a set of data to a normal distribution by finding the value of λ that minimizes the variation. This allows you to perform those calculations that require the data to be normally distributed, The Box-Cox transformation does not always convert the data to a normal distribution. You must check the transformation to ensure it worked.