Contenu du cours
Preprocessing Data
Preprocessing Data
Replace Numerical Missing Data with Values
In the previous chapter, you dropped all the records containing NaN. If everything was done correct, the shape of new dataframe was (183, 12) - we lost 709 rows out of 891!
To avoid data loss because of NaNs, use different approaches to replace the missing data. The choice depends on the amount of missing data, its distribution(one or multiple columns, homogeneous or not, etc.).
The popular approaches to deal with NaNs for numerical data are:
- replace with the mean value: good for the normal data distribution and when the number of NaNs is small.
- replace with the mode value: good for exponential distributions and a small amount of NaNs.
- replace with the max or min value: good if you are sure there are no outliers that may affect the result.
- replace with some const value: for example, 0 or 1, if the possible value is either 0 or 1.
To replace the NaNs, you can use fillna():
data.fillna(some_val) # replaces NaN with some_val # or data.fillna(some_val, inplace=True) # change the data in-place
Also, you can use replace(old_val, new_val) to replace not only NaNs, but any other values:
data['Age'].replace(np.nan, 0) # replaces NaN with 0 # or data['Age'].replace(np.nan, 0, inplace=True)
Do you remember that titanic dataset contains missing values in the Age column? Instead of dropping rows, let's think how to replace NaNs and save data.
Swipe to start coding
If share of NaNs is low enough, replace them with value - but which one? Do the following:
- Calculate the share of missing values in
Agecolumn. Round this value to 2 decimal places. - Build the histogram of
Agedistribution. Usematplotlib.pyplotand methodhist().
Solution
Well, hope you have the similar histogram:
If yes, move on to the next chapter to deal with NaNs.
Passez à un bureau pour une pratique réelleContinuez d'où vous êtes en utilisant l'une des options ci-dessousMerci pour vos commentaires !
Replace Numerical Missing Data with Values
In the previous chapter, you dropped all the records containing NaN. If everything was done correct, the shape of new dataframe was (183, 12) - we lost 709 rows out of 891!
To avoid data loss because of NaNs, use different approaches to replace the missing data. The choice depends on the amount of missing data, its distribution(one or multiple columns, homogeneous or not, etc.).
The popular approaches to deal with NaNs for numerical data are:
- replace with the mean value: good for the normal data distribution and when the number of NaNs is small.
- replace with the mode value: good for exponential distributions and a small amount of NaNs.
- replace with the max or min value: good if you are sure there are no outliers that may affect the result.
- replace with some const value: for example, 0 or 1, if the possible value is either 0 or 1.
To replace the NaNs, you can use fillna():
data.fillna(some_val) # replaces NaN with some_val # or data.fillna(some_val, inplace=True) # change the data in-place
Also, you can use replace(old_val, new_val) to replace not only NaNs, but any other values:
data['Age'].replace(np.nan, 0) # replaces NaN with 0 # or data['Age'].replace(np.nan, 0, inplace=True)
Do you remember that titanic dataset contains missing values in the Age column? Instead of dropping rows, let's think how to replace NaNs and save data.
Swipe to start coding
If share of NaNs is low enough, replace them with value - but which one? Do the following:
- Calculate the share of missing values in
Agecolumn. Round this value to 2 decimal places. - Build the histogram of
Agedistribution. Usematplotlib.pyplotand methodhist().
Solution
Well, hope you have the similar histogram:
If yes, move on to the next chapter to deal with NaNs.
Passez à un bureau pour une pratique réelleContinuez d'où vous êtes en utilisant l'une des options ci-dessousMerci pour vos commentaires !
Passez à un bureau pour une pratique réelleContinuez d'où vous êtes en utilisant l'une des options ci-dessous