Course Content

Preprocessing Data

1. Data Exploration

Data Extracting Explore the Dataset Data Types

2. Data Cleaning

Missing and Wrong Data Work with NaNs Replace Numerical Missing Data with Values Dealing with Missing Ages Replace Categorical Missing Data with Values Replace Missing Values with Interpolation Removing Duplicates Removing Outliers Visualizing Outliers

3. Data Validation

Types Conversion Data Binning

4. Normalization & Standardization

Data Normalization Data Standardization

5. Data Encoding

Label Encoding One Hot Encoding

Replace Numerical Missing Data with Values

In the previous chapter, you dropped all the records containing NaN. If everything was done correct, the shape of new dataframe was (183, 12) - we lost 709 rows out of 891!

To avoid data loss because of NaNs, use different approaches to replace the missing data. The choice depends on the amount of missing data, its distribution(one or multiple columns, homogeneous or not, etc.).

The popular approaches to deal with NaNs for numerical data are:

replace with the mean value: good for the normal data distribution and when the number of NaNs is small.
replace with the mode value: good for exponential distributions and a small amount of NaNs.
replace with the max or min value: good if you are sure there are no outliers that may affect the result.
replace with some const value: for example, 0 or 1, if the possible value is either 0 or 1.

To replace the NaNs, you can use fillna():


              123
            
data.fillna(some_val) # replaces NaN with some_val
# or
data.fillna(some_val, inplace=True) # change the data in-place

Also, you can use replace(old_val, new_val) to replace not only NaNs, but any other values:


              123
            
data['Age'].replace(np.nan, 0) # replaces NaN with 0
# or
data['Age'].replace(np.nan, 0, inplace=True)

Do you remember that titanic dataset contains missing values in the Age column? Instead of dropping rows, let's think how to replace NaNs and save data.

Task

If share of NaNs is low enough, replace them with value - but which one? Do the following:

Calculate the share of missing values in Age column. Round this value to 2 decimal places.
Build the histogram of Age distribution. Use matplotlib.pyplot and method hist().

Well, hope you have the similar histogram:

If yes, move on to the next chapter to deal with NaNs.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Thanks for your feedback!

Section 2. Chapter 3

Replace Numerical Missing Data with Values

In the previous chapter, you dropped all the records containing NaN. If everything was done correct, the shape of new dataframe was (183, 12) - we lost 709 rows out of 891!

The popular approaches to deal with NaNs for numerical data are:

replace with the mean value: good for the normal data distribution and when the number of NaNs is small.
replace with the mode value: good for exponential distributions and a small amount of NaNs.
replace with the max or min value: good if you are sure there are no outliers that may affect the result.
replace with some const value: for example, 0 or 1, if the possible value is either 0 or 1.

To replace the NaNs, you can use fillna():


              123
            
data.fillna(some_val) # replaces NaN with some_val
# or
data.fillna(some_val, inplace=True) # change the data in-place

Also, you can use replace(old_val, new_val) to replace not only NaNs, but any other values:


              123
            
data['Age'].replace(np.nan, 0) # replaces NaN with 0
# or
data['Age'].replace(np.nan, 0, inplace=True)

Do you remember that titanic dataset contains missing values in the Age column? Instead of dropping rows, let's think how to replace NaNs and save data.

Task

If share of NaNs is low enough, replace them with value - but which one? Do the following:

Calculate the share of missing values in Age column. Round this value to 2 decimal places.
Build the histogram of Age distribution. Use matplotlib.pyplot and method hist().

Well, hope you have the similar histogram:

If yes, move on to the next chapter to deal with NaNs.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Thanks for your feedback!

Section 2. Chapter 3

Replace Numerical Missing Data with Values

In the previous chapter, you dropped all the records containing NaN. If everything was done correct, the shape of new dataframe was (183, 12) - we lost 709 rows out of 891!

The popular approaches to deal with NaNs for numerical data are:

replace with the mean value: good for the normal data distribution and when the number of NaNs is small.
replace with the mode value: good for exponential distributions and a small amount of NaNs.
replace with the max or min value: good if you are sure there are no outliers that may affect the result.
replace with some const value: for example, 0 or 1, if the possible value is either 0 or 1.

To replace the NaNs, you can use fillna():


              123
            
data.fillna(some_val) # replaces NaN with some_val
# or
data.fillna(some_val, inplace=True) # change the data in-place

Also, you can use replace(old_val, new_val) to replace not only NaNs, but any other values:


              123
            
data['Age'].replace(np.nan, 0) # replaces NaN with 0
# or
data['Age'].replace(np.nan, 0, inplace=True)

Do you remember that titanic dataset contains missing values in the Age column? Instead of dropping rows, let's think how to replace NaNs and save data.

Task

If share of NaNs is low enough, replace them with value - but which one? Do the following:

Calculate the share of missing values in Age column. Round this value to 2 decimal places.
Build the histogram of Age distribution. Use matplotlib.pyplot and method hist().

Well, hope you have the similar histogram:

If yes, move on to the next chapter to deal with NaNs.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Thanks for your feedback!

In the previous chapter, you dropped all the records containing NaN. If everything was done correct, the shape of new dataframe was (183, 12) - we lost 709 rows out of 891!

The popular approaches to deal with NaNs for numerical data are:

replace with the mean value: good for the normal data distribution and when the number of NaNs is small.
replace with the mode value: good for exponential distributions and a small amount of NaNs.
replace with the max or min value: good if you are sure there are no outliers that may affect the result.
replace with some const value: for example, 0 or 1, if the possible value is either 0 or 1.

To replace the NaNs, you can use fillna():


              123
            
data.fillna(some_val) # replaces NaN with some_val
# or
data.fillna(some_val, inplace=True) # change the data in-place

Also, you can use replace(old_val, new_val) to replace not only NaNs, but any other values:


              123
            
data['Age'].replace(np.nan, 0) # replaces NaN with 0
# or
data['Age'].replace(np.nan, 0, inplace=True)

Do you remember that titanic dataset contains missing values in the Age column? Instead of dropping rows, let's think how to replace NaNs and save data.

Task

If share of NaNs is low enough, replace them with value - but which one? Do the following:

Calculate the share of missing values in Age column. Round this value to 2 decimal places.
Build the histogram of Age distribution. Use matplotlib.pyplot and method hist().

Well, hope you have the similar histogram:

If yes, move on to the next chapter to deal with NaNs.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Section 2. Chapter 3

Switch to desktop for real-world practiceContinue from where you are using one of the options below