Notice: This page requires JavaScript to function properly.
Please enable JavaScript in your browser settings or update your browser.Replace Numerical Missing Data with Values | Data Cleaning
Preprocessing Data
course content

Conteúdo do Curso

Preprocessing Data

Preprocessing Data

1. Data Exploration
Data ExtractingExplore the DatasetData Types
2. Data Cleaning
Missing and Wrong DataWork with NaNsReplace Numerical Missing Data with ValuesDealing with Missing AgesReplace Categorical Missing Data with ValuesReplace Missing Values with InterpolationRemoving DuplicatesRemoving OutliersVisualizing Outliers
3. Data Validation
Types ConversionData Binning
4. Normalization & Standardization
Data NormalizationData Standardization
5. Data Encoding
Label EncodingOne Hot Encoding

Replace Numerical Missing Data with Values

In the previous chapter, you dropped all the records containing NaN. If everything was done correct, the shape of new dataframe was (183, 12) - we lost 709 rows out of 891!

To avoid data loss because of NaNs, use different approaches to replace the missing data. The choice depends on the amount of missing data, its distribution(one or multiple columns, homogeneous or not, etc.).

The popular approaches to deal with NaNs for numerical data are:

  • replace with the mean value: good for the normal data distribution and when the number of NaNs is small.
  • replace with the mode value: good for exponential distributions and a small amount of NaNs.
  • replace with the max or min value: good if you are sure there are no outliers that may affect the result.
  • replace with some const value: for example, 0 or 1, if the possible value is either 0 or 1.

To replace the NaNs, you can use fillna():


              
123

            
data.fillna(some_val) # replaces NaN with some_val
# or
data.fillna(some_val, inplace=True) # change the data in-place
copy

Also, you can use replace(old_val, new_val) to replace not only NaNs, but any other values:


              
123

            
data['Age'].replace(np.nan, 0) # replaces NaN with 0
# or
data['Age'].replace(np.nan, 0, inplace=True)
copy

Do you remember that titanic dataset contains missing values in the Age column? Instead of dropping rows, let's think how to replace NaNs and save data.

Tarefa

If share of NaNs is low enough, replace them with value - but which one? Do the following:

  1. Calculate the share of missing values in Age column. Round this value to 2 decimal places.
  2. Build the histogram of Age distribution. Use matplotlib.pyplot and method hist().

Tarefa

If share of NaNs is low enough, replace them with value - but which one? Do the following:

  1. Calculate the share of missing values in Age column. Round this value to 2 decimal places.
  2. Build the histogram of Age distribution. Use matplotlib.pyplot and method hist().

Well, hope you have the similar histogram:

If yes, move on to the next chapter to deal with NaNs.

Mude para o desktop para praticar no mundo realContinue de onde você está usando uma das opções abaixo

Tudo estava claro?

Seção 2. Capítulo 3
toggle bottom row

Replace Numerical Missing Data with Values

In the previous chapter, you dropped all the records containing NaN. If everything was done correct, the shape of new dataframe was (183, 12) - we lost 709 rows out of 891!

To avoid data loss because of NaNs, use different approaches to replace the missing data. The choice depends on the amount of missing data, its distribution(one or multiple columns, homogeneous or not, etc.).

The popular approaches to deal with NaNs for numerical data are:

  • replace with the mean value: good for the normal data distribution and when the number of NaNs is small.
  • replace with the mode value: good for exponential distributions and a small amount of NaNs.
  • replace with the max or min value: good if you are sure there are no outliers that may affect the result.
  • replace with some const value: for example, 0 or 1, if the possible value is either 0 or 1.

To replace the NaNs, you can use fillna():


              
123

            
data.fillna(some_val) # replaces NaN with some_val
# or
data.fillna(some_val, inplace=True) # change the data in-place
copy

Also, you can use replace(old_val, new_val) to replace not only NaNs, but any other values:


              
123

            
data['Age'].replace(np.nan, 0) # replaces NaN with 0
# or
data['Age'].replace(np.nan, 0, inplace=True)
copy

Do you remember that titanic dataset contains missing values in the Age column? Instead of dropping rows, let's think how to replace NaNs and save data.

Tarefa

If share of NaNs is low enough, replace them with value - but which one? Do the following:

  1. Calculate the share of missing values in Age column. Round this value to 2 decimal places.
  2. Build the histogram of Age distribution. Use matplotlib.pyplot and method hist().

Tarefa

If share of NaNs is low enough, replace them with value - but which one? Do the following:

  1. Calculate the share of missing values in Age column. Round this value to 2 decimal places.
  2. Build the histogram of Age distribution. Use matplotlib.pyplot and method hist().

Well, hope you have the similar histogram:

If yes, move on to the next chapter to deal with NaNs.

Mude para o desktop para praticar no mundo realContinue de onde você está usando uma das opções abaixo

Tudo estava claro?

Seção 2. Capítulo 3
toggle bottom row

Replace Numerical Missing Data with Values

In the previous chapter, you dropped all the records containing NaN. If everything was done correct, the shape of new dataframe was (183, 12) - we lost 709 rows out of 891!

To avoid data loss because of NaNs, use different approaches to replace the missing data. The choice depends on the amount of missing data, its distribution(one or multiple columns, homogeneous or not, etc.).

The popular approaches to deal with NaNs for numerical data are:

  • replace with the mean value: good for the normal data distribution and when the number of NaNs is small.
  • replace with the mode value: good for exponential distributions and a small amount of NaNs.
  • replace with the max or min value: good if you are sure there are no outliers that may affect the result.
  • replace with some const value: for example, 0 or 1, if the possible value is either 0 or 1.

To replace the NaNs, you can use fillna():


              
123

            
data.fillna(some_val) # replaces NaN with some_val
# or
data.fillna(some_val, inplace=True) # change the data in-place
copy

Also, you can use replace(old_val, new_val) to replace not only NaNs, but any other values:


              
123

            
data['Age'].replace(np.nan, 0) # replaces NaN with 0
# or
data['Age'].replace(np.nan, 0, inplace=True)
copy

Do you remember that titanic dataset contains missing values in the Age column? Instead of dropping rows, let's think how to replace NaNs and save data.

Tarefa

If share of NaNs is low enough, replace them with value - but which one? Do the following:

  1. Calculate the share of missing values in Age column. Round this value to 2 decimal places.
  2. Build the histogram of Age distribution. Use matplotlib.pyplot and method hist().

Tarefa

If share of NaNs is low enough, replace them with value - but which one? Do the following:

  1. Calculate the share of missing values in Age column. Round this value to 2 decimal places.
  2. Build the histogram of Age distribution. Use matplotlib.pyplot and method hist().

Well, hope you have the similar histogram:

If yes, move on to the next chapter to deal with NaNs.

Mude para o desktop para praticar no mundo realContinue de onde você está usando uma das opções abaixo

Tudo estava claro?

In the previous chapter, you dropped all the records containing NaN. If everything was done correct, the shape of new dataframe was (183, 12) - we lost 709 rows out of 891!

To avoid data loss because of NaNs, use different approaches to replace the missing data. The choice depends on the amount of missing data, its distribution(one or multiple columns, homogeneous or not, etc.).

The popular approaches to deal with NaNs for numerical data are:

  • replace with the mean value: good for the normal data distribution and when the number of NaNs is small.
  • replace with the mode value: good for exponential distributions and a small amount of NaNs.
  • replace with the max or min value: good if you are sure there are no outliers that may affect the result.
  • replace with some const value: for example, 0 or 1, if the possible value is either 0 or 1.

To replace the NaNs, you can use fillna():


              
123

            
data.fillna(some_val) # replaces NaN with some_val
# or
data.fillna(some_val, inplace=True) # change the data in-place
copy

Also, you can use replace(old_val, new_val) to replace not only NaNs, but any other values:


              
123

            
data['Age'].replace(np.nan, 0) # replaces NaN with 0
# or
data['Age'].replace(np.nan, 0, inplace=True)
copy

Do you remember that titanic dataset contains missing values in the Age column? Instead of dropping rows, let's think how to replace NaNs and save data.

Tarefa

If share of NaNs is low enough, replace them with value - but which one? Do the following:

  1. Calculate the share of missing values in Age column. Round this value to 2 decimal places.
  2. Build the histogram of Age distribution. Use matplotlib.pyplot and method hist().

Well, hope you have the similar histogram:

If yes, move on to the next chapter to deal with NaNs.

Mude para o desktop para praticar no mundo realContinue de onde você está usando uma das opções abaixo
Seção 2. Capítulo 3
Mude para o desktop para praticar no mundo realContinue de onde você está usando uma das opções abaixo
We're sorry to hear that something went wrong. What happened?
some-alt