Summary  
This chapter shows how to handle missing values by using a fill method to replace NaNs with a computed statistic (e.g., the median) and apply the change in place.  

General domain of usage  
Data cleaning

Deleting missing values is not the only way to get rid of them. You can also replace all NaNs with a defined value, for instance, with the mean value of the column or with zeros. It can be useful in a lot of cases. You will learn this in the course <a href="https://codefinity.com/courses/v2/a849660e-ddfa-4033-80a6-94a1b7772e23" target="_blank"><svg width="1em" height="1em" viewBox="0 0 30 32" fill="none" xmlns="http://www.w3.org/2000/svg"><path fill="none" stroke="#098f67" style="stroke: var(--color1, #098f67)" stroke-linejoin="miter" stroke-linecap="round" stroke-miterlimit="4" stroke-width="2.4156" d="M17.289 17.305v0c-1.754 1.754-4.597 1.754-6.351 0l-7.76-7.759c-1.755-1.755-1.755-4.601 0-6.356v0c1.753-1.753 4.595-1.756 6.351-0.005l6.208 6.187"></path><path fill="none" stroke="#098f67" style="stroke: var(--color1, #098f67)" stroke-linejoin="miter" stroke-linecap="round" stroke-miterlimit="4" stroke-width="2.4156" d="M12.504 13.97v0c1.754-1.754 4.597-1.754 6.351 0l7.762 7.762c1.754 1.754 1.754 4.597 0 6.351v0c-1.754 1.754-4.597 1.754-6.351 0l-5.953-5.953"></path></svg> Learning Statistics with Python</a>.

Look at the example of filling missing values in the column `'Age'` with the median value of this column:

import pandas as pd
data = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/4bf24830-59ba-4418-969b-aaf8117d522e/titanic_2', index_col = 0)
data['Age'].fillna(value=data['Age'].median(), inplace=True)
print(data['Age'].isna().sum())

**Explanation:**

```python
.fillna(value=data['Age'].median(), inplace=True)
```
- `value = data['Age'].median()` - using the argument `value`, we tell the `.fillna()` method what to do with the `NaN` values. In this case, we applied the `.fillna()` method to the column `'Age'` and replaced all missing values with the **median** of the column;
- `inplace=True` - the argument we can use for saving changes.

import unittest
import pandas as pd
import io
import sys
import numpy as np


def _dynamic_test(test_case, condition, success_msg, failure_msg):
    if condition:
        test_case._testMethodName = success_msg
        test_case.assertTrue(True, success_msg)
    else:
        test_case._testMethodName = failure_msg
        test_case.fail(failure_msg)


class TestFillNaMean(unittest.TestCase):
    def test_age_filled_with_mean(self):
        """
        1. Check that all NaN values in the 'Age' column were replaced with the mean of that column.
        """
        import user_code

        url = "https://codefinity-content-media.s3.eu-west-1.amazonaws.com/4bf24830-59ba-4418-969b-aaf8117d522e/titanic_2"
        df_ref = pd.read_csv(url, index_col=0)

        # Calculate reference mean (excluding NaN)
        mean_age = df_ref["Age"].mean()

        # User result
        assert hasattr(user_code, "data"), "Variable 'data' not found."
        df_user = user_code.data.copy()

        # Condition 1: no NaN values in Age
        no_nans = df_user["Age"].isna().sum() == 0

        # Condition 2: filled values match mean where they were previously NaN
        mask_nan = df_ref["Age"].isna()
        filled_correctly = True
        if mask_nan.any():
            filled_correctly = np.allclose(
                df_user.loc[mask_nan, "Age"], mean_age, rtol=1e-7, atol=1e-9
            )

        condition = no_nans and filled_correctly
        _dynamic_test(
            self,
            condition,
            "All NaN values in 'Age' were correctly replaced with the mean of that column.",
            "The 'Age' column still has missing values or was not filled with the mean correctly."
        )


class TestCountNaN(unittest.TestCase):
    def test_age_nan_count_zero(self):
        """
        2. Check that the number of missing values in 'Age' is 0 and is printed.
        """
        import user_code

        # Check NaN count variable
        assert hasattr(user_code, "NaN"), "Variable 'NaN' not found."
        nan_value = user_code.NaN
        condition = isinstance(nan_value, (int, np.integer)) and nan_value == 0

        _dynamic_test(
            self,
            condition,
            "The variable storing the NaN count correctly equals 0.",
            "The variable with NaN count is incorrect or not 0."
        )


class TestOutput(unittest.TestCase):
    def test_output_print(self):
        """
        3. Check that the NaN count was printed to the output.
        """
        import user_code
        captured_output = io.StringIO()
        sys.stdout = captured_output
        print(user_code.NaN)
        sys.stdout = sys.__stdout__

        output_text = captured_output.getvalue().strip()
        condition = len(output_text) > 0
        _dynamic_test(
            self,
            condition,
            "The NaN count was printed successfully.",
            "The output is missing. Use print(NaN) to display the NaN count."
        )


if __name__ == "__main__":
    unittest.main()



test_code.py

This course contains a lot of useful functions for a future data analyst. You will learn different ways of extracting data and even set conditions on it. After it, you will be familiar with the methods of grouping data. Also, you will learn how to preprocess data. Each section has its data set so that the course will be gripping.

This section will teach you how to output specific columns by their titles or indices. Also, you will get acquainted with the ways you can select rows  by indices.

Here, you will learn how to extract data that has specific conditions. Also, you will learn how to combine them and even create your own.

In this section, you will expand your knowledge on setting different data conditions. You will learn to check if your data is in a defined list of values or between two values. You will also learn how to find the largest and smallest values.

This section is one of the most fascinating of the course. Here, you will learn how to group data in different ways. It will help you work as a data analyst to find out information on specific data groups.

This section is one of the most significant for a data analyst because if the data contains missing data values in the incorrect format, it will be impossible to work with. Thus, you will learn how to deal with such inappropriate values here. 

Filling In the Missing Values

Solution

Awesome!

Filling In the Missing Values

Solution