Lære Handling Special Characters and Encodings

When working with real-world text data, you often encounter special characters and encoding issues that can disrupt analysis and downstream processing. Text data may come from diverse sources, such as web scraping, legacy systems, or user-generated content, each introducing its own quirks. Common encoding problems include the presence of unexpected byte sequences, misinterpreted characters, and symbols that do not display correctly. For instance, characters like é, ü, or currency symbols may appear as garbled text if the encoding is not handled properly. Additionally, special characters—such as emojis, mathematical symbols, or non-printable control characters—can interfere with parsing, searching, or feature extraction. A robust data cleaning workflow must detect and resolve these issues to ensure consistency and reliability in your datasets.


              1234567891011121314151617181920212223242526
            
import pandas as pd

# Sample DataFrame with mixed encodings and special symbols
data = {
    "review": [
        "Great product! 👍",
        "Excelente calidad – muy útil.",
        "Terrible… would not buy again! 😡",
        "Preis: 20€",
        "Weird chars: \x93hello\x94 \u2013 test"
    ]
}
df = pd.DataFrame(data)

# Function to clean text: normalize encoding and remove non-ASCII characters
def clean_text(text):
    # Normalize unicode characters and encode to ASCII, ignoring errors
    cleaned = (
        text.encode("ascii", "ignore")
        .decode("ascii")
    )
    return cleaned

df["cleaned_review"] = df["review"].apply(clean_text)

print(df[["review", "cleaned_review"]])

Check the source encoding

Always know the encoding of your raw data files. Use "utf-8" as a default, but try "latin1" or "cp1252" if you encounter decoding errors;

Use Python's built-in tools

The open() function can take an encoding parameter. Try reading a problematic file with different encodings to see which one works;

Leverage pandas for file reading

When loading data with pd.read_csv(), specify the encoding parameter if you suspect issues;

Detect encoding with chardet

The chardet library can help guess the encoding of a file, though it's not always 100% accurate;

Normalize text

Use .encode("ascii", "ignore").decode("ascii") to strip out non-ASCII characters, or unicodedata.normalize() for more advanced normalization;

Replace or remove problematic characters

Use regular expressions or string methods to substitute or eliminate unwanted symbols.

Alt var klart?

Takk for tilbakemeldingene dine!

Seksjon 4. Kapittel 2

Spør AI

Spør om hva du vil, eller prøv ett av de foreslåtte spørsmålene for å starte chatten vår

Suggested prompts:

Can you explain how the clean_text function works in more detail?

What are some alternative ways to handle special characters instead of removing them?

How can I preserve accented characters like é or ü during cleaning?

Sveip for å vise menyen


              1234567891011121314151617181920212223242526
            
import pandas as pd

# Sample DataFrame with mixed encodings and special symbols
data = {
    "review": [
        "Great product! 👍",
        "Excelente calidad – muy útil.",
        "Terrible… would not buy again! 😡",
        "Preis: 20€",
        "Weird chars: \x93hello\x94 \u2013 test"
    ]
}
df = pd.DataFrame(data)

# Function to clean text: normalize encoding and remove non-ASCII characters
def clean_text(text):
    # Normalize unicode characters and encode to ASCII, ignoring errors
    cleaned = (
        text.encode("ascii", "ignore")
        .decode("ascii")
    )
    return cleaned

df["cleaned_review"] = df["review"].apply(clean_text)

print(df[["review", "cleaned_review"]])