Notice: This page requires JavaScript to function properly.
Please enable JavaScript in your browser settings or update your browser.Lemmatization | Stemming and Lemmatization
Introduction to NLP
course content

Course Content

Introduction to NLP

Introduction to NLP

1. Text Preprocessing Fundamentals
NLP and Its ApplicationsUnderstanding Text PreprocessingTokenizationTokenize a Sentence Tokenization Using Regular ExpressionsTokenize Using RegexRemoving Stop WordsApplying Text Preprocessing in Practice
2. Stemming and Lemmatization
StemmingStem the TokensLemmatizationPart of Speech TaggingLemmatization with POS Tagging
3. Basic Text Models
Overview of Vector Space ModelsKey Types of Vector Space ModelsBag of WordsCustomizing Bag of WordsCreate a Bag of WordsTF-IDFImplementing TF-IDF
4. Word Embeddings
Word Embeddings BasicsCBoW and Skip-gram Models (Optional)Implementing Word2VecCreate Word Embeddings

Lemmatization

Lemmatization vs Stemming

First, let's define what lemmatization is and how it is different from stemming.

Unlike stemming, which crudely chops off word endings, lemmatization considers the context and converts the word to its meaningful base form. For example, 'am', 'are', and 'is' are all lemmatized into 'be'. This approach can significantly reduce the size of the vocabulary (the number of unique words) in large text corpora, thereby increasing efficiency when training models.

On the other hand, while lemmatization is more accurate, it is also more computationally expensive and can be time-consuming with large datasets. Furthermore, for even better accuracy, performing morphological analysis and part of speech tagging is recommended before lemmatization.

Lemmatization with NLTK

The WordNet Lemmatizer, provided by the NLTK library, leverages the WordNet corpus to perform lemmatization.

When you use the WordNet Lemmatizer, it looks up the target word in the WordNet database to find the most appropriate lemma (base form) of the word.

As mentioned above, because words can have different meanings in different contexts (e.g., "running" as a verb vs. "running" as a noun), the lemmatizer may require you to specify the part of speech (e.g., verb, noun, adjective). This helps it select the correct lemma based on the word's role in a sentence.

Let's now take a look at an example:


              
12345678910

            
from nltk.stem import WordNetLemmatizer
import nltk
# Download the WordNet corpus
nltk.download('wordnet')
# Initialize the WordNet lemmatizer
lemmatizer = WordNetLemmatizer()
words = ["running", "bested"]
# Lemmatize words
lemmatized_words = [lemmatizer.lemmatize(word, 'v') for word in words]  # 'v' for verb
print("Lemmatized words:", lemmatized_words)
copy

As you can see, from the coding perspective, the approach is rather straightforward. Once again, in order to lemmatize words accurately across an entire corpus, it would be best to first perform part-of-speech (POS) tagging, which we will cover in the following chapter.

Task

Your task is to lemmatize the tokens, given the text string. Tokenization is already applied with stop words filtered out.

  1. Import the WordNet lemmatizer.
  2. Download the WordNet corpus.
  3. Initialize the WordNet lemmatizer.
  4. Lemmatize the tokens using list comprehension.

Task

Your task is to lemmatize the tokens, given the text string. Tokenization is already applied with stop words filtered out.

  1. Import the WordNet lemmatizer.
  2. Download the WordNet corpus.
  3. Initialize the WordNet lemmatizer.
  4. Lemmatize the tokens using list comprehension.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Section 2. Chapter 3
toggle bottom row

Lemmatization

Lemmatization vs Stemming

First, let's define what lemmatization is and how it is different from stemming.

Unlike stemming, which crudely chops off word endings, lemmatization considers the context and converts the word to its meaningful base form. For example, 'am', 'are', and 'is' are all lemmatized into 'be'. This approach can significantly reduce the size of the vocabulary (the number of unique words) in large text corpora, thereby increasing efficiency when training models.

On the other hand, while lemmatization is more accurate, it is also more computationally expensive and can be time-consuming with large datasets. Furthermore, for even better accuracy, performing morphological analysis and part of speech tagging is recommended before lemmatization.

Lemmatization with NLTK

The WordNet Lemmatizer, provided by the NLTK library, leverages the WordNet corpus to perform lemmatization.

When you use the WordNet Lemmatizer, it looks up the target word in the WordNet database to find the most appropriate lemma (base form) of the word.

As mentioned above, because words can have different meanings in different contexts (e.g., "running" as a verb vs. "running" as a noun), the lemmatizer may require you to specify the part of speech (e.g., verb, noun, adjective). This helps it select the correct lemma based on the word's role in a sentence.

Let's now take a look at an example:


              
12345678910

            
from nltk.stem import WordNetLemmatizer
import nltk
# Download the WordNet corpus
nltk.download('wordnet')
# Initialize the WordNet lemmatizer
lemmatizer = WordNetLemmatizer()
words = ["running", "bested"]
# Lemmatize words
lemmatized_words = [lemmatizer.lemmatize(word, 'v') for word in words]  # 'v' for verb
print("Lemmatized words:", lemmatized_words)
copy

As you can see, from the coding perspective, the approach is rather straightforward. Once again, in order to lemmatize words accurately across an entire corpus, it would be best to first perform part-of-speech (POS) tagging, which we will cover in the following chapter.

Task

Your task is to lemmatize the tokens, given the text string. Tokenization is already applied with stop words filtered out.

  1. Import the WordNet lemmatizer.
  2. Download the WordNet corpus.
  3. Initialize the WordNet lemmatizer.
  4. Lemmatize the tokens using list comprehension.

Task

Your task is to lemmatize the tokens, given the text string. Tokenization is already applied with stop words filtered out.

  1. Import the WordNet lemmatizer.
  2. Download the WordNet corpus.
  3. Initialize the WordNet lemmatizer.
  4. Lemmatize the tokens using list comprehension.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Section 2. Chapter 3
toggle bottom row

Lemmatization

Lemmatization vs Stemming

First, let's define what lemmatization is and how it is different from stemming.

Unlike stemming, which crudely chops off word endings, lemmatization considers the context and converts the word to its meaningful base form. For example, 'am', 'are', and 'is' are all lemmatized into 'be'. This approach can significantly reduce the size of the vocabulary (the number of unique words) in large text corpora, thereby increasing efficiency when training models.

On the other hand, while lemmatization is more accurate, it is also more computationally expensive and can be time-consuming with large datasets. Furthermore, for even better accuracy, performing morphological analysis and part of speech tagging is recommended before lemmatization.

Lemmatization with NLTK

The WordNet Lemmatizer, provided by the NLTK library, leverages the WordNet corpus to perform lemmatization.

When you use the WordNet Lemmatizer, it looks up the target word in the WordNet database to find the most appropriate lemma (base form) of the word.

As mentioned above, because words can have different meanings in different contexts (e.g., "running" as a verb vs. "running" as a noun), the lemmatizer may require you to specify the part of speech (e.g., verb, noun, adjective). This helps it select the correct lemma based on the word's role in a sentence.

Let's now take a look at an example:


              
12345678910

            
from nltk.stem import WordNetLemmatizer
import nltk
# Download the WordNet corpus
nltk.download('wordnet')
# Initialize the WordNet lemmatizer
lemmatizer = WordNetLemmatizer()
words = ["running", "bested"]
# Lemmatize words
lemmatized_words = [lemmatizer.lemmatize(word, 'v') for word in words]  # 'v' for verb
print("Lemmatized words:", lemmatized_words)
copy

As you can see, from the coding perspective, the approach is rather straightforward. Once again, in order to lemmatize words accurately across an entire corpus, it would be best to first perform part-of-speech (POS) tagging, which we will cover in the following chapter.

Task

Your task is to lemmatize the tokens, given the text string. Tokenization is already applied with stop words filtered out.

  1. Import the WordNet lemmatizer.
  2. Download the WordNet corpus.
  3. Initialize the WordNet lemmatizer.
  4. Lemmatize the tokens using list comprehension.

Task

Your task is to lemmatize the tokens, given the text string. Tokenization is already applied with stop words filtered out.

  1. Import the WordNet lemmatizer.
  2. Download the WordNet corpus.
  3. Initialize the WordNet lemmatizer.
  4. Lemmatize the tokens using list comprehension.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Lemmatization vs Stemming

First, let's define what lemmatization is and how it is different from stemming.

Unlike stemming, which crudely chops off word endings, lemmatization considers the context and converts the word to its meaningful base form. For example, 'am', 'are', and 'is' are all lemmatized into 'be'. This approach can significantly reduce the size of the vocabulary (the number of unique words) in large text corpora, thereby increasing efficiency when training models.

On the other hand, while lemmatization is more accurate, it is also more computationally expensive and can be time-consuming with large datasets. Furthermore, for even better accuracy, performing morphological analysis and part of speech tagging is recommended before lemmatization.

Lemmatization with NLTK

The WordNet Lemmatizer, provided by the NLTK library, leverages the WordNet corpus to perform lemmatization.

When you use the WordNet Lemmatizer, it looks up the target word in the WordNet database to find the most appropriate lemma (base form) of the word.

As mentioned above, because words can have different meanings in different contexts (e.g., "running" as a verb vs. "running" as a noun), the lemmatizer may require you to specify the part of speech (e.g., verb, noun, adjective). This helps it select the correct lemma based on the word's role in a sentence.

Let's now take a look at an example:


              
12345678910

            
from nltk.stem import WordNetLemmatizer
import nltk
# Download the WordNet corpus
nltk.download('wordnet')
# Initialize the WordNet lemmatizer
lemmatizer = WordNetLemmatizer()
words = ["running", "bested"]
# Lemmatize words
lemmatized_words = [lemmatizer.lemmatize(word, 'v') for word in words]  # 'v' for verb
print("Lemmatized words:", lemmatized_words)
copy

As you can see, from the coding perspective, the approach is rather straightforward. Once again, in order to lemmatize words accurately across an entire corpus, it would be best to first perform part-of-speech (POS) tagging, which we will cover in the following chapter.

Task

Your task is to lemmatize the tokens, given the text string. Tokenization is already applied with stop words filtered out.

  1. Import the WordNet lemmatizer.
  2. Download the WordNet corpus.
  3. Initialize the WordNet lemmatizer.
  4. Lemmatize the tokens using list comprehension.

Switch to desktop for real-world practiceContinue from where you are using one of the options below
Section 2. Chapter 3
Switch to desktop for real-world practiceContinue from where you are using one of the options below
We're sorry to hear that something went wrong. What happened?
some-alt