Words Frequency

The main goal of this chapter is to quantify the distribution of words across sentences within a given text. By determining how many sentences each unique word appears in, we aim to lay the foundation for calculating the Inverse Sentence Frequency (ISF) part of the TF-ISF score.

Setting Up a Counting Mechanism

Dictionary Initialization: We start by creating an empty dictionary named word_sentence_counts. This dictionary is designed to map each unique word to the number of sentences it appears in. The key-value pairs consist of the word as the key and its sentence occurrence count as the value.

Processing Each Sentence

Iterating Through Tokenized Sentences: The code loops through each sentence in the tokenized_sentences list, which contains sentences that have already been split into individual words (tokens).

Updating Word Counts

Word Presence Check: For every unique word in a sentence, the code checks if that word already exists in the word_sentence_counts dictionary.

• New Words: If a word is not found in the dictionary, it implies that this is the first sentence in which the word has been encountered. Consequently, the word is added to the dictionary with a count of 1;

• Existing Words: If the word is already in the dictionary, its count is incremented by 1, reflecting its appearance in an additional sentence.