Word Frequency: How to Count Words and Find Key Terms

What is word frequency?

Word frequency is a count of how many times each word occurs in a text, often shown as a percentage of all words. It is one of the simplest ways to analyse writing, and the basis of tag clouds, keyword lists and many text analysis methods.

How words are counted

1. Split the text into words, usually at spaces and punctuation.
2. Convert to one case so The and the match.
3. Decide how to treat numbers, hyphens and apostrophes.
4. Count each distinct word and sort by count.

Stop words

Common words such as the, of and and top almost every list. Removing them highlights the topic words. Whether to remove them depends on your goal.

Zipf law

In natural language, a word frequency is roughly inversely proportional to its rank. The most common word appears about twice as often as the second, and three times as often as the third. That is why a handful of words cover a large share of any text.

Counting in code

import re
from collections import Counter
words = re.findall(r"[\w'-]+", text.lower())
print(Counter(words).most_common(10))

Limits to remember

Plain counts treat run, runs and running as different words. Methods such as stemming merge them, but a simple counter does not. Short texts also give noisy results.

Try it

Use the word frequency counter for a full list, the most common words finder for a top list, or the stop word version to focus on topic words.