How to Remove Duplicate Lines

The short answer

The fastest way is an online tool: paste the list and copy the result. For a file you work with often, a text editor command, a spreadsheet function or a short command fits better. The table lists the common choices, and the sections below show how to use each one.

Online

Paste your text into remove duplicate lines, choose whether to ignore case and spaces, and copy the result. The tool also lists the lines that were repeated and how many times. If you only want to see the repeated values, use unique lines.

Excel and Google Sheets

In Excel, put the lines in one column, select it, and choose Data, then Remove Duplicates. Excel deletes later copies and keeps the first, and it reports how many it removed. In Microsoft 365 you can instead use a formula, which leaves your original list untouched:

=UNIQUE(A1:A100)

Google Sheets has the same UNIQUE function, and also Data, Data cleanup, Remove duplicates.

Notepad++

Use Edit, Line Operations, Remove Duplicate Lines to remove repeats anywhere in the file, or Remove Consecutive Duplicate Lines to remove only repeats that sit next to each other. The first command keeps the first copy of each line.

Sublime Text

Select the lines, then choose Edit, Permute Lines, Unique. VS Code has no built-in command for this. A common route is to paste the lines into an online tool, or to install an extension that removes duplicate lines.

Command line

Two commands do most of the work on Linux and macOS:

# keep the original order, remove all repeats
awk '!seen[$0]++' file.txt

# sort the lines and remove repeats
sort -u file.txt

# count how often each line appears
sort file.txt | uniq -c | sort -rn

The uniq command on its own only removes repeats that sit on neighbouring lines, so it needs a sorted file. In PowerShell:

Get-Content file.txt | Select-Object -Unique

In code

// JavaScript: keeps the first copy, keeps the order
const unique = [...new Set(text.split(/\r?\n/))].join("\n");

# Python
lines = open("file.txt").read().splitlines()
unique = list(dict.fromkeys(lines))

// PHP
$unique = array_values(array_unique(explode("\n", $text)));

-- SQL
SELECT DISTINCT line FROM list;

In Python, dict.fromkeys keeps the order, while turning the list into a plain set does not. In PHP, array_unique keeps the first copy and leaves gaps in the keys, which array_values fixes.

Why duplicates are not removed

• Trailing spaces. apple and apple are different lines. Trim every line first.
• Different case. Most tools treat Apple and apple as different unless you ask them to ignore case.
• Line endings. A file with Windows endings (CRLF) can leave a hidden carriage return on every line, so lines from two files never match.
• Invisible characters. A non-breaking space or a zero-width character makes two identical looking lines different.

Keep the first copy or the last?

Most tools keep the first copy. Keeping the last copy matters when later lines are newer, for example a log or a list of updates, because you want the latest entry for each key. The online tool lets you choose.

Related tools

Besides remove duplicate lines and unique lines, you can use remove empty lines, remove duplicate words, sort lines, reverse lines and shuffle lines. To check the length of a list, see the guide to counting lines.