Three meanings of unique
People ask for unique lines and mean different things. This tool covers all three:
- One copy of each line (distinct): every different line, shown once. This is the same as removing duplicates.
- Only lines that appear once: lines that have no copy anywhere in the list. A line that appears twice is dropped completely.
- Only lines that repeat: lines that appear more than once, each shown one time. Use it to find the duplicates in a list.
Example
For the list red, green, blue, red, yellow, green, red:
- One copy of each gives red, green, blue, yellow.
- Only lines that appear once gives blue and yellow.
- Only lines that repeat gives red and green.
Counts
Switch on Add counts to put the number of times each line appears in front of it, separated by a tab. Sort by count to see the most frequent lines first. The result pastes into a spreadsheet as two columns.
Options
- Ignore case treats Red and red as the same line.
- Trim spaces ignores spaces at the start and end of lines.
- Ignore empty lines leaves blank lines out of the count.
- Sort by first appearance, A to Z, Z to A, or by count.
How it compares with the uniq command
The Unix uniq command only compares lines that sit next to each other, so the file has to be sorted first. This tool compares every line with every other line, whatever the order, so no sorting is needed. The options match uniq -c for counts, uniq -u for lines that appear once, and uniq -d for lines that repeat.
Related tools
To simply delete the repeats and keep the rest of the list, use remove duplicate lines. Tidy blank lines with remove empty lines, count the words in a text with the word frequency counter, or order a list with sort lines. The remove duplicate lines guide shows other ways to do it.