Remove Duplicate Lines from a Text File in Linux with sort and uniq

Learn how to remove, inspect, count, and display duplicate lines in Linux using sort, uniq, pipelines, and safe output files.

Linux provides two simple commands for working with repeated lines in text files: sort and uniq. The usual pattern is sort FILE | uniq. Sorting places matching lines next to one another, and uniq keeps one copy of each adjacent group.

This lesson assumes that you can run commands in a terminal, name files, use the pipe operator (|), and redirect output with >. For more shell background, see Bourne Again Shell Bash.

What uniq Does

uniq processes repeated text lines. Its normal behavior is to retain one instance of each run of identical adjacent lines and write the result to standard output.

Standard output is the normal output stream produced by a command. You can view it in the terminal, send it through a pipe, or save it with output redirection. A file argument tells uniq where to read its input:

uniq FILE

When no output file is specified, the result goes to standard output. Save it with >:

uniq FILE > OUTPUT_FILE

A duplicate line is a line whose compared content matches another line. By default, comparison is exact: differences in uppercase and lowercase letters, spaces, tabs, and other characters matter.

Why uniq Usually Needs sort

uniq compares only adjacent lines, meaning lines next to one another in the input stream. It does not search the entire file for every matching line.

For example, given this file:

red
blue
red

The two red lines are separated by blue, so this command does not remove either one:

uniq colors.txt

To remove duplicates across the entire file, sort the input first:

sort colors.txt | uniq

A pipe is the shell operator |. It sends the standard output of the command on its left to the standard input of the command on its right. Here, sort orders the lines, making equal lines adjacent, and uniq emits one copy from each group.

Sorting changes the order of the data. The result is sorted rather than arranged in the original file order.

Basic Duplicate-Removal Workflow

Display unique lines

Use this command when you only need to inspect the result:

sort names.txt | uniq

The command reads names.txt, sorts its lines, and prints one copy of each distinct line.

Save the result in a new file

Use a separate destination when you want to preserve the source for review:

sort names.txt | uniq > names-unique.txt

The original names.txt remains unchanged. The cleaned result is stored in names-unique.txt.

Safety note: Do not use the source file as the direct redirection target in the same pipeline, such as sort names.txt | uniq > names.txt. The shell can truncate the destination before the command has finished reading it. Write to a temporary or clearly named file, verify it, and replace the original only afterward.

Replace the original only after checking

Preview the saved file before deciding whether it should replace the source:

sort names.txt | uniq > names-unique.txt
head names-unique.txt
wc -l names.txt names-unique.txt

wc -l compares line counts. The counts may differ because duplicates were removed. After checking the content and counts, use a separate file operation to replace the original if that is really required.

Common uniq Options

No option — Retain one line from each adjacent run. Typical use: sort FILE | uniq.

-d — Output each duplicated adjacent run once. Typical use: sort FILE | uniq -d.

-u — Output only lines that are not repeated in their adjacent run. After sorting, these are lines occurring exactly once.

-c — Prefix each distinct adjacent line with its occurrence count. Typical use: sort FILE | uniq -c.

Displaying Duplicated Lines with uniq -d

Use -d when you want to identify values that occur more than once rather than create a deduplicated file:

sort names.txt | uniq -d

Each duplicated value is displayed once. A value occurring five times still appears only once in this output. The input should be sorted first when matching lines may be located in different parts of the file.

This is different from ordinary duplicate removal:

sort FILE | uniq prints one copy of every distinct line.
sort FILE | uniq -d prints only lines whose count is greater than one.

Showing Lines That Occur Exactly Once

Use -u to print only lines that occur once after the input has been grouped:

sort names.txt | uniq -u

This is useful when you want to find values that have no duplicate. Without sorting, uniq -u only examines adjacent runs, so it cannot reliably identify globally unique lines in an unsorted file.

Counting Occurrences with uniq -c

The -c option prefixes each distinct line with its occurrence count:

sort names.txt | uniq -c

Example output might look like this:

      2 Alice
      1 Bob
      3 Carol

The counts are counts of consecutive occurrences in the input stream. Sorting first makes each value's total file frequency appear as one count.

Frequency output is useful for auditing data before removing duplicates. You can sort the count output to inspect the most frequent values, but remember that the count is printed as a field before the original line:

sort names.txt | uniq -c | sort -n

For a simple count-based filter, you can use awk:

sort names.txt | uniq -c | awk '$1 > 1 {print $2}'

This example prints the second whitespace-separated field for repeated values. It is best suited to lines without leading whitespace or complex field requirements. For general lines that may contain spaces, uniq -d is safer because it does not split the line into fields.

When to Use sort Before uniq

Duplicates already adjacent — Use uniq FILE. The matching lines are already grouped, so sorting is unnecessary if preserving their current order matters.

Duplicates scattered throughout the file — Use sort FILE | uniq. Equal lines become adjacent and are reduced to one copy.

Need to preserve original line order — The basic sort-and-uniq pattern is not suitable because sorting changes order. Use an order-preserving deduplication method instead, depending on your shell and processing tools.

sort -u as a Shortcut

sort can also remove duplicate lines directly with its -u option:

sort -u names.txt

This is often equivalent to:

sort names.txt | uniq

The explicit pipeline is valuable for learning and for combining sorting with modes such as uniq -d, uniq -u, or uniq -c.

Input and Output Details

File arguments and standard input

A command can read from a named file or from standard input. In this pipeline, sort names.txt reads a file and sends sorted text to standard output. The pipe connects that output to uniq's standard input.

sort names.txt | uniq

You can also provide input to uniq through a pipe from another command:

cat names.txt | uniq

This works when duplicates are already adjacent, but using cat here is unnecessary. Prefer the direct form uniq names.txt when no preceding processing is needed.

Blank lines

Blank lines are lines too. Consecutive blank lines are treated as repeated lines and can be reduced by uniq. With scattered blank lines, sorting groups them along with other identical lines:

sort notes.txt | uniq

Case and whitespace

These lines are not identical under normal comparison:

Linux
linux
Linux 
Linux<TAB>

The differences are case, a trailing space, and a tab. Inspect suspicious data carefully and normalize it before deduplication if those differences should not matter. Depending on the data, suitable sort or uniq options and preprocessing tools may help, but changing comparison rules can also change the meaning of the data.

Safe Verification Checklist

Choose a clearly named destination, such as names-unique.txt or a temporary file.
Run the sort-and-uniq pipeline into that destination.
Preview the result with a command such as head or less.
Compare appropriate line counts with wc -l.
Check whether sorting the lines is acceptable for your use case.
Only then replace the source, if replacement is required.

sort names.txt | uniq > names-unique.txt
head names-unique.txt
wc -l names.txt names-unique.txt

Troubleshooting

Duplicate-looking lines remain

Cause: The matching lines were not adjacent. uniq does not search the whole file.

Fix: Sort first:

sort FILE | uniq

The output order changed

Cause: sort orders the data so that matching lines can be grouped.

Fix: Accept the sorted output when order is unimportant. If original order must be preserved, use a different order-preserving method rather than the basic sort-and-uniq pattern.

Similar lines are not considered duplicates

Cause: The lines differ in case, spaces, tabs, or another character.

Fix: Inspect invisible characters and decide whether the data should be normalized before comparison. Do not remove meaningful formatting accidentally.

The output is empty or the source was damaged

Cause: Redirection used the wrong path or targeted the source file itself. Shell redirection can truncate a destination before a pipeline reads it.

Fix: Use a separate output file, verify it, and then rename or replace the original only after the result is confirmed.

uniq -d does not show every repeated occurrence

Cause: -d prints one representative line for each repeated group, not every copy.

Fix: Use -c when you need the number of occurrences:

sort FILE | uniq -c

Quick Reference

uniq FILE — reduce consecutive identical lines to one.
sort FILE | uniq — remove duplicates across the whole file, with sorted output.
sort FILE | uniq > OUTPUT_FILE — save the result without changing the source.
sort FILE | uniq -d — display each duplicated value once.
sort FILE | uniq -u — display values occurring exactly once.
sort FILE | uniq -c — display each distinct value with its frequency.
sort -u FILE — use sort's built-in unique-output mode.

The central rule is simple: use uniq for adjacent repeated lines, and use sort before uniq when duplicates may be separated throughout the file.