VMware ESXi and vSphere Cluster Management

Remove Duplicate Lines from a Text File with uniq in Linux

Learn how to remove, count, and inspect duplicate lines in Linux with uniq, sort, pipelines, redirection, and case-insensitive comparisons.

The Linux uniq command processes repeated text lines. It can remove repeated lines, count occurrences, show only duplicated values, or select values that occur once. Its most important rule is that it compares only neighboring lines.

For that reason, uniq is often combined with sort. Sorting places equal lines next to one another, allowing uniq to process duplicates throughout a file.

What uniq Does

uniq reads lines from a file or from standard input. Standard input is the input stream consumed by a command, usually supplied by a file, a redirection, or another command in a pipeline. Unless you specify an output file, uniq writes its result to standard output, the normal output stream that can be displayed, piped, or redirected.

uniq FILE

This command reads FILE and prints the result in the terminal. It collapses only consecutive equal lines.

You can also provide input through a pipeline. A pipeline uses the | operator to send one command's standard output to the next command's standard input.

some-command | uniq

For example, to deduplicate command locations returned by a shell lookup command:

command -v -a sh | sort | uniq

Adjacent Duplicates: The Essential Limitation

Adjacent duplicates are matching lines located directly next to one another in the input stream. uniq compares each line with the previous line, not with every line anywhere in the file.

alpha
beta
alpha

Running uniq on this input does not remove either copy of alpha, because the two copies are separated by beta.

alpha
alpha
beta

Here, uniq prints one alpha followed by beta. The first line in each consecutive duplicate group is retained.

Why sort Is Commonly Used

sort orders lines. When it runs before uniq, equal lines become adjacent, even if they were scattered through the original file.

sort names.txt | uniq

The usual sort | uniq workflow removes duplicate values regardless of their original positions, but it also changes the order of the data.

Remove Duplicate Lines

To remove duplicates anywhere in a line-based text file, use:

sort names.txt | uniq

The result contains one copy of each distinct line. The first line in each consecutive group produced by sort is retained.

For an input that is already grouped, sorting is not necessary:

uniq grouped.txt

Save the Result to a File

Use shell redirection to write standard output to a new file. Redirection uses operators such as >.

sort names.txt | uniq > names.unique.txt

This creates names.unique.txt and leaves names.txt unchanged. Some implementations also support an output-file argument for uniq, but a pipeline with explicit redirection is widely understood and convenient.

Use a separate file or a temporary-file workflow instead:

tmp=$(mktemp)
sort names.txt | uniq > "$tmp" && mv "$tmp" names.txt

Verify the temporary result before replacing the original when the data is important. A backup can also be created first:

cp names.txt names.txt.bak
sort names.txt | uniq > names.txt.tmp && mv names.txt.tmp names.txt

Display Only Duplicate Lines

The -d option displays one representative line for each repeated adjacent group.

sort names.txt | uniq -d

Sorting first is important when duplicates can appear in arbitrary positions. The command reports duplicated values, not every occurrence of those values. For example, three copies of alice produce one alice line with -d.

sort entries.txt | uniq -d

If you need every occurrence from repeated groups, -d alone is not the right result. On implementations that support it, -D prints all instances from repeated groups:

sort entries.txt | uniq -D

Useful uniq Options

Count Each Distinct Line

Use -c after sorting to produce a count for every distinct line:

sort entries.txt | uniq -c

A typical result might look like this:

      2 error
      5 info
      1 warning

Show Values That Occur Exactly Once

The -u option selects lines whose grouped count is one:

sort entries.txt | uniq -u

This is different from ordinary deduplication. sort | uniq keeps one copy of every value, while sort | uniq -u removes every value that appeared more than once.

Ignore Leading Fields or Characters

Sometimes a line contains a prefix that should not determine duplicate identity. The -f N option skips fields, and -s N skips characters, for comparison purposes.

sort records.txt | uniq -f 1

This compares lines after skipping the first whitespace-separated field. The exact meaning of fields and whitespace follows the implementation's uniq behavior, so consult the local manual for unusual input.

sort records.txt | uniq -s 8

This compares lines after skipping their first eight characters. If you skip a prefix for comparison, make sure the preceding sorting command uses a compatible comparison rule.

Case-Insensitive Deduplication

Default comparisons are usually case-sensitive. To treat capitalization variants as equivalent, use matching case-insensitive options in both commands:

sort -f users.txt | uniq -i

The -f option belongs to sort and means case-insensitive sorting there; the -i option belongs to uniq and means case-insensitive comparison. Both stages need compatible rules so that equal values are grouped correctly.

Sorting Choices and Comparison Consistency

sort and uniq should use compatible comparison rules. If sorting treats two lines as equivalent but uniq does not, or the reverse, the result may not match your intended definition of a duplicate.

Locale settings can affect collation and character comparisons. A locale is a set of environment settings that influences text ordering and comparison.

LC_ALL=C sort FILE | LC_ALL=C uniq

LC_ALL=C requests a predictable byte-oriented comparison that is useful for machine-oriented processing. It is not always the right choice for human-language sorting, so choose a locale based on the data and the desired behavior.

When duplicate identity depends on a selected field, sort by that key before grouping. For example, if the second whitespace-separated field identifies a record:

sort -k2,2 records.txt | uniq -f 1

For structured records, use a consistent field definition and consider tools such as awk when the identity rule is more complex than uniq's field or character skipping options.

Blank Lines and Invisible Differences

Blank lines are lines too. Adjacent blank lines are collapsed by uniq, and all blank lines are grouped together by a normal sort:

sort FILE | uniq

Lines that look identical may differ because of trailing spaces, tabs, carriage-return characters, or different encodings. Inspect nonprinting characters when deduplication does not behave as expected:

cat -vet FILE

After identifying the difference, normalize the input with an appropriate tool such as sed, tr, or awk, then sort and deduplicate it.

Common Troubleshooting Cases

Repeated Lines Remain

The matching lines are probably not adjacent. Use:

sort FILE | uniq

If sorting the file is not acceptable because original order matters, arrange or process records with a method designed to preserve first-seen order.

The Output Order Changed

sort orders the input before uniq sees it. Sort-based deduplication therefore changes the original order. If order must be preserved, use a different deduplication approach, such as an awk script that records previously seen lines.

Case Variants Remain Separate

Default comparisons are case-sensitive. Pair compatible options:

sort -f FILE | uniq -i

The Source File Became Empty

This usually happens when output is redirected to the same pathname as the input. Write to a temporary or separate file, inspect the result, and then replace the original if appropriate.

Duplicate Values Are Reported, but Not Every Occurrence

uniq -d prints one representative line from each repeated group. Use uniq -D where supported when all instances from repeated groups are required, or choose another processing method for portable behavior.

Quick Reference

# Collapse only adjacent duplicates
uniq FILE

# Remove duplicates anywhere, with sorted output
sort FILE | uniq

# Save unique sorted lines
sort FILE | uniq > FILE.unique

# Print one representative of each duplicated value
sort FILE | uniq -d

# Count each distinct value
sort FILE | uniq -c

# Print values that occur exactly once
sort FILE | uniq -u

# Compare case-insensitively
sort -f FILE | uniq -i

# Use a consistent byte-oriented locale
LC_ALL=C sort FILE | LC_ALL=C uniq

In summary, use uniq alone when repeated lines are already adjacent. Use sort | uniq when matching lines may be separated. Select -d, -u, or -c according to whether you need duplicated values, one-time values, or frequency counts. For more practice, see the Linux duplicate-line removal reference.