VMware ESXi and vSphere Cluster Management

Pipe Data Between Programs in Linux

Learn how Linux pipes connect one command's standard output to another command's standard input, with examples using ps, sort, tail, grep, and redirection.

What Is a Linux Pipe?

A pipe is a shell connection that sends one program's standard output directly to another program's standard input. The pipe operator is the vertical bar character: |.

A pipeline is a chain of two or more commands connected with pipe operators. Each command performs one step, allowing small utilities to process data together instead of requiring one large command.

command1 | command2

In this example, command1 produces data and command2 consumes it. Data moves from left to right.

Standard Input, Standard Output, and Standard Error

Linux commands commonly use three standard streams:

• Standard input, or stdin, is the default stream from which a program reads data.
• Standard output, or stdout, is the default stream for normal command results.
• Standard error, or stderr, is the stream for diagnostic and error messages.

A normal pipe forwards standard output only. It does not automatically forward standard error.

command1 | command2

Here, standard output from command1 becomes standard input for command2. Error messages from command1 normally still appear in the terminal. To send both normal output and errors through the pipe, redirect standard error into standard output first:

command1 2>&1 | command2

A pipe is different from saving output in an intermediate file. A pipe connects programs directly, while a file stores the result for later use.

Basic Pipe Syntax

command1 | command2

The shell evaluates the pipeline from left to right:

1. The left command runs and produces standard output.
2. The pipe connects that output to the next command's standard input.
3. The right command reads and processes the incoming data.

Whitespace around | is customary because it improves readability, but whitespace itself does not create the pipe. The pipe operator does.

command1|command2
command1 | command2

Both forms use a pipe, although the second form is easier to read.

Pipe a Process Listing to sort

ps displays process information. The -A option asks ps to list processes across the system.

ps -A

The output is a group of text lines. You can send those lines to sort:

ps -A | sort

ps -A produces the process-list text. sort receives that text through standard input and sorts the lines. It sorts the lines received from ps; it is not reading a process-list file directly.

Sort by a Specific Field

Use sort -k to select a field, also called a sort key. In the following example, -k 4 sorts according to the fourth whitespace-separated field. For this process-list example, that field is identified as the executable or command field.

ps -A | sort -k 4

Field-based sorting depends on the actual layout of the input. If the command output has a heading or different spacing, inspect the unpiped output before choosing a field number.

Build a Multi-Command Pipeline

A pipeline can have more than two stages. This example lists processes, sorts them by the fourth field, and displays the final ten lines:

ps -A | sort -k 4 | tail

The stages work as follows:

tail displays the end of its input. It displays ten lines by default. Select a different number with -n:

ps -A | sort -k 4 | tail -n 5

ps -A | sort -k 4 | tail -n 10

In the first command, only the final five lines are displayed. The second command explicitly requests ten lines.

Common Pipeline Roles

Many Unix utilities are designed to read text lines from standard input. A useful pipeline often gives each command one focused role:

• Generating data: commands such as ps produce text.
• Filtering: grep keeps lines that match a pattern.
• Sorting: sort orders input lines.
• Counting: wc counts lines, words, or bytes.
• Selecting output: head and tail display portions of the input.

For example, this pipeline filters lines containing error, without case sensitivity, and counts the matching lines:

grep -i "error" application.log | wc -l

grep reads the file and outputs matching lines. wc -l reads those lines from standard input and counts them.

Command documentation usually indicates whether a utility accepts standard input. Check its manual page when you are unsure:

man command-name

How Pipelines Behave

• Each command in a pipeline runs as a separate process.
• Commands can process streamed output, so you do not need to manually copy the complete result into another command.
• The displayed result normally comes from the last command in the pipeline.
• The commands can begin working while earlier commands are still producing data.

For example, in ps -A | sort -k 4 | tail, the terminal normally displays the output from tail, not separate final displays from ps and sort.

Pipeline Exit Status

In many shells, the exit status of a pipeline is normally the status of its final command. This can make a pipeline appear successful even when an earlier command failed.

In compatible shells, enable stricter behavior with:

set -o pipefail

With pipefail, the pipeline reports failure when a command in the pipeline fails, rather than considering only the final command. This is especially useful in shell scripts.

Pipes Compared with Redirection

Redirection changes where a command gets input or sends output. A pipe connects one running program to another; redirection usually connects a program to a file.

You can combine a pipeline with output redirection. In this example, the pipe connects ps and sort, and > saves the final result:

ps -A | sort -k 4 > sorted-processes.txt

You can also provide a file as standard input to the first command and then pipe its output onward:

command1 < input.txt | command2

Troubleshooting Pipelines

Error messages do not enter the next command

Cause: A normal pipe forwards standard output, while diagnostic messages use standard error.

Solution: Redirect standard error into standard output before the pipe when you intentionally want to process both streams.

command1 2>&1 | command2

The sort order is unexpected

Cause: The selected sort key may not match the input's field layout, or a heading line may be included.

Solution: Run the producing command by itself, inspect its columns, and then adjust the key or handle the heading before sorting.

tail displays less data than expected

Cause: tail intentionally limits output to the ending portion, ten lines by default.

Solution: Use tail -n with the required count, use head for the beginning, or remove the limiting stage.

An earlier failure is hidden

Cause: Many shells use the exit status of the last pipeline command by default.

Solution: Check individual commands when necessary, or use set -o pipefail in a compatible shell.

A command does not accept piped input

Cause: Not every command reads its primary input from standard input. Some require file arguments or special options.

Solution: Read the command's manual page. Look for standard-input support or an option that enables it. If necessary, use a temporary file or choose another compatible command.

Key Points

• A pipe uses | to connect one command's standard output to the next command's standard input.
• Data flows from left to right through a pipeline.
• A normal pipe transfers standard output, not standard error.
• Each pipeline stage is a separate process that can transform streamed text.
• ps -A | sort -k 4 | tail demonstrates generating, sorting, and selecting data.
• >, >>, and < perform file redirection rather than command-to-command piping.

For a focused reference to this subject, see piping data between Linux programs.