VMware ESXi and vSphere Cluster Management

Transmission Control Protocol (TCP) Explained

Learn how TCP provides reliable, ordered, connection-oriented communication through handshakes, sequence numbers, acknowledgments, flow control, headers, flags, and connection termination.

Transmission Control Protocol (TCP) is a transport-layer protocol in the TCP/IP suite. It allows applications on different hosts to exchange a reliable, ordered stream of bytes. TCP is widely used by web browsing, secure shell, email, file transfer, and database applications.

This lesson explains TCP's purpose, relationship with IP, connection lifecycle, three-way handshake, reliability mechanisms, flow and congestion control, header fields, options, termination, troubleshooting, and differences from UDP.

TCP's Purpose and Position in the Network Stack

TCP operates at Layer 4, the transport layer, of the OSI model. The transport layer provides communication between application processes rather than merely between hosts.

IP operates below TCP at the Internet layer of the TCP/IP model, commonly mapped to OSI Layer 3. IP delivers packets between host addresses using a best-effort service. It can route packets, but it does not guarantee delivery, ordering, duplicate prevention, or recovery from loss. TCP adds those controls between the applications at the two endpoints.

TCP, IP, Ports, and Encapsulation

TCP provides process-to-process communication. A host can run many network applications at once, so TCP uses 16-bit port numbers to identify services and processes. A server commonly listens on a known port, while a client normally uses an ephemeral, temporary source port.

A socket is an endpoint commonly identified by an IP address and a port number. A TCP conversation is identified using source IP address, source TCP port, destination IP address, and destination TCP port. This combination lets one host maintain many simultaneous connections.

Encapsulation adds headers as data moves down the stack:

1. An application produces application data, such as an HTTP request.
2. TCP accepts the byte stream, adds a TCP header, and creates a TCP segment.
3. IP adds an IP header containing source and destination IP addresses, creating an IP packet.
4. The link layer adds a frame header and trailer for transmission over a local network.

Application data
    becomes a TCP segment
        carried inside an IP packet
            carried inside a link-layer frame

At the receiving host, the headers are removed in reverse order. IP handles host-to-host delivery; TCP handles the application-to-application conversation carried by that delivery.

Connection-Oriented Communication

Before normal application data is exchanged, TCP creates a logical connection. Each endpoint records state such as sequence numbers, acknowledgment information, windows, and connection status. This state allows TCP to recognize missing, duplicated, delayed, or out-of-order data.

Conceptual TCP states include LISTEN for a server waiting for connections, SYN-SENT and SYN-RECEIVED during setup, ESTABLISHED during normal transfer, and closing states such as FIN-WAIT, CLOSE-WAIT, and TIME-WAIT.

Each endpoint selects an initial sequence number. The handshake confirms that both endpoints can communicate in both directions and synchronizes the sequence-number spaces before regular data transfer begins.

The TCP Three-Way Handshake

The three-way handshake establishes a TCP connection. Suppose a client connects to a web server using TCP port 443. The client uses an ephemeral source port, such as 51500, and the server listens on destination port 443.

1. SYN: The client sends a segment with the SYN flag set. It proposes its initial sequence number, such as 1000.
2. SYN-ACK: The server replies with both SYN and ACK set. It acknowledges the client's SYN with acknowledgment number 1001 and proposes its own initial sequence number, such as 7000.
3. ACK: The client acknowledges the server's SYN with acknowledgment number 7001. The connection can now enter ESTABLISHED state and carry application data.

Client: 51500                         Server: 443
  |------ SYN, Seq=1000 ------------------>| 
  |<----- SYN-ACK, Seq=7000, Ack=1001 -----|
  |------ ACK, Ack=7001 ------------------>| 
  |------ HTTPS application data --------->|

The SYN consumes one sequence-number position, which is why the acknowledgment for sequence number 1000 is 1001. A successful handshake confirms bidirectional reachability and sequence-number synchronization.

If no process is listening on the destination port, the host may respond with RST, producing an immediate connection refusal. If firewalls, routing problems, or packet loss silently prevent the exchange, the client commonly retransmits its SYN and eventually times out.

Reliable and Ordered Delivery

TCP numbers bytes, not simply segments. The sequence number identifies the position of the first payload byte in a segment. The acknowledgment number identifies the next byte the receiver expects.

TCP acknowledgments are generally cumulative. An acknowledgment of 3001 means that all bytes through 3000 have been received correctly and that byte 3001 is the next expected byte.

Ordered Reassembly

TCP can receive segments out of order and temporarily store them. For example, imagine these byte ranges:

1. Bytes 1–1000 arrive.
2. Bytes 2001–3000 arrive before bytes 1001–2000.
3. The receiver acknowledges the next missing byte, 1001, because the stream has a gap.
4. Bytes 1001–2000 arrive.
5. The receiver can now reassemble and deliver bytes 1–3000 in order.

The application normally receives an ordered byte stream rather than separate, visibly numbered segments.

Loss, Corruption, and Retransmission

A TCP checksum helps detect accidental corruption in a segment and relevant addressing information. A checksum detects a problem; it does not repair the data. TCP recovers by having the sender retransmit data that appears to be missing.

If an acknowledgment does not arrive before a retransmission timer expires, TCP retransmits. Repeated acknowledgments for the same missing byte range can also indicate a gap. After enough duplicate acknowledgments, TCP may perform fast retransmission without waiting for the timer.

Sender                         Receiver
  |--- bytes 1-1000 ----------->|
  |<-- ACK 1001 -----------------|
  |--- bytes 1001-2000 --X       |  segment lost
  |--- bytes 2001-3000 ---------->|  arrives out of order
  |<-- duplicate ACK 1001 -------|
  |--- retransmit 1001-2000 ---->|
  |<-- ACK 3001 ------------------|

TCP reliability is end-to-end. TCP can deliver bytes correctly to the receiving operating system, but it cannot guarantee that the application successfully parsed, stored, displayed, or acted on the content.

Flow Control and Sliding Windows

The receiver advertises a receive window, also called the advertised window. It tells the sender how much additional unacknowledged data the receiver currently has room to accept.

A sliding window allows multiple bytes or segments to be in transit before an acknowledgment arrives. As the receiver acknowledges data, the usable sending range moves forward, or slides. This is more efficient than sending one segment and waiting after every segment.

If the receiving application is slow or its buffers fill, the advertised window can become small. A window of zero tells the sender to stop sending normal data temporarily. When buffer space becomes available, the receiver sends a window update.

Congestion Control Overview

A reliable protocol must also avoid overwhelming routers and links. TCP therefore maintains a sender-side congestion window, which limits in-flight data based on its estimate of network capacity. The effective sending limit is influenced by both the congestion window and the receiver's advertised window.

• Slow start: The sender begins cautiously and increases its sending rate rapidly while the path appears healthy.
• Congestion avoidance: After reaching a threshold, the sender increases more conservatively.
• Loss detection: A timeout or pattern of duplicate acknowledgments can indicate congestion or loss.
• Rate reduction: The sender reduces its sending rate and then probes for available capacity again.

Exact behavior depends on the congestion-control algorithm implemented by the operating system and TCP stack. Different systems and versions may use different algorithms.

TCP Segment Header

A TCP header is at least 20 bytes long. Options can extend it to a maximum of 60 bytes. The header supports multiplexing, session control, reliability, flow control, and error detection.

TCP Flags

PSH and URG are often misunderstood. PSH is a delivery hint, not a guarantee that an individual application write maps to one segment. URG works with the urgent pointer and is uncommon in many modern applications.

TCP Options

• Maximum Segment Size (MSS): Negotiated during connection establishment. It states the largest TCP payload an endpoint is willing to receive in one segment.
• Window scaling: Extends the effective receive-window range beyond the basic 16-bit field, allowing larger windows on high-bandwidth or high-latency paths.
• Selective Acknowledgment (SACK): Identifies separate blocks of data that arrived successfully, helping the sender recover efficiently when multiple segments are lost.
• Timestamps: Commonly observed options that can assist with round-trip measurement and protection against old duplicate segments.

Options increase the header length and are not necessarily present in every segment. Padding ensures the header ends on the required alignment.

Data Transfer, Segmentation, MSS, and MTU

An application writes a byte stream to TCP. TCP divides that stream into segments according to implementation decisions, the negotiated MSS, available windows, and path conditions. The receiver uses sequence numbers to reconstruct the original stream.

MSS is the largest TCP payload an endpoint is willing to receive. MTU, or Maximum Transmission Unit, is the largest packet payload supported by a link-layer technology. MSS is therefore concerned with TCP payload, while MTU concerns the IP packet size supported by a link.

In general, TCP payload size must leave room for IP and TCP headers within the path MTU. A common Ethernet path with a 1500-byte IP MTU often uses an MSS of 1460 bytes for IPv4 TCP without additional headers, although actual values vary with addressing, options, tunneling, and path configuration.

TCP is full-duplex: data can travel in both directions simultaneously over one connection. Each direction has its own sequence and acknowledgment progress.

TCP Connection Lifecycle

TCP Connection Termination

A graceful TCP close is bidirectional. A typical orderly shutdown uses four logical messages because each direction closes separately:

1. Endpoint A sends FIN, stating that it will send no more data.
2. Endpoint B acknowledges A's FIN.
3. After finishing its own data, endpoint B sends FIN.
4. Endpoint A acknowledges B's FIN.

Client                                      Server
  |----------- FIN ------------------------->|
  |<---------- ACK --------------------------|
  |<---------- FIN --------------------------|
  |----------- ACK ------------------------->|
  |       client may enter TIME_WAIT         |

A half-close occurs when one endpoint has stopped sending but continues receiving. This is useful when one side has finished its request while still expecting a response.

The endpoint that actively closes commonly enters TIME_WAIT. This state allows delayed segments from the old connection to expire and gives the endpoint time to retransmit the final acknowledgment if necessary. TIME_WAIT is not automatically an error; many short-lived outbound connections can create many TIME_WAIT sockets.

An RST is different from FIN. FIN supports an orderly shutdown, while RST immediately aborts or rejects a connection and may discard data that has not been delivered to the application.

TCP Overhead and Tradeoffs

TCP's benefits require overhead: connection setup, headers, acknowledgments, retransmission timers, state tracking, receive buffers, and congestion-control behavior. A new connection can add latency before application data is exchanged, and loss recovery can delay delivery.

TCP is not inherently unsuitable for real-time applications. Suitability depends on whether the application values reliable ordered delivery more than bounded latency and how it handles delay or loss. Applications that need lower protocol overhead or can tolerate loss may choose UDP, while others may implement reliability above UDP.

Common TCP Applications and Ports

Port numbers are conventions that help identify services; they do not prove which application protocol is in use. An administrator can configure HTTPS, SSH, or a database service to listen on a nondefault port.

TCP Versus UDP

UDP applications can add their own reliability, ordering, retransmission, or rate control when required. The choice depends on application requirements rather than a simple rule that TCP is always better or UDP is always faster.

Observing TCP on Linux, Windows, and Wireshark

On Linux, display TCP sockets and their states with:

ss -tan

On Windows, view TCP connections, listening ports, states, and process IDs with:

netstat -ano -p tcp

Generate a TCP-based web connection from Linux or another system with curl:

curl -v https://example.com/

Capture TCP traffic on Linux without name resolution:

sudo tcpdump -nn -i any tcp

In Wireshark, useful display filters include:

tcp
tcp.flags.syn == 1
tcp.analysis.retransmission

The first filter shows TCP segments, the second locates connection-establishment attempts, and the third identifies retransmissions that Wireshark suspects from packet timing and sequence analysis.

TCP Troubleshooting Patterns

Connection Times Out

Likely causes: a firewall silently drops SYN packets or return traffic, routing is incorrect, the destination is unreachable, the server is down, or the path is impaired.

TCP evidence: repeated SYN packets with no SYN-ACK or RST response. Investigate addressing and routing, check firewall rules in both directions, and capture traffic at the client, firewall, and server when possible.

Connection Is Immediately Refused

Likely causes: no process is listening on the destination port, or a host or security device actively rejects the request.

TCP evidence: a SYN followed by RST, often with ACK set depending on the context. Verify that the server process is running, listening on the intended address and port, and configured correctly.

TCP Transfer Is Slow or Stalls

Likely causes: packet loss, a small or zero receive window, network congestion, high round-trip time, or path MTU problems.

TCP evidence: retransmissions, duplicate acknowledgments, zero-window advertisements, window updates, high RTT, or reduced throughput. Inspect captures and check interface errors, packet loss, link saturation, and MTU consistency.

Many Connections Are in TIME_WAIT

A high rate of short-lived outbound connections or ineffective connection reuse can produce many TIME_WAIT sockets. Determine whether the volume is expected, then review keep-alive behavior, connection pooling, and application reuse. Do not treat TIME_WAIT as an error by default.

Exam-Relevant Summary

• TCP is a Layer 4, connection-oriented, reliable, ordered, full-duplex byte-stream protocol.
• IP provides best-effort host-to-host delivery; TCP provides delivery control between application processes.
• Ports identify transport endpoints, and a socket is commonly an IP address plus port.
• The three-way handshake is SYN, SYN-ACK, ACK.
• TCP sequence numbers identify byte positions, and the acknowledgment number is the next expected byte.
• Cumulative acknowledgments, retransmissions, duplicate acknowledgments, and fast retransmission support reliability.
• The receive window provides flow control; the congestion window provides sender-side congestion control.
• The TCP header is at least 20 bytes and can reach 60 bytes with options.
• FIN closes one direction gracefully; RST aborts or rejects a connection.
• TIME_WAIT helps handle delayed segments and final acknowledgments after active closure.
• TCP and UDP solve different application problems; neither is universally superior.

Continue with Transmission Control Protocol (TCP) Explained for a structured review of TCP's transport-layer behavior.