TCP window-size scaling
TCP window scaling is a key design in the TCP protocol that improves network performance. In this document, we will dive into this topic to see how TCP window scale works in Linux. The following are some of the things we will cover:
- What is TCP window scaling?
- Benefits of TCP window scaling
- How to check TCP window scaling in Linux?
- How to enable TCP window scaling?
- Tips for TCP window scaling
- Example of TCP window scaling
What is TCP window scaling?
The TCP Window Scale option allows for a window size larger than 65K bytes by using a scale factor to multiply the window size value. This factor is set based on the maximum size of the receive buffer used by TCP sockets.
In the following example, the maximum size of the receive buffer is 4194304, so the scale factor is 7.
net.ipv4.tcp_rmem = 4096 87380 4194304
Note: The largest supported scaling factor is 14. This allows TCP window sizes of up to one Gigabyte.
We can get more information from here. https://www.ietf.org/rfc/rfc1323.txt
Benefits of TCP window scaling
The base window size cannot exceed 65535 bytes due to the limitations of the TCP header. This makes the window size too small.
This option improves network bandwidth a lot. We can get the window size up to 1GB.
Here’s the formula for calculating the maximum throughput of a single TCP connection:
Window size / (RTT latency in milliseconds / 1,000) = maximum bytes/second
This table shows the maximum megabytes/per second throughput of a single TCP connection.
After TCP window scaling is enabled, we can see that the bandwidth improves significantly.
How to Check TCP Window Scaling in Linux?
This is enabled by default in most Linux distributions. We can check it by using the following command.
cat /proc/sys/net/ipv4/tcp_window_scaling
If the output is 1, that means TCP window scaling is enabled.
If the output is 0, that means TCP window scaling is disabled.
How to Enable TCP Window Scaling?
We can use the following command to enable TCP window scaling.
sysctl -w net.ipv4.tcp_window_scaling=1
To make this reboot persistent, we can add the following line to /etc/sysctl.conf
net.ipv4.tcp_window_scaling=1
Tips for TCP Window Scaling
The window scaling option only appears in the first two packets of a TCP connection, and it cannot be changed afterward.
This only works by enabling this option on both the sender and receiver ends. This would not improve the performance of an existing connection since the TCP handshake was already completed. We need to restart the TCP connection to utilize this.
Window Scaling only works if both sides include it as an option during the initial 3-way handshake (even if their multipliers are different). The protocol recommends using the option even if the shift-count is 0 (meaning 2^0, which would multiply the window size by 1). That way, if one side doesn’t plan on using scaling, it doesn’t prevent the other side from taking advantage of the increased buffer space.
Example of TCP Window Scaling
The scp speed between our two US data centers was about 1-2MB/s. From the network packets, it turns out that this option was not enabled on one server.
After TCP window scaling is enabled, the scp speed reaches 40MB/s. This is a huge improvement.
Reference
https://cloud.google.com/architecture/tcp-optimization-for-network-performance-in-gcp-and-hybridhttps://datatracker.ietf.org/doc/html/rfc1323#page-8https://datatracker.ietf.org/doc/html/rfc7323