Chapter 4 TCP/IP Tunable Parameters

This section describes the TCP/IP tunable parameters. For information on kernel tunables, see Chapter 2, Solaris Kernel Tunables. For information on NFS tunables, see Chapter 3, NFS Tunable Parameters.

• IP Tunable Parameters

• TCP Tunable Parameters

• UDP Tunable Parameters

• Per-Route Metrics

Overview of Tuning TCP/IP Parameters

You can set all of the tuning parameters described in this chapter with the ndd command, except for the following two parameters that can only be set in the /etc/system file:

• tcp_conn_hash_size

• ipc_tcp_conn_hash_size

Use the following syntax to set TCP/IP parameters with the ndd command.

# ndd -set driver parameter

For example, the following ndd command disables IP forwarding.

# ndd -set /dev/ip ip_forwarding 0

See ndd(1M) for more information.

To set a TCP/IP parameter across system reboots, include the appropriate ndd command in a system startup script. Use the following guidelines to create a system startup script to include ndd commands:

• Create a script in the /etc/init.d directory and create links to it in the /etc/rc2.d, /etc/rc1.d, and /etc/rcS.d directories.

• The script should run between the existing S69inet and S72inetsvc scripts.

• Name the script with the S70 or S71 prefix. Scripts with the same prefix are run in some sequential way so it doesn't matter if there is more than one script with the same prefix.

• See the README file in the /etc/init.d directory for more information on naming run control scripts.

See “Run Control Scripts” in System Administration Guide, Volume 1 for more information on creating a startup script.

TCP/IP Parameter Validation

All of the TCP/IP parameters described in this section are checked to verify they fall in the parameter range, which is provided in each tunable section, except for the two parameters that can be set only in the /etc/system file described above. See the validation section for tcp_conn_hash_size and ipc_tcp_conn_hash_size for more information.

Internet Request for Comments (RFCs)

Internet protocol and standard specifications are described in RFC documents. You can get copies of RFCs by using anonymous ftp to the sri-nic.arpa machine. Browse RFC topics by viewing the rfc-index.txt file at this site.

IP Tunable Parameters

This section describes some of the IP tunable parameters.

ip_icmp_err_interval and ip_icmp_err_burst

Description

Control the rate of IP in generating IPv4 or IPv6 ICMP error messages. IP generates only up to ip_icmp_err_burst IPv4 or IPv6 ICMP error messages in any ip_icmp_err_interval. This parameter protects IP from denial of service attacks. Set ip_icmp_err_interval to 0 to disable IP to generate IPv4 or IPv6 ICMP error messages.

Default

100 milliseconds for ip_icmp_err_interval

10 for ip_icmp_err_burst

Range

0 - 99,999 milliseconds for ip_icmp_err_interval

1 - 99,999 for ip_icmp_err_burst

Dynamic?

Yes

When to Change

Change the parameter values if you need a higher error message generation rate for diagnostic purposes.

Commitment Level

Unstable

ip_forwarding and ip6_forwarding

Description

Control whether IP does IPv4 or IPv6 forwarding between interfaces. See also xxx:ip_forwarding below.

Default

0 (disabled)

Range

0 (disabled), 1 (enabled)

Dynamic?

Yes

When to Change

If IP forwarding is needed, enable it.

Commitment Level

Unstable

xxx:ip_forwarding

Description

Enables IPv4 forwarding for a particular xxx interface. The exact name of the parameter is interface-name:ip_forwarding. For example, two interfaces are hme0 and hme1. Their corresponding parameter names are:

hme0:ip_forwarding and hme1:ip_forwarding

Default

0 (disabled)

Range

0 (disabled), 1 (enabled)

Dynamic?

Yes

When to Change

If you need IPv4 forwarding, use this parameter to enable forwarding on a per-interface basis.

Commitment Level

Unstable

ip_respond_to_echo_broadcast and ip6_respond_to_echo_multicast

Description

Control whether IPv4 or IPv6 responds to broadcast ICMPv4 echo request or multicast ICMPv6 echo request.

Default

1 (enabled)

Range

0 (disabled), 1 (enabled)

Dynamic?

Yes

When to Change

If you do not want this behavior for security reasons, disable it.

Commitment Level

Unstable

ip_send_redirects and ip6_send_redirects

Description

Control whether IPv4 or IPv6 sends out ICMPv4 or ICMPv6 redirect messages. See also ip_forwarding and ip6_forwarding.

Default

1 (enabled)

Range

0 (disabled), 1 (enabled)

Dynamic?

Yes

When to Change

If you do not want this behavior for security reasons, disable it.

Commitment Level

Unstable

ip_forward_src_routed and ip6_forward_src_routed

Description

Control whether IPv4 or IPv6 forwards packets with source IPv4 routing options or IPv6 routing headers. See also ip_forwarding and ip6_forwarding.

Default

1 (enabled)

Range

0 (disabled), 1 (enabled)

Dynamic?

Yes

When to Change

If you do not want this behavior for security reasons, disable it.

Commitment Level

Unstable

ip_addrs_per_if

Description

The maximum number of logical interfaces associated with a real interface.

Default

256

Range

1 to 8192

Dynamic?

Yes

When to Change

Do not change the value. If more logical interfaces are required, increase the value, but recognize that this change might have a negative impact on IP's performance.

Commitment Level

Unstable

ip_strict_dst_multihoming and ip6_strict_dst_multihoming

Description

Determine whether a packet arriving on a non-forwarding interface can be accepted for an IP address that is not explicitly configured on that interface. If ip_forwarding is enabled, or xxx:ip_forwarding for the appropriate interfaces is enabled, then this parameter is ignored, because the packet is actually forwarded.

Refer to RFC 1122 3.3.2.4.

Default

0 (loose multihoming)

Range

0 = Off (loose multihoming)

1 = On (strict multihoming)

Dynamic?

Yes

When to Change

If a machine has interfaces that cross strict networking domains (for example, a firewall or a VPN node), set this variable to 1.

Commitment Level

Unstable

IP Tunable Parameters With Additional Cautions

Changing the following parameters is not recommended unless there are extenuating circumstances that are described with each parameter.

ip_ire_pathmtu_interval

Description

The interval in milliseconds when IP flushes the path maximum transfer unit (PMTU) discovery information, and tries to rediscover PMTU.

Refer to RFC 1191 on PMTU discovery.

Default

10 minutes

Range

5 seconds to 277 hours

Dynamic?

Yes

When to Change

Do not change this value.

Commitment Level

Unstable

ip_icmp_return_data_bytes and ip6_icmp_return_data_bytes

Description

When IPv4 or IPv6 sends an ICMPv4 or ICMPv6 error message, it includes the IP header of the packet that causes the error message. This parameter controls how many extra bytes of the packet beyond the IPv4 or IPv6 header to be included in the ICMPv4 or ICMPv6 error message.

Default

64 bytes

Range

8 to 65,536 bytes

Dynamic?

Yes

When to Change

Do not change the value. Including more information in an ICMP error message might help in diagnosing network problems. If this feature is needed, increase the value.

Commitment Level

Unstable

TCP Tunable Parameters

tcp_deferred_ack_interval

Description

The time-out value for TCP delayed acknowledgment (ACK) timer in milliseconds.

Refer to RFC 1122, 4.2.3.2.

Default

100 milliseconds

Range

1 millisecond to 1 minute

Dynamic?

Yes

When to Change

Do not increase this value to more than 500 milliseconds.

If in some circumstances, slow network links (less than 57.6 Kbps) with greater than 512 bytes maximum segment size (MSS) when the interval is short for receiving more than one TCP segment, increase the value.

Commitment Level

Unstable

tcp_deferred_acks_max

Description

The maximum number of TCP segments (in units of maximum segment size MSS for individual connections) received before an acknowledgment (ACK) is generated. If set to 0 or 1, it means no delayed ACKs, assuming all segments are 1 MSS long.

Note that for remote destinations (not directly connected), the maximum number is fixed to 2, no matter what this parameter is set to. The actual number is dynamically calculated for each connection. The value is the default maximum.

Default

8

Range

0 to 16

Dynamic?

Yes

When to Change

Do not change the value. In some circumstances, when the network traffic becomes very bursty because of the delayed ACK effect, decrease the value. Do not decrease this value below 2.

Commitment Level

Unstable

tcp_wscale_always

Description

If set to 1, TCP always sends SYN segment with the window scale option, even if the option value is 0. Note that if TCP receives a SYN segment with the window scale option, even if the parameter is set to 0, TCP responds with a SYN segment with the window scale option, and the option value is set according to the receive window size.

Refer to RFC 1323 for the window scale option.

Default

0 (disabled)

Range

0 (disabled), 1 (enabled)

Dynamic?

Yes

When to Change

If you want the window scale option in a high-speed network configuration, enable it.

Commitment Level

Unstable

tcp_tstamp_always

Description

If set to 1, TCP always sends SYN segment with the timestamp option. Note that if TCP receives a SYN segment with the timestamp option, TCP responds with a SYN segment with the timestamp option even if the parameter is set to 0.

Default

0 (disabled)

Range

0 (disabled), 1 (enabled)

Dynamic?

Yes

When to Change

In summary, if an accurate measurement of round trip time (RTT) and TCP sequence number wraparound is a problem, enable it.

Refer to RFC 1323 for more reasons to enable this option.

Commitment Level

Unstable

tcp_xmit_hiwat

Description

The default send window size in bytes. Refer to the following discussion of per-route metrics for setting a different value on a per route basis. See tcp_max_buf also.

Default

16,384 bytes

Range

4096 to 1,073,741,824

Dynamic?

Yes

When to Change

Note that this is the default value. An application can use setsockopt(3SOCKET) SO_SNDBUF to change the individual connection's send buffer.

Commitment Level

Unstable

tcp_recv_hiwat

Description

The default receive window size in bytes. Refer to the following discussion of per-route metrics for setting a different value on a per-route basis. See tcp_recv_hiwat_minmss and tcp_max_buf also.

Default

24,576

Range

2048 to 1,073,741,824

Dynamic?

Yes

When to Change

Note that this is the default value. An application can use setsockopt(3SOCKET) SO_RCVBUF to change the individual connection's receive buffer.

Commitment Level

Unstable

tcp_max_buf

Description

The maximum buffer size in bytes. It controls how large the send and receive buffers are set to by an application using setsockopt(3SOCKET).

Default

1,048,576

Range

8192 to 1,073,741,824

Dynamic?

Yes

When to Change

If TCP connections are being made in a high-speed network environment, increase the value to match the network link speed.

Commitment Level

Unstable

tcp_cwnd_max

Description

The maximum value of TCP congestion window (cwnd) in bytes.

Refer to RFC 1122 and RFC 2581 for more information on TCP congestion window.

Default

1,048,576

Range

128 to 1,073,741,824

Dynamic?

Yes

When to Change

This is the maximum value a TCP cwnd can grow to. Note that even if an application uses setsockopt(3SOCKET) to change the window size to a value higher than tcp_cwnd_max, the actual window used can never grow beyond tcp_cwnd_max. Thus, tcp_max_buf should be greater than tcp_cwnd_max in general.

Commitment Level

Unstable

tcp_slow_start_initial

Description

The maximum initial congestion window (cwnd) size in MSS of a TCP connection.

Refer to RFC 2414 on how initial congestion window size is calculated.

Default

4

Range

1 to 4

Dynamic?

Yes

When to Change

Do not change the value.

If the initial cwnd size causes network congestion under special circumstances, decrease the value.

Commitment Level

Unstable

Changes From Previous Release

See tcp_slow_start_initial for more information.

tcp_slow_start_after_idle

Description

The congestion window size in MSS of a TCP connection after it has been idled (no segment received) for a period of one retransmission timeout (RTO).

Refer to RFC 2414 for the calculation.

Default

4

Range

1 to 16,384

Dynamic?

Yes

When to Change

See tcp_slow_start_initial for more information.

Commitment Level

Unstable

tcp_sack_permitted

Description

If set to 2, TCP always sends SYN segment with the selective acknowledgment (SACK) permitted option. If TCP receives a SYN segment with a SACK-permitted option and this parameter is set to 1, TCP responds with a SACK-permitted option. If the parameter is set to 0, TCP does not send a SACK-permitted option, regardless of whether the incoming segment contains the SACK permitted option or not.

Refer to RFC 2018 for information on the SACK option.

Default

2 (active enabled)

Range

0 (disabled), 1 (passive enabled), 2 (active enabled)

Dynamic?

Yes

When to Change

SACK processing can improve TCP retransmission performance so it should be actively enabled. If, in some circumstances, the other side can be confused with the SACK option actively enabled, set the value to 1 so that SACK processing is enabled only when incoming connections allow SACK processing.

Commitment Level

Unstable

tcp_rev_src_routes

Description

If set to 0, TCP does not reverse the IP source routing option for incoming connections for security reasons. If set to 1, TCP does the normal reverse source routing.

Default

0 (disabled)

Range

0 (disabled), 1 (enabled)

Dynamic?

Yes

When to Change

If IP source routing is needed for diagnostic purposes, enable it.

Commitment Level

Unstable

tcp_time_wait_interval

Description

The time in milliseconds a TCP connection stays in TIME-WAIT state.

Refer to RFC 1122, 4.2.2.13 for more information.

Default

4 minutes

Range

1 second to 10 minutes

Dynamic?

Yes

When to Change

On a busy web server, there can be too many TCP connections in TIME-WAIT state, consuming too much memory. In this situation, you can decrease the value for performance reasons. Do not set the value lower than 60 seconds.

Refer to RFC 1122, 4.2.2.13 for more information.

Commitment Level

Unstable

tcp_conn_req_max_q

Description

The default maximum number of pending TCP connections for a TCP listener waiting to be accepted by accept(3SOCKET). See also tcp_conn_req_max_q0.

Default

128

Range

1 to 4,294,967,296

Dynamic?

Yes

When to Change

For applications such as web servers that might receive several connection requests, the default value might be increased to match the incoming rate.

Do not increase the parameter to a very large value. The pending TCP connections can consume excessive memory. And if an application is not fast enough to handle that many connection requests in a timely fashion because the number of pending TCP connections is too large, new incoming requests might be denied.

Note that increasing tcp_conn_req_max_q does not mean that applications can have that many pending TCP connections. Applications can use listen(3SOCKET) to change the maximum number of pending TCP connections for each socket. This parameter is the maximum an application can use listen() to set the number to. This means that even if this parameter is set to a very large value, the actual maximum number for a socket might be much less than tcp_conn_req_max_q, depending on the value used in listen().

Commitment Level

Unstable

tcp_conn_req_max_q0

Description

The default maximum number of incomplete (three-way handshake not yet finished) pending TCP connections for a TCP listener.

Refer to RFC 793 for more information on TCP three-way handshake. See also tcp_conn_req_max_q.

Default

1024

Range

0 to 4,294,967,296

Dynamic?

Yes

When to Change

For applications, such as web servers that might receive excessive connection requests, you can increase the default value to match the incoming rate.

The following explains the relationship between tcp_conn_req_max_q0 and the maximum number of pending connections for each socket.

When a connection request is received, TCP first checks if the number of pending TCP connections (three-way handshake is done) waiting to be accepted exceeds the maximum (N) for the listener. If the connections are excessive, the request is denied. If the number of connections is allowable, then TCP checks if the number of incomplete pending TCP connections exceeds the sum of N and tcp_conn_req_max_q0. If it does not, the request is accepted. Otherwise, the oldest incomplete pending TCP request is dropped.

Commitment Level

Unstable

Changes From Previous Release

See tcp_conn_req_max_q0 for more information.

tcp_conn_req_min

Description

The default minimum value of the maximum number of pending TCP connection requests for a listener waiting to be accepted. This is the lowest maximum value of listen(3SOCKET) an application can use.

Default

1

Range

1 to 1024

Dynamic?

Yes

When to Change

This can be a solution for applications that use listen(3SOCKET) to set the maximum number of pending TCP connections to a value too low. Increase the value to match the incoming connection request rate.

Commitment Level

Unstable

TCP Parameters Set in the /etc/system File

These parameters can be set only in the /etc/system file. After the file is modified, reboot the system.

The following entry sets tcp_conn_hash_size:

set tcp:tcp_conn_hash_size=1024

tcp_conn_hash_size

Description

Controls the hash table size in the TCP module for all TCP connections.

Data Type

Signed integer

Default

512

Range

512 to 1,073,741,824

Implicit

The value should be a power of 2.

Dynamic?

No. The parameter can only be changed at boot time.

Validation

If you set the parameter to a value that is not a power of 2, it is rounded up to the nearest power of 2.

When to Change

If the system consistently has tens of thousands of TCP connections, increase the value accordingly. With the default value, TCP performs well up to a few thousand active connections. Note that increasing the hash table size means more memory consumption so set an appropriate value to avoid wasting memory unnecessarily.

Commitment Level

Unstable

ipc_tcp_conn_hash_size

Description

Controls the hash table size in an IP module for all active (in ESTABLISHED state) TCP connections.

Data Type

Unsigned integer

Default

512

Range

512 to 2,147,483,648

Implicit

It should be a power of two.

Dynamic?

No. This parameter can only be changed at boot time.

Validation

If you set the parameter to a value that is not a power of 2, it is rounded up to the nearest power of two.

When to Change

If the system consistently has tens of thousands of active TCP connections, increase the value accordingly. With the default value, the system performs well up to a few thousand active connections. Note that increasing the hash table size means more memory consumption so set an appropriate value to avoid wasting memory unnecessarily.

Commitment Level

Unstable

TCP Parameters With Additional Cautions

Changing the following parameters is not recommended unless there are extenuating circumstances that are described with each parameter.

tcp_ip_abort_interval

Description

The default total retransmission timeout value for a TCP connection in milliseconds. For a given TCP connection, if TCP has been retransmitting for tcp_ip_abort_interval period of time and it has not received any acknowledgment from the other endpoint during this period, TCP closes this connection.

For TCP retransmission timeout (RTO) calculation, refer to RFC 1122, 4.2.3. See also tcp_rexmit_interval_max.

Default

8 minutes

Range

500 millisecond to 1193 hours

Dynamic?

Yes

When to Change

Do not change this value. See tcp_rexmit_interval_max for exceptions.

Commitment Level

Unstable

tcp_rexmit_interval_initial

Description

The default initial retransmission timeout (RTO) value for a TCP connection in milliseconds. Refer to the following discussion of per route metrics for setting a different value on a per-route basis.

Default

3 seconds

Range

1 millisecond to 20 seconds

Dynamic?

Yes

When to Change

Do not change this value. Lowering the value can result in unnecessary retransmissions.

Commitment Level

Unstable

tcp_rexmit_interval_max

Description

The default maximum retransmission timeout value (RTO) in milliseconds. The calculated RTO for all TCP connections cannot exceed this value. See also tcp_ip_abort_interval.

Default

60 seconds

Range

1 millisecond to 2 hours

Dynamic?

Yes

When to Change

Do not change the value in a normal network environment.

If in some special circumstances, the round trip time (RTT) for a connection is in the order of 10 seconds, you can change the value to a higher value. If you change this value, you should also change the tcp_ip_abort_interval parameter to match it. Change the value of tcp_ip_abort_interval to at least four times the value of tcp_rexmit_interval_max.

Commitment Level

Unstable

Changes From Previous Release

tcp_rexmit_interval_max

tcp_rexmit_interval_min

Description

The default minimum retransmission time-out (RTO) value in milliseconds. The calculated RTO for all TCP connections cannot be lower than this value. See also tcp_rexmit_interval_max.

Default

400 milliseconds

Range

1 millisecond to 20 seconds

Dynamic?

Yes

When to Change

Do not change the value in a normal network environment.

TCP's RTO calculation should be able to cope with most RTT fluctuations. If in some very special circumstances such that the round trip time (RTT) for a connection is in the order of 10 seconds, change to a higher value. If you change this value, you should change the tcp_rexmit_interval_max parameter to match it. You should change the value of tcp_rexmit_interval_max to at least eight times the value of tcp_rexmit_interval_min.

Commitment Level

Unstable

tcp_rexmit_interval_extra

Description

A constant added to the calculated retransmission time-out value (RTO) in milliseconds.

Default

0 milliseconds

Range

0 to 2 hours

Dynamic?

Yes

When to Change

Do not change the value.

When the RTO calculation fails to obtain a good value for a connection in some circumstances, you can change this value to avoid unnecessary retransmissions.

Commitment Level

Unstable

tcp_tstamp_if_wscale

Description

If this parameter is set to 1, and the window scale option is enabled for a connection, TCP also enables the timestamp option for that connection.

Default

1 (enabled)

Range

0 (disabled), 1 (enabled)

Dynamic?

Yes

When to Change

Do not change this value. In general, when TCP is used in high-speed network, protection against sequence number wraparound is essential, thus you need the timestamp option.

Commitment Level

Unstable

tcp_recv_hiwat_minmss

Description

Controls the default minimum receive window size. The minimum is tcp_recv_hiwat_minmss times the size of maximum segment size (MSS) of a connection.

Default

4

Range

1 to 65,536

Dynamic?

Yes

When to Change

Do not change the value. If changing it is necessary, do not change the value lower than 4.

Commitment Level

Unstable

tcp_compression_enabled

Description

If set to 1, protocol control blocks of TCP connections in TIME-WAIT state are compressed to reduce memory usage. If set to 0, no compression is done. See tcp_time_wait_interval also.

Default

1 (enabled)

Range

0 (disabled), 1 (enabled)

Dynamic?

Yes

When to Change

Do not turn off the compression mechanism.

Commitment Level

Unstable

UDP Tunable Parameters

This section describes some of the UDP tunable parameters.

udp_xmit_hiwat

Description

The default maximum UDP socket datagram size in bytes. See udp_max_buf for more information.

Default

8192 bytes

Range

4096 to 65,536

Dynamic?

Yes

When to Change

Note that an application can use setsockopt(3SOCKET) SO_SNDBUF to change the size for an individual socket. In general, you do not need to change the default value.

Commitment Level

Unstable

udp_recv_hiwat

Description

The default maximum UDP socket receive buffer size in bytes. See udp_max_buf for more information.

Default

8192 bytes

Range

4096 to 65,536

Dynamic?

Yes

When to Change

Note that an application can use setsockopt(3SOCKET) SO_RCVBUF to change the size for an individual socket. In general, you do not need to change the default value.

Commitment Level

Unstable

UDP Parameters with Additional Cautions

Changing the following parameters is not recommended unless there are extenuating circumstances that are described with each parameter.

udp_max_buf

Description

Controls how large send and receive buffers (in bytes) can be for a UDP socket.

Default

262,144 bytes

Range

65,536 to 1,073,741,824

Dynamic?

Yes

When to Change

Do not change the value. If this parameter is set to a very large value, UDP socket applications can consume too much memory.

Commitment Level

Unstable

Per-Route Metrics

In the Solaris 8 release, you can use the per-route metrics to associate some properties with IPv4 and IPv6 routing table entries.

For example, a system has two different network interfaces, fast ethernet interface and gigabit ethernet interface. The system default tcp_recv_hiwat is 24,576 bytes. This default is sufficient for the fast ethernet interface, but may not be sufficient for the gigabit ethernet interface.

Instead of increasing the system's default tcp_recv_hiwat, you can associate a different default TCP receive window size to the gigabit ethernet interface routing entry. By making this association, all TCP connections going through the route will have the increased receive window size.

Assuming IPv4, the following is in the routing table (netstat -rn).

192.123.123.0        192.123.123.4         U        1      4  hme0
192.123.124.0        192.123.124.4         U        1      4  ge0
default              192.123.123.1         UG       1      8

Do the following:

# route change -net 192.123.124.0 -recvpipe x

This means all connections going to the 192.123.124.0 network, which is on the ge0 link, use the receive buffer size x, instead of the default 24567 receive window size.

If the destination is in the a.b.c.d network, and there is no specific routing entry for that network, you can add a prefix route to that network and change the metric. For example:

# route add -net a.b.c.d 192.123.123.1 -netmask w.x.y.z
# route change -net a.b.c.d -recvpipe y

Note that the prefix route's gateway is the default router. Then all connections going to that network use receive buffer size y. If you have more than one interface, use the -ifp argument to specify which interface to use. This way, you can control which interface to use for specific destinations. Use the route(1M) get command to verify the metric.