vADC Docs

Tuning the Solaris operating system for Stingray Traffic Manager

by on ‎02-25-2013 03:07 AM (3,394 Views)

This document describes some Solaris operating system tunables you may wish to apply to a production Stingray Traffic Manager instance.  Note that the kernel tunables only apply to Stingray Traffic Manager software installed on a customer-provided Solaris instance; it does not apply to the Stingray Traffic Manager Virtual Appliance or Cloud instances.

Consider the tuning techniques in this document when:

  • Running Stingray on a severely-constrained hardware platform, or where Stingray should not seek to use all available resources;
  • Running in a performance-critical environment;
  • The Stingray host appears to be overloaded (excessive CPU or memory usage);
  • Running with very specific traffic types, for example, large video downloads or heavy use of UDP;
  • Any time you see unexpected errors in the Stingray event log or the operating system syslog that relate to resource starvation, dropped connections or performance problems

For more information on performance tuning, start with the Tuning Stingray Traffic Manager article.

Increasing the ephemeral port range

The ephemeral port range sets the upper limit on the the number of TCP connections a server can hold open to a particular IP address (including connections that have been closed and are in the TIME_WAIT state). Increasing it is recommended; doing so has no unwanted side effects on a typical system.

/usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 1024

/usr/sbin/ndd -set /dev/tcp tcp_largest_anon_port 65535 # This is usually the default value


Increasing the TCP listen queue


tcp_conn_req_max_q sets the maximum size of the TCP listen queue (the number of completed TCP connections waiting to be accept()ed), per port.


tcp_conn_req_max_q0 sets the maximum number of half-open TCP connections.


/usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q 1024

/usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q0 4096

If you expect a very high rate of new connections, you may find it beneficial to increase these again (by up to a factor of 10 or so).


TCP window scaling and timestamps

The following two tunables force window scaling on and enable TCP timestamps (a protection against sequence number wraparound). We recommend they both be enabled.

/usr/sbin/ndd -set /dev/tcp tcp_wscale_always 1
/usr/sbin/ndd -set /dev/tcp tcp_tstamp_if_wscale 1

TCP send/receive buffer size

The following increases the size of the TCP/Send receive buffer, enabling more efficient handling of connections.

/usr/sbin/ndd -set /dev/tcp tcp_max_buf 16777216

TCP window sizes

The following three tunables set the maximum size of the TCP congestion, transmit and receive windows respectively.

/usr/sbin/ndd -set /dev/tcp tcp_cwnd_max 8388608
/usr/sbin/ndd -set /dev/tcp tcp_xmit_hiwat 4000000
/usr/sbin/ndd -set /dev/tcp tcp_recv_hiwat 4000000

Duplicate Address Detection

Solaris' duplicate address detection can interfere with correct fail over of Traffic IP addresses. We recommend that the following tuning is applied to prevent this.

ndd -set /dev/arp arp_probe_count 0
ndd -set /dev/ip ip_dup_recovery 50

Solaris UltraSPARC T1/T2 systems

The following tunables have been found to increase performance on systems that use UltraSPARC T1 and T2 processors. We recommend that you consult Oracle for the recommended tunings for your Solaris release. The following lines should be added to /etc/system. The system will need to be rebooted before they take effect.

set ip:ip_soft_rings_cnt=4
set ip_squeue_soft_ring=1
set ip:ip_squeue_fanout=1
set ip:ip_squeue_bind=0
set hires_tick=1

TCP Fusion

During our testing on OpenSolaris we encountered some issues when the tcp_fusion option was enabled. We found that the kernel could incorrectly buffer large amounts of data, which could cause the machine in question to hang. We recommend that customers using this release of OpenSolaris turn off the tcp_fusion feature to avoid this issue.

To turn off this feature you can immediately turn it off with the following command:

# echo 'do_tcp_fusion/W 0' | mdb -kw

This will not persist across system reboots. If you want 'tcp_fusion' to be permanently disabled you will need to add the following line to '/etc/system':

set ip:do_tcp_fusion = 0

UltraSparc T2 Crypto Acceleration

The Sun UltraSPARC T1 and T2 processors contain on-board cryptographic routines that speed up SSL connections, and Stingray will use these routines if they are detected.

The T1 and T2 processors contain the ncp module which provides operations for the RSA algorithm. Additionally the T2 contains n2cp module which provides symmetric cipher support (such as AES, DES and 3DES) and digest algorithms (e.g. MD5 and SHA-1).

When Stingray starts up it will detect the presence of the ncp support and if it is present it will take full advantage of the performance improvements. In particular, don't configure PKCS#11 support on the Global Settings -> SSL Hardware support (this is used for external SSL hardware such as a PCI card or network device).

You can check Stingray is using the ncp support by running:

$ $ZEUSHOME/zxtm/bin/zeus.zxtm -vv

Version 5.0, Build date: May 20 2008 22:05:47

Compiled for platform: SunOS

Compression library      : zlib v1.2.3

Regex library            : PCRE v7.6

Crypto library           : Built-in PKCS#11: ncp/0 Crypto Accel Asym 1.0

Compiler                 : CC: Sun C++ 5.8 2005/10/13

XML library              : 2.6.23

XSLT library             : 10115

Note the "Crypto Library" line refers to ncp. If Stingray is reporting itself as using something else you should check that the ncp support is enabled using the cryptoadm list command.