vADC Docs

Tech Tip: Debugging Techniques for Performance Investigation

by on ‎02-21-2013 11:14 AM (2,058 Views)

dmesg: use dmesg to quickly display recent kernel messages.  Any warnings about resource starvation, overfilled kernel tables and the like should be addressed by appropriate kernel tuning.

vmstat: vmstat 3 is a quick and easy way to monitor CPU utilization.  On a well-utilized Stingray system, the user (us) and system (sy) CPU times will give a rough indication of the utilization, and the idle (id) time a rough indication of the spare capacity. Stingray workload is shared between user and system time; user time will predominate for a complex configuration or one that uses CPU-intensive operations (e.g. SSL, compression), and system time will predominate for a simple configuration with minimal traffic inspection.  The wait time (wa) should always be low.

User, System and Idle time is not a good indication of spare capacity because Stingray uses system resources as eagerly as possible, and it will operate more efficiently the more highly-loaded it is.  For example, even if a Stingray system is 25% utilized, it will likely be at less than 25% its total capacity.

/proc: use /proc/<pid> for a quick investigation of the state of Stingray processes – memory usage, number of open file descriptors, process limits

ethtool: use ethtool to query and configure the network interface hardware:

  • ethtool eth0 - to determine interface speed and negotiated options;
  • ethtool –S eth0 to dump statistics for a network interface.  Large numbers of retransmits, errors or collisions may indicate a faulty NIC or poor cabling, or a congested network;
  • ethtool –i eth0 to confirm the driver in use;
  • ethtool –k eth0 to list the offload features employed by the card.

tcpdump: use tcpdump to capture raw packets from named interfaces.  A tcpdump analysis can uncover unexpected problems such as slow closes and inappropriate use of tcp optimizations as well as application level problems.

trace: Stingray includes a wrapper (ZEUSHOME/zxtm/bin/trace) around the standard operating system system trace tools.  trace will output all system calls performed by the traced Stingray processes.