07-19-2012 11:02 AM
I was closely monitoring our ISL ports from Blade SAN switches to the DCX. There is constant increase in the tim_txcrd_z and tim_txcrd_z_vc values. I reset the port statistics for the ISL ports and found that the tim_txcrd_z value increases by 1500 per second on certain ISL ports. The tim_txcrd_z_vc values for the data VC's 2,3,4 and 5 increments anywhere from 0 to 1100 per second, there is no increment on the tim_rdy_pri values. All the ISL's are 8Gbps with QOS in AE state (we do not have QOS license) and the Long Distance mode disabled. There is no errors on the SAN switches indicating a possible SFP or cable issue. The servers connected to these SAN switches face occasional reset of the HBA's and application I/O response alerts.
I have been digging around for an clarification on these values with no luck.
Can anyone let me know whether I need to worry about the values and the rapid increase of the tim_txcrd_z and tim_txcrd_z_vc values?
Thanks and Regards,
07-26-2012 01:42 AM
In your case I wouldn't analyze it from a pure statistics point of view. (the tim_txcrd_z ticks 400000 times a second)
I would enable bottleneckmon first (http://seb-t.de/bneck). Also the maintenance provider for your SAN switches should be able to offer a performance analysis to find bottlenecks and their reasons in your SAN.
07-26-2012 06:13 AM
It is related to lack of buffers (BB credit zero).
There may be two reasons:
1) channel really can not cope with the loading and there are bottlenecks in fabric - need to monitor stat_ftx, stat_frx;
2) there is slow device in fabric, such as 2Gb port (it is more likely). In this case need to exclude device from passing through ISL. If you have a trunk on ISL then perhaps it is better to split into separate ISL, in this case will more Virtual Channels, slow traffic will go through one VC, other will be available for faster traffic.
10-03-2017 09:57 PM
what is the normal Count on the ftx an rtx is this normal or not.
stat64_ftx 1 top_int : Frames transmitted
944912424 bottom_int : Frames transmitted
stat64_frx 0 top_int : Frames received
3712718161 bottom_int : Frames received
10-13-2017 12:01 AM
the ftx and frx have to put in context current speed of link (4GB or 16GB etc), and also duration. Taking out of context it is hard to say. But these are not extreme, the frame transmit and receive roll over in 1-4 hours depedent on speed of the link.
Concerning the second statement from Andriy:
1. if you use the standard routing policy (Exchange Based), then only way to have traffic over one ISL instead of another is to use TI zoning;
2. Splitting a trunk of two ISLs into a seperate ISL will give you more routes, and potentially congestion might only hit one ISL (instead of all members of the trunk). The advantage of the trunk is that if the first member runs out of credits, the second take over, so you can hande larger peaks.
3. in general, if you have two ISL between two switches, do not trunk them; if you have 4 ISLs, then create two trunks of two...
10-16-2017 06:45 AM
10-16-2017 07:16 AM
let me clarif myself a little- if you have two links, and on both side you have switches (and not directors), then I would go for trunking. If you are doing director to director with two links, then the links should be on different blade (HA first). If you are doing director to switch, I would probably go for to different blades. If you are using multiple ASIC switch (4900, 5300, 6520), I would prefer 2 trunks of two.
It is an balance between performance and availabilty - but in setup with single ASIC switch, you can trunk, You only have one ASIC, and if it fails, the links goes. In your case, for, when you have slow drain device and needs more buffer credits, I would go for trunking (first).
The interesting discussion is between narrow (a lot) or fat trunks (a few) - assuming you are using EBR (Exchange Base Routing).
1. With a fat trunk, in case of a latency / slow drain leading to congestion more devices (using that trunk will be hit) - say we had 4 trunks (of 4), so 25% (one of four trunks) one VC will be hit...
2. With a narrow trunks, in the case congestion, say we have 8 trunk of 2, then 12,5% of traffic sharing the congested trunk. But you will have less of handline of peak.
In general, I prefer at least two (trunks) of 2 for larger switches since we can handle peaks of credit need, and also loose a link in a trunk (and replace it) with less impact. When you have 2 ISLs, dependent on hardware on each end.