Fibre Channel (SAN)

Reply
New Contributor
Posts: 2
Registered: ‎09-09-2009

Slow Draining Devices with tim_txcrd_z and tim_txcrd_z_vc on ISL ports

Hi All,

I was closely monitoring our ISL ports from Blade SAN switches to the DCX. There is constant increase in the tim_txcrd_z and tim_txcrd_z_vc values. I reset the port statistics for the ISL ports and found that the tim_txcrd_z value increases by 1500 per second on certain ISL ports. The  tim_txcrd_z_vc values for the data VC's 2,3,4 and 5 increments anywhere from 0 to 1100 per second, there is no increment on the tim_rdy_pri values. All the ISL's are 8Gbps with QOS in AE state (we do not have QOS license) and the Long Distance mode disabled. There is no errors on the SAN switches indicating a possible SFP or cable issue. The servers connected to these SAN switches face occasional reset of the HBA's and application I/O response alerts.

I have been digging around for an clarification on these values with no luck.

Can anyone let me know whether I need to worry about the values and the rapid increase of the tim_txcrd_z and tim_txcrd_z_vc values?

Thanks and Regards,

Sudharsan.V.S

Frequent Contributor
Posts: 141
Registered: ‎05-26-2009

Re: Slow Draining Devices with tim_txcrd_z and tim_txcrd_z_vc on ISL ports

In your case I wouldn't analyze it from a pure statistics point of view. (the tim_txcrd_z ticks 400000 times a second)

I would enable bottleneckmon first (http://seb-t.de/bneck). Also the maintenance provider for your SAN switches should be able to offer a performance analysis to find bottlenecks and their reasons in your SAN.

New Contributor
Posts: 3
Registered: ‎12-16-2011

Re: Slow Draining Devices with tim_txcrd_z and tim_txcrd_z_vc on ISL ports

It is related to lack of buffers (BB credit zero).

There may be two reasons:

1) channel really can not cope with the loading and there are bottlenecks in fabric - need to monitor stat_ftx, stat_frx;

2) there is slow device in fabric, such as 2Gb port (it is more likely). In this case need to exclude device from passing through ISL. If you have a trunk on ISL then perhaps it is better to split into separate ISL, in this case will more Virtual Channels, slow traffic will go through one VC, other will be available for faster traffic.

Contributor
Posts: 25
Registered: ‎05-16-2017

Re: Slow Draining Devices with tim_txcrd_z and tim_txcrd_z_vc on ISL ports

Dear Andriy,

what is the normal Count on the ftx an rtx is this normal or not.

 

stat64_ftx      1           top_int : Frames transmitted
                944912424   bottom_int : Frames transmitted
stat64_frx      0           top_int : Frames received
                3712718161  bottom_int : Frames received

 

regards

Ernst

Brocade Moderator
Posts: 414
Registered: ‎03-29-2011

Re: Slow Draining Devices with tim_txcrd_z and tim_txcrd_z_vc on ISL ports

Hi Ernst,

 

the ftx and frx have to put in context current speed of link (4GB or 16GB etc), and also duration.   Taking out of context it is hard to say.  But these are not extreme, the frame transmit and receive roll over in 1-4 hours depedent on speed of the link.

 

Concerning the second statement from Andriy:

 

1. if you use the standard routing policy (Exchange Based), then only way to have traffic over one ISL instead of another is to use TI zoning;

 

2. Splitting a trunk of two ISLs into a seperate ISL will give you more routes, and potentially congestion might only hit one ISL (instead of all members of the trunk).  The advantage of the trunk is that if the first member runs out of credits, the second take over, so you can hande larger peaks.

 

3. in general, if you have two ISL between two switches, do not trunk them; if you have 4 ISLs, then create two trunks of two...

 

 




If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution".


Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
New Contributor
Posts: 2
Registered: ‎12-10-2016

Re: Slow Draining Devices with tim_txcrd_z and tim_txcrd_z_vc on ISL ports

Martin just wanted to clarify point 2 & 3 please.
I have some slow drain devices causing tim_txcrd value increase as well. I have 2x ISL which I’ve trunked to use the additional credits as per your point 2. This seems to be softening the impact of the SD device (until budget allows me to replace it)
But you then say in general don’t trunk 2 ISL if that’s all you’ve got. I can’t find reference to this in any white paper or guide sorry.

Would trunking 2x ISL in this setup actually be detrimental to performance? Or is it just about giving it more routes with seperate ISL?

Cheers
Jason
Brocade Moderator
Posts: 414
Registered: ‎03-29-2011

Re: Slow Draining Devices with tim_txcrd_z and tim_txcrd_z_vc on ISL ports

Hi Jason,

 

let me clarif myself a little- if you have two links, and on both side you have switches (and not directors), then I would go for trunking. If you are doing director to director with two links, then the links should be on different blade (HA first). If you are doing director to switch, I would probably go for to different blades. If you are using multiple ASIC switch (4900, 5300, 6520), I would prefer 2 trunks of two.

 

It is an balance between performance and availabilty - but in setup with single ASIC switch, you can trunk, You only have one ASIC, and if it fails, the links goes.  In your case, for, when you have slow drain device and needs more buffer credits, I would go for trunking (first).

 

The interesting discussion is between narrow (a lot) or fat trunks (a few) - assuming you are using EBR (Exchange Base Routing).

 

1. With a fat trunk, in case of a latency / slow drain leading to congestion more devices (using that trunk will be hit) - say we had 4 trunks (of 4), so 25% (one of four trunks) one VC will be hit...

 

2. With a narrow trunks, in the case congestion, say we have 8 trunk of 2, then 12,5% of traffic sharing the congested trunk. But you will have less of handline of peak.

 

In general, I prefer at least two (trunks) of 2 for larger switches since we can handle peaks of credit need, and also loose a link in a trunk (and replace it) with less impact. When you have 2 ISLs, dependent on hardware on each end.




If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution".


Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
New Contributor
Posts: 2
Registered: ‎12-10-2016

Re: Slow Draining Devices with tim_txcrd_z and tim_txcrd_z_vc on ISL ports

Thank you very much for the replay Martin. It’s clarified and also confirmed the setup I’ve implemented in our environment.

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.

vADC is now Pulse Secure
Download FREE NVMe eBook