05-01-2015 07:42 AM
I hope someone in the discussion board is able to help me clarify a situation we have found in our SAN environment.
To keep things simple, we have a lot of computing, connected via an IBM SAN768B (Brocade DCX) to some IBM XIV systems.
The problem we find is that the XIV ports have a continuos increase rate of tim_txcrd_z at around 400 per minute when it's some workload.
Take into account that even when there is workload, we are really really far from the numbers indicated by IBM in terms of IOPS and throughtput, and in tests, we have been able to triple those with performance tests.
While working on this, I came across a number of posts (This one is brilliant for this) and articles in regards to FLOGI and BB_credits, and how the flow control works and with the command portcfgfportbuffers as an option to increase the number of buffer credits on a switch port.
So, the questions:
In regards to FLOGI and credit "negotiation", there is no such negotiation (at least for F ports) and that it's basically "This are the credits I have memory to keep" more or less.
how can I check the credits in both sides from the director?
Is the number of credits on the storage array port the one that can be seen in portloginshow?
XXXXXXXXXXXXX:FID128:admin> portloginshow 1/41 Type PID World Wide Name credit df_sz cos ===================================================== fe 6881c0 50:XX:XX:XX:XX:XX:XX:51 40 2048 8 scr=0x3 ff 6881c0 50:XX:XX:XX:XX:XX:XX:51 0 0 8 d_id=FFFFFC
If so, the port in the array gives 40 credits to the switch, but the switch gives back only 8.... is that a fair deal? I assume the RDY primitives don't count as a frame, or the storage would only be able to send RDYs
Finally, that would mean that the switch can have 40 frames in transit or waiting for response, and still I get zero credits situations quite a few times.
Is it possible that the port on the storage array can't take more due to a bottleneck on the way back to the switch?
In that situation, would using portcfgfportbuffers and changing the port up to 40 credits be a potential improvement?
could that change cause potential issues?
Should I learn to live with these zero credits situation?
There are no other errors on those ports, nor in any but one of the clients ports.
There are no other errors apparent in the switchs or fabrics nor in the Storage Arrays.
05-01-2015 03:06 PM
You are right about the "negotiation" - there is none. It's more an exchange of information.
If you don't have any performance problem at the moment, 400 is not very concerning. Polling every 2.5ms the max increase rate would be ~400k per second. Well of course that would be a total block of the link (resulting in a link reset), but 7 per second (400 per minute) is basically: out of 400000 times the mechanism looked on its buffer credits only 7 times it was zero.
Of course I like ports where all counters are zero and often that's really the case. But 7 per second is nothing I would worry too much about.
For portcfgfportbuffer: This command totally makes sense if you have a device connected over a longer distance (like the classical ISL-less SVC stretched cluster). There you would need extra buffers to span the distance. For normal local connections with 2-50m cables it's seldom that it really helps. I could imagine it to help in case of backpressure from a congestion bottleneck (not a slow drain device) somewhere down the path. A congestion bottleneck often delays frames only a short time. Extra buffers could help here a bit.
On local links there is also no need to keep it equal. The developers put the amount of buffers in their device according to their device's needs. If they think they only need 5 buffers the switch gets 5 credits. If they think 40 buffers might be better, the switch gets 40 credits. There is no need to set the switch to the same value, because it's the other direction.
Last not least: I don't think that you see the zero credit counter against the XIV port increasing, because the XIV cannot send something out in the reverse direction. The XIV is a plain target with a cache. I see little chance to turn it into a slow drain device due to back pressure from the other side. But again 400 per minute is not really a slow drain device.
05-10-2015 09:06 PM
Thank you for referencing my post. Much appreciated.
I agree with Seb here. The tim_txcrd_z gives more an indication of a well utilsed link but does not provide info on high_latency devices perse. If you see almost no traffic at all in the sense of number of frames/second but still see this counter increase there is a pretty high chance you will see frame-drop when these links get more busy. Thats when we get worried and start to investigate.
As for your remark on "fairness" of the number of credits between the switchport (8) and the array (40) I say that this is of no influence as the switch itself does not store frame for processing in any way. (There are exceptions when for instance encryption is in play or when a frame has to traverse an ISL which requires some scheduling and wait time for forwarding). An end-device which actually has to do something with this frame may need a larger buffer for offloading this to back-end processes and thus may incur a somewhat higher delay. The buffer count simply helps with this to keep a well utilised array in these cirsumstances.
Hope this helps.