11-04-2013 01:54 PM
We are receiving severe latency bottleneck warnings on an ISL port. How can I troubleshoot this? I have run SAN health against our fabrics and the only thing that I can see that is different on this port is that it is a locked G port (not sure if this has any bearing on the situation.) Any help/direction would be appreciated.
11-05-2013 08:04 AM
Locked G port only means it cannot become an FL port, so not of real concern.
using san health to Troubleshooting bottlenecks is not ideal.
Do you have BNA?
If not start with bottleneckmon on the CLI and/or look for slow drain devices in you fabric(s).
11-05-2013 10:21 AM
I really wasn't using San Health to troubleshoot, per se. Just happened to notice that it was the only port that was in that locked state, and it just happened to be the port with the bottleneck. I wasn't sure what that status was.
We do use CMCNE and I was told that BNA was wrapped up in there, and that is how the bottleneck was noticed in the first place via CMCNE.
I'm extremely green to this and was just wondering what to do next? The problem port is identified, how do I go about tracking down slow drain devices? How do I figure out what the issue is? Also, bottleneckmon is disabled on our switches.
11-05-2013 12:40 PM
CMCNE is EMC's version of BNA.
If you know what the problem port is you don't need to troubleshoot further.
My guess is you mean with problem port the ISL port.
Usually the port reporting it is not the one causing it.
So your ISL port is most likely the symptom or result of the issue.
A slow drain device is a device lwhich cannot handle trafic being sent to it fast enough.
For instance a 8GB storage port sents traffic faster than a 1GB host port can consume.
The 1GB host port acts as the slow drain because it is not giving back buffer credit back to the switch fast enough.
So the switch will hold the frames in buffer against the switchport the host is using.
In turn buffers at ISLport start to will up and buffers are not returned to the other end.
This happens on all ports along the IO path, unfortunately you'll see it most on the ISL as they process, in general, more traffic as other ports.
try to enable bottleneckmon, it could be helpful
11-05-2013 12:54 PM
Thanks for the reply! Makes a lot more sense now, and the situation you describe is quite similar to what is present in the environment: 8GB storage ports to 4GB host port. I will try and see if we can turn bottleneckmon; thanks for the suggestion on that.
This may be a dumb question, but when there is a situation in the environment as I described, a mismatch of speeds, can anything be done about that (to prevent the bottleneck, if indeed that is what is causing it) short of getting the 8GB host connections?
11-06-2013 08:37 AM - edited 11-06-2013 08:37 AM
Change your storage port to 4G ;)
You could tinker with QoS or Ingress Rate Limiting, reducing buffer credits on port (if possible on F ports)