Fibre Channel (SAN)

Reply
Contributor
Posts: 21
Registered: ‎06-09-2016

Bottlenecks

Initiated automatic bottleneck mitigation actions on slot 0 and port 10 Getting a lot of these on my 5470 Blade switch. It points to one specific Blade Server and port. Should I be worried?

Frequent Contributor
Posts: 107
Registered: ‎04-05-2011

Re: Bottlenecks

Hey Rob!

 

Do you have bottleneckmon enabled? what is FOS version ?

 

What is the exact message you are getting ?

Contributor
Posts: 21
Registered: ‎06-09-2016

Re: Bottlenecks

Here are the exact errors I'm getting on my 5470 Edge Switch:


Mon Jun 06 2016 21:16:37 GMT+00:00    Information    Concluded automatic bottleneck mitigation actions on slot 0 and port 10.    Switch    666    1    AN-1009    brocade_xxxx    
Mon Jun 06 2016 21:16:32 GMT+00:00    Information    Initiated automatic bottleneck mitigation actions on slot 0 and port 10    Switch    665    1    AN-1007    brocade_xxxx

I am also getting errors on my core 6505 switch:


Time    Level    Message    Service    Number    Count    Message ID    Switch    
Wed Jun 01 2016 21:01:17 EDT    Warning    Severe latency bottleneck detected at E-Port 1.    Switch    1116    1    AN-1010    

These two switches are connected with a single 8gb ISL link between them. I have bottleneckmon turned on

Kernel:     2.6.14.2
Fabric OS:  v6.4.2b4
Made on:    Thu Aug 9 21:10:00 2012
Flash:      Fri Jan 2 17:31:58 2015
BootProm:   1.0.9
brocade_BOTTOM:rspencer>

   

Frequent Contributor
Posts: 107
Registered: ‎04-05-2011

Re: Bottlenecks

[ Edited ]

A Latency bottleneck is a bad thing. Because it can cause impact on several ports, even port that is not directly related, specially if you have ISL, that is your case.

 

Does the port 0/10 on the 5740 and the port 1 on your core are related to the same ISL ? (check with islshow)

If the reported ports are both ends on the same ISL, look for errors incrementing in the porterrshow and portstatsshow logs. If all the physical related counters are ok, issue a nsshow our nscamshow and look for a line like "Slow Drain Device: YES"

 

[edit]

I recommend you to do the following, in sequence:

1 - get a supportshow

2 - clear the errors (slotstatsclear, statsclear)

3 - wait for a few hours

4 - save a new supporthow

 

After that, look again into porterrshow and portstatsshow logas and see if there are some higly incrementing counter.

[/edit]

Contributor
Posts: 21
Registered: ‎06-09-2016

Re: Bottlenecks

Port 10 corresponds to an IBM Blade with a Qlogic fiber card in it.

Frequent Contributor
Posts: 107
Registered: ‎04-05-2011

Re: Bottlenecks

Ah right, sorry for this miss.

 

Anyway, can you sare porterrshow and portstatsshow from your edge switch.

 

Also, can you provide islshow and porterrshow from your core switch? and a portshow from the port reported by the bottleneckmon message?

Contributor
Posts: 21
Registered: ‎06-09-2016

Re: Bottlenecks

[ Edited ]

login as:
rspencer@BROCADE_xxxxx's password:

-----------------------------------------------------------------
brocade_xxxxxx:r> porterrshow
          frames      enc    crc    crc    too    too    bad    enc   disc   lin                                         k   loss   loss   frjt   fbsy
       tx     rx      in    err    g_eof  shrt   long   eof     out   c3    fail                                             sync   sig
     ===========================================================================                                         ==============================
  0:   12.3m  28.3m   0      0      0      0      0      0      0      0      0                                               0      0      0      0
  1:   11.0m   9.1m   0      0      0      0      0      0      0      0      0                                               0      0      0      0
  2:    0      0      0      0      0      0      0      0      0      0      0                                               0      0      0      0
  3:    0      0      0      0      0      0      0      0      0      0      0                                               0      0      0      0
  4:    0      0      0      0      0      0      0      0      0      0      0                                               0      0      0      0
  5:    1.0m 565.6k   0      0      0      0      0      0      0      0      0                                               0      0      0      0
  6:    0      0      0      0      0      0      0      0      0      0      0                                               0      0      0      0
  7:    0      0      0      0      0      0      0      0      0      0      0                                               0      0      0      0
  8:  191.4m   1.2g   0      0      0      0      0      0      0      0      0                                               0      0      0      0
  9:    3.0g 570.1m   0      0      0      0      0      0      0      0      0                                               0      0      0      0
 10:  833.4m 412.7m   0      0      0      0      0      0      0    127      0                                               0      0      0      0
 11:  811.0m 590.5m   0      0      0      0      0      0      0      0      0                                               0      0      0      0
 12:   16.8m   4.9m   0      0      0      0      0      0      0      0      0                                               0      0      0      0
 13:    1.0m 590.1k   0      0      0      0      0      0      0      0      0                                               0      0      0      0
 14:    0      0      0      0      0      0      0      0      0      0      0                                               0      0      0      0
 15:    6.9m   9.2m   0      0      0      0      0      0      0      0      0                                               0      0      0      0
 16:    0      0      0      0      0      0      0      0      0      0      0                                               0      0      0      0
 17:    3.5g   3.9g   0      0      0      0      0      0      0      7.9k   0                                               0      0      0      0
 18:  800.1m   2.4g   0      0      0      0      0      0      0      6.6k   0                                               0      0      0      0
 19:    0      0      0      0      0      0      0      0      0      0      0                                               0      0      0      0

brocade_xxxxxSmiley Embarassed portstatsshow 10
stat_wtx                1766190275  4-byte words transmitted
stat_wrx                1557715701  4-byte words received
stat_ftx                833407266   Frames transmitted
stat_frx                412756836   Frames received
stat_c2_frx             0           Class 2 frames received
stat_c3_frx             412756836   Class 3 frames received
stat_lc_rx              0           Link control frames received
stat_mc_rx              0           Multicast frames received
stat_mc_to              0           Multicast timeouts
stat_mc_tx              0           Multicast frames transmitted
tim_rdy_pri             0           Time R_RDY high priority
tim_txcrd_z             236825107   Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc  0- 3:  0           0           0           0
tim_txcrd_z_vc  4- 7:  236825107   0           0           0
tim_txcrd_z_vc  8-11:  0           0           0           0
tim_txcrd_z_vc 12-15:  0           0           0           0
er_enc_in               0           Encoding errors inside of frames
er_crc                  0           Frames with CRC errors
er_trunc                0           Frames shorter than minimum
er_toolong              0           Frames longer than maximum
er_bad_eof              0           Frames with bad end-of-frame
er_enc_out              0           Encoding error outside of frames
er_bad_os               0           Invalid ordered set
er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout
er_tx_c3_timeout        127         Class 3 transmit frames discarded due to timeout
er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable
er_other_discard        0           Other discards
er_type1_miss           0           frames with FTB type 1 miss
er_type2_miss           0           frames with FTB type 2 miss
er_type6_miss           0           frames with FTB type 6 miss
er_zone_miss            0           frames with hard zoning miss
er_lun_zone_miss        0           frames with LUN zoning miss
er_crc_good_eof         0           Crc error with good eof
er_inv_arb              0           Invalid ARB
open                    0           loop_open
transfer                0           loop_transfer
opened                  0           FL_Port opened
starve_stop             0           tenancies stopped due to starvation
fl_tenancy              0           number of times FL has the tenancy
nl_tenancy              0           number of times NL has the tenancy
zero_tenancy            0           zero tenancy

login as: rspencer
rspencer@BROCADE-**filtered**-1's password:


BROCADE-**filtered**-1Smiley Embarassed islshow
  1:  0-> 17 10:00:00:05:1e:c7:ff:f1   4 brocade_TOP     sp:  8.000G bw:  8.000G
  2:  1-> 18 10:00:00:05:1e:c8:6a:79   3 brocade_BOTTOM  sp:  8.000G bw:  8.000G
  3:  2-> 17 10:00:00:05:33:40:dc:84   1 BCH2-TOP        sp:  8.000G bw:  8.000G
  4:  3-> 17 10:00:00:05:33:40:7b:d7   2 BCH2-BOTTOM     sp:  8.000G bw:  8.000G
  5: 11-> 11 10:00:50:eb:1a:6f:2f:3a   5 BROCADE--1   sp:  8.000G bw:  8.000G CR_RECOV
BROCADE-**filtered**-1:r>


BROCADE-**filtered**-1Smiley Embarassed porterrshow
          frames      enc    crc    crc    too    too    bad    enc   disc   link   loss   loss   frjt   fbsy  c3timeout    pcs
       tx     rx      in    err    g_eof  shrt   long   eof     out   c3    fail    sync   sig                 tx    rx    err
  0:    2.4g 799.0m   0     24      0      0      0      0      0     34      0      0      0      0      0     34      0      0
  1:    2.4g 805.4m   0      0      0      0      0      0      0    134      0      0      0      0      0    134      0      0
  2:  514.0m 254.0m   0      9      0      0      0      0      0      0      0      0      0      0      0      0      0      0
  3:  953.0m   1.6g   0      0      0      0      0      0      0     28      0      0      0      0      0      0     20      0
  4:    4.2g   1.5g   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
  5:    4.2g   1.5g   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
  6:    4.2g 339.1m   0      0      0      0      0      0      0     58      0      0      0      0      0      0     53      0
  7:  257.8m   1.2g   0      0      0      0      0      0      0    100      0      0      0      0      0      0     95      0
  8:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
  9:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 10:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 11:  313.3m   2.8g   0    397      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 12:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 13:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 14:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 15:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 16:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 17:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 18:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 19:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 20:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 21:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 22:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 23:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
BROCADE-**filtered**-1Smiley Embarassed

External Moderator
Posts: 4,973
Registered: ‎02-23-2004

Re: Bottlenecks

Rob,

 

this Command "bottleneckmon" cause more damage than you can imagine and is for nothing good. That for first.

 

I have said several times, and indeed AFAIK in FOS 8.x show as not longer available. I'm so happy!

 

back to you Q. I did not understand exactly what your trouble, can you please explain a bit more in details ?

 

thank you.

 

 

TechHelp24
Frequent Contributor
Posts: 141
Registered: ‎05-26-2009

Re: Bottlenecks

I cannot confirm this statement. Configured correctly bottleneckmon was (and for older FOS still is) a valuable tool to identify both congestion and latency bottlenecks especially on 8G platforms. 

Frequent Contributor
Posts: 141
Registered: ‎05-26-2009

Re: Bottlenecks

And by the way: You don't need bottleneckmon to be configured to get the "severe latency bottleneck" messages. If you get those, you know there is really a problem. And in the case above, there are actually even frames dropped due to timout against the blade. So the next step is to troubleshoot this very blade to find out why it's slow draining.

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.