For more details, please see ourCookie Policy.


Fibre Channel (SAN)

Reply
Occasional Contributor
Posts: 5
Registered: ‎05-11-2018
Accepted Solution

BE_PORT_LATENCY on DCX

I'm reciving alert with back end ports latency.

 

BE Port 6/85, Condition=ALL_BE_PORTS(BE_LATENCY_IMPACT==IO_PERF_IMPACT), Current Value:[BE_LATENCY_IMPACT,IO_PERF_IMPACT, 88.6% of 1 secs], RuleName=defALL_BE_PORTS_LATENCY_IMPACT, Dashboard Category=Fabric Performance Impact.

 

I have recived a bulk port alerts at once.

 

How can I find the root of issue?

How can I understood a BE port address 6/85?

Best regards,
Anton Karashov
Frequent Contributor
Posts: 83
Registered: ‎04-04-2018

Re: BE_PORT_LATENCY on DCX

[ Edited ]

Hello,

 

as i see you based on message its comming from MAPS monitoring. Based on message "ALL_BE_PORTS" means that the latency bottleneck has been detected on backen port of DCX.

 

What the backend port is si nice explained in MAPS admin guide:

 

A back-end port connects a core switching blade to a port or application blade (and vice versa). The primary task of back-end ports is to
route packets passing through a switch's ASICs. Switch (and consequently fabric) performance degrades when there are errors in backend
ports. MAPS error notification allows you to take corrective action earlier. In terms of monitoring system functionality, back-end
ports can be connected to ports within a fixed-port switch or to other blades within a Backbone chassis, and so their functionality is
different from front-end ports, which connect to devices outside of the switch.

Do you have credit recovery enabled ?

Please mark this response as "Accept as Solution" if it answers your question.

Marian Bezeg
Contributor
Posts: 53
Registered: ‎01-19-2018

Re: BE_PORT_LATENCY on DCX

Hi Anton :)

You can find the internal port numbers using the bladeportmap command.

Backend latencies could be caused by all the same reasons that are applicable to the normal ISL latencies. You may think that backend links are ISLs between the ASICs, like there is a small SAN fabric internal to the chassis.

Is it 8G or 16G DCX? Do you have the bottleneckmon setup? It would fix many internal issues on the flight. Have you seen any other messages around the alert time?
Occasional Contributor
Posts: 5
Registered: ‎05-11-2018

Re: BE_PORT_LATENCY on DCX

Yes, we have enabled credit recovery on the ISL trunks. And we have enabled QOS on the ISL.
Best regards,
Anton Karashov
Occasional Contributor
Posts: 5
Registered: ‎05-11-2018

Re: BE_PORT_LATENCY on DCX

I don't understand the output of bladeportmap.

CTL

BPt

UPt uplink port

SPt #frontend port-1:backend

MPt minis port #

CPt chip port#

IFi

GPt

M# (Minis)

C# (Chip)

Sw# -1:BackEnd, 0:FrontEnd

G#

ChID

PBPt:ISL(Backend port)

OID

 

 bladeportmap 6
miniS=0 chips/Minis=1 bports/chip=48
miniS=1 chips/Minis=1 bports/chip=48
minis_count=2 chip_count=2 port_count=96

CTL BPt  UPt  SPt  MPt  CPt  IFi  GPt  M#   C#   Sw#  G#   ChID       PBPt:ISL        OID
================================================================================================
ENB 0    793  25   0    0    1    0    0    0    2    6    CONDOR3    :No_Module      0x43628000
ENB 1    797  29   1    1    2    1    0    0    2    6    CONDOR3    :No_Module      0x43628001
ENB 2    281  9    2    2    3    2    0    0    2    6    CONDOR3    :No_Module      0x43628002
ENB 3    285  13   3    3    4    3    0    0    2    6    CONDOR3    :No_Module      0x43628003
ENB 4    789  21   4    4    5    4    0    0    2    6    CONDOR3    :No_Module      0x43628004
ENB 5    785  17   5    5    6    5    0    0    2    6    CONDOR3    :No_Module      0x43628005
ENB 6    277  5    6    6    7    6    0    0    2    6    CONDOR3    :No_Module      0x43628006
ENB 7    273  1    7    7    8    7    0    0    2    6    CONDOR3    :No_Module      0x43628007
ENB 8    -1   -1   8    8    13   0    0    0    -1   1804 CONDOR3    7/2:            0x43628008
ENB 9    -1   -1   9    9    13   1    0    0    -1   1804 CONDOR3    7/4:            0x43628009
ENB 10   -1   -1   10   10   15   2    0    0    -1   1806 CONDOR3    7/51:           0x4362800a
ENB 11   -1   -1   11   11   15   3    0    0    -1   1806 CONDOR3    7/52:           0x4362800b
ENB 12   -1   -1   12   12   13   4    0    0    -1   1804 CONDOR3    7/1:            0x4362800c
ENB 13   -1   -1   13   13   13   5    0    0    -1   1804 CONDOR3    7/3:            0x4362800d
ENB 14   -1   -1   14   14   15   6    0    0    -1   1806 CONDOR3    7/49:           0x4362800e
ENB 15   -1   -1   15   15   15   7    0    0    -1   1806 CONDOR3    7/50:           0x4362800f
ENB 16   -1   -1   16   16   21   0    0    0    -1   1812 CONDOR3    8/2:            0x43628010
ENB 17   -1   -1   17   17   18   1    0    0    -1   1809 CONDOR3    8/49:           0x43628011
ENB 18   -1   -1   18   18   18   2    0    0    -1   1809 CONDOR3    8/52:           0x43628012
ENB 19   -1   -1   19   19   21   3    0    0    -1   1812 CONDOR3    8/3:            0x43628013
ENB 20   -1   -1   20   20   21   4    0    0    -1   1812 CONDOR3    8/1:            0x43628014
ENB 21   -1   -1   21   21   21   5    0    0    -1   1812 CONDOR3    8/4:            0x43628015
ENB 22   -1   -1   22   22   18   6    0    0    -1   1809 CONDOR3    8/51:           0x43628016
ENB 23   -1   -1   23   23   18   7    0    0    -1   1809 CONDOR3    8/50:           0x43628017
ENB 24   276  4    24   24   25   0    0    0    2    9    CONDOR3    :No_Module      0x43628018
ENB 25   272  0    25   25   26   1    0    0    2    9    CONDOR3    :No_Module      0x43628019
ENB 26   788  20   26   26   27   2    0    0    2    9    CONDOR3    :No_Module      0x4362801a
ENB 27   784  16   27   27   28   3    0    0    2    9    CONDOR3    :No_Module      0x4362801b
ENB 28   280  8    28   28   29   4    0    0    2    9    CONDOR3    :No_Module      0x4362801c
ENB 29   284  12   29   29   30   5    0    0    2    9    CONDOR3    :No_Module      0x4362801d
ENB 30   792  24   30   30   31   6    0    0    2    9    CONDOR3    :No_Module      0x4362801e
ENB 31   796  28   31   31   32   7    0    0    2    9    CONDOR3    :No_Module      0x4362801f
ENB 32   -1   -1   32   32   33   0    0    0    -1   514  CONDOR3    1/2:            0x43628020
ENB 33   -1   -1   33   33   35   1    0    0    -1   562  CONDOR3    1/50:           0x43628021
ENB 34   -1   -1   34   34   35   2    0    0    -1   562  CONDOR3    1/49:           0x43628022
ENB 35   -1   -1   35   35   33   3    0    0    -1   514  CONDOR3    1/4:            0x43628023
ENB 36   -1   -1   36   36   33   4    0    0    -1   514  CONDOR3    1/3:            0x43628024
ENB 37   -1   -1   37   37   35   5    0    0    -1   562  CONDOR3    1/51:           0x43628025
ENB 38   -1   -1   38   38   35   6    0    0    -1   562  CONDOR3    1/52:           0x43628026
ENB 39   -1   -1   39   39   33   7    0    0    -1   514  CONDOR3    1/1:            0x43628027
ENB 40   -1   -1   40   40   41   0    0    0    -1   769  CONDOR3    2/1:            0x43628028
ENB 41   -1   -1   41   41   41   1    0    0    -1   769  CONDOR3    2/2:            0x43628029
ENB 42   -1   -1   42   42   41   2    0    0    -1   769  CONDOR3    2/3:            0x4362802a
ENB 43   -1   -1   43   43   41   3    0    0    -1   769  CONDOR3    2/4:            0x4362802b
ENB 44   -1   -1   44   44   48   4    0    0    -1   818  CONDOR3    2/50:           0x4362802c
ENB 45   -1   -1   45   45   48   5    0    0    -1   818  CONDOR3    2/51:           0x4362802d
ENB 46   -1   -1   46   46   48   6    0    0    -1   818  CONDOR3    2/52:           0x4362802e
ENB 47   -1   -1   47   47   48   7    0    0    -1   818  CONDOR3    2/49:           0x4362802f
ENB 48   795  27   0    0    1    0    1    1    2    18   CONDOR3    :No_Module      0x43628030
ENB 49   799  31   1    1    2    1    1    1    2    18   CONDOR3    :No_Module      0x43628031
ENB 50   283  11   2    2    3    2    1    1    2    18   CONDOR3    :No_Module      0x43628032
ENB 51   287  15   3    3    4    3    1    1    2    18   CONDOR3    :No_Module      0x43628033
ENB 52   791  23   4    4    5    4    1    1    2    18   CONDOR3    :No_Module      0x43628034
ENB 53   787  19   5    5    6    5    1    1    2    18   CONDOR3    :No_Module      0x43628035
ENB 54   279  7    6    6    7    6    1    1    2    18   CONDOR3    :No_Module      0x43628036
ENB 55   275  3    7    7    8    7    1    1    2    18   CONDOR3    :No_Module      0x43628037
ENB 56   -1   -1   8    8    12   0    1    1    -1   1851 CONDOR3    7/61:           0x43628038
ENB 57   -1   -1   9    9    10   1    1    1    -1   1849 CONDOR3    7/8:            0x43628039
ENB 58   -1   -1   10   10   10   2    1    1    -1   1849 CONDOR3    7/10:           0x4362803a
ENB 59   -1   -1   11   11   12   3    1    1    -1   1851 CONDOR3    7/56:           0x4362803b
ENB 60   -1   -1   12   12   10   4    1    1    -1   1849 CONDOR3    7/12:           0x4362803c
ENB 61   -1   -1   13   13   12   5    1    1    -1   1851 CONDOR3    7/62:           0x4362803d
ENB 62   -1   -1   14   14   10   6    1    1    -1   1849 CONDOR3    7/15:           0x4362803e
ENB 63   -1   -1   15   15   12   7    1    1    -1   1851 CONDOR3    7/58:           0x4362803f
ENB 64   -1   -1   16   16   22   0    1    1    -1   1861 CONDOR3    8/10:           0x43628800
ENB 65   -1   -1   17   17   18   1    1    1    -1   1857 CONDOR3    8/56:           0x43628801
ENB 66   -1   -1   18   18   18   2    1    1    -1   1857 CONDOR3    8/61:           0x43628802
ENB 67   -1   -1   19   19   22   3    1    1    -1   1861 CONDOR3    8/15:           0x43628803
ENB 68   -1   -1   20   20   22   4    1    1    -1   1861 CONDOR3    8/12:           0x43628804
ENB 69   -1   -1   21   21   22   5    1    1    -1   1861 CONDOR3    8/8:            0x43628805
ENB 70   -1   -1   22   22   18   6    1    1    -1   1857 CONDOR3    8/62:           0x43628806
ENB 71   -1   -1   23   23   18   7    1    1    -1   1857 CONDOR3    8/58:           0x43628807
ENB 72   278  6    24   24   25   0    1    1    2    21   CONDOR3    :No_Module      0x43628808
ENB 73   274  2    25   25   26   1    1    1    2    21   CONDOR3    :No_Module      0x43628809
ENB 74   790  22   26   26   27   2    1    1    2    21   CONDOR3    :No_Module      0x4362880a
ENB 75   786  18   27   27   28   3    1    1    2    21   CONDOR3    :No_Module      0x4362880b
ENB 76   282  10   28   28   29   4    1    1    2    21   CONDOR3    :No_Module      0x4362880c
ENB 77   286  14   29   29   30   5    1    1    2    21   CONDOR3    :No_Module      0x4362880d
ENB 78   794  26   30   30   31   6    1    1    2    21   CONDOR3    :No_Module      0x4362880e
ENB 79   798  30   31   31   32   7    1    1    2    21   CONDOR3    :No_Module      0x4362880f
ENB 80   -1   -1   32   32   33   0    1    1    -1   522  CONDOR3    1/10:           0x43628810
ENB 81   -1   -1   33   33   34   1    1    1    -1   570  CONDOR3    1/58:           0x43628811
ENB 82   -1   -1   34   34   34   2    1    1    -1   570  CONDOR3    1/61:           0x43628812
ENB 83   -1   -1   35   35   33   3    1    1    -1   522  CONDOR3    1/12:           0x43628813
ENB 84   -1   -1   36   36   33   4    1    1    -1   522  CONDOR3    1/15:           0x43628814
ENB 85   -1   -1   37   37   34   5    1    1    -1   570  CONDOR3    1/62:           0x43628815
ENB 86   -1   -1   38   38   34   6    1    1    -1   570  CONDOR3    1/56:           0x43628816
ENB 87   -1   -1   39   39   33   7    1    1    -1   522  CONDOR3    1/8:            0x43628817
ENB 88   -1   -1   40   40   44   0    1    1    -1   780  CONDOR3    2/12:           0x43628818
ENB 89   -1   -1   41   41   44   1    1    1    -1   780  CONDOR3    2/10:           0x43628819
ENB 90   -1   -1   42   42   44   2    1    1    -1   780  CONDOR3    2/15:           0x4362881a
ENB 91   -1   -1   43   43   44   3    1    1    -1   780  CONDOR3    2/8:            0x4362881b
ENB 92   -1   -1   44   44   48   4    1    1    -1   826  CONDOR3    2/58:           0x4362881c
ENB 93   -1   -1   45   45   48   5    1    1    -1   826  CONDOR3    2/62:           0x4362881d
ENB 94   -1   -1   46   46   48   6    1    1    -1   826  CONDOR3    2/61:           0x4362881e
ENB 95   -1   -1   47   47   48   7    1    1    -1   826  CONDOR3    2/56:           0x4362881f

It is the 16G DCX.

We have FPI enabled, he was disable bottleneck.

I want to find FE ports, wich create traffic to BE port (Mapping FE to BE).

Best regards,
Anton Karashov
Frequent Contributor
Posts: 83
Registered: ‎04-04-2018

Re: BE_PORT_LATENCY on DCX

Hello,

 

based on your output its DCX 8510 - 4. Only in this case you may have core blade in slot 6.

From your output latency is on 6/85, which means slot 6 and backend port 85. As you may see its mapped to port 1/62 on slot 1.

 

ENB 85   -1   -1   37   37   34   5    1    1    -1   570  CONDOR3    1/62:           0x43628815

If you will check portmap for slot 1 you will see to which ASIC is this CORE BLADE backend port connected. 

But there my knowledge about mapping ends...there is command sloterrshow where you may check errors but i still didnt decode that.

 

 

Please mark this response as "Accept as Solution" if it answers your question.

Marian Bezeg
Broadcom Moderator
Posts: 114
Registered: ‎04-27-2009

Re: BE_PORT_LATENCY on DCX

Hi,

 

I would watch out for FE FPI Messages at these times no find the contributing FE Port(s).

 

TL

If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution".

Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider.
Occasional Contributor
Posts: 5
Registered: ‎05-11-2018

Re: BE_PORT_LATENCY on DCX

Hi,
There is not FPI Messages at this time on the FE ports.
Best regards,
Anton Karashov
Occasional Contributor
Posts: 5
Registered: ‎05-11-2018

Re: BE_PORT_LATENCY on DCX

Hi!

FOS 7.4.1c

I did'n make statsclear.

The Alerts is rising time to time.

I found that the BE port going to slot 1.

root> bladeportmap 6

BPt - Back End port number

PBPt:ISL - Outgoing Back End slot number/port number

I have seen BB credit pick on the FE port 1/9 at this time. http://c2n.me/3U468o8

This is ISL to blade switch.

On the blade switch i have seen BB credit pick on the FE port 1/9.

This is the NPIV port (Power server) with one virtual port with pick in port send Mb/s and have one LUN

Best regards,
Anton Karashov
Frequent Contributor
Posts: 83
Registered: ‎04-04-2018

Re: BE_PORT_LATENCY on DCX

Hello, 

 

do you see any bottleneck on this NPIV port? Check portstatsshow about this.

It looks like that you have slowdrain device in your fabric which is consuming your BB credits.

 

Sometimes its hard do find out that, my advice is to clear stats on all switches and look on c3 TX/RX timeouts after some time.

 

 

Please mark this response as "Accept as Solution" if it answers your question.

Marian Bezeg

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.