Fibre Channel (SAN)

Reply
Occasional Contributor
Posts: 11
Registered: ‎03-18-2010
Accepted Solution

LW Gbic or dark fibre issue on Brocade 6510s

Hi all,

 

For the past couple of days i have been troubleshooting a very strange issue, after two days of testing I am thinking we have a bad batch of 8Gbps 10Km LW gbics (Brocaded branded and purchased through a major storage vendor).  I have configured many multi-site fabrics (up to 40Km) and never had an issue before. 

 

Here is a summary of the sites.

 

All switches are Brocade 6510's running FOS 7.4.1

All ISL ports set to hard 8Gbps and LE

 

Site A and Site B are about 5Km apart

We have 2 fibre paths from Vendor A and Vendor B

We have 2 pairs (4 cores) from each Vendor - First pair for IP and the second for FC

Vendor A is 5Km in length - as reported on the OTDR and Vendor B is 7km  - all well under 10Km

Site A is actually connect to another site (site C) via 25Km optics, and has been for years

Site A switches has the extended fabrics license, not that it matters for this setup

Site B switches only had an additional POD licence.

 

Configured Fabric A - No problem at all (5Km path)

Site A - SW01 => Vendor A fibre => Site B - SW01

sfpshow:

Site A - SW01 Rx Power: -5.7 (267uW) Tx Power: -1.1 (776uW)

Site B - SW01 Rx Power: -6.8 (208uW) Tx Power: -1.4 (717uW)

 

Fabric B - Solid Orange light on at Site A - No lights at Site B

Site A - SW02 => Vendor B fibre => Site B - SW02

sfpshow:

Site A - SW02 Rx Power: -15.5 (28uW) Tx Power: -1.9 (646uW)

Site B - SW02 Rx Power: -37 (0.2uW) Tx Power: -1.0 (792uW)

 

Troubleshooting so far

 - Swap the gbics with the working fabric - No change (Solid Orange light at site A)

 - Swap to use the IP fibre pair - No change (Solid Orange light at site A)

 - Have the working IP link use our FC pair - no issues at all

 - Replace patch leads with new ones - No change (Solid Orange light at site A) 

 - Clean all connectors  - No change (Solid Orange light at site A)

 - Stretch patch lead at Site B so Site A - SW01 merges with Site B SW02  - Merges fine (this tested the Site B - SW02 config, Port, Gbic and the patch lead at Site A)

 - This is just strange -  swap the send and receive cores at both ends (they are using SC connectors so it's nice and easy) i was expecting to see the orange light to move to Site B as i thought we had a stuffed core.  - No change (Solid Orange light at site A remained).

 - We have had a third party do a TOR( top of rack) => carrier room => long cable => carrier room => TOR  testing and we have a insertion loss of about 7.0 to 7.5 dB loss. within spec.

 - we grabbed an old Brocade 5000 we had at Site A with an old config on it and tried to merge with Site B - SW - No change (Solid Orange light at site A) 

 

So what i am asking and since i don't have access to any other 10Km gbics do you think the Tx power levels on these optics are too low?  The spec saids "Tx Average power: -8.4 to 0.5dB" so what we are seeing looks good on paper.

 

Any other troubleshooting ideas before i log a support call to assist?

 

Thanks for reading and for any responses

 

Craig

 

 

External Moderator
Posts: 4,929
Registered: ‎02-23-2004

Re: LW Gbic or dark fibre issue on Brocade 6510s

@craig.smith

 

can you post please "potcfgshow" both port's afected and site ?

 

and check if VC Link is set to 1

TechHelp24
Occasional Contributor
Posts: 11
Registered: ‎03-18-2010

Re: LW Gbic or dark fibre issue on Brocade 6510s

Hi,

 

Yes it is set to 1

 

Site A - SW2

xxxxxxxxxxxxx:admin>  portcfgshow 47
Area Number:              47
Octet Speed Combo:        1(16G|8G|4G|2G)
Speed Level:              8G
AL_PA Offset 13:          OFF
Trunk Port                ON
Long Distance             LE
VC Link Init              ON
Locked L_Port             OFF
Locked G_Port             OFF
Disabled E_Port           OFF
Locked E_Port             ON
ISL R_RDY Mode            OFF
RSCN Suppressed           OFF
Persistent Disable        OFF
LOS TOV mode              0(OFF)
NPIV capability           ON
QOS Port                  OFF
Port Auto Disable:        OFF
Rate Limit                OFF
EX Port                   OFF
Mirror Port               OFF
SIM Port                  OFF
Credit Recovery           ON
F_Port Buffers            OFF
E_Port Credits            OFF
Fault Delay:              0(R_A_TOV)
NPIV PP Limit:            126
NPIV FLOGI Logout:        OFF
CSCTL mode:               OFF
TDZ mode:                 OFF
D-Port mode:              OFF
D-Port over DWDM:         OFF
Compression:              OFF
Encryption:               OFF
FEC:                      ON
FEC via TTS:              OFF
Non-DFE:                  OFF

 

sfpshow 47

                                           Alarm                  Warn
                                    low         high       low         high
Temperature: 53      Centigrade      -40        90         -10         85
Current:     30.538  mAmps           8.000      80.000     10.000      70.000
Voltage:     3310.9  mVolts          2850.0     3800.0     2970.0      3630.0
RX Power:    -5.8    dBm (262.2uW)   4.5    uW  2238.7 uW  11.2   uW   1122.0 uW
TX Power:    -1.1    dBm (777.9 uW)  26.9   uW  2238.7 uW  67.6   uW   1122.0 uW

 

 

 

 

Site B - SW 2

 

xxxxxxxxxxxxxx:admin> portcfgshow 35
Area Number:              35
Octet Speed Combo:        1(16G|8G|4G|2G)
Speed Level:              8G
AL_PA Offset 13:          OFF
Trunk Port                ON
Long Distance             LE
VC Link Init              ON
Locked L_Port             OFF
Locked G_Port             OFF
Disabled E_Port           OFF
Locked E_Port             ON
ISL R_RDY Mode            OFF
RSCN Suppressed           OFF
Persistent Disable        OFF
LOS TOV mode              0(OFF)
NPIV capability           ON
QOS Port                  OFF
Port Auto Disable:        OFF
Rate Limit                OFF
EX Port                   OFF
Mirror Port               OFF
SIM Port                  OFF
Credit Recovery           ON
F_Port Buffers            OFF
E_Port Credits            OFF
Fault Delay:              0(R_A_TOV)
NPIV PP Limit:            126
NPIV FLOGI Logout:        OFF
CSCTL mode:               OFF
TDZ mode:                 OFF
D-Port mode:              OFF
D-Port over DWDM:         OFF
Compression:              OFF
Encryption:               OFF
FEC:                      ON
FEC via TTS:              OFF
Non-DFE:                  OFF

 

sfpshow 35

 

                                           Alarm                  Warn
                                    low         high       low         high
Temperature: 50      Centigrade      -40        90         -10         85
Current:     28.666  mAmps           8.000      80.000     10.000      70.000
Voltage:     3292.4  mVolts          2850.0     3800.0     2970.0      3630.0
RX Power:    -27.0   dBm (2.0  uW)   4.5    uW  2238.7 uW  11.2   uW   1122.0 uW
TX Power:    -1.0    dBm (786.3 uW)  26.9   uW  2238.7 uW  67.6   uW   1122.0 uW

 

 

from switchshow

 

Site A SW02 it saids:

Index Port Address Media Speed       State   Proto
==================================================

 47  47   072f00   id    8G       In_Sync     FC  LE

 

Site B SW02 it saids:

Index Port Address Media Speed       State   Proto
==================================================

35  35   032300   id    8G       No_Light    FC  LE

External Moderator
Posts: 4,929
Registered: ‎02-23-2004

Re: LW Gbic or dark fibre issue on Brocade 6510s

@craig.smith

 

can you please try "temporarely" follow:

 

on both site

 

portcfgpersistentdisable port_number

portcfgspeed = 4G

portcfgpersistentenable

 

BTW, can you please specified the SFP Part Number ?

 

did you perhaps use SFP with Part Number 57-1000027-01 ?

 

 

TechHelp24
Occasional Contributor
Posts: 11
Registered: ‎03-18-2010

Re: LW Gbic or dark fibre issue on Brocade 6510s

Hi again,

 

SFP details are (i have checked and they are both the same):

 

 sfpshow 35
Identifier:  3    SFP
Connector:   7    LC
Transceiver: 5401001200000000 2,4,8_Gbps SM lw Long_dist
Encoding:    1    8B10B
Baud Rate:   85   (units 100 megabaud)
Length 9u:   10   (units km)
Length 9u:   100  (units 100 meters)
Length 50u (OM2):  0    (units 10 meters)
Length 50u (OM3):  0    (units 10 meters)
Length 62.5u:0    (units 10 meters)
Length Cu:   0    (units 1 meter)
Vendor Name: BROCADE
Vendor OUI:  00:05:1e
Vendor PN:   57-1000027-02
Vendor Rev:  A
Wavelength:  1310 (units nm)
Options:     001a Loss_of_Sig,Tx_Fault,Tx_Disable
BR Max:      0
BR Min:      0
Serial No:   UDA215294198J02
Date Code:   150719
DD Type:     0x68
Enh Options: 0xfa
Status/Ctrl: 0xa2
Alarm flags[0,1] = 0x0, 0x40
Warn Flags[0,1] = 0x0, 0x40

 

 

These are the spfstatus when set to 4G Hard - still get the same result though

 

Site A - SW02

                                           Alarm                  Warn
                                    low         high       low         high
Temperature: 51      Centigrade      -40        90         -10         85
Current:     21.838  mAmps           8.000      80.000     10.000      70.000
Voltage:     3310.9  mVolts          2850.0     3800.0     2970.0      3630.0
RX Power:    -9.1    dBm (123.6uW)   4.5    uW  2238.7 uW  11.2   uW   1122.0 uW
TX Power:    -3.9    dBm (406.3 uW)  26.9   uW  2238.7 uW  67.6   uW   1122.0 uW

 

 

Site B - SW02

                                           Alarm                  Warn
                                    low         high       low         high
Temperature: 49      Centigrade      -40        90         -10         85
Current:     21.112  mAmps           8.000      80.000     10.000      70.000
Voltage:     3292.4  mVolts          2850.0     3800.0     2970.0      3630.0
RX Power:    -32.2   dBm (0.6  uW)   4.5    uW  2238.7 uW  11.2   uW   1122.0 uW
TX Power:    -3.9    dBm (406.1 uW)  26.9   uW  2238.7 uW  67.6   uW   1122.0 uW

 

I beleive these are self attenuating SFPs, have you every seen a problem with them choosing the wrong Tx power levels? is there a way to force the maximum power level?

 

I have left it set at 4G hard for the moment.

 

Craig.

 

External Moderator
Posts: 4,929
Registered: ‎02-23-2004

Re: LW Gbic or dark fibre issue on Brocade 6510s

Craig,

 

->I have left it set at 4G hard for the moment.

 

can you please confirm, because is not clear to me, work the link when Port's are setting at 4GB Fixed Speed?

 

I believe you have Buffer problem.

 

what say portbuffershow ?

 

with portbuffercalc you can calcutate the required buffer on the link.

TechHelp24
External Moderator
Posts: 4,929
Registered: ‎02-23-2004

Re: LW Gbic or dark fibre issue on Brocade 6510s

@craig.smith

 

I have another suspect, but should not affect the functionality, however when Extended fabric License is present, the License is active on all port, and is probable the Switch on Site B, don't accept some fabric.ops from Switch in Fabric A due a missing License.

 

try follow please, keep in mind this is a Trial license Valid till January 2, 2017

 

Extended Fabric License

 

MN44NXYaSm7DXQNZ39TM7HrREMRtMrRDLJFgXKMEQQDaaA7HRD

 

and then set again the port at 8G

 

what happened ?

 

TechHelp24
Occasional Contributor
Posts: 11
Registered: ‎03-18-2010

Re: LW Gbic or dark fibre issue on Brocade 6510s

Ports are definently set to 4G hard,  just remember we are looking at port 47 when looking at Site A

 

Site A - SW02

:admin> portbuffershow
User     Port     Lx     Max/Resv  Avg Buffer Usage & FrameSize    Buffer Needed     Link     Remaining
Port     Type    Mode    Buffers        Tx         Rx              Usage  Buffers   Distance  Buffers
----     ----    ----    -------   ----------------------------    ------ -------   --------- ----------
   0       F       -         8       - (  - )      - (  - )           8       -          -
   1       F       -         8       - (  - )      - (  - )           8       -          -
   2       F       -         8       - (  - )      - (  - )           8       -          -
   3       F       -         8       - (  - )      - (  - )           8       -          -
   4       F       -         8       - ( 904)      - ( 216)           8       -          -
   5       F       -         8       - (  76)       1(1176)           8       -          -
   6       F       -         8       - (  52)       1(1408)           8       -          -
   7       F       -         8       - (  - )      - (  - )           8       -          -
   8       F       -         8        1(1600)       1(2020)           8       -          -
   9       F       -         8       - (  - )      - (  - )           8       -          -
  10       F       -         8       - (  52)       1(1380)           8       -          -
  11       F       -         8        1(1944)       1(2032)           8       -          -
  12       F       -         8        1(1564)       1(1920)           8       -          -
  13       F       -         8       - (  - )      - (  - )           8       -          -
  14       F       -         8       - (  - )      - (  - )           8       -          -
  15       F       -         8       - (  - )      - (  - )           8       -          -
  16       F       -         8        1(2032)      - ( 820)           8       -          -
  17       F       -         8       - ( 172)       1(1796)           8       -          -
  18       F       -         8       - (  52)       1(1376)           8       -          -
  19       F       -         8        1(2000)       1(1612)           8       -          -
  20       F       -         8        1(1796)       1(2020)           8       -          -
  21       F       -         8       - (  52)       1(1500)           8       -          -
  22       F       -         8       - (  - )      - (  - )           8       -          -
  23       F       -         8       - (  - )      - (  - )           8       -          -
  24       F       -         8       - ( 256)      - ( 984)           8       -          -
  25       F       -         8       - (  52)       1(1240)           8       -          -
  26       F       -         8       - (  - )      - (  - )           8       -          -
  27       F       -         8       - (  - )      - (  - )           8       -          -
  28       F       -         8       - (  60)      - ( 464)           8       -          -
  29       F       -         8       - (  - )      - (  - )           8       -          -
  30       F       -         8       - (  - )      - (  - )           8       -          -
  31       F       -         8       - (  - )      - (  - )           8       -          -
  32       F       -         8       - (  - )      - (  - )           8       -          -
  33       F       -         8       - (  - )      - (  - )           8       -          -
  34               -         8       - (  - )      - (  - )           0       -          -
  35               -         8       - (  - )      - (  - )           0       -          -
  36       F       -         8       - (  - )      - (  - )           8       -          -
  37               -         8       - (  - )      - (  - )           0       -          -
  38               -         8       - (  - )      - (  - )           0       -          -
  39       E      LS       106       - (  56)      17(1388)         106     106        50km
  40               -         8       - (  - )      - (  - )           0       -          -
  41               -         8       - (  - )      - (  - )           0       -          -
  42               -         8       - (  - )      - (  - )           0       -          -
  43               -         8       - (  - )      - (  - )           0       -          -
  44               -         8       - (  - )      - (  - )           0       -          -
  45               -         8       - (  - )      - (  - )           0       -          -
  46               -         8       - (  - )      - (  - )           0       -          -
  47       U      LE        26       - (  - )      - (  - )          26       -          -       7596

 

:admin> portcfgshow 47
Area Number:              47
Octet Speed Combo:        1(16G|8G|4G|2G)
Speed Level:              4G
AL_PA Offset 13:          OFF
Trunk Port                ON
Long Distance             LE
VC Link Init              ON
Locked L_Port             OFF
Locked G_Port             OFF
Disabled E_Port           OFF
Locked E_Port             ON
ISL R_RDY Mode            OFF
RSCN Suppressed           OFF
Persistent Disable        OFF
LOS TOV mode              0(OFF)
NPIV capability           ON
QOS Port                  OFF
Port Auto Disable:        OFF
Rate Limit                OFF
EX Port                   OFF
Mirror Port               OFF
SIM Port                  OFF
Credit Recovery           ON
F_Port Buffers            OFF
E_Port Credits            OFF
Fault Delay:              0(R_A_TOV)
NPIV PP Limit:            126
NPIV FLOGI Logout:        OFF
CSCTL mode:               OFF
TDZ mode:                 OFF
D-Port mode:              OFF
D-Port over DWDM:         OFF
Compression:              OFF
Encryption:               OFF
FEC:                      ON
FEC via TTS:              OFF

 

 

Site B - SW 02

 

:admin> portcfgshow 35
Area Number:              35
Octet Speed Combo:        1(16G|8G|4G|2G)
Speed Level:              4G
AL_PA Offset 13:          OFF
Trunk Port                ON
Long Distance             LE
VC Link Init              ON
Locked L_Port             OFF
Locked G_Port             OFF
Disabled E_Port           OFF
Locked E_Port             ON
ISL R_RDY Mode            OFF
RSCN Suppressed           OFF
Persistent Disable        OFF
LOS TOV mode              0(OFF)
NPIV capability           ON
QOS Port                  OFF
Port Auto Disable:        OFF
Rate Limit                OFF
EX Port                   OFF
Mirror Port               OFF
SIM Port                  OFF
Credit Recovery           ON
F_Port Buffers            OFF
E_Port Credits            OFF
Fault Delay:              0(R_A_TOV)
NPIV PP Limit:            126
NPIV FLOGI Logout:        OFF
CSCTL mode:               OFF
TDZ mode:                 OFF
D-Port mode:              OFF
D-Port over DWDM:         OFF
Compression:              OFF
Encryption:               OFF
FEC:                      ON
FEC via TTS:              OFF
Non-DFE:                  OFF

:admin> portbuffershow
User     Port     Lx     Max/Resv  Avg Buffer Usage & FrameSize    Buffer Needed     Link     Remaining
Port     Type    Mode    Buffers        Tx         Rx              Usage  Buffers   Distance  Buffers
----     ----    ----    -------   ----------------------------    ------ -------   --------- ----------
   0               -         8       - (  - )      - (  - )           0       -          -
   1       F       -         8       - (  - )      - (  - )           8       -          -
   2       F       -         8       - (  - )      - (  - )           8       -          -
   3       F       -         8       - (  - )      - (  - )           8       -          -
   4               -         8       - (  - )      - (  - )           0       -          -
   5       F       -         8       - (  - )      - (  - )           8       -          -
   6       F       -         8       - (  - )      - (  - )           8       -          -
   7       F       -         8       - (  - )      - (  - )           8       -          -
   8               -         8       - (  - )      - (  - )           0       -          -
   9       F       -         8       - (  - )      - (  - )           8       -          -
  10       F       -         8       - (  - )      - (  - )           8       -          -
  11       F       -         8       - (  - )      - (  - )           8       -          -
  12               -         8       - (  - )      - (  - )           0       -          -
  13       F       -         8       - (  - )      - (  - )           8       -          -
  14       F       -         8       - (  - )      - (  - )           8       -          -
  15       F       -         8       - (  - )      - (  - )           8       -          -
  16               -         8       - (  - )      - (  - )           0       -          -
  17       F       -         8       - (  - )      - (  - )           8       -          -
  18       F       -         8       - (  - )      - (  - )           8       -          -
  19               -         8       - (  - )      - (  - )           0       -          -
  20               -         8       - (  - )      - (  - )           0       -          -
  21       F       -         8       - (  - )      - (  - )           8       -          -
  22       F       -         8       - (  - )      - (  - )           8       -          -
  23               -         8       - (  - )      - (  - )           0       -          -
  24               -         8       - (  - )      - (  - )           0       -          -
  25               -         8       - (  - )      - (  - )           0       -          -
  26               -         8       - (  - )      - (  - )           0       -          -
  27               -         8       - (  - )      - (  - )           0       -          -
  28               -         8       - (  - )      - (  - )           0       -          -
  29               -         8       - (  - )      - (  - )           0       -          -
  30               -         8       - (  - )      - (  - )           0       -          -
  31               -         8       - (  - )      - (  - )           0       -          -
  32               -         8       - (  - )      - (  - )           0       -          -
  33               -         8       - (  - )      - (  - )           0       -          -
  34               -         8       - (  - )      - (  - )           0       -          -
  35              LE        26       - (  - )      - (  - )           0       -          -
  36               -         8       - (  - )      - (  - )           0       -          -
  37               -         8       - (  - )      - (  - )           0       -          -
  38               -         8       - (  - )      - (  - )           0       -          -
  39               -         8       - (  - )      - (  - )           0       -          -
  40               -         8       - (  - )      - (  - )           0       -          -
  41               -         8       - (  - )      - (  - )           0       -          -
  42               -         8       - (  - )      - (  - )           0       -          -
  43               -         8       - (  - )      - (  - )           0       -          -
  44               -         8       - (  - )      - (  - )           0       -          -
  45               -         8       - (  - )      - (  - )           0       -          -
  46               -         8       - (  - )      - (  - )           0       -          -
  47               -         8       - (  - )      - (  - )           0       -          -       7694

 

I didn't think you had to set port buffers on ISL links when they are <10Km?

 

CRaig.

External Moderator
Posts: 4,929
Registered: ‎02-23-2004

Re: LW Gbic or dark fibre issue on Brocade 6510s

->I didn't think you had to set port buffers on ISL links when they are <10Km?

 

this is correct, in LE Mode the port need just to set at fixed speed to allocated correct Buffer, and from the output it's show correct.

 

did you tried to add a Trial License ?

TechHelp24
Occasional Contributor
Posts: 11
Registered: ‎03-18-2010

Re: LW Gbic or dark fibre issue on Brocade 6510s

Hi,

 

I have added the trail licence to the Site B - SW02 and set the ports back to 8Gb hard.

 

xxxxxxxxxxxxxxxxx:admin> licenseshow
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:
    First Ports on Demand license - additional 12 port upgrade license
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:
    Extended Fabric license
    Expiry Date 02/01/2017

 

No change at all

 

Site A - SW02

                                    low         high       low         high
Temperature: 52      Centigrade      -40        90         -10         85
Current:     30.450  mAmps           8.000      80.000     10.000      70.000
Voltage:     3310.9  mVolts          2850.0     3800.0     2970.0      3630.0
RX Power:    -5.9    dBm (259.8uW)   4.5    uW  2238.7 uW  11.2   uW   1122.0 uW
TX Power:    -1.1    dBm (777.1 uW)  26.9   uW  2238.7 uW  67.6   uW   1122.0 uW

 

Site B - SW02
                                    low         high       low         high
Temperature: 50      Centigrade      -40        90         -10         85
Current:     28.666  mAmps           8.000      80.000     10.000      70.000
Voltage:     3292.4  mVolts          2850.0     3800.0     2970.0      3630.0
RX Power:    -27.0   dBm (2.0  uW)   4.5    uW  2238.7 uW  11.2   uW   1122.0 uW
TX Power:    -1.0    dBm (794.6 uW)  26.9   uW  2238.7 uW  67.6   uW   1122.0 uW

 

 

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.