Fibre Channel (SAN)

Reply
Contributor
Posts: 29
Registered: ‎06-21-2011

DCX ISL to 48000 problems

I wonder if somebody code decipher the attached fabriclog entries?

The extracts show the beginning and end of an interuption on one ISL port. The ISL is formed from 2 links:

trunkshow -perf

  2:243->  0 10:00:00:05:1e:66:26:16 116 deskew 15 MASTER

     244->  1 10:00:00:05:1e:66:26:16 116 deskew 15

The two log extracts are showing (I believe) the link in one of the trunks going down and othe other taking over).

We have had problems with our ISLs effectively "flapping" and causing the traffic to re-route,this in turns seems to affect all traffic and slow things down to a crawl.

DCX fabriclog:

20:18:00.743675 SCN EPORT Change,old_m=1,g=0x0              A1,P0  A1,P0  0     NA - Is this the master link flipping over to the slave?

20:18:11.134546 AUTH start failed (Unknown Error);g=0x0     A1,P3  A1,P3  0     NA  - not known what this is indicating

20:18:11.134551 *Snd e-port change (1) (0)                  A1,NA  A1,NA  NA    NA

20:18:11.134572 SCN Port EPORT_CHANGE: Offline              A1,P3  A1,P0  1     NA - now it looks like a port going offline?

20:18:11.134576 *Snd port state: P0                         A1,P0  A1,P0  1     NA

20:18:11.134586 *Removing all nodes from port               A1,P0  A1,P0  1     NA

20:18:11.164381 SCN LR_PORT (0);g=0x8a6                     A1,P0  A1,P0  1     NA

20:18:11.164398 SCN Port Online;g=0x8a6                     A1,P0  A1,P1  1     NA

20:18:12.262392 *ELP Send:rev=2,flow=4,flen=288,size=372    A1,P1  A1,P1  1     0xc016

20:18:12.262426 ELP Drop > 6000 (msecs)                     A1,P1  A1,P1  1     0xb95f

20:18:12.262446 ELP Rcv:rev=2,flow=1,flen=80,size=164       A1,P1  A1,P1  1     0xb97d

20:18:12.262594 *ELP Snd RJT(wwn) ct prfrm, in prgs         A1,P1  A1,P1  1     0xb97d

20:18:12.266785 SCN LR_PORT (0);g=0x8a6                     A1,P1  A1,P1  1     NA

20:18:12.266877 ELP RJT Rcv, nt spt, clt mod                A1,P1  A1,P1  1     0xc016

20:18:12.267431 Send ELP requesting VC RDY + ARB mode       A1,NA  A1,NA  NA    NA - is this the link trying to re-establish

20:18:12.267452 *Start ELP node timer                       A1,P1  A1,P1  1     NA

20:18:12.267730 *ELP Send:rev=2,flow=1,flen=80,size=164     A1,P1  A1,P1  1     0xc084

20:18:12.267738 ELP Retry 1 times                           A1,P1  A1,P1  1     0xc084 - then timing out?

20:18:12.281396 ELP ACC Rcv:rev=2,flow=1,flen=80,size=164   A1,P1  A1,P2  1     0xc084

20:18:12.282267 Negotiated VC_RDY,qos=1,interop=0,rrdy=0    A1,P2  A1,P2  1     0xc084

20:18:12.282688 *Initiate Emt                               A1,P2  A1,I0  1     NA

20:18:12.282919 *EMT Send                                   A1,I0  A1,I0  1     0xc049

20:18:12.297125 *EMT ACC Rcv                                A1,I0  A1,I1  1     0xc049

20:18:12.297130 *EMT next, NONE                             A1,NA  A1,NA  NA    NA

20:18:12.297209 *ETP next, NONE                             A1,NA  A1,NA  NA    NA

20:18:12.297355 *ETP Send                                   A1,I1  A1,I1  1     0xc042

20:18:12.297362 ETP Sent for mport 0, of 0 1 0 0            A1,I1  A1,I2  1     NA

20:18:12.321244 *Snd slv port (1) (0) (1)                   A1,NA  A1,NA  NA    NA - And is this the MATER / SLAVE changing back and the link coming back?

20:18:12.321263 ETP Rcv ACC set mport 0                     A1,I2  A1,I3  1     0xc042

20:18:12.321291 *ETP next, NONE                             A1,NA  A1,NA  NA    NA

20:18:12.421664 SCN T_PORT                                  A1,I3  A1,I4  1

N/A
Posts: 1
Registered: ‎03-19-2013

Re: DCX ISL to 48000 problems

This is looks like a there is some issue with the trunck configuration issue, can you validate the master and slave. and port speeds configured.

paste the trunck debug output, for further investigation if needed.

Valued Contributor
Posts: 761
Registered: ‎06-11-2010

Re: DCX ISL to 48000 problems

Hi,

How are the ports configured? please, paste the output of portcfgshow for each port.

are there errors in these ports? please, paste the output of portshow and portstatsshow for these ports.

I do not know the reason why the port could go offline, but when it comes back online, the events shown are normal quite normal when to ports are negotiating to become an ISL. In this case, the ELP process is taking place and both engs send over their parameters and the remote end reject them if it is not able to support them. When any of them send a set of parameters supported by both ends, an ACC is sent and the ISL is established. Once the ISL is up, the trunk process starts.

rgds

Contributor
Posts: 29
Registered: ‎06-21-2011

Re: DCX ISL to 48000 problems

From 48000:

Index:            243
---------------- +---
Speed              4G
Fill Word          0
AL_PA Offset 13    ..
Trunk Port         ON
Long Distance      ..
VC Link Init       ..
Locked L_Port      ..
Locked G_Port      ..
Disabled E_Port    ..
Locked E_Port      ..
ISL R_RDY Mode     ..
RSCN Suppressed    ..
Persistent Disable ..
LOS TOV enable     ..
NPIV capability    ON
NPIV PP Limit     126
QOS E_Port         ..
Mirror Port        ..
Fport Buffers      ..
Port Auto Disable  ..

  where AE:QoSAutoEnable, AN:AutoNegotiate, ..:OFF, NA:NotApplicable, ??:INVALID,

LM:L0.5
RELN_PRO_SAN002:mowatw> portcfgshow -i 244
Index:            244
---------------- +---
Speed              4G
Fill Word          0
AL_PA Offset 13    ..
Trunk Port         ON
Long Distance      ..
VC Link Init       ..
Locked L_Port      ..
Locked G_Port      ..
Disabled E_Port    ..
Locked E_Port      ..
ISL R_RDY Mode     ..
RSCN Suppressed    ..
Persistent Disable ..
LOS TOV enable     ..
NPIV capability    ON
NPIV PP Limit     126
QOS E_Port         ..
Mirror Port        ..
Fport Buffers      ..
Port Auto Disable  ..

From DCX:

Index:              0
---------------- +---
Speed              4G
Fill Word          0
AL_PA Offset 13    ..
Trunk Port         ON
Long Distance      ..
VC Link Init       ..
Locked L_Port      ..
Locked G_Port      ..
Disabled E_Port    ..
Locked E_Port      ..
ISL R_RDY Mode     ..
RSCN Suppressed    ..
Persistent Disable ..
LOS TOV enable     ..
NPIV capability    ON
NPIV PP Limit     126
QOS E_Port         AE
EX Port            ..
Mirror Port        ..
Rate Limit         ..
Credit Recovery    ON
Fport Buffers      ..
Port Auto Disable  ..
CSCTL mode         ..

  where AE:QoSAutoEnable, AN:AutoNegotiate, ..:OFF, NA:NotApplicable, ??:INVALID,

LM:L0.5
RELN_PRO_SAN004:mowatw> portcfgshow -i 1
Index:              1
---------------- +---
Speed              4G
Fill Word          0
AL_PA Offset 13    ..
Trunk Port         ON
Long Distance      ..
VC Link Init       ..
Locked L_Port      ..
Locked G_Port      ..
Disabled E_Port    ..
Locked E_Port      ..
ISL R_RDY Mode     ..
RSCN Suppressed    ..
Persistent Disable ..
LOS TOV enable     ..
NPIV capability    ON
NPIV PP Limit     126
QOS E_Port         AE
EX Port            ..
Mirror Port        ..
Rate Limit         ..
Credit Recovery    ON
Fport Buffers      ..
Port Auto Disable  ..
CSCTL mode         ..

Valued Contributor
Posts: 761
Registered: ‎06-11-2010

Re: DCX ISL to 48000 problems

Hi,

the 48000 has QoS disabled by default whilst DCX has it AE, you can directly disable it in the DCX. The same happens with Credit Recovery, but these facts does not seem to be related with the ISL going offline.

Please, post the output of portstatsshow and portshow for these ports and we may see something that can explain this behavior. On the other hand, is there any event in fabriclog prior to

20:18:00.743675 SCN EPORT Change,old_m=1,g=0x0

rgds

Contributor
Posts: 29
Registered: ‎06-21-2011

Re: DCX ISL to 48000 problems

DCX:

RELN_PRO_SAN004:mowatw> portshow -i 0
portIndex:   0
portName:
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x10000903    PRESENT ACTIVE E_PORT T_PORT T_MASTER G_PORT U_PORT LOGICAL_ONLINE LOGIN
LocalSwcFlags: 0x0
portType:  17.0
portState: 1    Online
Protocol: FC
portPhys:  6    In_Sync         portScn:   16   E_Port    Trunk master port
port generation number:    0
state transition count:    0

portId:    740000
portIfId:    43120038
portWwn:   20:00:00:05:1e:66:26:16
portWwn of device(s) connected:

Distance:  normal
portSpeed: 4Gbps

LE domain: 0
FC Fastwrite: OFF
Interrupts:        0          Link_failure: 0          Frjt:         0
Unknown:           0          Loss_of_sync: 0          Fbsy:         0
Lli:               1          Loss_of_sig:  0
Proc_rqrd:         6145       Protocol_err: 0
Timed_out:         0          Invalid_word: 0
Rx_flushed:        0          Invalid_crc:  0
Tx_unavail:        0          Delim_err:    0
Free_buffer:       0          Address_err:  0
Overrun:           0          Lr_in:        0
Suspended:         0          Lr_out:       0
Parity_err:        0          Ols_in:       0
2_parity_err:      0          Ols_out:      0
CMI_bus_err:       0

Port part of other ADs: No

RELN_PRO_SAN004:mowatw> portshow -i 1
portIndex:   1
portName: ISL to reln-pro-san002
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x4903        PRESENT ACTIVE E_PORT T_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN LED
LocalSwcFlags: 0x0
portType:  17.0
portState: 1    Online
Protocol: FC
portPhys:  6    In_Sync         portScn:   128  Unknown   Trunk port
port generation number:    2214
state transition count:    2

portId:    740100
portIfId:    43120038
portWwn:   20:01:00:05:1e:66:26:16
portWwn of device(s) connected:

Distance:  normal
portSpeed: 4Gbps

LE domain: 0
FC Fastwrite: OFF
Interrupts:        0          Link_failure: 0          Frjt:         0
Unknown:           0          Loss_of_sync: 0          Fbsy:         0
Lli:               0          Loss_of_sig:  0
Proc_rqrd:         0          Protocol_err: 0
Timed_out:         0          Invalid_word: 0
Rx_flushed:        0          Invalid_crc:  0
Tx_unavail:        0          Delim_err:    0
Free_buffer:       0          Address_err:  0
Overrun:           0          Lr_in:        0
Suspended:         0          Lr_out:       0
Parity_err:        0          Ols_in:       0
2_parity_err:      0          Ols_out:      0
CMI_bus_err:       0

48000:

RELN_PRO_SAN002:mowatw> portshow -i 243
portIndex: 243
portName:
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x10004903    PRESENT ACTIVE E_PORT T_PORT T_MASTER G_PORT U_PORT LOGICAL_ONLINE LOGIN LED
LocalSwcFlags: 0x0
portType:  10.0
portState: 1    Online
Protocol: FC
portPhys:  6    In_Sync         portScn:   16   E_Port    Trunk master port
port generation number:    0
state transition count:    0

portId:    6af300
portIfId:    43a2002c
portWwn:   20:f3:00:05:1e:36:0d:b4
portWwn of device(s) connected:

Distance:  normal
portSpeed: 4Gbps

LE domain: 0
FC Fastwrite: OFF
Interrupts:        1          Link_failure: 0          Frjt:         0
Unknown:           0          Loss_of_sync: 0          Fbsy:         0
Lli:               1          Loss_of_sig:  0
Proc_rqrd:         16788      Protocol_err: 0
Timed_out:         0          Invalid_word: 7
Rx_flushed:        0          Invalid_crc:  0
Tx_unavail:        0          Delim_err:    0
Free_buffer:       0          Address_err:  0
Overrun:           0          Lr_in:        0
Suspended:         0          Lr_out:       0
Parity_err:        0          Ols_in:       0
2_parity_err:      0          Ols_out:      0
CMI_bus_err:       0

Port part of other ADs: No

RELN_PRO_SAN002:mowatw> portshow -i 244
portIndex: 244
portName: ISL to reln-pro-san004
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x4903        PRESENT ACTIVE E_PORT T_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN LED
LocalSwcFlags: 0x0
portType:  10.0
portState: 1    Online
Protocol: FC
portPhys:  6    In_Sync         portScn:   128  Unknown   Trunk port
port generation number:    420
state transition count:    2

portId:    6af400
portIfId:    43a2002c
portWwn:   20:f4:00:05:1e:36:0d:b4
portWwn of device(s) connected:

Distance:  normal
portSpeed: 4Gbps

LE domain: 0
FC Fastwrite: OFF
Interrupts:        0          Link_failure: 0          Frjt:         0
Unknown:           0          Loss_of_sync: 0          Fbsy:         0
Lli:               0          Loss_of_sig:  0
Proc_rqrd:         0          Protocol_err: 0
Timed_out:         0          Invalid_word: 0
Rx_flushed:        0          Invalid_crc:  0
Tx_unavail:        0          Delim_err:    0
Free_buffer:       0          Address_err:  0
Overrun:           0          Lr_in:        0
Suspended:         0          Lr_out:       0
Parity_err:        0          Ols_in:       0
2_parity_err:      0          Ols_out:      0
CMI_bus_err:       0

Port part of other ADs: No

Contributor
Posts: 29
Registered: ‎06-21-2011

Re: DCX ISL to 48000 problems

48000:

RELN_PRO_SAN002:mowatw> portstatsshow -i 244

port:  244
=========
stat_wtx                2991866092  4-byte words transmitted
stat_wrx                1333303156  4-byte words received
stat_ftx                103944721   Frames transmitted
stat_frx                28452342    Frames received
stat_cdr_frx            0           Class 2 frames received
stat_c3_frx             28452301    Class 3 frames received
stat_lc_rx              28          Link control frames received
stat_mc_rx              0           Multicast frames received
stat_mc_to              0           Multicast timeouts
stat_mc_tx              0           Multicast frames transmitted
tim_rdy_pri             0           Time R_RDY high priority
tim_txcrd_z             6408        Time BB credit zero (2.5Us ticks)
er_enc_in               0           Encoding errors inside of frames
er_crc                  0           Frames with CRC errors
er_trunc                0           Frames shorter than minimum
er_toolong              0           Frames longer than maximum
er_bad_eof              0           Frames with bad end-of-frame
er_enc_out              0           Encoding error outside of frames
er_bad_os               0           Invalid ordered set
er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout
er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout
er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable
er_other_discard        0           Other discards
er_zone_discard         0           Class 3 frames discarded due to zone mismatch
er_crc_good_eof         0           Crc error with good eof
er_inv_arb              0           Invalid ARB
open                    0           loop_open
transfer                0           loop_transfer
opened                  0           FL_Port opened
starve_stop             0           tenancies stopped due to starvation
fl_tenancy              0           number of times FL has the tenancy
nl_tenancy              0           number of times NL has the tenancy
zero_tenancy            0           zero tenancy
RELN_PRO_SAN002:mowatw> portstatsshow -i 243

port:  243
=========
stat_wtx                3152986284  4-byte words transmitted
stat_wrx                2196705328  4-byte words received
stat_ftx                958117347   Frames transmitted
stat_frx                3477919327  Frames received
stat_cdr_frx            0           Class 2 frames received
stat_c3_frx             3470495779  Class 3 frames received
stat_lc_rx              4337541     Link control frames received
stat_mc_rx              0           Multicast frames received
stat_mc_to              0           Multicast timeouts
stat_mc_tx              0           Multicast frames transmitted
tim_rdy_pri             8           Time R_RDY high priority
tim_txcrd_z             6408        Time BB credit zero (2.5Us ticks)
er_enc_in               0           Encoding errors inside of frames
er_crc                  8           Frames with CRC errors
er_trunc                0           Frames shorter than minimum
er_toolong              0           Frames longer than maximum
er_bad_eof              0           Frames with bad end-of-frame
er_enc_out              7           Encoding error outside of frames
er_bad_os               912         Invalid ordered set
er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout
er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout
er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable
er_other_discard        150         Other discards
er_zone_discard         0           Class 3 frames discarded due to zone mismatch
er_crc_good_eof         0           Crc error with good eof
er_inv_arb              0           Invalid ARB
open                    0           loop_open
transfer                0           loop_transfer
opened                  0           FL_Port opened
starve_stop             0           tenancies stopped due to starvation
fl_tenancy              0           number of times FL has the tenancy
nl_tenancy              0           number of times NL has the tenancy
zero_tenancy            0           zero tenancy

RELN_PRO_SAN004:mowatw> portstatsshow -i 0

port:  0
=========
stat_wtx                2126584049  4-byte words transmitted
stat_wrx                547369207   4-byte words received
stat_ftx                1278054836  Frames transmitted
stat_frx                113847611   Frames received
stat_c2_frx             0           Class 2 frames received
stat_c3_frx             113844033   Class 3 frames received
stat_lc_rx              1806        Link control frames received
stat_mc_rx              0           Multicast frames received
stat_mc_to              0           Multicast timeouts
stat_mc_tx              0           Multicast frames transmitted
tim_rdy_pri             0           Time R_RDY high priority
tim_txcrd_z             80149680    Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc  0- 3:  0           0           77005106    409375
tim_txcrd_z_vc  4- 7:  2732394     2805        0           0
tim_txcrd_z_vc  8-11:  0           0           0           0
tim_txcrd_z_vc 12-15:  0           0           0           0
er_enc_in               0           Encoding errors inside of frames
er_crc                  0           Frames with CRC errors
er_trunc                0           Frames shorter than minimum
er_toolong              0           Frames longer than maximum
er_bad_eof              0           Frames with bad end-of-frame
er_enc_out              0           Encoding error outside of frames
er_bad_os               1           Invalid ordered set
er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout
er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout
er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable
er_other_discard        0           Other discards
er_type1_miss           0           frames with FTB type 1 miss
er_type2_miss           0           frames with FTB type 2 miss
er_type6_miss           0           frames with FTB type 6 miss
er_zone_miss            0           frames with hard zoning miss
er_lun_zone_miss        0           frames with LUN zoning miss
er_crc_good_eof         0           Crc error with good eof
er_inv_arb              0           Invalid ARB
open                    0           loop_open
transfer                0           loop_transfer
opened                  0           FL_Port opened
starve_stop             0           tenancies stopped due to starvation
fl_tenancy              0           number of times FL has the tenancy
nl_tenancy              0           number of times NL has the tenancy
zero_tenancy            0           zero tenancy

DCX:

RELN_PRO_SAN004:mowatw> portstatsshow -i 1

port:  1
=========
stat_wtx                1323987190  4-byte words transmitted
stat_wrx                3013823629  4-byte words received
stat_ftx                28433938    Frames transmitted
stat_frx                104049003   Frames received
stat_c2_frx             0           Class 2 frames received
stat_c3_frx             104042795   Class 3 frames received
stat_lc_rx              3101        Link control frames received
stat_mc_rx              0           Multicast frames received
stat_mc_to              0           Multicast timeouts
stat_mc_tx              0           Multicast frames transmitted
tim_rdy_pri             0           Time R_RDY high priority
tim_txcrd_z             80177113    Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc  0- 3:  0           0           77032539    409375
tim_txcrd_z_vc  4- 7:  2732394     2805        0           0
tim_txcrd_z_vc  8-11:  0           0           0           0
tim_txcrd_z_vc 12-15:  0           0           0           0
er_enc_in               0           Encoding errors inside of frames
er_crc                  0           Frames with CRC errors
er_trunc                0           Frames shorter than minimum
er_toolong              0           Frames longer than maximum
er_bad_eof              0           Frames with bad end-of-frame
er_enc_out              0           Encoding error outside of frames
er_bad_os               0           Invalid ordered set
er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout
er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout
er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable
er_other_discard        0           Other discards
er_type1_miss           0           frames with FTB type 1 miss
er_type2_miss           0           frames with FTB type 2 miss
er_type6_miss           0           frames with FTB type 6 miss
er_zone_miss            0           frames with hard zoning miss
er_lun_zone_miss        0           frames with LUN zoning miss
er_crc_good_eof         0           Crc error with good eof
er_inv_arb              0           Invalid ARB
open                    0           loop_open
transfer                0           loop_transfer
opened                  0           FL_Port opened
starve_stop             0           tenancies stopped due to starvation
fl_tenancy              0           number of times FL has the tenancy
nl_tenancy              0           number of times NL has the tenancy
zero_tenancy            0           zero tenancy

Contributor
Posts: 29
Registered: ‎06-21-2011

Re: DCX ISL to 48000 problems

Looking at the outputs of the statsshow, the tim_txcrd_z values don't look favourable for the dcx.

Valued Contributor
Posts: 761
Registered: ‎06-11-2010

Re: DCX ISL to 48000 problems

The DCX side looks fine to me, it is the 48000 side that looks strange.

DCX:

tim_txcrd_z             80177113    Time TX Credit Zero (2.5Us ticks)

tim_txcrd_z_vc  0- 3:  0           0           77032539    409375

tim_txcrd_z_vc  4- 7:  2732394     2805        0           0

tim_txcrd_z_vc  8-11:  0           0           0           0

tim_txcrd_z_vc 12-15:  0           0           0           0

48000:

tim_txcrd_z             6408        Time BB credit zero (2.5Us ticks)

What FOS code is running in each switch?

Contributor
Posts: 29
Registered: ‎06-21-2011

Re: DCX ISL to 48000 problems

both DCX and 48000 are running on 6.4.3

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.

vADC is now Pulse Secure
Download FREE NVMe eBook