Fibre Channel (SAN)

Reply
Occasional Contributor
Posts: 8
Registered: ‎02-01-2012

Port random timeouts/resets on ISL over DWDM

I have two DCX directors connection through the Cisco DWDM 15454SDH over 4Gb and 2Gb ports, two different providers distances 15 and 20 km.DWDM is the card I think TDM with ports with speeds 1,2,4 Gb. On DCX is using Exchange-based policy routing. The 2Gb ports everything works all rights, the 4Gb occurs resets and timeouts irregular short order of tenths of seconds in an interval of several hours to days. In statistics DCX port (mini 8Gb SFP) to discover errors er_enc_out, er_bad_os, er_tx_c3_timeout.
FOS is a version on the DCX v6.4.2a. I experimented with various modes of long-distance LD, LS VC, ISL_RDY, Fill Word, etc.,  but no effect ..

Valued Contributor
Posts: 931
Registered: ‎12-30-2009

Re: Port random timeouts/resets on ISL over DWDM

Have you ruled out the physical layer yet?

i.e.

swap sfp's

swap the dwdm links (if you can)

swap patchcable

swap ports

etc.

Occasional Contributor
Posts: 8
Registered: ‎02-01-2012

Re: Port random timeouts/resets on ISL over DWDM

I changed the cable, reaching a new direct connection between DCX-DWDM SFP, he used another slot / port in the chassis, no effect.

Failures and Resets are seeing on both ports only 4Gb at other random times..

Valued Contributor
Posts: 931
Registered: ‎12-30-2009

Re: Port random timeouts/resets on ISL over DWDM

So you didn't swap the SFP?

And if your reduce the speed, does the ISL become stable?

Occasional Contributor
Posts: 8
Registered: ‎02-01-2012

Re: Port random timeouts/resets on ISL over DWDM

Thank you,


we exchanged SFP cables, slots, tuned parameters LD / LS long distance extended fabric, various parameters including various distance connections, set fixed speed, but errors remain. On the side of DWDM is promoted to the latest IOS version. A similar problem is already described in the current firmware Reles Notes, but it should cover greater distances - TSB 2010-088-A.

Best regards

Super Contributor
Posts: 635
Registered: ‎04-12-2010

Re: Port random timeouts/resets on ISL over DWDM

Hi,

did you have configured the port as normal E-Port or E-Port in LE without QoS enabled?

This is only a guess.

Did you notice that the TSB is not listed in FOS 7.0?

Regards,

Andreas

Occasional Contributor
Posts: 8
Registered: ‎02-01-2012

Re: Port random timeouts/resets on ISL over DWDM

Hi,

ports are configured as E-port with QOS disable(recomended for TDM connections). Setting port in LE has bad throughput, we have LS/LD mode with Extended Fabric Feature, performance and latency is optimal.

Settings of ISL port:

portcfgshow 1/0
Area Number:              0
Speed Level:              4G
Fill Word:                0(Idle-Idle)
AL_PA Offset 13:          OFF
Trunk Port                OFF
Long Distance             LS
VC Link Init              OFF
Desired Distance          45 Km
Reserved Buffers          96
Locked L_Port             OFF
Locked G_Port             ON
Disabled E_Port           OFF
Locked E_Port             ON
ISL R_RDY Mode            ON
RSCN Suppressed           OFF
Persistent Disable        OFF
LOS TOV enable            OFF
NPIV capability           OFF
QOS E_Port                OFF
Port Auto Disable:        OFF
Rate Limit                OFF
EX Port                   OFF
Mirror Port               OFF
Credit Recovery           OFF
F_Port Buffers            OFF
Fault Delay:              0(R_A_TOV)
NPIV PP Limit:            126
CSCTL mode:               OFF

we also tried firmware version 7.x, but the behavior was the same port, also used DCFM does not ..

Best Regards

LT.

Valued Contributor
Posts: 761
Registered: ‎06-11-2010

Re: Port random timeouts/resets on ISL over DWDM

do you get these errors in both ends? Could you please attach the output of portstatsshow of them?

Occasional Contributor
Posts: 8
Registered: ‎02-01-2012

Re: Port random timeouts/resets on ISL over DWDM

Hi,

I waited for the presence of suitable error. The error counter always appears on the side where the problem originated ..

On DCX 1:

fabriclog -s

Time Stamp      Input and *Action                           S, P   Sn,Pn  Port  Xid
===================================================================================
12:50:55.594776 TA_SCN Port Offline                         A1,P3  A1,P0  0     NA
12:50:55.594780 *Snd port state: P0                         A1,P0  A1,P0  0     NA
12:50:55.676841 TA_SCN Port Offline                         A1,P0  A1,P0  0     NA
12:50:55.683344 SCN Port Offline;g=0x890                    A1,P0  A1,P0  0     NA
12:50:55.683351 *Removing all nodes from port               A1,P0  A1,P0  0     NA
12:50:55.706950 SCN LR_PORT (0);g=0x890                     A1,P0  A1,P0  0     NA
12:50:55.827803 SCN Port Online;g=0x890                     A1,P0  A1,P1  0     NA
12:50:55.828130 *ELP Send:rev=2,flow=2,flen=20,size=104     A1,P1  A1,P1  0     0x97c6
12:50:55.975136 ELP Rcv:rev=2,flow=2,flen=20,size=104       A1,P1  A1,P1  0     0x2464
12:50:55.975759 *ELP Snd ACC:rev=2,flow=2,flen=20,size=164  A1,P1  A1,P2  0     0x2464
12:50:55.975905 ELP RJT Rcv, ct prfrm, in prgs              A1,P2  A1,P2  0     0x97c6
12:50:56.069384 SCN AC_PORT                                 A1,P2  A1,ESC 0     NA
12:50:56.070247 ESC Rcv cnt=3,o=-1 e=2 vc=-1 f=0            A1,ESC A1,ESC 0     0x2466
12:50:56.070394 *ESC Snd ACC cnt=3,o=-1 e=2 vc=-1 f=0       A1,ESC A1,P3  0     0x2466
12:50:56.072138 *EFP Send:p=0x2,w=0x331e2d00,l=48,v=0       A1,P3  A1,P3  0     0x97c7
12:50:56.072915 EFP Rcv                                     A1,P3  A1,P3  0     0x2469
12:50:56.072947 EFP pld:p=0x2,w=0x331e2d00,l=48             A1,P3  A1,P3  0     0x2469
12:50:56.073213 *EFP Snd ACC:p=0x2,w=0x331e2d00,l=48,v=0    A1,P3  A1,P3  0     0x2469
12:50:56.073276 *Snd Nbr Wwn 10:00:00:05:33:1e:2d:00        A1,P3  A1,P3  0     NA
12:50:56.073294 *Snd Nbr port 0                             A1,NA  A1,NA  NA    NA
12:50:56.073299 *Snd port state: P3                         A1,P3  A1,P3  0     NA
12:50:56.073304 *Snd nbr esc (etiz:2)                       A1,P3  A1,P3  0     NA
12:50:56.073310 *Snd port flag: 0x120                       A1,P3  A1,P3  0     NA
12:50:56.073649 SCN E_PORT                                  A1,P3  A1,P3  0     NA
12:50:56.073769 EFP ACC Rcv                                 A1,P3  A1,P3  0     0x97c7
12:50:56.073786 EFP pld:p=0x2,w=0x331e2d00,l=48             A1,P3  A1,P3  0     0x97c7
..

09:09:02.681924 TA_SCN Port Offline                         A1,P3  A1,P0  0     NA
09:09:02.681928 *Snd port state: P0                         A1,P0  A1,P0  0     NA
09:09:02.753346 TA_SCN Port Offline                         A1,P0  A1,P0  0     NA
09:09:02.753950 SCN Port Offline;g=0x89a                    A1,P0  A1,P0  0     NA
09:09:02.753956 *Removing all nodes from port               A1,P0  A1,P0  0     NA
09:09:02.771415 SCN LR_PORT (0);g=0x89a                     A1,P0  A1,P0  0     NA
09:09:02.817774 SCN Port Online;g=0x89a                     A1,P0  A1,P1  0     NA
09:09:02.818097 *ELP Send:rev=2,flow=2,flen=20,size=104     A1,P1  A1,P1  0     0xbced
09:09:02.974733 ELP Rcv:rev=2,flow=2,flen=20,size=104       A1,P1  A1,P1  0     0x311c
09:09:02.975419 *ELP Snd ACC:rev=2,flow=2,flen=20,size=164  A1,P1  A1,P2  0     0x311c
09:09:02.975574 ELP RJT Rcv, ct prfrm, in prgs              A1,P2  A1,P2  0     0xbced
09:09:03.069535 SCN AC_PORT                                 A1,P2  A1,ESC 0     NA
09:09:03.070475 ESC Rcv cnt=3,o=-1 e=2 vc=-1 f=0            A1,ESC A1,ESC 0     0x3121
09:09:03.070620 *ESC Snd ACC cnt=3,o=-1 e=2 vc=-1 f=0       A1,ESC A1,P3  0     0x3121
09:09:03.072358 *EFP Send:p=0x2,w=0x331e2d00,l=48,v=0       A1,P3  A1,P3  0     0xbcf2
09:09:03.086472 EFP Rcv                                     A1,P3  A1,P3  0     0x3122
09:09:03.086652 EFP pld:p=0x2,w=0x331e2d00,l=48             A1,P3  A1,P3  0     0x3122
09:09:03.086769 *EFP Snd ACC:p=0x2,w=0x331e2d00,l=48,v=0    A1,P3  A1,P3  0     0x3122
09:09:03.086838 *Snd Nbr Wwn 10:00:00:05:33:1e:2d:00        A1,P3  A1,P3  0     NA
09:09:03.086857 *Snd Nbr port 0                             A1,NA  A1,NA  NA    NA
09:09:03.086862 *Snd port state: P3                         A1,P3  A1,P3  0     NA
09:09:03.086866 *Snd nbr esc (etiz:2)                       A1,P3  A1,P3  0     NA
09:09:03.086872 *Snd port flag: 0x120                       A1,P3  A1,P3  0     NA
09:09:03.087978 SCN E_PORT                                  A1,P3  A1,P3  0     NA
09:09:03.087997 EFP ACC Rcv                                 A1,P3  A1,P3  0     0xbcf2
09:09:03.088015 EFP pld:p=0x2,w=0x331e2d00,l=48             A1,P3  A1,P3  0     0xbcf2

> portshow 1/0
portIndex:   0
portName: ISL_BCxCh
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x10005903    PRESENT ACTIVE E_PORT G_PORT LOGICAL_ONLINE RRDY_MODE LOGIN LED
LocalSwcFlags: 0x0
portType:  17.0
portState: 1    Online
Protocol: FC
portPhys:  6    In_Sync         portScn:   16   E_Port    Flow control mode 2
port generation number:    2202
state transition count:    226

portId:    510000
portIfId:    43120008
shareIfId:   43121020
portWwn:   20:00:00:05:33:1b:bb:00
portWwn of device(s) connected:

Distance:  static (desired = 45 Km)
portSpeed: N4Gbps

LE domain: 0
FC Fastwrite: OFF
Interrupts:        0          Link_failure: 3          Frjt:         0
Unknown:           0          Loss_of_sync: 0          Fbsy:         0
Lli:               20         Loss_of_sig:  0
Proc_rqrd:         17499      Protocol_err: 0
Timed_out:         0          Invalid_word: 13
Rx_flushed:        0          Invalid_crc:  0
Tx_unavail:        0          Delim_err:    0
Free_buffer:       0          Address_err:  0
Overrun:           0          Lr_in:        8
Suspended:         0          Lr_out:       1
Parity_err:        0          Ols_in:       1
2_parity_err:      0          Ols_out:      4
CMI_bus_err:       0

Port part of other ADs: No

> portstatsshow 1/0
stat_wtx                1455752100  4-byte words transmitted
stat_wrx                2353816921  4-byte words received
stat_ftx                307222412   Frames transmitted
stat_frx                1865893799  Frames received
stat_c2_frx             0           Class 2 frames received
stat_c3_frx             1865876295  Class 3 frames received
stat_lc_rx              7623        Link control frames received
stat_mc_rx              0           Multicast frames received
stat_mc_to              0           Multicast timeouts
stat_mc_tx              0           Multicast frames transmitted
tim_rdy_pri             675         Time R_RDY high priority
tim_txcrd_z             163152      Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc  0- 3:  60          0           163092      0
tim_txcrd_z_vc  4- 7:  0           0           0           0
tim_txcrd_z_vc  8-11:  0           0           0           0
tim_txcrd_z_vc 12-15:  0           0           0           0
er_enc_in               0           Encoding errors inside of frames
er_crc                  0           Frames with CRC errors
er_trunc                0           Frames shorter than minimum
er_toolong              0           Frames longer than maximum
er_bad_eof              0           Frames with bad end-of-frame
er_enc_out              13          Encoding error outside of frames
er_bad_os               4           Invalid ordered set
er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout
er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout
er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable
er_other_discard        0           Other discards
er_type1_miss           0           frames with FTB type 1 miss
er_type2_miss           0           frames with FTB type 2 miss
er_type6_miss           0           frames with FTB type 6 miss
er_zone_miss            0           frames with hard zoning miss
er_lun_zone_miss        0           frames with LUN zoning miss
er_crc_good_eof         0           Crc error with good eof
er_inv_arb              0           Invalid ARB
open                    0           loop_open
transfer                0           loop_transfer
opened                  0           FL_Port opened
starve_stop             0           tenancies stopped due to starvation
fl_tenancy              0           number of times FL has the tenancy
nl_tenancy              0           number of times NL has the tenancy
zero_tenancy            0           zero tenancy

On DCX 2:

fabriclog -s

Time Stamp      Input and *Action                           S, P   Sn,Pn  Port  Xid
===================================================================================
12:50:55.677386 TA_SCN Port Offline                         D0,P3  D0,P0  0     NA
12:50:55.677390 *Snd port state: P0                         D0,P0  D0,P0  0     NA
12:50:55.904652 TA_SCN Port Offline                         D0,P0  D0,P0  0     NA
12:50:56.052901 SCN Port Offline;g=0xc70                    D0,P0  D0,P0  0     NA
12:50:56.052907 *Removing all nodes from port               D0,P0  D0,P0  0     NA
12:50:56.053718 SCN LR_PORT (0);g=0xc70                     D0,P0  D0,P0  0     NA
12:50:56.053736 ELP Rcv:rev=2,flow=2,flen=20,size=104       D0,P0  D0,P0  0     0x97c6
12:50:56.053754 *ELP ONLINE timer started                   D0,P0  D0,P0  0     0x97c6
12:50:56.053781 SCN Port Online;g=0xc70                     D0,P0  D0,P1  0     NA
12:50:56.054099 *ELP Send:rev=2,flow=2,flen=20,size=104     D0,P1  D0,P1  0     0x2464
12:50:56.054110 *ELP ONLINE timer process                   D0,P1  D0,P1  0     0x97c6
12:50:56.054312 *ELP Snd RJT(wwn) ct prfrm, in prgs         D0,P1  D0,P1  0     0x97c6
12:50:56.055923 ELP ACC Rcv:rev=2,flow=2,flen=20,size=104   D0,P1  D0,P2  0     0x2464
12:50:56.149212 SCN AC_PORT                                 D0,P2  D0,ESC 0     NA
12:50:56.149407 *ESC Send cnt=3,o=-1 e=2 vc=-1 f=0          D0,ESC D0,ESC 0     0x2466
12:50:56.150454 ESC ACC Rcv cnt=3,o=-1 e=2 vc=-1 f=0        D0,ESC D0,ESC 0     0x2466
12:50:56.152216 *EFP Send:p=0x2,w=0x331e2d00,l=48,v=0       D0,P3  D0,P3  0     0x2469
12:50:56.152318 EFP Rcv                                     D0,P3  D0,P3  0     0x97c7
12:50:56.152348 EFP pld:p=0x2,w=0x331e2d00,l=48             D0,P3  D0,P3  0     0x97c7
12:50:56.152459 *EFP Snd ACC:p=0x2,w=0x331e2d00,l=48,v=0    D0,P3  D0,P3  0     0x97c7
12:50:56.152537 *Snd Nbr Wwn 10:00:00:05:33:1b:bb:00        D0,P3  D0,P3  0     NA
12:50:56.152556 *Snd Nbr port 0                             D0,NA  D0,NA  NA    NA
12:50:56.152561 *Snd port state: P3                         D0,P3  D0,P3  0     NA
12:50:56.152566 *Snd nbr esc (etiz:2)                       D0,P3  D0,P3  0     NA
12:50:56.152572 *Snd port flag: 0x120                       D0,P3  D0,P3  0     NA
12:50:56.153244 SCN E_PORT                                  D0,P3  D0,P3  0     NA
12:50:56.153262 EFP ACC Rcv                                 D0,P3  D0,P3  0     0x2469
12:50:56.153278 EFP pld:p=0x2,w=0x331e2d00,l=48             D0,P3  D0,P3  0     0x2469
12:50:56.558530 *ELP ONLINE timer removed                   D0,P3  D0,P3  0     NA
....

09:09:03.372977 TA_SCN Port Offline                         D0,P3  D0,P0  0     NA
09:09:03.372981 *Snd port state: P0                         D0,P0  D0,P0  0     NA
09:09:03.651954 TA_SCN Port Offline                         D0,P0  D0,P0  0     NA
09:09:03.652472 SCN Port Offline;g=0xc78                    D0,P0  D0,P0  0     NA
09:09:03.652478 *Removing all nodes from port               D0,P0  D0,P0  0     NA
09:09:03.653290 SCN LR_PORT (0);g=0xc78                     D0,P0  D0,P0  0     NA
09:09:03.653308 SCN Port Online;g=0xc78                     D0,P0  D0,P1  0     NA
09:09:03.653639 *ELP Send:rev=2,flow=2,flen=20,size=104     D0,P1  D0,P1  0     0x311c
09:09:03.653666 ELP Rcv:rev=2,flow=2,flen=20,size=104       D0,P1  D0,P1  0     0xbced
09:09:03.653868 *ELP Snd RJT(wwn) ct prfrm, in prgs         D0,P1  D0,P1  0     0xbced
09:09:03.655466 ELP ACC Rcv:rev=2,flow=2,flen=20,size=104   D0,P1  D0,P2  0     0x311c
09:09:03.749322 SCN AC_PORT                                 D0,P2  D0,ESC 0     NA
09:09:03.749510 *ESC Send cnt=3,o=-1 e=2 vc=-1 f=0          D0,ESC D0,ESC 0     0x3121
09:09:03.750524 ESC ACC Rcv cnt=3,o=-1 e=2 vc=-1 f=0        D0,ESC D0,ESC 0     0x3121
09:09:03.765594 *EFP Send:p=0x2,w=0x331e2d00,l=48,v=0       D0,P3  D0,P3  0     0x3122
09:09:03.765634 EFP Rcv                                     D0,P3  D0,P3  0     0xbcf2
09:09:03.765663 EFP pld:p=0x2,w=0x331e2d00,l=48             D0,P3  D0,P3  0     0xbcf2
09:09:03.765773 *EFP Snd ACC:p=0x2,w=0x331e2d00,l=48,v=0    D0,P3  D0,P3  0     0xbcf2
09:09:03.765837 *Snd Nbr Wwn 10:00:00:05:33:1b:bb:00        D0,P3  D0,P3  0     NA
09:09:03.765856 *Snd Nbr port 0                             D0,NA  D0,NA  NA    NA
09:09:03.765861 *Snd port state: P3                         D0,P3  D0,P3  0     NA
09:09:03.765866 *Snd nbr esc (etiz:2)                       D0,P3  D0,P3  0     NA
09:09:03.765871 *Snd port flag: 0x120                       D0,P3  D0,P3  0     NA
09:09:03.766644 SCN E_PORT                                  D0,P3  D0,P3  0     NA
09:09:03.766663 EFP ACC Rcv                                 D0,P3  D0,P3  0     0x3122
09:09:03.766681 EFP pld:p=0x2,w=0x331e2d00,l=48             D0,P3  D0,P3  0     0x3122

> portshow 1/0
portIndex:   0
portName: ISL_BCxCh
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x10005903    PRESENT ACTIVE E_PORT G_PORT LOGICAL_ONLINE RRDY_MODE LOGIN LED
LocalSwcFlags: 0x0
portType:  17.0
portState: 1    Online
Protocol: FC
portPhys:  6    In_Sync         portScn:   16   E_Port    Flow control mode 2
port generation number:    3192
state transition count:    226

portId:    520000
portIfId:    43120008
shareIfId:   43121020
portWwn:   20:00:00:05:33:1e:2d:00
portWwn of device(s) connected:

Distance:  static (desired = 45 Km)
portSpeed: N4Gbps

LE domain: 0
FC Fastwrite: OFF
Interrupts:        0          Link_failure: 1          Frjt:         0
Unknown:           0          Loss_of_sync: 0          Fbsy:         0
Lli:               14         Loss_of_sig:  0
Proc_rqrd:         17583      Protocol_err: 0
Timed_out:         0          Invalid_word: 5
Rx_flushed:        0          Invalid_crc:  0
Tx_unavail:        0          Delim_err:    0
Free_buffer:       0          Address_err:  0
Overrun:           0          Lr_in:        4
Suspended:         0          Lr_out:       7
Parity_err:        0          Ols_in:       3
2_parity_err:      0          Ols_out:      4
CMI_bus_err:       0

Port part of other ADs: No

> portstatsshow 9/0
stat_wtx                87091866    4-byte words transmitted
stat_wrx                321479768   4-byte words received
stat_ftx                781375079   Frames transmitted
stat_frx                698766398   Frames received
stat_c2_frx             0           Class 2 frames received
stat_c3_frx             698758178   Class 3 frames received
stat_lc_rx              4111        Link control frames received
stat_mc_rx              0           Multicast frames received
stat_mc_to              0           Multicast timeouts
stat_mc_tx              0           Multicast frames transmitted
tim_rdy_pri             184         Time R_RDY high priority
tim_txcrd_z             2020        Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc  0- 3:  0           0           2020        0
tim_txcrd_z_vc  4- 7:  0           0           0           0
tim_txcrd_z_vc  8-11:  0           0           0           0
tim_txcrd_z_vc 12-15:  0           0           0           0
er_enc_in               0           Encoding errors inside of frames
er_crc                  0           Frames with CRC errors
er_trunc                0           Frames shorter than minimum
er_toolong              0           Frames longer than maximum
er_bad_eof              0           Frames with bad end-of-frame
er_enc_out              4           Encoding error outside of frames
er_bad_os               2           Invalid ordered set
er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout
er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout
er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable
er_other_discard        0           Other discards
er_type1_miss           0           frames with FTB type 1 miss
er_type2_miss           0           frames with FTB type 2 miss
er_type6_miss           0           frames with FTB type 6 miss
er_zone_miss            0           frames with hard zoning miss
er_lun_zone_miss        0           frames with LUN zoning miss
er_crc_good_eof         0           Crc error with good eof
er_inv_arb              0           Invalid ARB
open                    0           loop_open
transfer                0           loop_transfer
opened                  0           FL_Port opened
starve_stop             0           tenancies stopped due to starvation
fl_tenancy              0           number of times FL has the tenancy
nl_tenancy              0           number of times NL has the tenancy
zero_tenancy            0           zero tenancy

Regards

LT.

lle
New Contributor
Posts: 2
Registered: ‎06-14-2011

Re: Port random timeouts/resets on ISL over DWDM

Do you have the right SFPs on both sides?

sfpshow on both switches and see.

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.

vADC is now Pulse Secure
Download FREE NVMe eBook