Fibre Channel (SAN)

Reply
Contributor
Posts: 39
Registered: ‎12-08-2011

Link_failure

when i do a portshow on my fc port I see this, does this indicate a link or hardware failure?

 

dcx2:FID128:root> portshow 2/23
portIndex:  87
portName: 
portHealth: HEALTHY 
 
Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x24b03       PRESENT ACTIVE F_PORT G_PORT U_PORT NPIV LOGICAL_ONLINE LOGIN NOELP LED ACCEPT FLOGI
LocalSwcFlags: 0x0
portType:  17.0
portState: 1    Online   
Protocol: FC
portPhys:  6    In_Sync         portScn:   32   F_Port    
port generation number:    2152
state transition count:    13         
 
portId:    155700
portIfId:    4322001e
portWwn:   20:57:00:05:33:57:44:00
portWwn of device(s) connected:
        20:00:00:25:b5:11:0b:1f
        20:00:00:25:b5:11:0b:9f
        20:00:00:25:b5:11:0b:7f
        20:00:00:25:b5:11:0b:bf
        20:00:00:25:b5:11:0b:fe
        20:00:00:25:b5:11:0b:ef
        20:1c:54:7f:ee:b5:c6:80
Distance:  normal
portSpeed: N8Gbps
 
LE domain: 0
FC Fastwrite: OFF
Interrupts:        0          Link_failure: 3          Frjt:         0          
Unknown:           377        Loss_of_sync: 7          Fbsy:         0          
Lli:               93         Loss_of_sig:  10         
Proc_rqrd:         68440      Protocol_err: 0          
Timed_out:         0          Invalid_word: 1728881    
Rx_flushed:        0          Invalid_crc:  0          
Tx_unavail:        0          Delim_err:    0          
Free_buffer:       0          Address_err:  0          
Overrun:           0          Lr_in:        4          
Suspended:         0          Lr_out:       3          
Parity_err:        0          Ols_in:       3          
2_parity_err:      0          Ols_out:      4          
CMI_bus_err:       0          
 
 
 
 
Contributor
Posts: 39
Registered: ‎12-08-2011

Re: Link_failure

and this

 

 

dcx2:FID128:root> portshow 8/23
portIndex: 215
portName:
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x24b03 PRESENT ACTIVE F_PORT G_PORT U_PORT NPIV LOGICAL_ONLINE LOGIN NOELP LED ACCEPT FLOGI
LocalSwcFlags: 0x0
portType: 17.0
portState: 1 Online
Protocol: FC
portPhys: 6 In_Sync portScn: 32 F_Port
port generation number: 2150
state transition count: 15

portId: 15d700
portIfId: 4382001e
portWwn: 20:d7:00:05:33:57:44:00
portWwn of device(s) connected:
20:00:00:25:b5:11:0b:8f
20:00:00:25:b5:11:0b:5f
20:00:00:25:b5:11:0b:cf
20:00:00:25:b5:11:0b:0f
20:00:00:25:b5:11:0b:6f
20:00:00:25:b5:11:0b:4f
20:1b:54:7f:ee:b5:c6:80
Distance: normal
portSpeed: N8Gbps

LE domain: 0
FC Fastwrite: OFF
Interrupts: 0 Link_failure: 4 Frjt: 0
Unknown: 645 Loss_of_sync: 27 Fbsy: 0
Lli: 98 Loss_of_sig: 30
Proc_rqrd: 413670 Protocol_err: 0
Timed_out: 0 Invalid_word: 797647
Rx_flushed: 0 Invalid_crc: 0
Tx_unavail: 0 Delim_err: 0
Free_buffer: 0 Address_err: 0
Overrun: 0 Lr_in: 6
Suspended: 0 Lr_out: 4
Parity_err: 0 Ols_in: 4
2_parity_err: 0 Ols_out: 5
CMI_bus_err: 0

Contributor
Posts: 39
Registered: ‎12-08-2011

Re: Link_failure

diagshow:

 

Diagnostics Status: Sun Mar 16 16:20:39 2014

Slot: 2 UPORTS
Port BPort Diag Active Speed FrTX FrRX LLI Errs Loopback
2/0 56 OK DN 8G Auto -- -- --
2/1 57 OK DN 8G Auto -- -- --
2/2 59 OK DN 8G Auto -- -- --
2/3 58 OK DN 8G Auto -- -- --
2/4 60 OK DN 8G Auto -- -- --
2/5 61 OK DN 8G Auto -- -- --
2/6 63 OK DN 8G Auto -- -- --
2/7 62 OK DN 8G Auto -- -- --
2/8 16 OK DN 8G Auto -- -- --
2/9 17 OK DN 8G Auto -- -- --
2/10 19 OK DN 8G Auto -- -- --
2/11 18 OK DN 8G Auto -- -- --
2/12 20 OK DN 8G Auto -- -- --
2/13 21 OK DN 8G Auto -- -- --
2/14 23 OK DN 8G Auto -- -- --
2/15 22 OK DN 8G Auto -- -- --
2/16 24 OK DN 8G Auto -- -- --
2/17 25 OK DN 8G Auto -- -- --
2/18 27 OK DN 8G Auto -- -- --
2/19 26 OK DN 8G Auto -- -- --
2/20 28 OK DN 8G Auto -- -- --
2/21 29 OK DN 8G Auto -- -- --
2/22 31 OK DN 8G Auto -- -- --
2/23 30 OK UP 8G Auto 1823026950 248043035 1728881
2/24 64 OK DN 8G Auto -- -- --
2/25 65 OK DN 8G Auto -- -- --
2/26 67 OK DN 8G Auto -- -- --
2/27 66 OK DN 8G Auto -- -- --
2/28 68 OK DN 8G Auto -- -- --
2/29 69 OK UP 8G Auto 1886491129 834282991 668420
2/30 71 OK DN 8G Auto -- -- --
2/31 70 OK DN 8G Auto -- -- --
2/32 72 OK UP 8G Auto 1793705475 4259765430 139571
2/33 73 OK UP 4G Auto 292298249 3643542693 3
2/34 75 OK UP 4G Auto 1681905406 3866540566 305496
2/35 74 OK DN 8G Auto -- -- --
2/36 76 OK DN 8G Auto -- -- --
2/37 77 OK DN 8G Auto -- -- --
2/38 79 OK DN 8G Auto -- -- --
2/39 78 OK DN 8G Auto -- -- --
2/40 32 OK DN 8G Auto -- -- --
2/41 33 OK DN 8G Auto -- -- --
2/42 35 OK UP 4G Auto 2710271523 1392938361 14
2/43 34 OK DN 8G Auto -- -- --
2/44 36 OK DN 8G Auto -- -- --
2/45 37 OK DN 8G Auto -- -- --
2/46 39 OK DN 8G Auto -- -- --
2/47 38 OK DN 8G Auto -- -- --

Total Diag Frames Tx: 800
Total Diag Frames Rx: 800

Slot: 8 UPORTS
Port BPort Diag Active Speed FrTX FrRX LLI Errs Loopback
8/0 56 OK UP 4G Auto 3068476965 1677092200 2852
8/1 57 OK DN 8G Auto -- -- --
8/2 59 OK DN 8G Auto -- -- --
8/3 58 OK DN 8G Auto -- -- --
8/4 60 OK DN 8G Auto -- -- --
8/5 61 OK DN 8G Auto -- -- --
8/6 63 OK UP 8G Auto 29638509 14013233 582596
8/7 62 OK DN 8G Auto -- -- --
8/8 16 OK DN 8G Auto -- -- --
8/9 17 OK DN 8G Auto -- -- --
8/10 19 OK UP 4G Auto 607 108 0
8/11 18 OK UP 4G Auto 2080410589 2312253700 13
8/12 20 OK UP 8G Auto 2846749872 1544554319 2416
8/13 21 OK DN 8G Auto -- -- --
8/14 23 OK UP 8G Auto 1639825540 2667874216 1458548
8/15 22 OK UP 4G Auto 593154142 1177327792 2307570
8/16 24 OK UP 4G Auto 3898306237 2747105761 0
8/17 25 OK DN 8G Auto -- -- --
8/18 27 OK DN 8G Auto -- -- --
8/19 26 OK UP 4G Auto 836951007 4264096104 16
8/20 28 OK DN 8G Auto -- -- --
8/21 29 OK DN 8G Auto -- -- --
8/22 31 OK DN 8G Auto -- -- --
8/23 30 OK UP 8G Auto 935533110 2232716263 797647
8/24 64 OK UP 4G Auto 3966358398 1928475535 0
8/25 65 OK DN 8G Auto -- -- --
8/26 67 OK DN 8G Auto -- -- --
8/27 66 OK UP 8G Auto 1314763214 23126552 90
8/28 68 OK DN 8G Auto -- -- --
8/29 69 OK UP 4G Auto 899419283 3638516596 0
8/30 71 OK UP 4G Auto 2862713440 2204495516 0
8/31 70 OK DN 8G Auto -- -- --
8/32 72 OK UP 4G Auto 3552400717 4258290284 0
8/33 73 OK UP 4G Auto 772370041 3964023239 0
8/34 75 OK UP 8G Auto 4267602519 4035745389 129525
8/35 74 OK UP 4G Auto 4019481025 47215083 0
8/36 76 OK DN 8G Auto -- -- --
8/37 77 OK DN 8G Auto -- -- --
8/38 79 OK UP 4G Auto 1889949406 3081918001 295940
8/39 78 OK UP 4G Auto 486201681 2055011623 176370
8/40 32 OK DN 8G Auto -- -- --
8/41 33 OK UP 4G Auto 631288759 3408067344 94464
8/42 35 OK DN 8G Auto -- -- --
8/43 34 OK DN 8G Auto -- -- --
8/44 36 OK UP 4G Auto 1696956891 335721461 3
8/45 37 OK UP 4G Auto 1530495808 3596473303 4180112
8/46 39 OK UP 8G Auto 170599381 135594176 639303
8/47 38 OK DN 8G Auto -- -- --

Total Diag Frames Tx: 800
Total Diag Frames Rx: 800

Slot: 3 UPORTS
Port BPort Diag Active Speed FrTX FrRX LLI Errs Loopback
3/0 32 OK DN 8Gbps -- -- --
3/1 33 OK DN 8Gbps -- -- --
3/2 34 OK DN 8Gbps -- -- --
3/3 35 OK DN 8Gbps -- -- --
3/4 36 OK DN 8Gbps -- -- --
3/5 37 OK DN 8Gbps -- -- --
3/6 38 OK DN 8Gbps -- -- --
3/7 39 OK DN 8Gbps -- -- --
3/8 72 OK DN 8Gbps -- -- --
3/9 73 OK DN 8Gbps -- -- --
3/10 74 OK DN 8Gbps -- -- --
3/11 75 OK DN 8Gbps -- -- --
3/12 76 OK DN 8Gbps -- -- --
3/13 77 OK DN 8Gbps -- -- --
3/14 78 OK DN 8Gbps -- -- --
3/15 79 OK DN 8Gbps -- -- --

Total Diag Frames Tx: 320
Total Diag Frames Rx: 320

Slot: 6 UPORTS
Port BPort Diag Active Speed FrTX FrRX LLI Errs Loopback
6/0 32 OK DN 8Gbps -- -- --
6/1 33 OK DN 8Gbps -- -- --
6/2 34 OK DN 8Gbps -- -- --
6/3 35 OK DN 8Gbps -- -- --
6/4 36 OK DN 8Gbps -- -- --
6/5 37 OK DN 8Gbps -- -- --
6/6 38 OK DN 8Gbps -- -- --
6/7 39 OK DN 8Gbps -- -- --
6/8 72 OK DN 8Gbps -- -- --
6/9 73 OK DN 8Gbps -- -- --
6/10 74 OK DN 8Gbps -- -- --
6/11 75 OK DN 8Gbps -- -- --
6/12 76 OK DN 8Gbps -- -- --
6/13 77 OK DN 8Gbps -- -- --
6/14 78 OK DN 8Gbps -- -- --
6/15 79 OK DN 8Gbps -- -- --

Central Memory: OK
Total Diag Frames Tx: 320
Total Diag Frames Rx: 320
dcx2:FID128:root>

 

 

what are the LLI  Errs?

Valued Contributor
Posts: 528
Registered: ‎03-20-2011

Re: Link_failure

lli is low level interrupts, whatever it means Smiley Happy

 

so it looks like you have a blade chassis attached and something is going wrong. you'd better look at the errors with portstatsshow. i'd also login into the access gateway switch and look what's going in there. from experience, the copper ports connected to the blades are the first suspects as the source of all errors.

Contributor
Posts: 39
Registered: ‎12-08-2011

Re: Link_failure

I do not have an access gateway but ere are the portstatsshow on the 4 FC ports

 

dcx2:FID128:root> portstatsshow 2/23

stat_wtx                3240493428  4-byte words transmitted

stat_wrx                4120706335  4-byte words received

stat_ftx                3462882141  Frames transmitted

stat_frx                1371763577  Frames received

stat_c2_frx             0           Class 2 frames received

stat_c3_frx             1371763577  Class 3 frames received

stat_lc_rx              0           Link control frames received

stat_mc_rx              0           Multicast frames received

stat_mc_to              0           Multicast timeouts

stat_mc_tx              0           Multicast frames transmitted

tim_rdy_pri             0           Time R_RDY high priority

tim_txcrd_z             15460       Time TX Credit Zero (2.5Us ticks)

tim_txcrd_z_vc  0- 3:  0           0           0           0        

tim_txcrd_z_vc  4- 7:  0           15460       0           0         

tim_txcrd_z_vc  8-11:  0           0           0           0        

tim_txcrd_z_vc 12-15:  0           0           0           0        

er_enc_in               0           Encoding errors inside of frames

er_crc                  0           Frames with CRC errors

er_trunc                0           Frames shorter than minimum

er_toolong              0           Frames longer than maximum

er_bad_eof              0           Frames with bad end-of-frame

er_enc_out              1728881     Encoding error outside of frames

er_bad_os               1902230     Invalid ordered set

er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout

er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout

er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable

er_other_discard        36          Other discards

er_type1_miss           0           frames with FTB type 1 miss

er_type2_miss           0           frames with FTB type 2 miss

er_type6_miss           0           frames with FTB type 6 miss

er_zone_miss            0           frames with hard zoning miss

er_lun_zone_miss        0           frames with LUN zoning miss

er_crc_good_eof         0           Crc error with good eof

er_inv_arb              0           Invalid ARB

open                    0           loop_open

transfer                0           loop_transfer

opened                  0           FL_Port opened

starve_stop             0           tenancies stopped due to starvation

fl_tenancy              0           number of times FL has the tenancy

nl_tenancy              0           number of times NL has the tenancy

zero_tenancy            0           zero tenancy

dcx2:FID128:root> portstatsshow 8/23

stat_wtx                2663765255  4-byte words transmitted

stat_wrx                3677335610  4-byte words received

stat_ftx                2027301314  Frames transmitted

stat_frx                3549592438  Frames received

stat_c2_frx             0           Class 2 frames received

stat_c3_frx             3549249091  Class 3 frames received

stat_lc_rx              160458      Link control frames received

stat_mc_rx              0           Multicast frames received

stat_mc_to              0           Multicast timeouts

stat_mc_tx              0           Multicast frames transmitted

tim_rdy_pri             44          Time R_RDY high priority

tim_txcrd_z             8673523     Time TX Credit Zero (2.5Us ticks)

tim_txcrd_z_vc  0- 3:  8451111     0           99773       108343   

tim_txcrd_z_vc  4- 7:  0           14296       0           0        

tim_txcrd_z_vc  8-11:  0           0           0           0        

tim_txcrd_z_vc 12-15:  0           0           0           0        

er_enc_in               0           Encoding errors inside of frames

er_crc                  0           Frames with CRC errors

er_trunc                0           Frames shorter than minimum

er_toolong              0           Frames longer than maximum

er_bad_eof              0           Frames with bad end-of-frame

er_enc_out              797647      Encoding error outside of frames

er_bad_os               855209      Invalid ordered set

er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout

er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout

er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable

er_other_discard        4           Other discards

er_type1_miss           0           frames with FTB type 1 miss

er_type2_miss           0           frames with FTB type 2 miss

er_type6_miss           0           frames with FTB type 6 miss

er_zone_miss            0           frames with hard zoning miss

er_lun_zone_miss        0           frames with LUN zoning miss

er_crc_good_eof         0           Crc error with good eof

er_inv_arb              0           Invalid ARB

open                    0           loop_open

transfer                0           loop_transfer

opened                  0           FL_Port opened

starve_stop             0           tenancies stopped due to starvation

fl_tenancy              0           number of times FL has the tenancy

nl_tenancy              0           number of times NL has the tenancy

zero_tenancy            0           zero tenancy

dcx2:FID128:root>

 

----------------------------------------------------------------

 

dcx1:FID128:admin> portstatsshow 2/23

stat_wtx                2472858760  4-byte words transmitted

stat_wrx                1254815634  4-byte words received

stat_ftx                2209445671  Frames transmitted

stat_frx                3483937250  Frames received

stat_c2_frx             0           Class 2 frames received

stat_c3_frx             3483915404  Class 3 frames received

stat_lc_rx              10923       Link control frames received

stat_mc_rx              0           Multicast frames received

stat_mc_to              0           Multicast timeouts

stat_mc_tx              0           Multicast frames transmitted

tim_rdy_pri             341         Time R_RDY high priority

tim_txcrd_z             1088068     Time TX Credit Zero (2.5Us ticks)

tim_txcrd_z_vc  0- 3:  3234        0           931403      7147     

tim_txcrd_z_vc  4- 7:  0           146284      0           0        

tim_txcrd_z_vc  8-11:  0           0           0           0        

tim_txcrd_z_vc 12-15:  0           0           0           0        

er_enc_in               0           Encoding errors inside of frames

er_crc                  0           Frames with CRC errors

er_trunc                0           Frames shorter than minimum

er_toolong              0           Frames longer than maximum

er_bad_eof              0           Frames with bad end-of-frame

er_enc_out              770377      Encoding error outside of frames

er_bad_os               706460      Invalid ordered set

er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout

er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout

er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable

er_other_discard        19          Other discards

er_type1_miss           0           frames with FTB type 1 miss

er_type2_miss           0           frames with FTB type 2 miss

er_type6_miss           0           frames with FTB type 6 miss

er_zone_miss            0           frames with hard zoning miss

er_lun_zone_miss        0           frames with LUN zoning miss

er_crc_good_eof         0           Crc error with good eof

er_inv_arb              0           Invalid ARB

open                    0           loop_open

transfer                0           loop_transfer

opened                  0           FL_Port opened

starve_stop             0           tenancies stopped due to starvation

fl_tenancy              0           number of times FL has the tenancy

nl_tenancy              0           number of times NL has the tenancy

zero_tenancy            0           zero tenancy

dcx1:FID128:admin> portstatsshow 8/23

stat_wtx                1737188603  4-byte words transmitted

stat_wrx                1042960656  4-byte words received

stat_ftx                3317111599  Frames transmitted

stat_frx                1440113498  Frames received

stat_c2_frx             0           Class 2 frames received

stat_c3_frx             1440113498  Class 3 frames received

stat_lc_rx              0           Link control frames received

stat_mc_rx              0           Multicast frames received

stat_mc_to              0           Multicast timeouts

stat_mc_tx              0           Multicast frames transmitted

tim_rdy_pri             0           Time R_RDY high priority

tim_txcrd_z             870         Time TX Credit Zero (2.5Us ticks)

tim_txcrd_z_vc  0- 3:  0           0           0           0        

tim_txcrd_z_vc  4- 7:  0           870         0           0        

tim_txcrd_z_vc  8-11:  0           0           0           0        

tim_txcrd_z_vc 12-15:  0           0           0           0        

er_enc_in               0           Encoding errors inside of frames

er_crc                  0           Frames with CRC errors

er_trunc                0           Frames shorter than minimum

er_toolong              0           Frames longer than maximum

er_bad_eof              0           Frames with bad end-of-frame

er_enc_out              0           Encoding error outside of frames

er_bad_os               3           Invalid ordered set

er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout

er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout

er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable

er_other_discard        2           Other discards

er_type1_miss           0           frames with FTB type 1 miss

er_type2_miss           0           frames with FTB type 2 miss

er_type6_miss           0           frames with FTB type 6 miss

er_zone_miss            0           frames with hard zoning miss

er_lun_zone_miss        0           frames with LUN zoning miss

er_crc_good_eof         0           Crc error with good eof

er_inv_arb              0           Invalid ARB

open                    0           loop_open

transfer                0           loop_transfer

opened                  0           FL_Port opened

starve_stop             0           tenancies stopped due to starvation

fl_tenancy              0           number of times FL has the tenancy

nl_tenancy              0           number of times NL has the tenancy

zero_tenancy            0           zero tenancy

dcx1:FID128:admin>

 

Valued Contributor
Posts: 761
Registered: ‎06-11-2010

Re: Link_failure

hi there,

 

er_bad_os and er_enc_out can indicate a HW issue. please, clear the stats (commands statsclear and slotstatsclear) and 24 hours after get a new output of portshow and portstatsclear.

Valued Contributor
Posts: 528
Registered: ‎03-20-2011

Re: Link_failure

you see, anything that is being sent through the fibre should be either an ordered set or a frame. in your case, we can see lots of errors of these two kinds: 1) bad ordered set, 2) error outside of frame. so what means "outside of frame"? i might be wrong, but if you miss the "start of frame" - then the rest of the frame is detected as "errors outside of frame". all this basically means that these four ports transmit lots of "garbage". since it is not very likely that all connected sfps and/or cables became bad, i suspect that you have some "system" kind of the failure, when all four ports connected to the same (or same kind) software npiv solution (since you said this is not access gateway) are giving the same errors. i believe you should seek support from the vendor who is supporting your device(s) connected to these four ports. i'd first suspect hba firmware and drivers...

Valued Contributor
Posts: 528
Registered: ‎03-20-2011

Re: Link_failure

isn't it cisco ucs?
Contributor
Posts: 39
Registered: ‎12-08-2011

Re: Link_failure

yes it is ucs. any idea? any suggestions

 

thanks

Contributor
Posts: 26
Registered: ‎09-11-2012

Re: Link_failure

Run the following and post here:

sfpshow -all | grep Rx

switchshow

porterrshow

portcfgshow

 

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.