Fibre Channel (SAN)

Reply
Occasional Contributor
Posts: 7
Registered: ‎04-24-2008

Link Reset on port cused by buffer credit limit ?

Hi all !

I have Brocade 4900 with 5.3 FOS

4900 connect two SUN server with one 3511 storage. Connect is local , 10-15m cable

Periodically at both server registered error , typical as -

"Aug 5 17:16:50 e25k-a /scsi_vhci/ssd@g600c0ff0000000000067d9751cbff900 (ssd493): Command Timeout on path /pci@7d,6000"

Aug 5 17:17:20 e25k-b-a /scsi_vhci/ssd@g600c0ff0000000000067d953ed9c0f002 (ssd410): Command Timeout on path /pci @7d,600000/SUNW,qlc@1/fp@0,0 (fp5

67d9 - it's 3511 WWN

At this moment on Brocade port registered "Link Reset"

17:16:23.241 INTR pstate 22 LR1

17:16:23.242 INTR pstate 22 AC

17:16:23.242 PORT scn 22 11 00000000,00000000,00000002

22 port - 3511

Also I view tim_txcrd_z counter on this port - it's grow very fast

I think what Buffer credit on 22 port is settled and port reset as result

I do not understand the reasons why it occurs .

Summary traffic on 22 not very high - 50-60 Mb/s

Question - In what cases occurs LR and why can tim_txcrd_z to grow quickly ?

It is a known problem or not ?

WBR Andrew A. "Ouk"

Contributor
Posts: 26
Registered: ‎07-26-2009

Re: Link Reset on port cused by buffer credit limit ?

We have now exactly the same in ISL (4Gbit/s, 22km) between two 5100 in one fabric:

from portlogshowport:

14:29:19.266 INTR       pstate    0  LR1
14:29:19.267 INTR       pstate    0   AC
14:29:19.267 PORT       scn       0   11  00000000,00000000,00000001
14:29:19.769 INTR       pstate    0  LR1
14:29:19.770 INTR       pstate    0   AC
14:29:19.770 PORT       scn       0   11  00000000,00000000,00000001

from the Solaris-host:

Jul 20 14:29:19 xtcup01 scsi: WARNING: /scsi_vhci/disk@g600a0b8000478ad40000fc804e03f541 (sd8):
Jul 20 14:29:19 xtcup01         SCSI transport failed: reason 'tran_err': retrying command

The weird thing is that the other fabric (configured identically) in error-free...

Did You get any feedback on this? I'm totally puzzled....

Regular Contributor
Posts: 201
Registered: ‎11-24-2009

Re: Link Reset on port cused by buffer credit limit ?

Hi Andrew,

in your scenario the likely suspect is the storage device which is not capable to handle that many frames your host is trying to send. Either the controller or its ports are not capable enough, or there is not enough back-end devices to handle I/O.

Hope this helps,

Linar

Regular Contributor
Posts: 201
Registered: ‎11-24-2009

Re: Link Reset on port cused by buffer credit limit ?

Hi miika.pulkkinen,

in your case I'd look at longhaul link first, check sfpShow and portErrShow/portShow stats on the ISL.

Hope this helps,

Linar

Contributor
Posts: 26
Registered: ‎07-26-2009

Re: Link Reset on port cused by buffer credit limit ?

Hmm... These are the abnormities (in bold) what I can see:

MCR5001:admin> portstatsshow 0
stat_wtx                160383461   4-byte words transmitted
stat_wrx                1800538247  4-byte words received
stat_ftx                2105113505  Frames transmitted
stat_frx                3203656904  Frames received
stat_c2_frx             0           Class 2 frames received
stat_c3_frx             3201809290  Class 3 frames received
stat_lc_rx              1015114     Link control frames received
stat_mc_rx              0           Multicast frames received
stat_mc_to              0           Multicast timeouts
stat_mc_tx              0           Multicast frames transmitted
tim_rdy_pri             330417      Time R_RDY high priority
tim_txcrd_z             47669672    Time TX Credit Zero (2.5Us ticks)
tim_txcrd_z_vc  0- 3:  3592        0           47666080    0       
tim_txcrd_z_vc  4- 7:  0           0           0           0        
tim_txcrd_z_vc  8-11:  0           0           0           0        
tim_txcrd_z_vc 12-15:  0           0           0           0        
er_enc_in               0           Encoding errors inside of frames
er_crc                  0           Frames with CRC errors
er_trunc                0           Frames shorter than minimum
er_toolong              0           Frames longer than maximum
er_bad_eof              0           Frames with bad end-of-frame
er_enc_out              0           Encoding error outside of frames
er_bad_os               19224245    Invalid ordered set
er_rx_c3_timeout        0           Class 3 receive frames discarded due to timeout
er_tx_c3_timeout        0           Class 3 transmit frames discarded due to timeout
er_c3_dest_unreach      0           Class 3 frames discarded due to destination unreachable
er_other_discard        0           Other discards
er_type1_miss           0           frames with FTB type 1 miss
er_type2_miss           0           frames with FTB type 2 miss
er_type6_miss           0           frames with FTB type 6 miss
er_zone_miss            0           frames with hard zoning miss
er_lun_zone_miss        0           frames with LUN zoning miss
er_crc_good_eof         0           Crc error with good eof
er_inv_arb              18          Invalid ARB
open                    0           loop_open
transfer                0           loop_transfer
opened                  0           FL_Port opened
starve_stop             0           tenancies stopped due to starvation
fl_tenancy              0           number of times FL has the tenancy
nl_tenancy              0           number of times NL has the tenancy
zero_tenancy            0           zero tenancy

and here:

MCR5001:admin> portshow 0
portName: MCR5001p0_G065002p0
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x10004907    PRESENT ACTIVE E_PORT G_PORT LOGICAL_ONLINE LOGIN LED
portType:  17.0
POD Port: Port is licensed
portState: 1    Online  
Protocol: FC
portPhys:  6    In_Sync         portScn:   16   E_Port   
port generation number:    0
state transition count:    0         

portId:    010000
portIfId:    4302000f
portWwn:   20:00:00:05:1e:65:d9:f9
portWwn of device(s) connected:

Distance:  auto (desired = 25 Km, actual = 22 Km)
portSpeed: 4Gbps

LE domain: 0
FC Fastwrite: OFF
Interrupts:        0          Link_failure: 0          Frjt:         0         
Unknown:           0          Loss_of_sync: 0          Fbsy:         0         
Lli:               3627       Loss_of_sig:  0         
Proc_rqrd:         1842899    Protocol_err: 0         
Timed_out:         0          Invalid_word: 0         
Rx_flushed:        0          Invalid_crc:  0         
Tx_unavail:        0          Delim_err:    0         
Free_buffer:       0          Address_err:  0         
Overrun:           0          Lr_in:        7         
Suspended:         0          Lr_out:       1215      
Parity_err:        0          Ols_in:       0         
2_parity_err:      0          Ols_out:      0         
CMI_bus_err:       0         

Port part of other ADs: No

Regular Contributor
Posts: 201
Registered: ‎11-24-2009

Re: Link Reset on port cused by buffer credit limit ?

Hi miika.pulkkinen,

is that a direct dark fiber link or an xWDM? As it looks more like a misconfigured link parameters.

Would you mind posting sfpShow for that port also?

Linar

Contributor
Posts: 26
Registered: ‎07-26-2009

Re: Link Reset on port cused by buffer credit limit ?

Ok, it's a 4Gbit/s DWDM-link and sfpshow says the following:

MCR5001:admin> sfpshow 0
Identifier:  3    SFP
Connector:   7    LC
Transceiver: 150c402001000000 100,200,400_MB/s M5,M6 sw Inter_dist
Encoding:    1    8B10B
Baud Rate:   42   (units 100 megabaud)
Length 9u:   0    (units km)
Length 9u:   0    (units 100 meters)
Length 50u:  15   (units 10 meters)
Length 62.5u:7    (units 10 meters)
Length Cu:   0    (units 1 meter)
Vendor Name: BROCADE        
Vendor OUI:  00:05:1e
Vendor PN:   57-1000013-01  
Vendor Rev:  A  
Wavelength:  850  (units nm)
Options:     0032 Loss_of_Sig,Tx_Disable
BR Max:      0  
BR Min:      0  
Serial No:   VAF108460000MJS
Date Code:   081112 
DD Type:     0x68
Enh Options: 0xd8
Status/Ctrl: 0x10
Alarm flags = 0x0, 0x0
Warn Flags = 0x0, 0x0
                                          Alarm                  Warn
                                      low        high       low         high
Temperature: 36      Centigrade     -25         95         -20         90
Current:     7.018   mAmps          1.000       17.000     2.000       14.000
Voltage:     3288.4  mVolts         2700.0      3900.0     2900.0      3700.0
RX Power:    -4.4    dBm (359.4 uW) 10.0   uW   1259.0 uW  15.8   uW   794.0  uW
TX Power:    -4.1    dBm (385.5 uW) 67.0   uW   631.0  uW  79.0   uW   631.0  uW

Regular Contributor
Posts: 201
Registered: ‎11-24-2009

Re: Link Reset on port cused by buffer credit limit ?

If that is a DWDM you might want to try R_RDY mode on the ISL (portCfgIslMode).

Linar

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.