For more details, please see ourCookie Policy.


Fibre Channel (SAN)

Reply
Occasional Contributor
Posts: 7
Registered: ‎12-12-2017
Accepted Solution

Help interpretting the following errors

I've setup storage replication with two Brocade 300s (located in two separate buildings) and noticed the following errors:

 

On the downstream side (ports 1,3 connected to SAN, port 18 connected to destination Brocade 300):

 

          frames      enc    crc    crc    too    too    bad    enc   disc   link   loss   loss   frjt   fbsy  c3timeout    pcs
       tx     rx      in    err    g_eof  shrt   long   eof     out   c3    fail    sync   sig                  tx    rx     err
  1:    7.5m  36.2m   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
  3:    1.4m   7.0m   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 18:   36.8m   2.5m   9      2      1      0      0      1      0      0      0      0      0      0      0      0      0      0

 

On the upstream side (ports 0,1 connected to SAN, port 7 connected to source Brocade 300):

 

          frames      enc    crc    crc    too    too    bad    enc   disc   link   loss   loss   frjt   fbsy  c3timeout    pcs
       tx     rx      in    err    g_eof  shrt   long   eof     out   c3    fail    sync   sig                  tx    rx     err
  0:    1.1m   1.1m   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
  1:   36.1m   1.8m   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
  7:    2.5m  36.8m  11     11     11      0      0      0      0      0      0      0      0      0      0      0      0      0

 

I'm concerned with ports 18 and 7; they are configured as long distance and run at 8Gbps.

 

For port 18, I have the following information:

 

portIndex:  18
portName: port18
portHealth: Fabric vision license not present. Please install the license and retry the operation.

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x1000490b    PRESENT ACTIVE E_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN LED
LocalSwcFlags: 0x0
portType:  18.0
POD Port: Port is licensed
portState: 1    Online
Protocol: FC
portPhys:  6    In_Sync         portScn:   16   E_Port
port generation number:    68
state transition count:    10

portId:    0a0300
portIfId:    43020003
portWwn:  
portWwn of device(s) connected:

Distance:  standard <= 10km
portSpeed: N8Gbps

Credit Recovery: Inactive
LE domain: 0
Peer beacon: Off
FC Fastwrite: OFF
Interrupts:        0          Link_failure: 0          Frjt:         0
Unknown:           0          Loss_of_sync: 0          Fbsy:         0
Lli:               1          Loss_of_sig:  0
Proc_rqrd:         34171      Protocol_err: 0
Timed_out:         0          Invalid_word: 9
Rx_flushed:        0          Invalid_crc:  1
Tx_unavail:        0          Delim_err:    0
Free_buffer:       0          Address_err:  0
Overrun:           0          Lr_in:        0
Suspended:         0          Lr_out:       0
Parity_err:        0          Ols_in:       0
2_parity_err:      0          Ols_out:      0
CMI_bus_err:       0

Port part of other ADs: No

 

 

 

On port 7:

 

portIndex:   7
portName: port7
portHealth: Fabric vision license not present. Please install the license and retry the operation.

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x10004907    PRESENT ACTIVE E_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN LED
LocalSwcFlags: 0x0
portType:  18.0
POD Port: Port is licensed
portState: 1    Online
Protocol: FC
portPhys:  6    In_Sync         portScn:   16   E_Port
port generation number:    60
state transition count:    58

portId:    010700
portIfId:    43020010
portWwn:  
portWwn of device(s) connected:

Distance:  standard <= 10km
portSpeed: N8Gbps

Credit Recovery: Inactive
LE domain: 0
Peer beacon: Off
FC Fastwrite: OFF
Interrupts:        0          Link_failure: 0          Frjt:         0
Unknown:           0          Loss_of_sync: 0          Fbsy:         0
Lli:               0          Loss_of_sig:  0
Proc_rqrd:         34207      Protocol_err: 0
Timed_out:         0          Invalid_word: 11
Rx_flushed:        0          Invalid_crc:  11
Tx_unavail:        0          Delim_err:    0
Free_buffer:       0          Address_err:  0
Overrun:           0          Lr_in:        0
Suspended:         0          Lr_out:       0
Parity_err:        0          Ols_in:       0
2_parity_err:      0          Ols_out:      0
CMI_bus_err:       0

Port part of other ADs: No

 

 

Any advice where the source of the error could potentially be?  I have replaced the fiber, SFP (even tried different fiber between the two buildings) but still run into the same errors.

 

Also, the number of errors are small....should I even be concerned at all?  In replication tests, I do see 5.5Gbps throughput...

 

Thanks.

Broadcom Moderator
Posts: 467
Registered: ‎03-29-2011

Re: Help interpretting the following errors

Hi,

 

both port (with errors) are loggin good eof errors, so the switch detected a frame corruption (and encoding errors as well as one bad end-of-frame (eof)).  Two more questions:

 

1. What is the FOS version on your 300s?

2. I assume that you are using your own fibre and no CDWM or DWDM link. 

a. There is a longwave SFP in port 18 and 7, right.

b. And otherwise, you are only using port 1,3 respectively 0,3 in switches?

 

 




If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution".


Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
Occasional Contributor
Posts: 7
Registered: ‎12-12-2017

Re: Help interpretting the following errors

Thanks Martin,

 

Please bear with me as I am still a novice with FC implementations.

 

1) FOS is v7.4.2a

2a) yes

2b) That is correct.   1,3 and 0,1 are respectively connected to the SANs at each site.   18 and 7 transfer the replicated SAN data (uni-directional) between the two sites.   No other ports are being used.

 

I've been running more tests and here are the results after pushing more data:

 

          frames      enc    crc    crc    too    too    bad    enc   disc   link   loss   loss   frjt   fbsy  c3timeout    pcs
       tx     rx      in    err    g_eof  shrt   long   eof     out   c3    fail    sync   sig                  tx    rx     err
  1:    7.9m  52.5m   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
  3:    1.5m   7.0m   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
 18:   53.1m   2.9m  11      4      3      0      0      1      0      0      0      0      0      0      0      0      0      0

          frames      enc    crc    crc    too    too    bad    enc   disc   link   loss   loss   frjt   fbsy  c3timeout    pcs
       tx     rx      in    err    g_eof  shrt   long   eof     out   c3    fail    sync   sig                  tx    rx     err
  0:    1.1m   1.1m   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
  1:   52.4m   2.1m   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
  7:    2.9m  53.1m  31     31     31      0      0      0      0      0      0      0      0      0      0      0      0      0

 

At this point, I'm not even sure where to start troubleshooting.

 

Broadcom Moderator
Posts: 467
Registered: ‎03-29-2011

Re: Help interpretting the following errors

Hi Marvin,

 

so this look like sfp / cable error - but you have already replace / check those, if I am remember correctly?




If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution".


Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
Occasional Contributor
Posts: 7
Registered: ‎12-12-2017

Re: Help interpretting the following errors

[ Edited ]

Yes, I have replaced the SFPs and fiber cable (and cleaned them numerous times).   With all the replacements (including connecting to different fiber strands between the buildings), I still get consistent errors.

 

I have more spare cables which I can try.   But I'll have to order more SFPs.   Considering the cost of the SFPs, I wanted to see if there were configuration changes I may have missed.

 

I'll also drop the 8Gb to 4Gb and see if that improves the situation.

Broadcom Moderator
Posts: 467
Registered: ‎03-29-2011

Re: Help interpretting the following errors

Hi Marvin,

 

drop down to 4GB if you do not need the 8GB speed for now and see if that reduces the error. 

 

kind regards,




If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution".


Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
Occasional Contributor
Posts: 7
Registered: ‎12-12-2017

Re: Help interpretting the following errors

Reducing the throughput from 8Gb (auto-negotiate) to 4Gb seems to have gotten rid of the errors.

 

Considering our SAN changes are minimal, I might be able to get away wtih 4Gb for our storage replication.

 

But why would 4Gb be error free compared to 8Gb?

 

Thanks.

Broadcom Moderator
Posts: 415
Registered: ‎08-31-2009

Re: Help interpretting the following errors

Hello,

 

What fiber quality are you using (OM type)?

What is the distance between both switches?

 

 

Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
Occasional Contributor
Posts: 7
Registered: ‎12-12-2017

Re: Help interpretting the following errors

It is a single mode fiber, OS2 (Corning SM OS2 LC-LC) between the two switches.

 

The distance between the two switches/datacenters is approximately 400m.   Both source and destination ports have been configured as long distance (LE<==10KM).

 

 

Broadcom Moderator
Posts: 467
Registered: ‎03-29-2011

Re: Help interpretting the following errors

Hi Marvin,

 

there have been issue in the past with Longwave SW SFPs (8GB) and interference with shortwave SFPs on 300.  And placement of SFPs, but your FOS level should correct for that.  

 

On a side note, if the distance is 400m, you do not need to use LE mode (that is for 5km to 10km). And if you go back to L0 (standard) you will also be using all VC (2,3,4,5) instead of having them collapsed to VC2. 




If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution".


Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.