Fibre Channel (SAN)

Reply
Highlighted
New Contributor
Posts: 4
Registered: ‎05-12-2017
Accepted Solution

Troubleshooting C3-1030s on DCX8510-8

All - I'm working on an issue with Brocade Support and NetApp. We are experiencing a number of [C3-1030] errors on one of our backbone switches (one of four). Here's a snippet of the log:

 
2017/05/09-19:05:18, [C3-1030], 51555, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x76, last:0xf0000068 thresh2:0x90000064.

2017/05/09-20:55:19, [C3-1030], 51572, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0xe0, last:0xf0000067 thresh2:0x90000064.

2017/05/09-23:00:20, [C3-1030], 51591, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x147, last:0xf0000065 thresh2:0x90000064.

2017/05/10-01:00:22, [C3-1030], 51609, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x1b9, last:0xf000006b thresh2:0x90000064.

2017/05/10-03:05:30, [C3-1030], 51628, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x221, last:0xf0000066 thresh2:0x90000064.

2017/05/10-05:10:31, [C3-1030], 51647, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x28e, last:0xf0000067 thresh2:0x90000064.

2017/05/10-06:50:32, [C3-1030], 51664, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x2fa, last:0xf0000067 thresh2:0x90000064.

2017/05/10-09:15:33, [C3-1030], 51686, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x370, last:0xf000006e thresh2:0x90000064.

2017/05/10-11:30:34, [C3-1030], 51706, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x3dd, last:0xf0000069 thresh2:0x90000064.

2017/05/10-13:30:35, [C3-1030], 51724, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x454, last:0xf000006a thresh2:0x90000064.

2017/05/10-15:45:37, [C3-1030], 51748, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x4c2, last:0xf0000066 thresh2:0x90000064.

2017/05/10-19:16:01, [C3-1030], 51779, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x79, last:0xf000006a thresh2:0x90000064.

2017/05/10-22:36:02, [C3-1030], 51808, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0xe1, last:0xf0000065 thresh2:0x90000064.

2017/05/11-04:41:02, [C3-1030], 51858, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x14b, last:0xf0000067 thresh2:0x90000064.

2017/05/11-12:31:28, [C3-1030], 51924, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x1b5, last:0xf0000065 thresh2:0x90000064.

 

 

I'm new to Brocade so I'm unfamiar with the SerDes tuning that Brocade has suggested, and so is my company and the Ops staff that are working on the issue with me. We have already swapped the core blade and a port blade and have manually tuned the card but the errors persist. 

 

Questions: 

1) What commands should be used to troubleshoot and isolate the issue?

2) What other steps should we take (with Brocade) to resolve? 

3) Does the error provided by the CP help us narrow it down to a particular ASIC, and as such specfic external ports that may or may not be causing the issue. 

 

 

 

New Contributor
Posts: 4
Registered: ‎05-12-2017

Re: Troubleshooting C3-1030s on DCX8510-8

Further to the above I found this, a snippet from the errdump:

017/05/11-04:41:02, [C3-1030], 51858, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x14b, last:0xf0000067 thresh2:0x90000064.

2017/05/11-12:31:28, [C3-1030], 51924, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x1b5, last:0xf0000065 thresh2:0x90000064.

2017/05/12-02:46:46, [C3-1030], 52049, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x73, last:0xf0000066 thresh2:0x90000064.

2017/05/12-07:36:48, [C3-1030], 52092, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0xd8, last:0xf0000065 thresh2:0x90000064.

2017/05/12-12:26:51, [C3-1030], 52134, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S5,P-1(70): Internal CRC with good EOF errors exceeded threshold, tuning is required. current:0x13f, last:0xf0000067 thresh2:0x90000064.

 

 

 

And for the portlog (for the last two days)

 


5-11-2017

12:31:26.936 PORT       Rx3     840   20  02fffffc,00c548c0,0156ffff,01000000

12:31:26.936 FCPH       read    840   20  02fffffc,00c548c0,00000000,00000000,0156707e

12:31:26.936 FCPH       seq     840   14  20290000,0156707e,00000757,0000001c,00000000

12:31:26.937 nsd        ctin    840    fc  00010118,00c548c0

12:31:26.937 nsd        ctout   840    fc  00018002,00000000

12:31:26.937 FCPH       write   840   20  03c548c0,00fffffc,00000000,00000000,0156707e

12:31:26.937 FCPH       seq     840   14  20980000,0156707e,000007bc,0004001c,00000000

12:31:26.937 PORT       Tx3     840   20  03c548c0,00fffffc,0156707e,01000000

12:32:13.623 cald       ctin      0    fa  00030649,000000c5,00000000

 

5-12-2017

12:26:47.889 PORT       Rx3     840   20  02fffffc,00c548c0,00c4ffff,01000000

12:26:47.889 FCPH       read    840   20  02fffffc,00c548c0,00000000,00000000,00c42580

12:26:47.889 FCPH       seq     840   14  20290000,00c42580,00000757,0000001c,00000000

12:26:47.889 nsd        ctin    840    fc  00010118,00c548c0

12:26:47.889 nsd        ctout   840    fc  00018002,00000000

12:26:47.889 FCPH       write   840   20  03c548c0,00fffffc,00000000,00000000,00c42580

12:26:47.889 FCPH       seq     840   14  20980000,00c42580,000007bc,0004001c,00000000

12:26:47.889 PORT       Tx3     840   20  03c548c0,00fffffc,00c42580,01000000

12:27:49.417 PORT       Rx3     840   20  02fffffc,00c548c0,0105ffff,01000000

 

 

The same port index (840) is referenced during the two events at the shown times, and we know it is the same internal port and C3-ASIC?

The port is:

> portshow -i 840

portIndex: 840

portName: primsgmbox02_hba1 (MS Exchange server).

portHealth: HEALTHY

 

Interrupts:        0          Link_failure: 0          Frjt:         0

Unknown:           0          Loss_of_sync: 0          Fbsy:         0

Lli:               0          Loss_of_sig:  0

Proc_rqrd:         1180       Protocol_err: 0

Timed_out:         0          Invalid_word: 0

Rx_flushed:        0          Invalid_crc:  0

Tx_unavail:        0          Delim_err:    0

Free_buffer:       0          Address_err:  0

Overrun:           0          Lr_in:        0

Suspended:         0          Lr_out:       0

Parity_err:        0          Ols_in:       0

2_parity_err:      0          Ols_out:      0

CMI_bus_err:       0

 

 

Does this mean there could be some link between port 840 and the C3-1030 errors we're seeing or is purely correlation? I'm just trying to understand if the backend CRCs are related to incoming/outgoing traffic happening on the switch at that point in time. 

Brocade Moderator
Posts: 251
Registered: ‎08-31-2009

Re: Troubleshooting C3-1030s on DCX8510-8

Hello,

 

What is the FOS release installed on the unit?

Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
Brocade Moderator
Posts: 63
Registered: ‎04-27-2009

Re: Troubleshooting C3-1030s on DCX8510-8

Hi,

 

the internal CRCs have no correlation to any external port, they are purely internal.

Also the portlogdump of that port does just look like Device <-> NameServer (fffffc) communication.

 

I think manual tuning with support is necessary.

 

TL

If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution".

Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider.
New Contributor
Posts: 4
Registered: ‎05-12-2017

Re: Troubleshooting C3-1030s on DCX8510-8

Thanks Thomas, this is the approach we're taking with Brocade. I have a change request in flight for this weekend and have been tracking the number of internal CRCs every day. They are quite low (2-4 per day), but I'm hopeful we can get to almost none. Thanks again for your feedback, I appreciate. - Brent

New Contributor
Posts: 4
Registered: ‎05-12-2017

Re: Troubleshooting C3-1030s on DCX8510-8

The switch in question is running 7.4.0a. Thank you for asking. We are also planning to upgrade the version of FOS on this switch once we've tuned it with Brocade. Thanks for checking!!

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.