Fibre Channel (SAN)

Reply
Occasional Contributor
Posts: 11
Registered: ‎10-08-2014

Updated to 7.4.1e and now one of my switches is showing Hard_flt errors..

Hi all,

       did post this elsewhere but appears to have been removed so reposting with redaction in case this was the reason it was removed previously.. updated a pair of old Brocade 300 switches to 7.4.1e over the last week from 6.4.2. One of the switches was already having issues with cpu resets and now appears to have multiple ports either in NoSync or Hard_Flt or L_Port status. All of these were fine 24 hours ago. I am guessing this is some sort of hardware failure but any advice on further troubleshooting welcome. Already tried disabling and enabling ports to no avail..

 

 

 

Index Port Address Media Speed State Proto
==================================================
0 0 0a1700 id 8G Online FC F-Port xx:xx:xx:xx:xx:xx:xx:xx
1 1 0a1500 id 8G No_Sync FC
2 2 0a1300 id N8 Hard_Flt FC
3 3 0a1100 id N8 Online FC F-Port  xx:xx:xx:xx:xx:xx:xx:xx
4 4 0a1600 id N8 Hard_Flt FC
5 5 0a1400 id N8 Online FC F-Port  xx:xx:xx:xx:xx:xx:xx:xx
6 6 0a1200 id N8 Hard_Flt FC
7 7 0a1000 id N8 Online FC F-Port  xx:xx:xx:xx:xx:xx:xx:xx
8 8 0a0f00 id N8 Online FC F-Port  xx:xx:xx:xx:xx:xx:xx:xx
9 9 0a0d00 id 4G Online FC L-Port 1 public
10 10 0a0b00 id N4 In_Sync FC
11 11 0a0900 id N8 In_Sync FC
12 12 0a0e00 id N4 Online FC F-Port  xx:xx:xx:xx:xx:xx:xx:xx
13 13 0a0c00 id N8 Online FC F-Port  xx:xx:xx:xx:xx:xx:xx:xx
14 14 0a0a00 id N8 Online FC F-Port  xx:xx:xx:xx:xx:xx:xx:xx
15 15 0a0800 id N8 In_Sync FC
16 16 0a0700 id N8 Online FC L-Port
17 17 0a0500 id N8 In_Sync FC
18 18 0a0300 id N8 Online FC F-Port  xx:xx:xx:xx:xx:xx:xx:xx
19 19 0a0100 id N8 Online FC F-Port  xx:xx:xx:xx:xx:xx:xx:xx
20 20 0a0600 id N8 Online FC F-Port  xx:xx:xx:xx:xx:xx:xx:xx
21 21 0a0400 id N8 No_Light FC
22 22 0a0200 -- N8 No_Module FC

 

All ports 0-20 were fine post update but this switch had had a few spontaneous reboots prior to the firmware update. I am guessing these all now point toward hardware issues?

 

the fabriclog appears to be showing some port flapping but these devices were fine previously so i am wondering if it's actually the switch which is at fault

 

08:46:28.534534 *Removing all nodes from port D0,P0 D0,P0 13 NA
08:46:28.709195 SCN LR_PORT(0);g=0x5df0 D0,P0 D0,P0 13 NA
08:46:28.709253 SCN Port Online; g=0x5df0,isolated=0 D0,P0 D0,P1 13 NA
08:46:28.709423 Port Elp engaged D0,P1 D0,P0 13 NA
08:46:28.709500 *Removing all nodes from port D0,P0 D0,P0 13 NA
08:46:28.709643 SCN Port F_PORT D0,P1 D0,P0 13 NA
08:46:30.284077 SCN Port Offline;g=0x5df2 D0,P0 D0,P0 15 NA
08:46:30.284096 *Removing all nodes from port D0,P0 D0,P0 15 NA
08:46:30.898566 *Removing all nodes from port D0,P0 D0,P0 5 NA
08:46:30.898708 SCN Port F_PORT D0,P0 D0,P0 5 NA
08:46:31.333173 SCN Port Offline;g=0x5df4 D0,P0 D0,P0 11 NA
08:46:31.333193 *Removing all nodes from port D0,P0 D0,P0 11 NA
08:46:32.593991 SCN Port Offline;g=0x5df6 D0,P0 D0,P0 13 NA
08:46:32.594009 *Removing all nodes from port D0,P0 D0,P0 13 NA
08:46:32.845692 SCN LR_PORT(0);g=0x5df6 D0,P0 D0,P0 13 NA
08:46:32.845741 SCN Port Online; g=0x5df6,isolated=0 D0,P0 D0,P1 13 NA
08:46:32.845913 Port Elp engaged D0,P1 D0,P0 13 NA
08:46:32.845993 *Removing all nodes from port D0,P0 D0,P0 13 NA
08:46:32.846137 SCN Port F_PORT D0,P1 D0,P0 13 NA
08:46:33.282719 SCN Port Offline;g=0x5df8 D0,P0 D0,P0 15 NA
08:46:33.282737 *Removing all nodes from port D0,P0 D0,P0 15 NA
08:46:34.334477 SCN Port Offline;g=0x5dfa D0,P0 D0,P0 11 NA
08:46:34.334498 *Removing all nodes from port D0,P0 D0,P0 11 NA
08:46:35.857793 SCN Port Offline;g=0x5dfc D0,P0 D0,P0 14 NA
08:46:35.857812 *Removing all nodes from port D0,P0 D0,P0 14 NA
08:46:36.111614 SCN LR_PORT(0);g=0x5dfc D0,P0 D0,P0 14 NA
08:46:36.111661 SCN Port Online; g=0x5dfc,isolated=0 D0,P0 D0,P1 14 NA
08:46:36.111835 Port Elp engaged D0,P1 D0,P0 14 NA
08:46:36.111915 *Removing all nodes from port D0,P0 D0,P0 14 NA
08:46:36.112060 SCN Port F_PORT D0,P1 D0,P0 14 NA
08:46:36.290822 SCN Port Offline;g=0x5dfe D0,P0 D0,P0 15 NA
08:46:36.290842 *Removing all nodes from port D0,P0 D0,P0 15 NA
08:46:36.653851 SCN Port Offline;g=0x5e00 D0,P0 D0,P0 13 NA
08:46:36.653872 *Removing all nodes from port D0,P0 D0,P0 13 NA
08:46:36.868517 SCN LR_PORT(0);g=0x5e00 D0,P0 D0,P0 13 NA
08:46:36.868572 SCN Port Online; g=0x5e00,isolated=0 D0,P0 D0,P1 13 NA
08:46:36.868744 Port Elp engaged D0,P1 D0,P0 13 NA
08:46:36.868823 *Removing all nodes from port D0,P0 D0,P0 13 NA
08:46:36.868966 SCN Port F_PORT D0,P1 D0,P0 13 NA
08:46:37.335304 SCN Port Offline;g=0x5e02 D0,P0 D0,P0 11 NA
08:46:37.335323 *Removing all nodes from port D0,P0 D0,P0 11 NA
08:46:39.290901 SCN Port Offline;g=0x5e04 D0,P0 D0,P0 15 NA
08:46:39.290920 *Removing all nodes from port D0,P0 D0,P0 15 NA
08:46:40.342782 SCN Port Offline;g=0x5e06 D0,P0 D0,P0 11 NA
08:46:40.342801 *Removing all nodes from port D0,P0 D0,P0 11 NA
08:46:40.712943 SCN Port Offline;g=0x5e08 D0,P0 D0,P0 13 NA
08:46:40.712962 *Removing all nodes from port D0,P0 D0,P0 13 NA
08:46:40.929195 SCN LR_PORT(0);g=0x5e08 D0,P0 D0,P0 13 NA
08:46:40.929251 SCN Port Online; g=0x5e08,isolated=0 D0,P0 D0,P1 13 NA
08:46:40.929420 Port Elp engaged D0,P1 D0,P0 13 NA
08:46:40.929499 *Removing all nodes from port D0,P0 D0,P0 13 NA
08:46:40.929643 SCN Port F_PORT D0,P1 D0,P0 13 NA
08:46:42.293870 SCN Port Offline;g=0x5e0a D0,P0 D0,P0 15 NA
08:46:42.293890 *Removing all nodes from port D0,P0 D0,P0 15 NA
08:46:42.489625 *Removing all nodes from port D0,P0 D0,P0 3 NA
08:46:42.489772 SCN Port F_PORT D0,P0 D0,P0 3 NA
08:46:43.347249 SCN Port Offline;g=0x5e0c D0,P0 D0,P0 11 NA
08:46:43.347268 *Removing all nodes from port D0,P0 D0,P0 11 NA
08:46:43.986892 *Removing all nodes from port D0,P0 D0,P0 19 NA
08:46:43.988040 SCN Port F_PORT D0,P0 D0,P0 19 NA
08:46:44.211151 *Removing all nodes from port D0,P0 D0,P0 18 NA
08:46:44.211298 SCN Port F_PORT D0,P0 D0,P0 18 NA
08:46:44.771924 SCN Port Offline;g=0x5e0e D0,P0 D0,P0 13 NA
08:46:44.771943 *Removing all nodes from port D0,P0 D0,P0 13 NA
08:46:45.119880 SCN LR_PORT(0);g=0x5e0e D0,P0 D0,P0 13 NA
08:46:45.119929 SCN Port Online; g=0x5e0e,isolated=0 D0,P0 D0,P1 13 NA
08:46:45.120102 Port Elp engaged D0,P1 D0,P0 13 NA
08:46:45.120179 *Removing all nodes from port D0,P0 D0,P0 13 NA
08:46:45.120324 SCN Port F_PORT D0,P1 D0,P0 13 NA
08:46:45.293326 SCN Port Offline;g=0x5e10 D0,P0 D0,P0 15 NA
08:46:45.293346 *Removing all nodes from port D0,P0 D0,P0 15 NA
08:46:46.347961 SCN Port Offline;g=0x5e12 D0,P0 D0,P0 11 NA
08:46:46.347980 *Removing all nodes from port D0,P0 D0,P0 11 NA

 

 

If I want to replace the chassis I assume I'll need to arrange for a licence transfer of the existing licences which I am guessing will be a vendor specific process

 

So I am guessing you'll say replace the switch but thought it was worth asking.

 

thanks

Adam.

Brocade Moderator
Posts: 278
Registered: ‎08-31-2009

Re: Updated to 7.4.1e and now one of my switches is showing Hard_flt errors..

Hello,

 

It is looking like to be an Hardware issue. Do you see specific issue like Asic messages into errdump:

errdump --all

 

Concerning the replacement process, yes you have to open a ticket to your OEM for potential extra analysis and Hardware replacement. They will provide you the new licenses based on the licenses transfer from the defect switch.

Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
Occasional Contributor
Posts: 11
Registered: ‎10-08-2014

Re: Updated to 7.4.1e and now one of my switches is showing Hard_flt errors..

Haha yes i think it may be a hardware problem

 

 

2017/05/05-05:18:09, [C2-1002], 1831, FFDC | CHASSIS, ERROR, Brocade300, Port 2 chip faulted due to an internal error.
2017/05/05-05:18:17, [C2-1002], 1833, FFDC | CHASSIS, ERROR, Brocade300, Port 4 chip faulted due to an internal error.
2017/05/05-05:19:05, [C2-1002], 1839, FFDC | CHASSIS, ERROR, Brocade300, Port 6 chip faulted due to an internal error.
2017/05/05-09:15:02, [C2-1002], 1848, FFDC | CHASSIS, ERROR, Brocade300, Port 2 chip faulted due to an internal error.

 

 :-(

 

 

 

 

 

Brocade Moderator
Posts: 278
Registered: ‎08-31-2009

Re: Updated to 7.4.1e and now one of my switches is showing Hard_flt errors..

Yes this indicate an HW issue.

Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
Occasional Contributor
Posts: 11
Registered: ‎10-08-2014

Re: Updated to 7.4.1e and now one of my switches is showing Hard_flt errors..

thanks. I will contact my vendor for advice on getting the licences transferred. Do you know if there is a KB article which describes how to transfer the configuration from one switch to another. I am guessing configdownload or similar? I think I would plan to have the replacement hardware sent to us first so i can apply the customer config and replacement licences before sending the kit to site but I've not had to do this before.

 

thanks

 

 

Brocade Moderator
Posts: 278
Registered: ‎08-31-2009

Re: Updated to 7.4.1e and now one of my switches is showing Hard_flt errors..

KB article 000001240 describes the process.

Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"
Occasional Contributor
Posts: 11
Registered: ‎10-08-2014

Re: Updated to 7.4.1e and now one of my switches is showing Hard_flt errors..

Do you have a link to the KB article. I tried searching for that ID on the support portal but it didnt yield anything.

 

many thanks

Adam

Brocade Moderator
Posts: 278
Registered: ‎08-31-2009

Re: Updated to 7.4.1e and now one of my switches is showing Hard_flt errors..

Attached is the KB article.

Any and all information provided by me is not reviewed, approved or endorsed by Brocade and is provided solely as a convenience for Brocade customers. All systems and all networks are different and unique. If you have a service affecting network problem, please open a TAC service request for service through Brocade, or through your OEM equipment provider. If this provided you with a solution to this issue, please mark it with the button at the bottom "Accept as solution"

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.