Ethernet Switches & Routers

Reply
Occasional Contributor
Posts: 14
Registered: ‎02-09-2011

FCX switches are shutting down occasionally

Actually the better phrase we've coined is " going dark" (all front panel leds are dark, no console output, all links are down but power supply fans are running and power supply leds are either green or red).  To restore network just unplug both power supplies, count to 15 and plug back in.

Has anyone experienced similar issues?  Any ideas on what is happening?

There appears to be no ryhme or reason for these outages.  I believe this is a power issue but I'm not sure why the PS runs but the switch chassis is dark.  There is one location at which the UPS has died and is now unprotected.  I'll get a call/txt from my contact in the building that there has been a power event.  Usually there will be at least one switch down in that location's closet.

For those whom have been blessed with not having this issue; a little background.

1.  We have roughly 150 fcx switches bought at multiple times. 

2.  Some of these switches have been on UPS's others not.  Ups's vary from apc 1400's to Matrix 5000.

3.  In a closet with 10 switches this issues may effect only one FCX.  These can be stacked or single installs.

4.  In mixed closets only FCX switches will be effected.  I've never seen and SX, FES, FGS have this issue.

5.  The issue may not effect the same switch each time.

6.  FCX can be single or dual PS setup

We returned approx 10 switches at brocade request and some of the replacement switches have been hit with this issue.

Thanks for your input.

External Moderator
Posts: 4,973
Registered: ‎02-23-2004

Re: FCX switches are shutting down occasionally

I suspected following:

1) to high Temperature

2) electricity problem, like incorrect grounding

TechHelp24
Occasional Contributor
Posts: 14
Registered: ‎02-09-2011

Re: FCX switches are shutting down occasionally

Thanks Techhelp24,  Since I can't give an across the board answer (I forgot to list that these switches are distributed to 40+ sites and mutiple techs( me included) have been dispatched to address these outages).

1) to high Temperature --

  These switches are configured with syslog and monitored for heat.  I've seen nothing in the logs or notifications.  Upon entering the rooms there is no apparent AC issue (rooms are kept 72F-78F)

2) electricity problem, like incorrect grounding  --

Do you mean the PS themselves or the chassis?  Some of these switches have been plugged into UPS's which report no grounding issue and company electricians have veified the circuits.  The racks are connected to a grounding bar and overhead ladder, overhead ladder is also connected to grounding bar.  Grounding bar is then connected to panel or straight to earth ground.   I had equipment ground verified at the first couple locations but since it didn't appear related haven't done it lately.    I'll randomly recheck this but I won't get back to these sites till next week

Thank you

External Moderator
Posts: 4,973
Registered: ‎02-23-2004

Re: FCX switches are shutting down occasionally

I've checked several Release Notes, and nothing found about such behavior or any such Bug / Defect is or was present.

I'm sorry, I have no other idea what can caused such occasionally shutdown.

At this point, i would procede with the Suppor tand open a call by Brocade.

TechHelp24
Super Contributor
Posts: 1,087
Registered: ‎12-13-2009

Re: FCX switches are shutting down occasionally

I have not seen this issue before.  A bit of a lame answer is to upgrade the switches to 7.2.02d (not e). and see if the issue happens again.  A better answer would be to get the Brocade TAC involed with the issue, ask them if a crash dump can be done or the like.

Thanks

Michael.

Occasional Contributor
Posts: 14
Registered: ‎02-09-2011

Re: FCX switches are shutting down occasionally

Thanks mschipp,  We just updated all the fcx's to 7.2.0.2d within the last month.  We'll see if that fixes the issue.  Since it is random we'll just need to monitor for the next year.

TAC and our SE has been involved in this issue for the last year or so.  We thought the firmware upgrade might fix the issue but I couldn't get an answer on what it addresses, how to tell if it is installed already, or if it is included in a later release.

Super Contributor
Posts: 1,087
Registered: ‎12-13-2009

Re: FCX switches are shutting down occasionally

Hi Swhall,

      We thought the firmware upgrade might fix the issue but I couldn't get an answer on what it addresses, how to tell if it is installed already, or if it is included in a later release.

     Sorry not sure what you mean by "how to tell if it is installed already, or if it is included in a later release."

The release notes of 7.2.02d should give you most (but not all) of what was fixed in that release.

Do you mean how to tell if 7.2.02d has been deployed?  Sorry no idea on "or if it is included in a later release" unless you mean the fix for this issue.

Thanks

Michael.

Occasional Contributor
Posts: 14
Registered: ‎02-09-2011

Re: FCX switches are shutting down occasionally

Sorry for the confusing response the 7.2.0.2d code is installed.  When I reference firmware I'm talking about the POE firmware upgrade.  Since our issue is related to the communication between the power supplies and the switch I thought this upgdate may address our issue.

In testing we could install the POE firmware release but noticed the following issues:

1.  There is no way to tell the version of the firmware release on the switch (whether stacked or single).  Installing the firmware does not change the version number reported by the switch.  SE and I both argued with TAC about this but they could see why the version number was important.

2.  Related to number one,

           How do I tell if a recently purchased switch has the new firmware?

             How do I tell if a RMA'd replacemnt switch has the new firmware installed?

             How do I verify if one of my techs has installed the firmware (or not)

3.  Does it address my issue or am I just wasting time on an upgrade that fixes a problem I'm not experencing.

I asked TAC if the firmware release may be rolled into a later release of the code.  They couldn't tell me yes or no...  But I think this was related to the fact noone knew what the POE firmware release addressed.

Super Contributor
Posts: 1,087
Registered: ‎12-13-2009

Re: FCX switches are shutting down occasionally

Hi swhall,

     Ok good questions.  Bad answers follow

Have you upgrade the switches to 7.2.02d (not just the PoE firmware?

How do you tell if the PoE firmware has been apply - no idea - the release notes do not say.  Suggest you ask the TAC on how to tell.  There must be some output from a 'show tech'  that stats this.

     I could not see any anything related to your issue in the release notes for 7.2.02.d.

The TAC must be able to show you how to tell if the PoE fireware has been apply.

Thanks

Michael.

Occasional Contributor
Posts: 14
Registered: ‎02-09-2011

Re: FCX switches are shutting down occasionally

Yes the switches are at 7.2.0.2.d.

The issue with the POE firmware is that TAC doesn't know.  I've talked to multiple engineers and my SE has called his contacts.  No one could/would give us an answer.

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.