03-14-2016 06:47 PM
I currently have a Fortigate device connected to a VDX cluster using VCS Chassis mode via SR-MM Fiber SFPs, hopefully someone else has had similar issues.
When the FortiGate edge device reboots, the line-protocol goes down in the VDX, however it stays up on the fortigate. This also happens if i hard-cycle the fortigate edge device by removing power to the unit and then booting it back up over a minute later. Even though the fortigate device thinks that there is connectivity still, but TX and RX counters do not increase.
The interface may be brought up by disconnecting and reconnecting the fiber cable (from either SFP, switch or firewall end), shut/no-shut on the switch, or power cycling the switch.
The interface is not brought up by power-cycling the firewall again, shut/no-shut the interface on the firewall.
It doesnt look like the Issue is SFP related, as this issue does not occur when the switches are connected to other edge devices (tested both the fortigate and the VDX by connecting them to separate devices.
interface TenGigabitEthernet 12/0/31
no fabric isl enable
no fabric trunk enable
show int te 12/0/31
TenGigabitEthernet 12/0/31 is up, line protocol is down (link protocol down)
Hardware is Ethernet, address is 50eb.1aff.1f25
Current address is 50eb.1aff.1f25
Pluggable media present
Interface index (ifindex) is 51943276574
MTU 2500 bytes
LineSpeed Actual : Nil
LineSpeed Configured : Auto, Duplex: Full
Priority Tag disable
IPv6 RA Guard disable
Last clearing of show interface counters: 4d04h09m
Queueing strategy: fifo
0 packets, 0 bytes
Unicasts: 0, Multicasts: 0, Broadcasts: 0
64-byte pkts: 0, Over 64-byte pkts: 0, Over 127-byte pkts: 0
Over 255-byte pkts: 0, Over 511-byte pkts: 0, Over 1023-byte pkts: 0
Over 1518-byte pkts(Jumbo): 0
Runts: 0, Jabbers: 0, CRC: 0, Overruns: 0
Errors: 0, Discards: 0
1003 packets, 132396 bytes
Unicasts: 0, Multicasts: 1003, Broadcasts: 0
Errors: 0, Discards: 0
Input 0.000000 Mbits/sec, 0 packets/sec, 0.00% of line-rate
Output 0.000000 Mbits/sec, 0 packets/sec, 0.00% of line-rate
Time since last interface status change: 00:02:58
ROGDC1PFW (global) $ diag hardware deviceinfo nic port6
Description :FortiASIC NP6 Adapter
Driver Name :FortiASIC Unified NPU Driver
PCI Slot :01:00.0
Major ID :5
Minor ID :0
lif id :5
lif oid :133
netdev oid :133
netdev flags :1203
phy name :port6
========== Link Status ==========
netdev status :up
============ Counters ===========
Rx Pkts :0
Rx Bytes :0
Tx Pkts :0
Tx Bytes :0
Host Rx Pkts :0
Host Rx Bytes :0
Host Rx dropped :0
Host Tx Pkts :0
Host Tx Bytes :0
Host Tx dropped :0
06-30-2016 07:54 PM
I am having a similar issue. Here is my setup:
VDX6940 switch with Brocade 40GB SR4 optical QSFPs
Dell servers with Mellanox ConnectX-3 dual port 40GB NICs using Brocade 40GB SR4 optical QSFPs
Win 2012R2 OS on server
1 or 2 meter fiber patches between switch and servers
Rebooting the server will randomly adminDown an interface on the switch. Sometimes I can fix the problem by doing a no shut on the interface. Other times I have to phyiscally pull the SFP from the server NIC to reset the link.
I am leaning towards something with the NICs but would like to debug on the switch. Anyone have a good set of commands to monitor the physical layer on an interface?
07-18-2016 06:23 AM
I have a very similar issue that was discovered with our setup, as well. I have contacted Brocade Support and have an open ticket with them regarding this issue. They are currently investigating this in a lab environment. I'll certainly keep everyone here informed as to our progress.
We have an ICX6610 running VRRP with another ICX6610, sitting as edge devices on an 8 switch ethernet fabric in logical-chassis mode (VDX 6740's). When the primary 6610 node is rebooted, the interface to the VDX shows up-up on the 6610, but the VDX port shows up-down. The problem can be fixed by bouncing the interface on either side. Unfortunately, our VRRP setup requires this communication for the switches to notify one another that they are both back up, so our environment gets into a master/master VRRP condition when the primary reboots fully, and we lose all layer 3 communication until the bounce of the port takes place.
I'm curious as to the NOS versions everyone is using on the VDX's. We're still back on 4.1.3a.
07-20-2016 09:22 AM
Follow up update!
Problem fixed. turns out it the firmware on the brocade VDX switch causing the issue. downgrading to 5.0.2a seems to have fixed the problem. (we have since upgraded to 6.0.2b, no problems here either)