Ethernet Fabric (VDX, CNA)

Reply
mrw
New Contributor
Posts: 2
Registered: ‎08-10-2015

Link-state down after edge device reboot

I currently have a Fortigate device connected to a VDX cluster using VCS Chassis mode via SR-MM Fiber SFPs, hopefully someone else has had similar issues.

 

When the FortiGate edge device reboots, the line-protocol goes down in the VDX, however it stays up on the fortigate. This also happens if i hard-cycle the fortigate edge device by removing power to the unit and then booting it back up over a minute later. Even though the fortigate device thinks that there is connectivity still, but TX and RX counters do not increase.

 

The interface may be brought up by disconnecting and reconnecting the fiber cable (from either SFP, switch or firewall end), shut/no-shut on the switch, or power cycling the switch.

 

The interface is not brought up by power-cycling the firewall again, shut/no-shut the interface on the firewall. 

 

It doesnt look like the Issue is SFP related, as this issue does not occur when the switches are connected to other edge devices (tested both the fortigate and the VDX by connecting them to separate devices.

 

 

interface TenGigabitEthernet 12/0/31

description rb12-31_fg8409-6

no fabric isl enable

no fabric trunk enable

no shutdown

!

 

 

show int te 12/0/31

TenGigabitEthernet 12/0/31 is up, line protocol is down (link protocol down)

Hardware is Ethernet, address is 50eb.1aff.1f25

    Current address is 50eb.1aff.1f25

Pluggable media present

Description: rb12-31_fg8409-6

Interface index (ifindex) is 51943276574

MTU 2500 bytes

LineSpeed Actual     : Nil

LineSpeed Configured : Auto, Duplex: Full

Priority Tag disable

IPv6 RA Guard disable

Last clearing of show interface counters: 4d04h09m

Queueing strategy: fifo

Receive Statistics:

    0 packets, 0 bytes

    Unicasts: 0, Multicasts: 0, Broadcasts: 0

    64-byte pkts: 0, Over 64-byte pkts: 0, Over 127-byte pkts: 0

    Over 255-byte pkts: 0, Over 511-byte pkts: 0, Over 1023-byte pkts: 0

    Over 1518-byte pkts(Jumbo): 0

    Runts: 0, Jabbers: 0, CRC: 0, Overruns: 0

    Errors: 0, Discards: 0

Transmit Statistics:

    1003 packets, 132396 bytes

    Unicasts: 0, Multicasts: 1003, Broadcasts: 0

    Underruns: 0

    Errors: 0, Discards: 0

Rate info:

    Input 0.000000 Mbits/sec, 0 packets/sec, 0.00% of line-rate

    Output 0.000000 Mbits/sec, 0 packets/sec, 0.00% of line-rate

Time since last interface status change: 00:02:58

 

 

ROGDC1PFW (global) $ diag hardware deviceinfo nic port6

Description     :FortiASIC NP6 Adapter

Driver Name     :FortiASIC Unified NPU Driver

Name            :np6_0

PCI Slot        :01:00.0

irq             :16

Board           :FGT300d

SN              :REDACTED

Major ID        :5

Minor ID        :0

lif id          :5

lif oid         :133

netdev oid      :133

netdev flags    :1203

Current_HWaddr  :90:6c:ac:34:5f:05

Permanent_HWaddr:90:6c:ac:34:5f:05

phy name        :port6

bank_id         :1

phy_addr        :0x09

lane            :9

sw_port         :0

sw_np_port      :0

vid_phy[6]      :[0x07][0x00][0x00][0x00][0x00][0x00]

vid_fwd[6]      :[0x00][0x00][0x00][0x00][0x00][0x00]

oid_fwd[6]      :[0x00][0x00][0x00][0x00][0x00][0x00]

========== Link Status ==========

Admin           :up

netdev status   :up

autonego_setting:1

link_setting    :1

link_speed      :1000

link_duplex     :0

Speed           :1000

Duplex          :Full

link_status     :Up

rx_link_status  :1

int_phy_link    :0

local_fault     :0

local_warning   :0

remote_fault    :0

============ Counters ===========

Rx Pkts         :0

Rx Bytes        :0

Tx Pkts         :0

Tx Bytes        :0

Host Rx Pkts    :0

Host Rx Bytes   :0

Host Rx dropped :0

Host Tx Pkts    :0

Host Tx Bytes   :0

Host Tx dropped :0

New Member
Posts: 1
Registered: ‎02-03-2012

Re: Link-state down after edge device reboot

I am having a similar issue. Here is my setup:

 

VDX6940 switch with Brocade 40GB SR4 optical QSFPs

Dell servers with Mellanox ConnectX-3 dual port 40GB NICs using Brocade 40GB SR4 optical QSFPs

Win 2012R2 OS on server

1 or 2 meter fiber patches between switch and servers

 

Rebooting the server will randomly adminDown an interface on the switch. Sometimes I can fix the problem by doing a no shut on the interface. Other times I have to phyiscally pull the SFP from the server NIC to reset the link.

 

I am leaning towards something with the NICs but would like to debug on the switch. Anyone have a good set of commands to monitor the physical layer on an interface?

 

 

Visitor
Posts: 1
Registered: ‎07-18-2016

Re: Link-state down after edge device reboot

I have a very similar issue that was discovered with our setup, as well.  I have contacted Brocade Support and have an open ticket with them regarding this issue.  They are currently investigating this in a lab environment.  I'll certainly keep everyone here informed as to our progress.

 

We have an ICX6610 running VRRP with another ICX6610, sitting as edge devices on an 8 switch ethernet fabric in logical-chassis mode (VDX 6740's).  When the primary 6610 node is rebooted, the interface to the VDX shows up-up on the 6610, but the VDX port shows up-down.  The problem can be fixed by bouncing the interface on either side.  Unfortunately, our VRRP setup requires this communication for the switches to notify one another that they are both back up, so our environment gets into a master/master VRRP condition when the primary reboots fully, and we lose all layer 3 communication until the bounce of the port takes place.

 

I'm curious as to the NOS versions everyone is using on the VDX's.  We're still back on 4.1.3a.

mrw
New Contributor
Posts: 2
Registered: ‎08-10-2015

Re: Link-state down after edge device reboot

Follow up update!

Problem fixed. turns out it the firmware on the brocade VDX switch causing the issue. downgrading to 5.0.2a seems to have fixed the problem. (we have since upgraded to 6.0.2b, no problems here either)  

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.

vADC is now Pulse Secure
Download FREE NVMe eBook