Ethernet Fabric (VDX, CNA)

Reply
da
Occasional Contributor
Posts: 9
Registered: ‎01-07-2013

VRRP-E Virtual Gateway IP ICMP unresponsive / heavily delayed responsetime

I am currently experiencing a strange issue on a pair of VDX 6740s in logical-chassis mode running version 4.1.2ac.

 

The issue I am seeing is that the VRRP-E Virtual Gateway is practically unresponsive when you attempt to ping it. This was not an issue, when I put in production. The cluster has been up for over 3 months and it seems that the performance has deteriorated since then.

 

As one might suspect, this causes some issues for the devices using the gateway address for HA purposes and so on.

 

Below are excerpts from a ping session towards the VRRP-E Virtual IP address, which clearly shows an issue. All the while I have a ping session towards the actual IP address of the VRRP-E Master (192.168.208.2), which responds in less than 1 ms. Also, pings going through the Virtual Gateway seems to work with only a few to no issues.

 

Virtual IP address = 192.168.208.1

Rbridge 1 = .2

Rbridge 2 = .3

 

C:\Users\klestrup>ping -t 192.168.208.1 -w 20000

Pinging 192.168.208.1 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 192.168.208.1: bytes=32 time=4240ms TTL=64
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 192.168.208.1: bytes=32 time=2420ms TTL=64
Reply from 192.168.208.1: bytes=32 time=3694ms TTL=64
Reply from 192.168.208.1: bytes=32 time=4797ms TTL=64
Request timed out.
Request timed out.
Reply from 192.168.208.1: bytes=32 time=2444ms TTL=64
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 192.168.208.1: bytes=32 time=4679ms TTL=64
Reply from 192.168.208.1: bytes=32 time=5298ms TTL=64
Reply from 192.168.208.1: bytes=32 time=4597ms TTL=64
Reply from 192.168.208.1: bytes=32 time=5396ms TTL=64
Request timed out.
Reply from 192.168.208.1: bytes=32 time=7198ms TTL=64
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 192.168.208.1: bytes=32 time=8162ms TTL=64

 

Below is the configuration part for the ve interfaces on both VCS nodes.

 

rbridge-id 1

interface Ve 208

ip address 192.168.208.2/24
no shutdown
vrrp-extended-group 208
virtual-ip 192.168.208.1
track tengigabitethernet 1/0/33 priority 20
enable
preempt-mode
priority 120

 

rbridge-id 2

interface Ve 208
ip address 192.168.208.3/24
no shutdown
vrrp-extended-group 208
virtual-ip 192.168.208.1
track tengigabitethernet 1/0/33 priority 20
enable
preempt-mode
priority 110

 

As far as I know this config should be good and working.

 

Below is an excerpt of the show vrrp detail output.

 

rbridge-id 1

VRID 208
Interface: Ve 208; Ifindex: 1207959760
Mode: VRRPE
Admin Status: Enabled
Description :
Address family: IPv4
Authentication type: No Authentication
State: Master
Session Master IP Address: Local
Backup Router(s):
Virtual IP(s): 192.168.208.1
Virtual MAC Address: 02e0.52d0.cdd0
Configured Priority: 120 (default: 100); Current Priority: 120
**filtered** interval: 1 sec (default: 1 sec)
Preempt mode: ENABLE (default: DISABLED)
Advertise-backup: DISABLE (default: DISABLED)
Backup **filtered** interval: 60 sec (default: 60 sec)
Short-path-forwarding: Disabled
Revert-Priority: unset; SPF Reverted: No
Hold time: 0 sec (default: 0 sec)
Master Down interval: 4 sec
Trackport:
Port(s) Priority Port Status
======= ======== ===========
TenGigabitEthernet 1/0/33 20 Up

Global Statistics:
==================
Checksum Error : 0
Version Error : 0
VRID Invalid : 0

Session Statistics:
===================
Advertisements : Rx: 98, Tx: 9556010
Gratuitous ARP : Tx: 4778010
Session becoming master : 5
Advts with wrong interval : 0
Prio Zero pkts : Rx: 0, Tx: 0
Invalid Pkts Rvcd : 0
Bad Virtual-IP Pkts : 0
Invalid Authenticaton type : 0
Invalid TTL Value : 0
Invalid Packet Length : 0
VRRPE backup advt recvd : 0

 

rbridge-id 2

VRID 208
Interface: Ve 208; Ifindex: 1207959760
Mode: VRRPE
Admin Status: Enabled
Description :
Address family: IPv4
Authentication type: No Authentication
State: Backup
Session Master IP Address: 192.168.208.2
Virtual IP(s): 192.168.208.1
Virtual MAC Address: 02e0.52d0.cdd0
Configured Priority: 110 (default: 100); Current Priority: 110
**filtered** interval: 1 sec (default: 1 sec)
Preempt mode: ENABLE (default: DISABLED)
Advertise-backup: DISABLE (default: DISABLED)
Backup **filtered** interval: 60 sec (default: 60 sec)
Short-path-forwarding: Disabled
Revert-Priority: unset; SPF Reverted: No
Hold time: 0 sec (default: 0 sec)
Master Down interval: 4 sec
Trackport:
Port(s) Priority Port Status
======= ======== ===========
TenGigabitEthernet 2/0/33 20 Up

Global Statistics:
==================
Checksum Error : 0
Version Error : 0
VRID Invalid : 0

Session Statistics:
===================
Advertisements : Rx: 9551137, Tx: 98
Gratuitous ARP : Tx: 53
Session becoming master : 4
Advts with wrong interval : 0
Prio Zero pkts : Rx: 0, Tx: 0
Invalid Pkts Rvcd : 0
Bad Virtual-IP Pkts : 0
Invalid Authenticaton type : 0
Invalid TTL Value : 0
Invalid Packet Length : 0
VRRPE backup advt recvd : 0

 

There are multiple interfaces on the device (10 VE's to be exact) and each has their own subnet and VRID associated. The issue exists on all of the VE interfaces.

 

I am troubleshooting a larger issue with slow traffic for the users and this is what pops into view.

 

Any and all advice is welcome!

da
Occasional Contributor
Posts: 9
Registered: ‎01-07-2013

Re: VRRP-E Virtual Gateway IP ICMP unresponsive / heavily delayed responsetime

I have now shut down rbridge 2. This slightly improved the ping statistics, but not by much. I still had many seconds where the pings would just time out and when I got a response it was with a delay of several seconds as before.

 

I then rebooted rbridge 1 and when it came back up the pings were somewhat stable afterwards. I still see a few spikes, but no timeouts. My only concern now is how long until it behaves as it did before.

 

Pinging 192.168.208.1 with 32 bytes of data:
Reply from 192.168.208.1: bytes=32 time=2ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=15ms TTL=64
Reply from 192.168.208.1: bytes=32 time=10ms TTL=64
Reply from 192.168.208.1: bytes=32 time=5ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=13ms TTL=64
Reply from 192.168.208.1: bytes=32 time=7ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64
Reply from 192.168.208.1: bytes=32 time=1ms TTL=64

Ping statistics for 192.168.208.1:
Packets: Sent = 28, Received = 28, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 1ms, Maximum = 15ms, Average = 2ms

 

I have a similar setup at another customers site where the only real difference is that the VDX hardware aren't 6740s, but instead 6720s. They have never experienced an issue like this.

Regular Visitor
Posts: 1
Registered: ‎06-23-2014

Re: VRRP-E Virtual Gateway IP ICMP unresponsive / heavily delayed responsetime

Hi!

 

We are also running NOS 4.1.2ac and experiencing the exact same issues. We've raised a TAC case and it was determined that this behaviour is due to a known issue tracked as DEFECT000513620 (High RTT when pinging VRRP VIP). The issue can happen every time a backup (who has learnt ARP for VIP) changes into SPF/backup or Master due to a race condition between ARP packets and the software modules trying to delete/add the same prefix from the hardware.

 

The bug is supposed to be fixed in NOS 4.1.3 (available since yesterday).

 

Has anyone already tested NOS 4.1.3 and can confirm that the issue has actually been fixed?

 

Kind regards,

Stefan

Contributor
Posts: 47
Registered: ‎08-03-2015

Re: VRRP-E Virtual Gateway IP ICMP unresponsive / heavily delayed responsetime

Below configuration is working at our datacenter since 1 year

 

rbridge-id 1
 interface Ve 2
  ip ospf area 0
  ip ospf passive
  ip proxy-arp
  ip address 10.105.1.6/28
  no shutdown
  vrrp-extended-group 2
   virtual-ip 10.105.1.4
   enable
   preempt-mode
   priority 200
   advertise-backup
   short-path-forwarding
  !
 !
!

rbridge-id 2
 interface Ve 2
  ip ospf area 0
  ip ospf passive
  ip proxy-arp
  ip address 10.105.1.5/28
  no shutdown
  vrrp-extended-group 2
   virtual-ip 10.105.1.4
   enable
   preempt-mode
   advertise-backup
   short-path-forwarding
  !
 !

______________________
Umair Khan Patel
https://in.linkedin.com/in/patelumairkhan

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.