Ethernet Fabric (VDX, CNA)

Reply
Contributor
Posts: 66
Registered: ‎02-25-2016

Ethernet discards between VDX6720 and Juniper EX

[ Edited ]

Hello,

 

I moved some working circuits from EX-switch to a VDX6720 in the middle and now I am seeing ethernet discards continously on the VDX-side. I am unclear about the reason because the ports are not congested and there is only one tagged vlan on both side. 

 

I noticed that turning on flowcontrol tx on / rx on on the port-channel interfaces will lead to less or zero discarded frames, but the incoming EX-traffic will drop 50%, thats of course unacceptable for a 30G port-channel.  

 

VDX-Interface:

 

 

Port-channel 79 is up, line protocol is up
Interface index (ifindex) is 671088719
Minimum number of links to bring Port-channel up is 1
MTU 2500 bytes
LineSpeed Actual     : 30000 Mbit
Allowed Member Speed : 10000 Mbit
Priority Tag disable
IPv6 RA Guard disable
Last clearing of show interface counters: 04:25:45
Queueing strategy: fifo
Receive Statistics:
    10760999230 packets, 15928897747609 bytes
    Unicasts: 10760993249, Multicasts: 5968, Broadcasts: 13
    64-byte pkts: 244911, Over 64-byte pkts: 36086062, Over 127-byte pkts: 33063667
    Over 255-byte pkts: 12757063, Over 511-byte pkts: 145739540, Over 1023-byte pkts: 7774019178
    Over 1518-byte pkts(Jumbo): 2759088809
    Runts: 0, Jabbers: 0, CRC: 0, Overruns: 0
    Errors: 0, Discards: 46599
Transmit Statistics:
    4737892630 packets, 391893699181 bytes
    Unicasts: 4737802225, Multicasts: 55403, Broadcasts: 35002
    Underruns: 0
    Errors: 0, Discards: 16
Rate info:
    Input 7850.593532 Mbits/sec, 661841 packets/sec, 26.17% of line-rate
    Output 179.163084 Mbits/sec, 292339 packets/sec, 0.60% of line-rate
Time since last interface status change: 1d10h51m

QOS Receive length:

 

 

show qos rcv-queue interface port-channel 79 
Port-channel member interfaces:
Interface TenGigabitEthernet 10/0/43
  In-use 252 bytes, Total buffer 203616 bytes
  376 packets dropped
          In-use        Max
  TC       Bytes      Bytes
  -------------------------
    0          0        252
    1          0        252
    2          0        252
    3          0     184023
    4          0        252
    5          0        252
    6          0        252
    7          0      18081
Port-channel member interfaces:
Interface TenGigabitEthernet 11/0/43
  In-use 0 bytes, Total buffer 203616 bytes
  25319 packets dropped
          In-use        Max
  TC       Bytes      Bytes
  -------------------------
    0          0        252
    1          0        252
    2          0        252
    3          0     184023
    4          0        252
    5          0        252
    6          0        252
    7          0      18081
Port-channel member interfaces:
Interface TenGigabitEthernet 11/0/48
  In-use 0 bytes, Total buffer 203616 bytes
  22402 packets dropped
          In-use        Max
  TC       Bytes      Bytes
  -------------------------
    0          0        252
    1          0        252
    2          0        252
    3        252     184023
    4          0        252
    5          0        252
    6          0        252
    7          0      18081

 

VDX-configuration

 

core-10# show running-config interface Port-channel 79
interface Port-channel 79
 vlag ignore-split
 load-balance src-dst-ip
 switchport
 switchport mode trunk
 switchport trunk allowed vlan all
 switchport trunk tag native-vlan
 spanning-tree shutdown
 no shutdown

 show running-config interface TenGigabitEthernet 10/0/43
interface TenGigabitEthernet 10/0/43
 no fabric isl enable
 no fabric trunk enable
 channel-group 79 mode active type standard
 lacp timeout long
 no shutdown

 

 

Juniper

 

Physical interface: ae0, Enabled, Physical link is Up
  Interface index: 128, SNMP ifIndex: 501, Generation: 131
  Link-level type: Ethernet, MTU: 1514, Speed: 30Gbps, BPDU Error: None, MAC-REWRITE Error: None,
  Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Minimum links needed: 1,
  Minimum bandwidth needed: 0
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x0
  Last flapped   : 2016-05-01 00:50:32 UTC (1d 10:55 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :      328138528266306            203572408 bps
   Output bytes  :    11694463197392924           8632712448 bps
   Input  packets:        3620659481646               316914 pps
   Output packets:        7925403568290               726163 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Giants: 0, Policed discards: 0,
    Resource errors: 0
  Output errors:
    Carrier transitions: 7, Errors: 0, Drops: 0, MTU errors: 0, Resource errors: 0

  Logical interface ae0.0 (Index 148) (SNMP ifIndex 829) (Generation 300)
    Flags: SNMP-Traps 0x0 Encapsulation: ENET2
    Statistics        Packets        pps         Bytes          bps
    Bundle:
        Input :       2491631          0     232620380            0
        Output:       2129191          0     326406472            0
    Link:
      xe-0/1/0.0
      xe-4/1/0.0
      xe-0/1/2.0
    LACP info:        Role     System             System      Port    Port  Port 
                             priority          identifier  priority  number   key 
      xe-0/1/0.0     Actor        127         127      10     1
      xe-0/1/0.0   Partner      32768      32768    5163    79
      xe-4/1/0.0     Actor        127        127      11     1
      xe-4/1/0.0   Partner      32768       32768    5675    79
      xe-0/1/2.0     Actor        127  2      127       2     1
      xe-0/1/2.0   Partner      32768  0    32768    5680    79
    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx 
      xe-0/1/0.0            125583        4203            0            0
      xe-4/1/0.0            125585        4203            0            0
      xe-0/1/2.0             10626         360            0            0
    Marker Statistics:   Marker Rx     Resp Tx   Unknown Rx   Illegal Rx
      xe-0/1/0.0                 0           0            0            0
      xe-4/1/0.0                 0           0            0            0
      xe-0/1/2.0                 0           0            0            0
    Protocol eth-switch, Generation: 339, Route table: 0
      Flags: Trunk-Mode
Juniper-configuration:
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 unit 0 family ethernet-switching port-mode trunk
set interfaces ae0 unit 0 family ethernet-switching vlan members VDX
set interfaces xe-0/1/0 ether-options 802.3ad ae0
set interfaces xe-4/1/0 ether-options 802.3ad ae0
set interfaces xe-0/1/2 ether-options 802.3ad ae0

 

 

Regards
Jörg Kost
Visitor
Posts: 1
Registered: ‎10-27-2008

Re: Ethernet discards between VDX6720 and Juniper EX

I'm seeing sort of the same thing have you gotten any answers on this?

Contributor
Posts: 66
Registered: ‎02-25-2016

Re: Ethernet discards between VDX6720 and Juniper EX

Hello,

 

I got my issues resolved by adding more bandwidth to my top switches. These top switches were connected with vLAGs and the traffic (lots of unicasts IP traffic, each flow is balanced in l2 and l3) from the EX was passing through the VDX to the top switches. The corresponding LAGs were also distributed over several VDXs switches. 

 

Now what happens, that the VDX code seems to conserve as much bandwidth as possible on the fabric ISL ports between the VDX switches and may underutilise it. When one of the top switches had a local LAG member on the same VDX that is receiving the packet from the EX, the VDX will  try to use this local connection and may avoid sending packets to a remote LAG member on a different fabric member. By adding more local vLAG ports, the discards went to zero. 

 

This is "what I saw" and my interpretation of load-sharing. Maybe some Brocade employee can clarify this, too.  Juniper has a similar feature and their documentation says: local-bias (You should enable local link bias if you want to conserve VCP bandwidth by always forwarding egress unicast traffic on a LAG bundle out of a local link.)

 

I did not find a similar expression in the NOS configuration guide. Maybe a similar thing can be archived by adding different load-balance statements  per rbridge member. 

 

i hope i could help you further.

Regards
Jörg Kost

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.

vADC is now Pulse Secure
Download FREE NVMe eBook