Ethernet Switches & Routers

Reply
Occasional Contributor
Posts: 6
Registered: ‎02-05-2013

MLX Multicast and Broadcast CPU protection

I wanted to protect our MLX's  from multicast and broadcast storms that could be generated from customers.

I found that there is no storm-control command build in the software. What I attempted to do was the following:

access-list 400 permit ffff.ffff.ffff ffff.ffff.ffff any any etype any

access-list 401 permit any 0100.5e00.0000 ffff.ff00.0000 any etype any

interface ethernet 2/20

rate-limit output access-group 400 8144 1250

rate-limit output access-group 401 8144 1250

rate-limit input access-group 400 8144 1250

rate-limit input access-group 401 8144 1250

Then I build a loop between the MLX and several other switches. When there is a loop, I can see the lp-cpu utilization go up to 95%, I see NP ingress dropped packets. I also see that the only counters rising are the counters on the output rate-limiter:

telnet@MLX16-TEST#             show rate-limit counters interface 2/20

interface e 2/20

rate-limit strict-acl

rate-limit input access-group 400 8144 1250

  Fwd:        0                       Drop:  0 bytes

  Re-mark:    0                       Total: 0 bytes

rate-limit input access-group 401 8144 1250

  Fwd:        0                       Drop:  0 bytes

  Re-mark:    0                       Total: 0 bytes

rate-limit output access-group 400 8144 1250

  Fwd:        0                       Drop:  0 bytes

  Re-mark:    0                       Total: 0 bytes

rate-limit output access-group 401 8144 1250

  Fwd:        905692                  Drop:  82969609072 bytes

  Re-mark:    0                       Total: 82970514764 bytes

I also tried vlan-cpu protection, but this did not lower the cpu usage of the line-card either.

show cpu-utilization lp

SLOT  #:          LP CPU UTILIZATION in  %:

               in 1 second:  in 5 seconds:  in 60 seconds: in 300 seconds:

     2:        96            96             10              2

     5:        1             1              1               1

show int eth 2/20

GigabitEthernet2/20 is up, line protocol is up

  STP Root Guard is disabled, STP BPDU Guard is disabled

  Hardware is GigabitEthernet, address is 001b.ed25.6943 (bia 001b.ed25.6943)

  Configured speed auto, actual 1Gbit, configured duplex fdx, actual fdx

  Member of VLAN 1 (untagged), 1 L2 VLANS (tagged), port is in dual mode (default vlan), port state is Forwarding

  STP configured to ON, Priority is level0, flow control enabled

  Priority force disabled, Drop precedence level 0, Drop precedence force disabled

  dhcp-snooping-trust configured to OFF

  mirror disabled, monitor disabled

  LACP BPDU ForwardingSmiley Very Happyisabled

  Not member of any active trunks

  Not member of any configured trunks

  Port name is EX4200-HA-TEST-01_ge-0/0/0

  MTU 9216 bytes, encapsulation ethernet

  Cluster L2 protocol forwarding enabled

  300 second input rate: 47296450 bits/sec, 86958 packets/sec, 6.12% utilization

  300 second output rate: 1025 bits/sec, 1 packets/sec, 0.00% utilization

  939945640 packets input, 60280672838 bytes, 0 no buffer

  Received 909319886 broadcasts, 30625753 multicasts, 1 unicasts

  0 input errors, 0 CRC, 0 frame, 0 ignored

  0 runts, 0 giants

  NP received 940352943 packets, Sent to TM 30740227 packets

  NP Ingress dropped 909612992 packets

  1602 packets output, 110860 bytes, 0 underruns

  Transmitted 0 broadcasts, 916 multicasts, 686 unicasts

  0 output errors, 0 collisions

  NP transmitted 1607 packets, Received from TM 14050030 packets

The line card connected to the loop is the following:

SL 5: NI-MLX-1Gx48-T-A 48-port 10/100/1000Base-T MRJ21 Module (Serial #: K9SA41E0FR, Part #: 60-1001405-05)

Boot     : Version 5.1.0aT175 Copyright (c) 1996-2009 Brocade Communications Systems, Inc.

Compiled on Jan  8 2011 at 07:35:52 labeled as xmlprm05100a

The sw version I am running is 5.1.0aT165

I was wondering what other Brocade users are doing to mitigate multicast and broadcast storms? Or, perhaps, I am forgetting to turn on a feature or missing some configuration command.

Occasional Contributor
Posts: 6
Registered: ‎02-05-2013

Re: MLX Multicast and Broadcast CPU protection

Interfaces on 80+% utilization:

telnet@MLX16-TEST#show int eth 2/18

GigabitEthernet2/18 is up, line protocol is up

  STP Root Guard is disabled, STP BPDU Guard is disabled

  Hardware is GigabitEthernet, address is 001b.ed25.6941 (bia 001b.ed25.6941)

  Configured speed auto, actual 1Gbit, configured duplex fdx, actual fdx

  Member of VLAN 1 (untagged), 1 L2 VLANS (tagged), port is in dual mode (default vlan), port state is Forwarding

  STP configured to ON, Priority is level0, flow control enabled

  Priority force disabled, Drop precedence level 0, Drop precedence force disabled

  dhcp-snooping-trust configured to OFF

  mirror disabled, monitor disabled

  LACP BPDU ForwardingSmiley Very Happyisabled

  Not member of any active trunks

  Not member of any configured trunks

  Port name is EX4200-HA-TEST-03_ge-0/0/0

  MTU 9216 bytes, encapsulation ethernet

  Cluster L2 protocol forwarding enabled

  300 second input rate: 645661306 bits/sec, 1186873 packets/sec, 83.55% utilization

  300 second output rate: 6649 bits/sec, 12 packets/sec, 0.00% utilization

  1336114291 packets input, 87220217516 bytes, 0 no buffer

  Received 909319891 broadcasts, 426794310 multicasts, 90 unicasts

  0 input errors, 0 CRC, 0 frame, 0 ignored

  0 runts, 0 giants

  NP received 1336417144 packets, Sent to TM 1305238351 packets

  NP Ingress dropped 31179045 packets

  34426 packets output, 2830232 bytes, 0 underruns

  Transmitted 0 broadcasts, 33740 multicasts, 686 unicasts

  0 output errors, 0 collisions

  NP transmitted 34428 packets, Received from TM 29834514 packets

telnet@MLX16-TEST#show int eth 2/19

GigabitEthernet2/19 is up, line protocol is up

  STP Root Guard is disabled, STP BPDU Guard is disabled

  Hardware is GigabitEthernet, address is 001b.ed25.6942 (bia 001b.ed25.6942)

  Configured speed auto, actual 1Gbit, configured duplex fdx, actual fdx

  Member of VLAN 1 (untagged), 1 L2 VLANS (tagged), port is in dual mode (default vlan), port state is Forwarding

  STP configured to ON, Priority is level0, flow control enabled

  Priority force disabled, Drop precedence level 0, Drop precedence force disabled

  dhcp-snooping-trust configured to OFF

  mirror disabled, monitor disabled

  LACP BPDU ForwardingSmiley Very Happyisabled

  Not member of any active trunks

  Not member of any configured trunks

  Port name is EX4200-HA-TEST-02_ge-0/0/0

  MTU 9216 bytes, encapsulation ethernet

  Cluster L2 protocol forwarding enabled

  300 second input rate: 637914860 bits/sec, 1172634 packets/sec, 82.55% utilization

  300 second output rate: 74 bits/sec, 0 packets/sec, 0.00% utilization

  1333930595 packets input, 87071729059 bytes, 0 no buffer

  Received 909319890 broadcasts, 424610015 multicasts, 690 unicasts

  0 input errors, 0 CRC, 0 frame, 0 ignored

  0 runts, 0 giants

  NP received 1334142961 packets, Sent to TM 1440228 packets

  NP Ingress dropped 1332703381 packets

  28643 packets output, 2430558 bytes, 0 underruns

  Transmitted 1 broadcasts, 28642 multicasts, 0 unicasts

  0 output errors, 0 collisions

  NP transmitted 28644 packets, Received from TM 56589 packets

telnet@MLX16-TEST#show int eth 2/20

GigabitEthernet2/20 is up, line protocol is up

  STP Root Guard is disabled, STP BPDU Guard is disabled

  Hardware is GigabitEthernet, address is 001b.ed25.6943 (bia 001b.ed25.6943)

  Configured speed auto, actual 1Gbit, configured duplex fdx, actual fdx

  Member of VLAN 1 (untagged), 1 L2 VLANS (tagged), port is in dual mode (default vlan), port state is Forwarding

  STP configured to ON, Priority is level0, flow control enabled

  Priority force disabled, Drop precedence level 0, Drop precedence force disabled

  dhcp-snooping-trust configured to OFF

  mirror disabled, monitor disabled

  LACP BPDU ForwardingSmiley Very Happyisabled

  Not member of any active trunks

  Not member of any configured trunks

  Port name is EX4200-HA-TEST-01_ge-0/0/0

  MTU 9216 bytes, encapsulation ethernet

  Cluster L2 protocol forwarding enabled

  300 second input rate: 637493527 bits/sec, 1171859 packets/sec, 82.49% utilization

  300 second output rate: 6292 bits/sec, 11 packets/sec, 0.00% utilization

  1336812676 packets input, 87267710286 bytes, 0 no buffer

  Received 909319886 broadcasts, 427492789 multicasts, 1 unicasts

  0 input errors, 0 CRC, 0 frame, 0 ignored

  0 runts, 0 giants

  NP received 1337053086 packets, Sent to TM 427421137 packets

  NP Ingress dropped 909632143 packets

  5517 packets output, 377080 bytes, 0 underruns

  Transmitted 0 broadcasts, 4831 multicasts, 686 unicasts

  0 output errors, 0 collisions

  NP transmitted 5520 packets, Received from TM 24093519 packets

Frequent Contributor
Posts: 160
Registered: ‎08-07-2009

Re: MLX Multicast and Broadcast CPU protection

The attached Best Practice for Denial of Service attack Prevention doc may provide some helpful detail.

Also, the "Protecting against Denial of Service Attacks" section of the NetIron Config Guide is an additional

resource. See link below:

http://www.brocade.com/downloads/documents/html_product_manuals/NI_05400a_CFG/wwhelp/wwhimpl/js/html/wwhelp.htm#href=DoS_Protection.71.1.html

Occasional Contributor
Posts: 6
Registered: ‎02-05-2013

Re: MLX Multicast and Broadcast CPU protection

The question is to protect against multicast and broadcast traffic. The configuration posted is not doing that on our MLX16, the input rate-limiter are not working with the layer 2 access-lists. When I build a loop, the entire platform suffers.

Also tried alternatvie configuration:

policy-map RATE-LIMIT-BUM-MIN

  cir 8144 cbs 1250

!

interface ethernet 11/10

rate-limit output access-group 400 policy-map RATE-LIMIT-BUM-MIN

rate-limit output access-group 401 policy-map RATE-LIMIT-BUM-MIN

rate-limit input access-group 400 policy-map RATE-LIMIT-BUM-MIN

rate-limit input access-group 401 policy-map RATE-LIMIT-BUM-MIN

!

access-list 400 permit ffff.ffff.ffff ffff.ffff.ffff any any etype any

access-list 401 permit any 0100.5e00.0000 ffff.ff00.0000 any etype any

!

This did not produce any result either.

I allready made a TAC case for this. So far their conclusion is that the config is working in their lab using an MLX8, but they could not (after various web sessions) determine why it was not functioning on our platform.

They did indicate that part of the problem was that the CPU was being flooded with arp-requests during my storm. That part is tackled using 'ip rate-limit arp policy-map Limitarp'

A solution could be ' transparent-hw-flooding' which disables source address learning on the vlan and will forward the traffic in hardware. Als vlan-cpu protection could be helpfull. However, in our environment this will not be helpfull as we want to deliver customers a vlan with a ve interface in it.

Still waiting on them to resolve the not being able to rate-limit or police the number of multicast/broadcast traffic.

Contributor
Posts: 28
Registered: ‎07-25-2013

Re: MLX Multicast and Broadcast CPU protection

Adding 'broadcast limit #' and 'multicast limit #' helps a little.

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.