Brocade XMR not load sharing

[isaacm]

I have a Brocade XMR (System Mode: XMR Chassis: MLXe 16-slot) on an edge network to various ISPs, and I've recently added a second 10G port to our peering. With the addition of the second port and the second static route on the router, I'd expected it to load share to the next hop automatically between the 2. Unfortunately, this did not happen. When both static routes are on the XMR, all traffic going to the next hop address prefers just one of the interfaces. The 2 static routes pointing to this one next-hop address are exactly the same in regards to administrative distance, etc. and are both in the IP routing table. I've also played with the "load-balance hash-diversify" command, but with no luck

Below is the code that I've running, the entries in the routing table to the next-hop IP, and the output rate for each of the 2 interfaces in question.

! show version

SL M1: NI-XMR-MR Management Module Active (Serial #: S12536G00H, Part #:    35524-601A):
Boot     : Version 5.2.0T165 Copyright (c) 1996-2009 Brocade Communications Systems, Inc.
Compiled on Jun  7 2011 at 17:06:32 labeled as xmprm05200
(523645 bytes) from boot flash
Monitor  : Version 5.2.0T165 Copyright (c) 1996-2009 Brocade Communications Systems, Inc.
Compiled on Jun  7 2011 at 17:05:54 labeled as xmb05200
(523655 bytes) from code flash
IronWare : Version 5.2.0bT163 Copyright (c) 1996-2009 Brocade Communications Systems, Inc.
Compiled on Sep 27 2011 at 16:56:32 labeled as xmr05200b         
(7645102 bytes) from Primary                                    
Board ID : 00 MBRIDGE Revision : 32                              
916 MHz Power PC processor 7447A (version 8003/0101) 166 MHz bus 
512 KB Boot Flash (MX29LV040C), 32 MB Code Flash (MT28F128J3)    
2048 MB DRAM INSTALLED                                           
2048 MB DRAM ADDRESSABLE                                         
Active Management uptime is 37 days 23 hours 57 minutes 21 seconds
==========================================================================
SL M2: NI-XMR-MR Management Module Standby (Serial #: S12535G004, Part #:    35524-601A):
Boot     : Version 5.2.0T165 Copyright (c) 1996-2009 Brocade Communications Systems, Inc.
Compiled on Jun  7 2011 at 17:06:32 labeled as xmprm05200        
(523645 bytes) from boot flash                                  
Monitor  : Version 5.2.0T165 Copyright (c) 1996-2009 Brocade Communications Systems, Inc.
Compiled on Jun  7 2011 at 17:05:54 labeled as xmb05200          
(523655 bytes) from code flash                                  
IronWare : Version 5.2.0bT163 Copyright (c) 1996-2009 Brocade Communications Systems, Inc.
Compiled on Sep 27 2011 at 16:56:32 labeled as xmr05200b         
(7645102 bytes) from Primary                                    
Board ID : 00 MBRIDGE Revision : 32                              
916 MHz Power PC processor 7447A (version 8003/0101) 166 MHz bus 
512 KB Boot Flash (MX29LV040C), 32 MB Code Flash (MT28F128J3)    
2048 MB DRAM INSTALLED                                           
2048 MB DRAM ADDRESSABLE                                         
Standby Management uptime is 37 days 23 hours 56 minutes 29 seconds
==========================================================================
SL 1: BR-MLX-10Gx8-X 8-port 10GbE (X) Module (Serial #: BQQ0429G01D, Part #: 60-1002031-08)
Boot     : Version 5.2.0T175 Copyright (c) 1996-2009 Brocade Communications Systems, Inc.
Compiled on Jun  7 2011 at 17:06:54 labeled as xmlprm05200       
(514997 bytes) from boot flash                                  
Monitor  : Version 5.2.0T175 Copyright (c) 1996-2009 Brocade Communications Systems, Inc.
Compiled on Jun  7 2011 at 17:07:18 labeled as xmlb05200         
(515819 bytes) from code flash                                  
IronWare : Version 5.2.0bT177 Copyright (c) 1996-2009 Brocade Communications Systems, Inc.
Compiled on Sep 27 2011 at 17:02:42 labeled as xmlp05200b        
(6519265 bytes) from Primary                                    
FPGA versions:                                                   
WARN: Invalid PBIF Version = 1.04, Build Time = 2/11/2011 10:0:00
                                                                 
Valid XPP Version = 3.08, Build Time = 2/7/2011 10:1:00          
                                                                 
X10G2MAC 0                                                       
X10G2MAC 1                                                       
X10G2MAC 2                                                       
X10G2MAC 3                                                       
1333 MHz MPC 8541 (version 8021/0022) 533 MHz bus                
512 KB Boot Flash (MX29LV040C), 64 MB Code Flash (MT28F256J3)    
2048 MB DRAM, 8 KB SRAM, 0 Bytes BRAM                            
PPCR0: 0K entries CAM, 0K PRAM, 0K AGE RAM                       
PPCR1: 0K entries CAM, 0K PRAM, 0K AGE RAM                       
LP Slot 1 uptime is 36 days 20 hours 41 minutes 22 seconds       
==========================================================================
SL 3: BR-MLX-10Gx8-X 8-port 10GbE (X) Module (Serial #: BQQ0429G01H, Part #: 60-1002031-08)
Boot     : Version 5.2.0T175 Copyright (c) 1996-2009 Brocade Communications Systems, Inc.
Compiled on Jun  7 2011 at 17:06:54 labeled as xmlprm05200       
(514997 bytes) from boot flash                                  
Monitor  : Version 5.2.0T175 Copyright (c) 1996-2009 Brocade Communications Systems, Inc.
Compiled on Jun  7 2011 at 17:07:18 labeled as xmlb05200         
(515819 bytes) from code flash                                  
IronWare : Version 5.2.0bT177 Copyright (c) 1996-2009 Brocade Communications Systems, Inc.
Compiled on Sep 27 2011 at 17:02:42 labeled as xmlp05200b        
(6519265 bytes) from Primary                                    
FPGA versions:                                                   
WARN: Invalid PBIF Version = 1.04, Build Time = 2/11/2011 10:0:00
                                                                 
Valid XPP Version = 3.08, Build Time = 2/7/2011 10:1:00          
                                                                 
X10G2MAC 0                                                       
X10G2MAC 1                                                       
X10G2MAC 2                                                       
X10G2MAC 3                                                       
1333 MHz MPC 8541 (version 8021/0022) 533 MHz bus                
512 KB Boot Flash (MX29LV040C), 64 MB Code Flash (MT28F256J3)    
2048 MB DRAM, 8 KB SRAM, 0 Bytes BRAM                            
PPCR0: 0K entries CAM, 0K PRAM, 0K AGE RAM                       
PPCR1: 0K entries CAM, 0K PRAM, 0K AGE RAM                       
LP Slot 3 uptime is 37 days 23 hours 57 minutes 1 seconds        
==========================================================================
All show version done

! show ip route 69.16.191.8

Type Codes - B:BGP D:Connected I:ISIS O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
ISIS Codes - L1:Level-1 L2:Level-2
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
        Destination        Gateway         Port          Cost          Type Uptime
1       69.16.191.8/32     69.16.190.249   eth 1/2       1/1           S    11h13m
        69.16.191.8/32     209.197.5.153   eth 3/4       1/1           S    11h13m

! sh int eth 1/2 | i rate
  30 second input rate: 1596511589 bits/sec, 324767 packets/sec, 16.48% utilization
  30 second output rate: 4897237430 bits/sec, 545201 packets/sec, 49.84% utilization

! sh int eth 3/4 | i rate
  30 second input rate: 1425200177 bits/sec, 314311 packets/sec, 14.75% utilization
  30 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization

Is there something I'm missing to have the router load share between these 2 ports? Any help is greatly appreciated.

Isaac

[isaacm]

I've been able to find the answer to my own question. After working with Brocade's TAC and our SE, we found that in order to do any kind of load sharing between multiple ports, you actually have to include the "maximum-paths" command in your BGP configuration. The reason we found was that adding multiple static routes to a next-hop address wasn't going to work, since the final destinations were beyond just that next hop. Being that this is various Internet traffic, the Brocade router is going to use the BGP table, and not the IP routing table. And because BGP chooses just 1 "best path" by default, that is all that is going to be used. Once we added the maximum-path command and increased it to "2", we began to see traffic go out of both physical interfaces.

I hope this helps anyone else that runs into this problem, since this is the first time myself or any of my colleagues have ever seen this.

Good luck

Isaac

[SKT]

typical LB methods i have seen is MAC based and IP based. By making "maximum paths=2" it sounds like the old  "round robin" method.

just my view though