Campus Networks

Campus Network Infrastructure, Best Practices-HyperEdge Architecture Design with Mixed Stacks

by nshaw on ‎03-25-2013 11:30 AM - edited on ‎08-06-2014 08:43 AM by (2,873 Views)

Contents

Preface

Networks are becoming more intelligent, utilizing smarter devices and faster links. Brocade’s HyperEdge™ Architecture addresses growing network needs simplifying management, reducing complexity and lowering cost. An innovative feature of the HyperEdge Archi tecture is the mixed stack option available with the Brocade ICX Series of switches.

 

Overview

Networking needs are evolving at an increasingly more dynamic pace. As next generation data centers move towards virtualizing more data, and wireless requirements stretch the wired infrastructure past their limits with the progression to 802.11n and the pending approval of 802.11ac, the campus network must evolve to keep up with today’s challenges to support user mobility and application agility. Previously, a growing network meant significantly increased capital and operational expenses. The HyperEdge Architecture provides a network that will grow with you without overburdening your IT staff or your budget. It supports this vision by integrating innovative new features with existing network technologies, improving support for mobility, security and rapid application deployment, while simplifying network management and deployment.

 

BestPractice_MixedStack-HyperEdgeWireWirelessFeatures.jpg

  HyperEdge Architecture Features for Wired and Wireless Campus Networks (click to enlarge)

 

An innovative feature of the HyperEdge Architecture is mixed stacking available with the Brocade ICX Switch family. A mixed stack eliminates the requirement that all switches in the stack have the same features and functionality. There are two ways to add mixed stacks to the network. The first is to add non-premium switches, such as the Brocade ICX 6450 Switch, to a stack of premium switches, such as the Brocade ICX 6610 switch when adding more ports to the stack. The second is to add premium ICX 6610 switches to a stack of non-premium ICX 6450 switches to enable advanced layer 3 features for the stack. Both approaches reduce the initial cash outlay and reduce the overall TCO.

 

The HyperEdge Architecture moves away from the traditional core/distribution/access layer topology toward a collapsed core/edge topology where the edge collapses the distribution/access layers into a single stack providing a cost-effective alternative to the more complicated traditional topology.

 

MixedStack_CompareCoreDistAcctoCoreEdge.jpg

 

  Comparison of Core/Distribution/Access and Core/Edge with HyperEdge Architecture (click to enlarge)

 

While there are reasons for a core/distribution/access topology, a one size network does not meet all requirements. Mixed stacking simplifies operation and configuration, and improves performance by eliminating bottlenecks in stack links while reducing cost.

 

Purpose of This Document

This document covers how to design campus networks using HyperEdge mixed stacks. A base design and alternates are provided with key CLI commands.

 

Audience

Network architects and designers who want a better understanding of how to design and apply mixed stacking to wired and wireless campus networks.

 

Objectives

One of the key benefits of the HyperEdge Architecture is the simplification of the network. The key objective of this paper is to:

  • Show the benefits of the mixed stacks.
  • Provide design templates to simplify management and scaling of the network.
  • Provide an overview of how to configure a mixed stack.

Related Documents

The following documents are valuable resources for the designer. In addition, any Brocade release notes that have been published for the FastIron, NetIron and Mobility operating systems should be reviewed.

References

 

About Brocade

Brocade® (NASDAQ: BRCD) networking solutions help the world’s leading organizations transition smoothly to a world where applications and information reside anywhere. This vision is designed to deliver key business benefits such as unmatched simplicity, non-stop networking, application optimization, and investment protection.Innovative Ethernet and storage networking solutions for data center, campus, and service provider networks help reduce complexity and cost while enabling virtualization and cloud computing to increase business agility.To help ensure a complete solution, Brocade partners with world-class IT companies and provides comprehensive education, support, and professional services offerings. (www.brocade.com)

 

Key Contributors

The content in this guide was developed by the following key contributors

.Lead Architect: Nadia Shaw, Technical Marketing Engineer

 

Document History

Date                  Version        Description

2013-04-08        1.0                  Initial Release

 

HyperEdge Reference Architecture

 

The campus reference architecture supports wired and wireless devices using both the traditional core/distribution/access topology and the core/edge topology with innovative mixed stacking as shown below.  

BestPractice_MixedStack-RefArchitecture.jpg

   Campus Network Reference Architecture (click to enlarge)

Mixed stacking is a flexible feature with several stack topology options, as shown below.MixedStack_AlternateStackTopologies.jpg  

   Alternate Mixed Stack Topologies (click to enlarge)

References

 

Design

A mixed stack includes one or two ICX 6610 switches. If two switches are used, they are the active and standby units for the mixed stack and also act as the Layer 2 / Layer 3 boundary within the stack. One or more ICX 6450 switches connect to the ICX 6610 switches with 10 GbE ports. A group of ICX 6450 switches is called a sub-stack. The ICX 6610 switches use 40 GbE QSFP ports to form a stack. There are a total of four QSFP ports; two are 40 GbE, and two are 4x10 GbE. All 40 GbE ports use the same cables. See the ICX Series and FastIron Configuration Guide in the References section for more information.

A single ICX 6610 can be used in a mixed stack, but it is not recommended since there is no switch redundancy in case of a switch failure. Two ICX 6610 switches provide resiliency so either can be off-line and device traffic still flows through the stack. 

 

MixedStack_Pair6610w4x40GE.jpg

   A pair of ICX 6610 Switches With four, 40 GbE Stack Links (click to enlarge)

 

Multiple ICX 6450 switches use 10 GbE stacking ports to connect to the ICX 6610 switches to form a sub-stack. The maximum number of switches in a mixed stack is eight so a sub-stack can contain up to six switches if there are two ICX 6610 switches. The sub-stack is connected in either a linear or ring topology to the core units. For better resiliency, the sub-stack should connect to both ICX 6610 switches.

 

MixedStack_6450LinearStack.jpg

   Four ICX 6450 Switches in a Linear Sub-stack Using 10 GbE Ports (click to enlarge)

 

MixedStack_6450RingW6610.jpg 

   Six ICX 6450 Switches in a Ring with 10 GbE ports (click to enlarge)

 

Within a mixed stack, up to two core units and six peripheral units are supported. When a stack trunk is formed the stacking bandwidth to the ICX 6450 switches is 20 GbE in each direction – stacking ports are no longer wasted as uplink ports.

 

By combining the ICX 6610 and the ICX6450 in a mixed stack, you can collapse the aggregation/distribution layer into an Edge layer providing simple management, active-active links, and the sharing of advanced features with all switches in the mixed stack.

 

Mixed stacking supports different stacking link topologies, but only the supported topologies should be used. Additionally, not all supported topologies are validated topologies. In the event that you configure a stack that is supported, but not validated, the mixed stack may continue to work without fault, but the master unit will print a warning after election.

 

References

 

Best Practices

The following are best practices for designing mixed stacks with the ICX Series of switches using the FastIron operating system.

  • Both ring and linear (Bus) stacking topologies are supported; nonetheless, a ring topology is preferred, since a ring topology adds redundancy in case of a link failure. This diagram below shows both a ring (left side) and a linear topology (right side). The ring allows for greater redundancy, which is vital for network uptime. A linear topology can be deployed where temporary orphan sub-stacks can be tolerated. Link sub-stacks to both the active and standby controllers.

 

        MixedStack_RingTopologyLinearTopology.jpg 

Left, Ring Topology; Right, Linear Topology (click to enlarge)

 

  • Limit the number of VLANs in the peripheral units to reduce broadcast and multicast packet flooding.
  • Limit sub-stack unit count due to limited 10GbE bandwidth.
  • Currently each mixed stack supports a total of up to two ICX 6610s and up to six ICX 6450s, for a total of eight devices.
  • Only the ICX 6610 switches can be the active and standby controllers. In the event that both ICX 6610 switches fail, the ICX 6450 sub-stack may continue to function in an orphan state before information times out. In the case of a single ICX 6610, there can only be one active controller and no standby. During ICX 6610 failure scenarios, the ICX6450s will function in an orphan state before time out. This second topology is not recommended due to the potential impact on a network that requires high-resiliency.
  • Traffic between the ICX 6610 switches will not flow to the sub-stack switches unless there is a stacking link failure between the ICX 6610 switches.
  • Broadcast and multicast packets will flood the ICX 6610 switches. They will only go through peripheral ports when necessary.
  • Each ICX 6610 can have up to two stacking links to sub-stacks.
  • FastIron 8.00.0 is the minimum requirement to support mixed stacking.

Licensing

The following license is required on the ICX6450 if there are multiple ICX6450s in a ring or linear formation (additional notes follow the SKU):

  • ICX6450-10G-LIC-POD
    • This license will enable ports 2 and 4 of the ICX 6450 to utilize the 10GbE capacity.
    • Since the peripheral port(s) connecting from the ICX 6610 can only be 10 GbE, the receiving port(s) on the ICX 6450 must be 10GbE as well. In the event that there is only one ICX6450 used in a sub-stack, the default 10GbE ports can be used and no additional license is required once two of the ICX 6450 ports are 10GbE by default. To create a stack with ICX6450s connected to each other, additional 10GbE licenses are required.
    • Each 10GbE port on the ICX6450 requires a separate license if you plan on utilizing the full 10GbE stacking/uplink bandwidth.
  • No additional licenses are required to support mixed stacking, although it makes good sense to run a premium or advanced license to maximize the added benefits of mixed stacking, such as Distributed Services. Note that although you do not need premium or advanced licenses, ALL units (core and peripheral) must be running the router image for mixed stacking to work. The following licenses are recommended:
    • ICX6610-ADV-LIC-SW or ICX6610-PREM-LIC-SW
      • All units in the ICX6610 core require a separate license if you plan on using premium or advanced features. Details on licensing can be found in the ICX and FastIron Configuration Guide. All core units must have the same image license if you plan on running premium or advanced features. The ICX 6450 does not require additional software licensing.
      • ICX6610-LIC-POD
        • Each POD license supplies 4 ports of 10GbE for the SFP+ links on face of the ICX 6610. If you are running 10GbE on the SFP+ ports on the ICX6610, you will need the corresponding licenses for the 10GbE ports on the ICX6450 as well.

Topology

Different stacking topologies are possible for mixed stacking, but not all make sense nor are recommended. Two topologies are shown, Best Practice Design and Alternate Design. The preferred topology is the Best Practice Design.

 

Best Practice Design

 

MixedStack_RecommendedStackingMixedStacks .jpg  

  Recommended Stacking Topology for Mixed Stacks (click to enlarge)

 

This is the topology for mixed stacking. It allows for the maximum utilization of links when supporting redundant stacking of the ICX 6610 switches, and full bandwidth for stacking links to the ICX 6450 sub-stacks.

It has two ICX 6610 switches for redundancy providing resilience and automatic failover. Between the two ICX 6610s, one to four QSFP stacking cables are used to form a stack between the ICX 6610 switches. It is recommended that at least two cables are used in case of cable or port failure. See the ICX Series and FastIron Configuration Guide in References section for more information.

 

There are two separate sub- stacks, each connected to the two ICX 6610 switches forming two rings. Each sub-stack should ideally connect to each ICX 6610 for redundancy.

 

Components

ICX 6610

All SKUs of the ICX 6610 support mixed stacking

ICX 6450

All SKUs of the ICX 6450 support mixed stacking

References

 

Alternate Design

Due to cost constraints, some deployments may not initially support full redundancy of the ICX 6610 switches in the mixed stack. This alternate design can be used but the following behavior will result.

  • During failover scenarios, without an additional ICX 6610, the ICX 6450 switches will become orphan units. This means they will function on their own in an isolated stack until information times out. This can greatly affect the network as all devices that connect to the sub-stack will have limited connectivity. The cost of this is downtime
  • During failover, the ICX 6450 switches in the sub-stack will have a linear topology. Additional breaks to the linear stack links may occur and create additional orphaned sub-stacks.

In the alternate design where only one ICX 6610 is used in the mixed stack, a ring topology should be employed as shown below. If any link between two ICX 6450 switches fails, another route to the ICX 6610 can be found. If any link fails, then the remaining ICX 6450 units can form one or two sub-stacks.

 

MixedStack_AlternateStackingMixedStacks .jpg  

   Alternate Design for Mixed Stack Topology (click to enlarge)

 

Upgrading/Downgrading Software Image in the Stack

 

Upgrading from pre-8.0 to 8.0 software/Downgrading from 8.0

 

If you update a switch with a pre-8.0 existing configuration, the configuration will be converted to work in 8.0 and located as the startup-config.txt. The old pre-8.0 configuration will be saved as startup_config.legacy and saved as a backup.In the event you wish to revert back to pre-8.0 software, the startup configuration will revert back to the startup_config.legacy file. If no startup_config.legacy file exists, then the 8.0 startup_config will be deleted completely. If you wish to downgrade, you will need to initiate the downgrade command as follows:

 

------

ICX6610-24P Router#downgrade_to <version>

  This operation will delete the current configuration. Are you sure? (enter 'y' or 'n'):

------

 

Upgrading from 8.0 to post-8.0 software

 

When utilizing the manifest upgrade commend , the command will first install images to all core ICX 6610 units in the stack. Next, it will install the ICX 6450 image to all peripheral units in the stack. Once all relevant images have been installed, the feature will notify the user that the upgrade is down and prompt the user to reload the system for the new images to take effect.

 

For additional information regarding software upgrade and downgrade, please refer to the ICX and FastIron Configuration Guide.

 

Verifying your Stack

When troubleshooting the HyperEdge Domain/stack, you should verify the following:

 

Do a “show stack” command to see where the stacking errors are occurring

  1. If show stackdemonstrates that the stacking capabilities are missing, verify:
    1. All cables are attached properly and all units are powered on.
    2. You have enabled stacking
  2. If show stackdemonstrates that the ICX 6450s are not a part of the stack, verify:
    1. Peripheral ports have been configured properly
    2. Stacking has been enabled properly on the peripheral units

Do a “show run

  1. Verify that the cabling matches the output of “show run
    1. If you manually configured all of the devices in the stack, verify that the devices are in the correct location in the stack.

 

SUMMARY

By designing an enterprise network with the HyperEdge Architecture in mind, you will discover how you can create a network that is increasingly effortless. Mixed stacking, in combination with Multi-Chassis Trunking (MCT) and Virtual Routing and Forwarding (VRF), can help ease your burdening network deployment needs. Utilizing innovations such as Distributed Services, Consolidated Management and Active-active links, your network will be prepared to handle the bandwidth hungry needs of our current and future technology eras, and at a cost-effective price point.

 

Appendix A: Base Design Mixed Stack Configuration

The following shows the configuration of a mixed stack of eight switches, two ICX 6610 and six ICX 6450, using the Base Design.

 

----------

ICX6610-24F Router#sh run

Current configuration:

ver 08.0.00q045T7f3

stack unit 1

  module 1 icx6610-24f-sf-port-management-module

  module 2 icx6610-qsfp-10-port-160g-module

  module 3 icx6610-8-port-10g-dual-mode-module

  priority 128

  stack-trunk 1/2/1 to 1/2/2

  stack-trunk 1/2/6 to 1/2/7

  stack-port 1/2/1 1/2/6

  peri-port 1/3/5

  peri-port 1/3/6

stack unit 2

  module 1 icx6610-24-port-management-module

  module 2 icx6610-qsfp-10-port-160g-module

  module 3 icx6610-8-port-10g-dual-mode-module

  stack-trunk 2/2/1 to 2/2/2

  stack-trunk 2/2/6 to 2/2/7

  stack-port 2/2/1 2/2/6

  peri-port 2/3/7

  peri-port 2/3/8

stack unit 3

  module 1 icx6450-48p-poe-port-management-module

  module 2 icx6450-sfp-plus-4port-40g-module

  stack-port 3/2/1 3/2/3

  connect 1/3/6

  connect 5/2/3

stack unit 4

  module 1 icx6450-24-port-management-module

  module 2 icx6450-sfp-plus-4port-40g-module

  stack-port 4/2/1 4/2/3

  connect 1/3/5

  connect 5/2/1

stack unit 5

  module 1 icx6450-48p-poe-port-management-module

  module 2 icx6450-sfp-plus-4port-40g-module

  stack-port 5/2/1 5/2/3

  connect 3/2/3

  connect 4/2/3

stack unit 6

  module 1 icx6450-48p-poe-port-management-module

  module 2 icx6450-sfp-plus-4port-40g-module

  stack-port 6/2/1 6/2/3

  connect 2/3/7

  connect 7/2/3

stack unit 7

  module 1 icx6450-48p-poe-port-management-module

  module 2 icx6450-sfp-plus-4port-40g-module

  stack-port 7/2/1 7/2/3

  connect 6/2/3

  connect 8/2/3

stack unit 8

  module 1 icx6450-48p-poe-port-management-module

  module 2 icx6450-sfp-plus-4port-40g-module

  no legacy-inline-power

  stack-port 8/2/1 8/2/3

  connect 2/3/8

  connect 7/2/1

stack enable

stack rconsole-off

stack mac 748e.f834.1ea0

vlan 1 name DEFAULT-VLAN by port

ip route 10.20.0.0/16 10.20.79.1

ip route 131.1.1.0/24 132.1.1.1

ip route 131.1.2.0/24 132.1.2.1

ip route 131.1.50.0/24 132.1.50.1

hitless-failover enable

interface management 1

no ip dhcp-client enable

ip address 10.20.79.35 255.255.255.128

end

----------

The following shows the stack configuration for a mixed stack of two ICX 6610 and six ICX 6450 switches.

----------

ICX6610-24F Router(config)#sh stack

alone: standalone, D: dynamic config, S: static config

ID   Type         Role Mac Address    Pri State   Comment                

1  S ICX6610-24F  active  748e.f834.1ea0 128 local   Ready

2  S ICX6610-24   standby 748e.f834.3318   0 remote Ready

3  S ICX6450-48P  member  748e.f882.eac0   0 remote Ready

4  D ICX6450-48P  member  748e.f8d2.8540   0 remote Ready

5  D ICX6450-48P  member  748e.f8d2.c200   0 remote Ready

6  D ICX6450-48P  member 748e.f8d2.d5c0   0 remote  Ready

7  D ICX6450-24   member 748e.f882.e4a0   0 remote  Ready

8  D ICX6450-48P  member  748e.f882.ee00   0 remote Ready

    active       standby

     +---+        +---+

=2/6| 1 |2/1==2/6| 2 |2/1=

|   +---+ +---+   |

|                        |

|------------------------|

    active standby             

      ---         +---+ +---+        +---+         ---               

     ( 1 )3/6--2/1| 4 |2/3--2/1| 5 |2/3--2/3| 6 |2/1--3/8( 2 )              

      ---         +---+        +---+        +---+         ---              

    standby                                             active              

      ---         +---+        +---+        +---+         ---               

     ( 2 )3/7--2/1| 3 |2/3--2/3| 8 |2/1--2/3| 7 |2/1--3/5( 1 )              

      ---         +---+        +---+        +---+         ---              

Will assign standby in 12 sec due to all ready

Standby u2 - protocols ready, can failover

Current stack management MAC is 748e.f834.1ea0

----------

Contributors