Data Center

VCS Ethernet Fabric Formation and Brocade ISL Trunks

by on ‎12-10-2012 02:20 AM (3,486 Views)

I promised in an earlier blog on VCS Ethernet Fabric and FCoE traffic to describe Brocade ISL Trunking.  To do that, we need to quickly review port types and how VCS Technology automatically creates inter-switch links (ISL) between VDX switches when forming an Ethernet fabric.


VCS Port Types

In a VDX switch, a port can be used in one of two ways, an edge port or a fabric port.  A port is acting as one of these, but never both. Fabric ports are part of the Ethernet fabric.  Edge ports are not.  Edge ports connect to devices or classic Ethernet switches that don’t join the Ethernet fabric.  Traffic from edge ports can be forwarded across the Ethernet fabric to any other VDX switch and exit the fabric on any other edge port.  To be clear, there is a single physical port in a VDX switch.  The role a port is operating in, edge or fabric, is automatically decided as described below.  VDX has no “special ports” as is the case with many switch stacking products commonly available today.  That’s one of the nice features about VCS Technology.


VCS Ethernet Fabric Formation

A VCS Ethernet Fabric will automatically form between VDX 6720 switches by default. Here is a quick rundown on how a VCS Ethernet Fabric forms between VDX switches.  Recall that the VCS control plane is distributed across all switches in the fabric. Protocols are designed to aid the formation of an Ethernet fabric with the minimum of user configuration. This is how it works:


Fabric Identification

Each VCS fabric is identified by a VCS fabric ID, or VFID, so VDX switches with the same VFID can form a fabric.


Neighbor Discovery

This involves two steps:


  1. Discover if the neighbor is a Brocade VDX switch.
  2. Discover if the neighbor switch is able to join an Ethernet fabric. Only switches able to join this fabric (advertize the same VFID) are allowed to join.

The neighbor switch port automatically transitions to being a “fabric port” when neighbor discovery is successful. Otherwise, it transitions to an “edge port”, so it will not participate in the Ethernet fabric. Each fabric port is assigned a MAC address and each switch discovers the neighbor’s fabric port MAC address during the message exchange during neighbor discovery.  The discovery protocol uses the LLC frame format and a well known control plane MAC address.


Fabric Formation

As was previously mentioned, a VCS Ethernet fabric is based on TRILL and uses TRILL frames.  Let me summarize the process for forming an Ethernet fabric.


Switch ID Allocation

A unique switch ID is assigned to each switch within the fabric. In the initial release, the RBridge ID (RID) used in the header of TRILL frames is set equal to the switch ID. In a subsequent release this will occur automatically.


Fabric Port Interface Configuration

As mentioned earlier, a physical interface (port) on a VDX switch can be an edge or a fabric port but not both at the same time. A fabric port has specific configuration settings assigned to the physical interface.  Two fabric ports on neighboring switches exchange their configuration information. The default values are sufficient for them to form an inter-switch link (ISL) automatically.  There are fabric port specific attributes that can be manually configured, but this is not required as the default settings enable automatic ISL formation between two neighbor VDX switches.


Topology Discovery

After the RID is assigned to a switch, the link state routing protocol starts forming adjacencies and collects topology information from its neighbors. Today, the link state routing used is Fabric Shortest Path First (FSPF)


The TRILL standard includes reference to modifying an existing routing protocol as noted in the charter for the TRILL working group:


“Note: The TRILL working group is not chartered to develop a new routing protocol or to make substantial modifications to an existing routing protocol. If, during the requirements definition and selection phase, the TRILL working group discovers that no existing routing protocol will meet their needs, we will need to re-assess the TRILL WG charter to determine how/if this work should proceed.”


The TRILL working group ultimately selected IS-IS as the layer 3 link state routing protocol to modify for use at layer 2. The layer 2 extensions to IS-IS are separate from the TRILL standard and are being finalized by IETF. Now, one standard can be substantially completed, as TRILL is, while referencing another standard that is pending ratification, as the modifications to IS-IS are. In the future, today’s VCS Ethernet Fabric will connect to switches that support TRILL and the IS-IS Layer 2 extensions by upgrading the firmware. This is normally how new standards and enhancements are provided by vendors to existing products.



Create a Tree for Broadcast/Unknown Unicast/Multi-cast Frames

A broadcast tree is created for the distribution of broadcast traffic across the fabric.


Now, let’s look at a unique feature of the VCS implementation of an Ethernet fabric, automatic Brocade trunk formation.


VCS Brocade ISL Trunk Formation Process

A VCS Ethernet Fabric includes Brocade trunks which use hardware for frame forwarding across the physical links in the trunk. These are dynamically formed between two adjacent switches and do not require any user intervention or configuration. The only configuration command is “fabric trunking <on|off>”.  The default is “on” so you can manually turn off the trunk feature if you wish, (but, I doubt you will want to that ;-) ).


Unlike a software based LAG formed by LACP, the Brocade trunk is hardware-based which means it can distribute traffic evenly across the physical links on a frame by frame basis without involving any hashing scheme.  The performance is outstanding with extremely high utilization.  We will provide some speeds and feeds in a future blog.


These are the requirements for a trunk to automatically form between two switches in a VCS Ethernet Fabric:

  • On the VDX 6720 switch, the maximum is eight ports per trunk-group for an total of 80 Gbps of bandwidth
  • All trunk ports have to reside on the same ASIC hardware boundaries, aka “port groups”, on either side of the link.
  • You can have multiple Brocade trunks between two neighbor switches


Now, you can have more than one Brocade trunk between two switches and these are treated as separate paths by ECMP when calculating shortest path routes.  Chip Copper provides a concise description of how ECMP works within the VCS Ethernet Fabric with IP and FCoE traffic.  And, as I mentioned in an earlier post about VCS Technology and FCoE, ECMP lowers latency and avoids bottlenecks for all traffic transiting the VCS Ethernet Fabric automatically so there’s no extra configuration required to take advantage of this.  Very simple and very valuable.


I hope this is helpful.  In my next blog, I’ll talk in more detail about how VCS Technology implements Equal Cost, Multi-Path (ECMP) to optimize flows across an Ethernet fabric reducing latency and avoiding congestion.


Thanks for dropping by.