Recently, Ivan posted an interested blog, “Brocade VCS Fabric Has Almost Perfect Load Balancing” about Brocade ISL Trunking, one of the great features of the VCS Ethernet Fabric. Ivan was initially skeptical about some of the claims we made and decided to dig deeper. At his request, I provided him more information including reference to some patents awarded back in the last decade covering frame trunking technology for Fibre Channel. In his post, Ivan said in part,
“I also added that I would be mightily impressed if they’d actually solved intra-flow packet scheduling. … I do owe [Brocade] a huge WOW”.
Cool. Thanks for the kind words Ivan.
I’d like to flesh out how Brocade ISL Trunking works at a more detailed level. You can read about how automatic ISL formation works and a high-level description of how Brocade ISL Trunks automatically form in an earlier blog of mine, “VCS Ethernet Fabric Formation and Brocade ISL Trunks”. If you haven’t read it, please do as it’s the starting point for what comes next.
How Brocade ISL Trunks Provide “Almost Perfect Load Balancing”
Figure 1 shows two VDX 6720-60, 60 port VDX switches. Everything I’m going to describe works the same with the VDX 6720-24, 24 port switch except that the number of ports in an ASIC Port Group is 12, not 10.
My earlier blog described automatic ISL formation. After that completes, the following process is used to automatically form one or more Brocade ISL Trunks.
Certain conditions have to be met for ISLs to become candidates for forming a Brocade ISL Trunk.
All ISL ports must be in a single ASIC Port Group on each switch. In figure 1, Port Groups are shown as the numbered blue rectangles.
A maximum of eight ISL links can participate in a Brocade ISL Trunk. This is indicated by (A) in figure 1 where a single Brocade ISL Trunk automatically formed with either ISL connections. (B) shows the other two ISL in the 10 port Port Group on the VDX 6720-60 switch will form a second Brocade ISL Trunk since they also are in a single Port Group in each switch.
Note, each Brocade ISL Trunk is a separate logical path as far as Equal Cost Multi-path (ECMP) is concerned. See my earlier blog on how ECMP works when multiple Brocade ISL Trunks exist between switches.
If some ISL ports are in different Port Groups on one switch, as shown by (C ) and (D) in figure 1 where the right hand switch has ISL ports in two different Port Groups, then multiple Brocade ISL Trunks automatically form so that ports on both sides of the ISL are in a single Port Group in each switch.
Once these rules are applied, a list of ISL candidates for inclusion in a Brocade ISL Trunk is available.
Next, the amount of skew in each ISL is determined to ensure the skew variation across all ISL is acceptable. Skew largely results from differences in the path length of the links. Here are the steps.
Hardware primitives measure the skew in each of the ISL connections. Since 10 GE is full duplex, each switch port has two paths, one for Transmit (TX) and one for Receive (RX), as shown below.
Figure 2. Full duplex paths in an ISL link
Each port transmits the primitives to the neighbor port which receives it. Both switches do this since there can be different skew values on the two paths in the link.
The switch on the RX side, decides if the skew variance for all the ISL links is within the acceptable range.
ISL paths that pass the skew test are automatically added to a Brocade ISL Trunk.
Each of the ISL has the appropriate amount of de-skew applied to eliminate the skew variance.
As shown by (E) and (F) in figure 1, ISLs in the same Port Group can form separate Brocade ISL Trunks when the skew variance is too large. The (E) trunk has shorter cables so those ISL form a Brocade ISL Trunk while the (F) trunk has longer cables so those ISL form a second Brocade ISL trunk.
At this point, frames are striped across the available ISL paths. This works as follows.
The TX side does the frame striping across the ISL connections. The load balancing/striping algorithm can handle variable frame sizes.
The scheduler monitors the TX side hardware to determine if a member port in the Brocade ISL Trunk is ready to transmit so it can efficiently utilize all available ISL links to their maximum bandwidth. The scheduler is able to achieve near linear bandwidth scalability over any number of links up to the maximum of eight.
To assist in ensuring in order delivery of Ethernet frames, a shim is added on the TX side and removed on the RX side by the re-sequencer. The re-sequencer also utilizes the skew information to ensure in order frame delivery.
The result is what Ivan called “Almost Perfect Load Balancing”. The practical value to the customer includes
Very high bandwidth utilization within the Brocade ISL Trunk with linear scalability
Very fast link resiliency
Very simple to use as Brocade ISL Trunks automatically form whenever possible and automatically handle variations in length and Port Group association of ISLs.
There is no need to understand applications and their flows to tune various hash algorithms and then retune them as the mix of applications changes.
Supports forwarding any Ethernet traffic including TRILL, DCB, FCoE and BPDU frames.
And, yes, you can turn off automatic Brocade ISL Trunk formation if you choose.
There are some other points I want to make based on several of the questions posted to Ivan’s blog.
The unique technology supporting Brocade ISL Trunks was first implemented in the last decade specifically for Fibre Channel traffic. Fibre Channel link rates traditionally increased by a factor of two every two years (1 Gbps to 2 Gbps, etc). Therefore, we needed an efficient way to scale ISL bandwidth as a 10x link rate was not readily available (Note, a 10 Gbps link rate was defined by ANSI and did have limited deployment, but that’s another story.). Applications generate storage traffic and the applications are very sensitive to congestion and the resulting latency or worse case, dropped frames that result from over-subscription. Brocade Trunks (as we called the Fiber Channel version) were a very successful solution providing linear scalability of bandwidth very cost-effectively.
Storage networks can include sustained high bandwidth flows (think streaming backup sessions with sequential, large block IO) mixed with spikes of lower bandwidth traffic. Brocade Trunks worked very well to ensure full bandwidth utilization across multiple ISL links in this environment. Today, Ethernet in the data center is subjected to the same conditions as more storage traffic (iSCSI, FCoE) uses Ethernet and virtual machine and storage movement introduce large sustained flows into the network. Brocade ISL Trunks are the evolution of Brocade Trunks optimized for Ethernet traffic. We included it in VCS to solve similar problems in Ethernet fabrics that we previously solved in Fibre Channel fabrics.
We have continued to refine the underlying frame trunking technology. Today, Brocade ISL Trunks use a shim in the frame for even better support of in order frame delivery for 10 GE links.
The maximum length of a Brocade ISL Trunk is quite a bit longer than the maximum cable length for 10 GE in the data center, so for all practical purposes, we don’t have a length limitation. Pick the optics and cable you need and you can use Brocade ISL Trunks in a VCS Ethernet Fabric.
Brocade ISL Trunks support variable frame sizes from 64 bytes to 9KB (Jumbo Frame) with linear scalability.
Brocade ISL Trunks are not a “cure all” for every problem in Ethernet, but are well suited for the data center which is what it is designed for. Extending the ISL trunk over WAN distances was not the use case. There are other solutions. That doesn’t lessen the value of Brocade ISL Trunks as this feature is built into VCS Ethernet Fabrics, at no additional cost, so it’s a great value.