Load sharing and load balancing are two functions that are often confused when discussing SAN performance. When balancing loads, the idea is to balance the “potential” flows over the available paths. When sharing loads, the idea is to distribute the “actual” flows over the available paths. Brocade provides methods for performing both functions and today’s blog discusses the differences between those methods in the context of FICON environments.
FICON environments demand performance and require robustness. These requirements necessitate the need to implement load shaping in cascaded fabric designs. The Brocade software feature that supports this capability is called Lossless (as in Lossless DLS and Lossless DPS) and is one of several load balancing and load sharing features in FOS.
Load sharing and load balancing across ISLs is an implementation dependent behavior; however, there are characteristics of FICON that influence the choices a customer can deploy in any particular environment. In addition, Fibre Channel switch vendors are constantly improving the methods to support newer technologies, more configurations, and additional environments. With that said, the various methods used by Brocade implementations are described below.
"Basic load balancing" is a technique executed at Fabric Login time. For every device that logs into the switch, an exit port assignment is made for each domain in the fabric. This assignment is fixed for the duration that the device exists in the fabric and is only changed if a fabric event occurs affecting the assignment (i.e. the assigned ISL leaves the fabric, the destination domain leaves the fabric, an intermediate domain or ISL leaves the fabric, etc). The balancing criteria used in this scenario are to balance the total "number" of devices across the available ISLs. For example, if twelve devices login and there are three ISLs, this method assigns four devices to each ISL. Note, however, that actual traffic flow to the destination domain served by the ISLs is not considered in the assignment.
Although an effort is made to maintain a balance between the "count of entries" per ISL, this method suffers when either the actual load happens to land on a few ISLs or when fabric events occur that force the flows onto a subset of the ISLs. For example, if two of four ISLs between a pair of domains fail, all of the flows will be moved to the remaining ISLs in order to maintain connectivity. However, when the two failed ISLs are repaired, the flows are not rebalanced and only new logins are assigned to the repaired ISLs. This basic method is used by all Fibre Channel switch vendors with very minor differences between the implementations.
The Brocade software feature that supports this method is called Port Based Routing.
"Dynamic load balancing" is provided by some vendors and adjusts the "count of entries" assigned to each ISL whenever a fabric event occurs. This mechanism addresses the imbalance that can occur when ISLs fail and repair. Still, actual loads are not considered when balancing the "count of entries" and only the number of devices logged in assigned to each ISL are balanced. In addition, the changes made to balance the "count of entries" are done immediately and without consideration for the activity of the ingress devices. Thus, it is possible to encounter discarded frames or out of order frames when a balance event is executed.
The Brocade software feature that supports this method is called Dynamic Load Sharing or DLS (yes, a bit of a misnomer, but that's the feature name ;-).
Load sharing occurs when the switch spreads the actual flow levels between a pair of domains across the available ISLs. There are several techniques that are used to support this capability and they differ in their methods for determining assignment changes.
"Hardware based trunks" combine a series of ISLs together to form a virtual ISL that has an aggregated bandwidth equal to the sum of the bandwidths of each individual ISL. Typically, this balancing is handled at the port ASIC level and is executed at the frame level. Some implementations spread the frames across each ISL in a round-robin fashion, others spread the frames in a random fashion, and still others spread the frames in an overflow or flooding fashion. Each mechanism is a function of the characteristics of the hardware and generally requires the lengths of each ISL to be similar (i.e. within a specified tolerance of each other - sometimes known as "skew"). They may also require the ISLs to be attached to ports that are serviced by the same port ASIC.
The Brocade hardware feature that supports hardware trunks is called Trunking and uses either the round-robin method (2G and 4G products) or the flooding method (8G products).
"Software based trunks" combine a group of ISLs together to aggregate the bandwidth between a pair of domains. The aggregated bandwidth is again equal to the sum of the bandwidths of the individual ISLs, but the methods of balancing the bandwidth between the ISLs differs from the techniques used for hardware trunks. This method uses programmatic techniques plus hardware support to share the traffic and, as such, is limited to sharing algorithms that balance flows or exchanges instead of frames. In this context, a flow is defined as the communication from a particular source (SID) to a particular destination (DID) and an exchange is the communication from a particular source (SID) to a particular destination (DID) using a particular exchange (OXID). Both of these methods are "proactive" operations that achieve balance by assigning flows or exchanges to different ISLs. The exchange method generally achieves a greater degree of balance between the ISLs because exchanges are short lived and can be assigned randomly or by convention (i.e. round-robin). Flow based assignments are fixed for the duration of the connection (i.e. for as long as the source and destination are in the fabric).
The Brocade software feature that supports this method is called Dynamic Path Selection (DPS).
Due to the wide range of load balancing and load sharing methods, a number of operational differences should be considered when deploying any particular technique for an application or environment. Most methods offered today are sufficient for distributed or open systems environments where fabric events have less of an impact on the operations of the storage area network. Many of the dynamic techniques are problematic for mainframe environments because the method used to move or change assignments impacts frame delivery, which is captured by the upper layer protocol and reported as service events or errors. For this reason, the above implementations have been enhanced to allow assignments and changes for sharing and/or balancing reasons are done without impacting frame delivery.
In summary, Brocade's hardware trunking uses the "flooding" method for 8G based products such as the DCX. Thus, when the bandwidth demand is low, only the first ISL in the Trunk would be used (until the bandwidth demand exceeded 8G then the next ISL would be used). Also, Port Based Routing is used as the default load balancing and load sharing method. Once Lossless DLS or Lossless DPS is turned on then the dynamic methods would modify the assignments accordingly. Lossless DPS is currently being evaluated for FICON certification and the initial testing has been demonstrated to work very well in a high demand/high bandwidth end customer deployments.
I hope you’ve enjoyed this discussion and keep those cards and letters coming. We’d love to hear from you. Till next time, let’s keep those bits flowing.