Buffer-to-buffer credit management affects performance over distances; therefore, allocating a sufficient number of buffer credits for long-distance traffic is essential to performance. With FOS 7.1 Brocade introduced an enhancement to our FOS Extended Fabrics Feature: TheportCfgLongDistance CLI command now includes the option to configure the number of buffers by using the-frameSizeoption command along with the-distanceoption. Prior to FOS 7.1, the only option was the–distance option.
Buffer Credit Background and Review
To prevent a target device (either host or storage) from being overwhelmed with frames, the Fibre Channel architecture provides flow control mechanisms based on a system of credits. Each of these credits represents the ability of the device to accept additional frames. If a recipient issues no credits to the sender, no frames can be sent. Pacing the transport of subsequent frames on the basis of this credit system helps prevent the loss of frames and reduces the frequency of entire Fibre Channel sequences needing to be retransmitted across the link.
Because the number of buffer credits available for use within each port group is limited, configuring buffer credits for extended links may affect the performance of the other ports in the group used for core-to-edge connections. You must balance the number of long-distance ISL connections and core-to-edge ISL connections within a switch.Buffer-to-buffer (BB) credit flow control is implemented to limit the amount of data that a port may send, and is based on the number and size of the frames sent from that port. Buffer credits represent finite physical-port memory. Within a fabric, each port may have a different number of buffer credits. Within a connection, each side may have a different number of buffer credits.
Buffer-to-buffer flow control is flow control between adjacent ports in the I/O path, for example, transmission control over individual network links. A separate, independent pool of credits is used to manage buffer-to-buffer flow control. A sending port uses its available credit supply and waits to have the credits replenished by the port on the opposite end of the link. These buffer credits are used by Class 2 and Class 3 services and rely on the Fibre Channel Receiver-Ready (R_RDY) primitive to be sent by the receiving link port to the sender. The rate of frame transmission is regulated by the receiving port, and is based on the availability of buffers to hold received frames. (If Virtual Channel technology is in use, the VC_RDY or EXT_VC control word is used instead of the R_RDY control word to manage buffer credits. A future blog post will focus on Virtual Channels).
Upon arriving at a receiver, a frame goes through several steps. It is received, deserialized, decoded, and is stored in a receive buffer where it is processed by the receiving port. If another frame arrives while the receiver is processing the first frame, a second receive buffer is needed to hold this new frame. Unless the receiver is capable of processing frames as fast as the transmitter is capable of sending them, it is possible for all of the receive buffers to fill up with received frames. At this point, if the transmitter should send another frame, the receiver will not have a receive buffer available and the frame is lost. Buffer-to-buffer flow control provides consistent and reliable frame delivery of information from sender to receiver.
Optimal buffer credit allocation
The optimal number of buffer credits is determined by the distance (frame delivery time), the processing time at the receiving port, the link signaling rate, and the size of the frames being transmitted. As the link speed increases, the frame delivery time is reduced and the number of buffer credits must be increased to obtain full link utilization, even in a short-distance environment.
For each frame that is transferred, the hardware at the other end must acknowledge that the frame has been received before a successful transmission occurs. This flow requires enough capacity in the hardware to allow continuous transmission of frames on the link, while waiting for the acknowledgment to be sent by the receiver at the other end.
As the distance between switches and the link speed increases, additional buffer credits are required for the ports used for long-distance connections. Distance levels define how buffer credits are allocated and managed for extended ISLs. Buffer credits are managed from a common pool available to a group of ports on a switch. The buffer credit can be changed for specific applications or operating environments, but it must be in agreement among all switches to allow formation of the fabric.
Smaller frame sizes need more buffer credits. In FOS 7.1, two CLI commands are available to help you determine whether you need to allocate more buffer credits to handle the average frame size. TheportBufferShowcommand calculates the average frame size. TheportBufferCalc command uses the average frame size with the speed and link distance to determine the number of buffer credits needed.
Configuring buffers using actual frame size
You can configure the number of buffers by using the-frameSizeoption of theportCfgLongDistancecommand along with the-distanceoption. Fabric OS calculates the number of buffers from the -frameSize option value according to the following formula:
If you enter a frame size of 1024, Fabric OS will allocate almost twice as many buffers as for the maximum frame size of 2048. The-frameSizeoption value is persistent across reboots and HA failover.
switch:admin> portcfglongdistance 2/35 LS 1 –distance 100 –framesize 1024
Calculating the number of buffers required given the distance, speed, and frame size
If you know the distance, speed, and frame size for a given port, you can use the portBufferCalc command to calculate the number of buffers required. If you omit the distance, speed, or frame size, the command uses the currently configured values for the port. Given the buffer requirement and port speed, you can use the same distance and frame size values when using theportCfgLongDistance command.
To determine the number of buffers required, perform the following steps:
Connect to the switch and log in using an account assigned to the admin role.
Enter the portBufferCalc command and provide values for the distance, port speed, and frame size.
The following example calculates the number of buffers required for an 8-Gbps port on a 100-km link with an average frame size of 512 bytes.