Buffer credits and their application to SAN design is one of the most misunderstood aspects in Fibre Channel, especially when applied to FICON environments. User’s have many questions about the optimal use of buffer credits. What’s the right number of credits? Are they the same for all ports? If a few credits are good are more even better? How many credits do I need to retire? What we find is that most of the time the answer to these questions is usually, “it depends.” YAAAAKKKK! That’s about the last answer I want to hear, but truth be told it’s the best answer we can give.
In order to fully understand buffer credits and ultimately to answer these burning questions, we need to look at the role of buffer credits in Fibre Channel. The primary function of buffer credits is flow control. They prevent the transmitting device from overrunning the receiving device. To do this, the devices must exchange their buffer credit values during the login process. Each device tells the other how many buffers it has to store incoming frames then the devices use this value to limit the number of frames transmitted from the sending device to the receiving device. As the transmitter sends frames, it decrements the value (the credit count) and stops sending frames when the count reaches zero. The receiver helps the transmitter to know when it can start sending again by responding with an acknowledgement primitive called Receiver Ready or R_RDY when it has dispatched the frame. When the transmitter receives the R_RDY, it increments the credit count. Thus, in a perfectly tuned system, the transmitter is sending its last frame and decrementing the credit count to zero just as it is receiving an R_RDY from the receiver.
If the only purpose of buffer credits was to manage flow control then we would only need one to maintain a steady state between a transmitter and a receiver. But since fiber lengths can be long and frame sizes are limited, we also need buffer credits to maintain the capacity of the link over distance. In fact, one of the most significant factors in determining the number of buffer credits is the length of the link. The longer the link the more credits are required to maintain the capacity of the link. Of course, the speed of transmission and the size of the frames transmitted also play a role in determining the number credits needed. In this way, we see that the optimal buffer credit value is determined in three dimensions – length of the link, speed of the link, and size of the frames. Hence, a reasonable rule of thumb for determining buffer credits is link speed in Gigabits times distance in kilometers divided by frame size in kilobytes then add one for good measure. For example, to determine how many buffer credits are needed for a 4G link at 10Km assuming 2KB frames, we get ((4*10)/2) + 1 = 21 buffer credits.
Hey, to this point, this buffer credit stuff doesn’t seem so hard, so what’s the big deal? The devil, of course, is in the details and the detail we care about is the frame size. It’s often quite difficult to determine the size of every frame other than to assume the maximum size of a Fibre Channel frame, which happens to be 2KB. However, some frames are full of data and others carry only a small amount of data or control information. Thus, the “average” frame size becomes a much better value for use in our three-dimensional rule of thumb. In FICON, a typical 4K data transfer requires between three and four frames to complete. When we factor in a couple of frames for the data and one or two for the commands, we end up with an “average” frame size closer to 1K per frame. Therefore, our three-dimensional rule of thumb can be reduced to a two-dimensional simplification of rate times distance plus one.
I’ve spent a bit of time reducing buffer credits to a simple rule; however, such rules can be an over simplification of the needs of any particular environment. For those with a greater interest in understanding the behavior of buffer credits, you can attend my session with Lou Ricci (IBM) at SHARE in Seattle (shameless celebrity plug so noted) or you can checkout our presentation from SHARE in Denver here.
I hope you’ve enjoyed this discussion and keep those cards and letters coming. We’d love to hear from you. Till next time, let’s keep those bits flowing.