06-15-2012 03:48 AM
Firstly some background.
We have 2 sites and 2 pairs of dedicated (for Ficon) 48K's each of which has two trunk groups to its reciprocal 48k at the other site (Via dwdm)
Each trunk group has 4 ports which are 4GB.
portcfglongdistance is LS (long static) mode.
Fabric o/s is V6.4.2a
We have dynamic path selection as shown by the D in the topologyshow output here:-
2 domain(s) in the fabric; Local Domain ID: 200
Path Count: 2
Out Port: 2/16
In Ports: 2/4 2/5 2/11 2/12 2/13 1/24 2/20 2/21 2/22 2/23 2/24 2/25
Total Bandwidth: 16.000 Gbps
Bandwidth Demand: 287 %
Out Port: 1/16
In Ports: 1/4 1/5 1/8 1/9 1/10 1/11 1/12 1/13 2/8 2/9 2/10 1/20 1/21 1/22 1/23 1/25
Total Bandwidth: 16.000 Gbps
Bandwidth Demand: 312 %
So i would expect the load to be balanced between both trunk groups.
However, The Master port on each trunkgroup seems to get most of the workload, where i would expect to see a more even flow across all 4 ports on each group.
Output below is from the porterrshow for frames tx & rx for about the last 18 hours.
128: 101.1m 1.1g
129: 572.0k 29
130: 17.0m 37.1k
131: 367.7m 9.9m
144: 1.9g 2.4g
145: 20.1k 2.2m
146: 5.2m 65.4m
147: 411.6m 1.1g
1:128->128 10:00:00:05:1e:36:36:76 201 deskew 16 MASTER
130->130 10:00:00:05:1e:36:36:76 201 deskew 18
129->129 10:00:00:05:1e:36:36:76 201 deskew 16
131->131 10:00:00:05:1e:36:36:76 201 deskew 15
2:144->144 10:00:00:05:1e:36:36:76 201 deskew 16 MASTER
147->147 10:00:00:05:1e:36:36:76 201 deskew 16
146->146 10:00:00:05:1e:36:36:76 201 deskew 15
145->145 10:00:00:05:1e:36:36:76 201 deskew 17
We are getting tim_rdy_pri & tim_txcrd_z on the ISL's so am in the process of increasing the number of buffer credits available.
It seems a bit pointless to ramp up buffer credits across all 4 ports per trunk group if the Master is going to hog most of the load.
Why isnt the port usage per trunk group more even?
i expect i'm missing something but it's not leaping out at me from the manuals!
Any input gratefully received.
06-15-2012 04:08 AM
--->>>The Master port on each trunkgroup seems to get most of the workload,....
---->>>.....where i would expect to see a more even flow across all 4 ports on each group.
I facing the same issue around 6 Mouth ago, and this is official answer from Brocade Third Level Support.
This is a normal behavior, because DPS in some case by Long Distance, is not able to "handle/balancing" the Traffic.
06-18-2012 05:13 AM
Remember that the output in porterrshow (frames tx/rx) is based on 32-bit counters (wrap after 4.2g). (you can also use portstats64show x/x output that is a 64-bit counter).
The load-balancing algorithm within a trunk has always done it this way since the 4Gbps platforms and the “master” assignment has nothing to do with the load-balancing of frames.
Not until the first ISL port is fully utilized the second port will be used and so on… (is done in a pre-determined way based on internal bport numbers).
You should configure the same settings across all ports within the trunk and as its FICON, I would at least set the LS distance to about 3x the actual distance to compensate for less frame size (more buffer credits).