How to maximize Probe utilization with Brocade Packet Broker
byJude Vedam05-08-201608:18 PM - edited 05-12-201609:12 AM
Are you running probes on x86 server platform? Are you unhappy about not being able to utilize your hardware that runs probes to its fullest potential? Do you end up spending more CPU cycles when you attempt to increase the probe utilization by spraying traffic across your CPUs? If you answered ‘yes’ to these questions and you have been searching for an answer, your quest is over. Brocade packet broker solution can help you maximize your probe utilization. Let me tell you how...
Across the globe, several leading mobile operators are either building home grown probes, or testing software-based probes from vendors. Either way, they are trying to leverage x86 server platform that comes equipped with several CPU cores, tons of memory and 40G NIC cards. Even if we assume these modern probing software workloads are multi-threaded, which takes advantage of multi-core CPU architecture, the packets need to be ‘sprayed’ across CPU cores to make them work in parallel.
Simple Load balancing
One way to spray the traffic across several CPU cores is to rely on the hashing feature available on modern NIC cards, such as the ones from Mellanox or Intel. This feature is called Receive Side Scaling. These cards are capable of hashing the traffic using up to 5-tuples to select a queue in multi-queue buffer. By making different CPU cores serve different receive queues on the NIC, traffic can be sprayed across CPU cores. However there is a problem with this approach.
Take a look at the typical user plane probing points. Most common are S1-U or SGi. S1-U tapping point results in GTP-U traffic coming into Probes and plain IP traffic in latter case. Compared to plain IP traffic, S1-U traffic carries more information as it contains TEID and Outer IP addresses. Using these fields, a probe could compute KPI and extract metadata on a per-bearer basis.
When GTP-U traffic is load balanced using simple 5-tuple-based hashing within the probe hardware, the traffic from one subscriber could be sent to different CPU cores at different times. As outer IP and or TEID could change thanks to mobility, link aggregation between eNodeB and Serving Gateways, and so on. (Please see my previous blog for more information.) As a result, KPI/Metadata pertaining to subscriber or eNodeB is available in multiple CPUs that would require data structure access across CPUs and/or correlation of records at analytics level. Both require additional computing resources.
Similar problem persists when SGI (plain IP) traffic is load balanced by a NIC card. This is because one subscriber might have more than one IP address assigned. In this case, one subscriber session might end up at different CPUs. Additionally, Private IP address assigned to IMSI data session could be reused across PGW domains leading to same IP address sessions actually belonging to different subscribers being processed by the same CPU core.
Brocade Packet Broker (BPB) solves this problem with two-level correlation. Two-level correlation helps maximize the probe hardware utilization. At the first level, BPB ensures the traffic belonging to subscriber(s) is sent on the same probe ports irrespective of the outer IP change or TEID change. At the second level, within the probe port, a smaller number of subscribers is tagged with specific VLAN ID. For example, if a probe port carries 300k subscriber sessions, it may use 30 different VLAN IDs to tag 10k subscribers per ID. Now, Probe can use VLAN ID as hashing key to select a receive queue. In this case, each CPU core will receive just 10k subscriber session traffic. Since BPB correlates traffic and assigns VLAN ID, the traffic belonging to subscriber(s) will always reach the same CPU core, avoiding the need to lookup data structure across CPU boundary. This way, a Probe can maximize CPU utilization. Probe can increase the utilization linearly with the incremental processing resource.
BPB can apply this two-level correlation to S11 and S1-U traffic, RADIUS and SGi traffic, and GTP-Cv1 and GTP-U traffic simultaneously.
In summary, BPB helps the probe scale linearly by making every packet tagged using session identifier of interest (IMSI or IMEI or APN or others). This makes the Probe based on software workload scale up as well as scale out, and hence optimize the resource utilization for operator.