on 02-29-201609:48 PM - last edited on 03-02-201602:08 PM by ShelbyKhan
In my last blog, I have discussed how a software defined visibility network could open up exciting applications for mobile operators. In this post, I would like to touch upon some typical operational challenges faced by implementation engineers and network operations staff when deploying and supporting network packet brokers.
Operators often have a high-level understanding of their monitoring needs, which is derived from their end use case. However the high level requirement needs to be programmed into packet broker in terms of interface, VLAN-id, MPLS label, IP addresses, ports, protocol id and so on. This requires up to date knowledge of the network. Contrary to popular belief, monitoring engineers often end up with inadequate traffic being monitored, due to wrong or inadequate rules which affects the very purpose of the monitoring exercise. Let me unpack this and see why this is so.
a. The Network is Complex
When we discuss about networks we tend to draw a flattened, simplistic model as below. This kind of picture represents logical interfaces as physical connection between various elements. For example, MME has 3 different logical interfaces shown as physical lines.
However, the actual EPC in a datacenter could be implemented as below with bunch of physical lines carrying several logical interfaces. Layers of switching and routing elements provide redundant interconnection path between the network devices. The redundant connections also help to load balance across devices. To complicate the things even further, these elements could be present in a geographically dispersed datacenters. Often one or many VLAN identifiers identify logical interfaces.
b. The Network is Dynamic
During a normal day of operation as the traffic condition goes thru peaks and troughs, the link and device utilization also track the variation. As the traffic varies the inbuilt load balancing logic of various network elements sends packets on different physical links. During the course of operation, the cards fail, links break, transceiver fail, devices fail routinely. When such failure event occurs, the traffic takes different links to circumvent the failure. The network devices get augmented, retired and taken out temporarily for maintenance.
TAPs are deployed on a physical link and they are going to replicate the traffic irrespective of logical interfaces or VLAN identifiers or protocol. Thanks to the above said reasons, packets and specific traffic flows are likely to be seen in different links at different times.
c. The Network is managed by several teams
The monitoring team that manages visibility infrastructure including packet broker may not be responsible for the day to-day operations of networks and solving some of the issues outlined in (b). When parameters such as VLAN-Id/IP addresses and so on changes, the monitoring team should adjust the rules in their packet broker to reflect the most up to date network configurations. Often this is easier said than done. It is not uncommon to see the network changes are tracked using service tickets, shared spreadsheets and email chains. When the rules are not up to date, they tend to lose some of the traffic of interest.
Brocade's Smart Packet Broker packs a nifty tool called ‘Network learning Module’ which scans all the packets and builds the detailed topology of the underlying network in run time. This tool provides accurate network topology in terms of VLAN Ids, IP addresses, Port and Protocols etc. The info provided by the tool is very useful in quickly implementing use case and also handy in debugging when the network condition changes. At last, Monitoring team can independently and accurately setup rules.