For more details, please see ourCookie Policy.

Service Providers

Evolving SP WAN Architectures; how do routers keep up?

by pmoyer on ‎08-24-2012 01:01 PM - last edited on ‎10-28-2013 11:06 PM by bcm1 (1,692 Views)

There are some interesting developments going on in service provider networks these days, to say the least. Besides all the OpenFlow and SDN hoopla, which is quite the hot topic right now, traditional SP backbone networks are undergoing their own natural evolution. A few of these trends could impact router architecture in terms of density, scalability and feature set.


Today, most SP networks rely heavily on MPLS technologies. One primary driver for using MPLS over the last decade or so is network convergence. Most SP networks are multi-service networks and MPLS provides this multi-service capability. In a MPLS-based, multi-service




SP network there are two primary roles for routers; the Provider-Edge (PE) or Label-Edge Router (LER) router at the edge of the network and the Provider (P) or Label-Switch Router (LSR) router in the core of the network.


All IP/MPLS routers, both the PE/LERs and the P/LSRs, need to run many different protocols, each with its own unique design and deployment challenge. There is an IGP (OSPF or ISIS), an EGB (by default, MP-BGP) and an MPLS signaling protocol (RSVP or LDP). An IGP is needed in any IP network for internal reachability. In an MPLS network, the IGP also provides information that is required by RSVP and LDP. BGP is required in IP transit networks for external reachability and also provides routing and signaling information in MPLS networks, particularly in MPLS VPN networks. That’s a lot of protocol state being required in all these IP/MPLS routers.


BTW: Does all this sound complicated yet? This is why highly skilled IP/MPLS network engineers and architects are in high demand!


From a forwarding plane performance perspective, it should surprise no-one that Internet traffic growth continues unabated. Over-The-Top (OTT) video and other services are fueling this growth. Core routers that scale to high density 10GbE & 100GbE interfaces, when combined with high performance switch fabric technologies, allow the forwarding capacity of SP networks to continue to scale to meet this demand.


SP WAN Super-Core Architecture

BGP provides external reachability in transit networks but do all IP/MPLS routers really require this external routing information? If all the transit routers forward packets based on an IP lookup, then the answer is clearly yes. But since the P/LSR routers in an MPLS network forward packets based on a label lookup, then why do they still need to run BGP and maintain all that external routing state?


The Internet IPv4 routing table is massive; as of the time of this writing it’s at somewhere around 430k BGP routes! The IPv6 routing table is relatively small in comparison at round 10k BGP routes, but it continues to grow as IPv6 is becoming more widely deployed. It’s difficult to predict how many IPv4 or IPv6 routes there will be in the Internet in 2 years, let alone in 5 or more years.


To address some of the challenges in density, performance and capacity, a simplified “super-core” network architecture has been developing over the recent years. The definition of “super-core” may vary slightly depending on who you ask, but in general it reflects an “inner” core network that is architected differently than the rest of the network. Brocade’s general definition of a super-core architecture is based on a few different factors; one being the high density and scalability requirements of the super-core router and the other being a reduced control plane and route table capacity requirement of the super-core router. A reduced route table capacity requirement is achieved by removing the requirement to run BGP on these super-core routers. Since LSRs in the core network only forward packets by examining the MPLS label information, they do not look into the IP packet header. So, these routers do not need to know about all the external Internet routing information that is distributed by BGP; hence, they no longer need to run BGP. This can be a beautiful thing! One definition of such a network is a “BGP-free core” network. In this type of network, BGP only runs at the edges of the network where external Internet routing information is required. Makes sense, right?


In terms of router architecture, there are always trade-offs that must be made when designing a high-end core router. Please see additional detailed information on router technology trade-offs in previous blogs; here, here, and here. As discussed in those blogs, a router vendor can push technology only so far. Design decisions must be made on which capabilities are more important to SP customers. Capabilities can vary rather dramatically depending on what role in the network the router is performing. At the edge of the network, the LER needs high edge port density and a scalable BGP implementation to carry all those external Internet routes. The LSR in the core however, has different requirements in terms of density, scalability and capacity. The core is the aggregation region of all those high speed interfaces! Today’s core routers need to be more focused on high density interfaces and high forwarding performance.


So, to accommodate this super-core architecture trend that we are seeing in the largest SP networks, higher density 10GbE and 100GbE interfaces are required. This pushes today’s technology to its limits in terms of silicon density and performance. The interesting angle to this is that since the routing table capacity of the super-core LSR is reduced by not running BGP on these routers, vendors have the choice to optimize products for higher density and performance without having to also optimize for forwarding table size.


SP WAN Public IP Offload Architecture

Another trend we are seeing in the more progressive SPs is what we are calling an “IP Offload” architecture. This is basically where “commodity” Internet transit traffic is forwarded over a simplified, higher capacity yet cost efficient network; which runs in parallel to the traditional multi-service MPLS backbone. What some providers are discovering is that while Internet transit traffic continues to grow exponentially, the revenues associated with this traffic are not keeping pace. In addition, enterprise VPN and other services are not growing as rapidly and this is where a large portion of the SPs revenues are derived. This presents a dilemma for the SP; to continue to invest in and upgrade the capacity of the multi-service MPLS network (which carries both Internet and enterprise VPN traffic) or to do something different.


One answer is to move the high growth Internet traffic to a simpler and more cost effective, higher capacity core network while maintaining the enterprise VPN and other services on the traditional MPLS network.  Since the traffic that is being moved to this IP Offload network is Internet transit traffic (eg. Public facing), we can refer to this as a “Public IP Offload” strategy.  The requirements for the higher capacity offload network are focused around density and performance. Again, high density 10GbE and 100GbE interfaces are required in this type of offload network, similar to the super-core network previously described. Seeing a trend yet?


SP WAN Private IP Offload Architecture

Somewhat similar to the public IP offload strategy, we are seeing an Inter-Data Center offload strategy emerge. I refer to this as a “Private IP Offload” strategy; as within a large Internet facing enterprise network, the inter-DC traffic is seeing unusually high growth rates. This is somewhat of a recent trend that is being driven by cloud networking, big data, and active-active data center architectures. DC network operators that have traditionally optimized their network for “north-south” traffic patterns are now seeing massive growth in “east-west” traffic patterns. As active-active data center architectures emerge, “east-west” often translates to “inter-DC”.


Somewhat of a similar dilemma exists here that was previously described. In this case, the high growth inter-DC traffic is often internal enterprise traffic that is not related to public, Internet facing customers where the enterprise receives a majority of its revenue. So, in this case the “commodity” inter-DC traffic utilizes the same multi-service MPLS infrastructure as the Internet facing customer traffic. A similar answer can be found by moving this high growth traffic onto a simplified, higher capacity and more cost effective infrastructure. An inter-DC backbone network can carry this traffic while the Internet facing traffic is maintained on the traditional MPLS backbone. And as one can imagine, the router requirements for this type of network are high density 10GbE and 100GbE interfaces. One attribute of this private IP offload architecture that differs from the public IP offload architecture is that since this traffic is internal to the enterprise, this backbone should not need to carry the same large routing tables as a real transit network would be required to carry.




Brocade MLXe IP/MPLS Router Platform

Brocade offers industry leading core router performance and density; combined with the “right” feature set for IP and MPLS applications. The MLXe product line is our flagship carrier-class IP/MPLS router. It performs equally well at many roles within the SP network; be it LER, LER aggregation, LSR and even DC applications. The software feature set is very comprehensive and scalable, which enables this platform to perform different roles within the SP network depending on how the MLXe is configured.


In addition to its advanced and flexible software feature set, the MLXe chassis supports exceptionally high density 10GbE and 100GbE interfaces. The forwarding performance of the largest MLXe chassis, the MLXe-32, exceeds 15Tbps! The advanced hardware architecture includes a fully distributed, non blocking CLoS switch fabric design. One particularly interesting capability of this platform is its ability to support 64x10GbE interfaces in a single Link-Aggregation (LAG) bundle. Do the math for the size of that LAG interface!


The MLXe product line meets the requirements of traditional multi-service MPLS backbones, as well as the emerging requirements of the SP super-core backbone and the public & private IP offload network that are described in this blog.


So, as I previously stated – these are some interesting times in the service provider market. Internet traffic growth continues unabated and network architecture solutions are emerging to meet this growth demand; while not exacerbating the revenue challenge that SPs are struggling with. There is a strong drive to simplify networks to reduce CAPEX and OPEX. As cloud computing solutions become widely deployed, traffic patterns internal and external to the data center are changing from being north-south focused to being east-west focused. As new network architecture solutions emerge to solve these challenges, Brocade will continue to innovate to lead the industry in terms of performance, density and capacity requirements for the largest SP networks. So, stay tuned to this space for some exciting developments in this area.