I recently returned from IETF 87 and would like to provide a brief synopsis of the event. First, who would have expected the weather to be that hot in Berlin? It hit 96F on Sunday, the first day of the event!
Overall, it was a very well attended event with a very relevant agenda. My first point to make is that the SDN trend continues to gain interest and acceptance. If you’ve been following IETF activity over the last 12 – 18 months or so, you know that this trend started kind of slow but is gaining more traction at each of the IETF events. The future meeting schedule can be found here, and if you’d like a quick review of the IETF 86 event, you can go here.
The NVO3 WG and SDN RG meetings continue to draw the most relevance to the SDN problem space. However, the L3VPN WG and to a lesser extent, the I2RS & PCE WGs, also have correlated activities worth following if SDN is of interest to you.
So, here are the brief updates, organized by WG/RG.
The SDN Research Group kicked off the Monday morning sessions. This RG is chaired by our Brocade SP CTO, Dave Meyer. Dave started the session with a thought provoking presentation that set the bar early. The session included some other quite interesting presentations; such as an SDN-enabled IXP, a presentation on how NFV fits with other IETF SDN activities, and how I2RS and SDN are related (or not).
This was a *very* well attended session, as it was the last time. With the advent of cloud computing and private/public/hybrid data-center clouds, this is where most of that activity is taking place in terms of defining the data-center virtualization problem space, its requirements and eventually the solution space.
The NVO3 architecture team provided an update on their activities. Brocade Principal Engineer, Jon Hudson, is part of this architecture team. This was quite an interactive session. They recommended to promote the current “as-is” VXLAN and NVGRE ID’s to Informational RFC status. This is primarily to document these technologies since they are already implemented and deployed. In addition, other IETF activities relate back to these drafts, so now they are Informational RFCs that can be referenced.
The NVO3 reference architecture that is being developed will identify the key system components and describe how they fit together. This architecture phase will then drive the requirements definition work; and that will then feed into a gap analysis. Some key definitions are being defined; such as, NVE (Network Virtualization Edge) and NVA (Network Virtualization Authority). Some critical ‘on-the-wire’ protocols are being flushed out, such as the NVE to NVA protocol and the Inter-NVA protocol. Finally, consensus on the need for both a push and a pull model of control plane distribution was agreed upon.
The L3VPN sessions are always worth attending, and more recently so since this WG has become closely correlated to the NVO3 activities. Besides the usual discussions of MPLS-related internet drafts and technologies, this session included a very active discussion on how the activities in this WG remains aligned (or not) with the NVO3 activities. This discussion was very much needed, since the last few L3VPN WG sessions included some internet drafts that overlap in some sense with the NVO3 charter. Most of this discussion was around which problem areas should remain in the L3VPN WG and which problem areas should perhaps move to the NVO3 WG. As an example, there was a good amount of consensus that technologies “inside” the DC should belong in the NVO3 WG. However, technologies around “inter-DC” solutions should perhaps remain in this WG. Well, this makes a large assumption that the inter-DC solutions and technologies are based on L3VPN solutions. L3VPN solutions could clearly be one answer to the inter-DC problem, but it’s not the only answer. That’s where it gets fuzzy.
I think one way to clarify this is that if the applicability statement in an ID includes intra-DC problems, then it should be part of the NVO3 WG. If protocol extensions to L3VPN solutions are needed for a particular solution, then clearly that work must be done in the L3VPN WG. But if an ID touches on inter-DC problems, then perhaps it needs to be presented in both WGs; at least until it gets further flushed out. Clear as mud yet?
Another WG that continues to generate a fair amount of activity and interest is PCE. This is also an area of IETF work that is somewhat related to the SDN solution space. This WG is focused on how to enhance traffic-engineering decisions in MPLS networks.
I had two key take-away’s from this session. One is that this WG has quickly moved from working on solutions that only “recommend” traffic-engineered LSPs to the network to also now including solutions that actually “instantiate” those LSPs into the network. In other words, the solutions being discussed here include both the centralized TE control-plane and the actual distributed data-plane. The other important take-away is there is more activity around providing PCE redundancy; that is, how to provide multiple PCE databases for the network and how to keep them synchronized. This is a hard problem to solve and in some sense can help flush out the entire “logically centralized” notion in SDN.
The session ended with an interactive discussion that I thought was quite interesting; well actually, almost amusing. The topic was about whether Auto-BW mechanisms should be pushed back into the network nodes so they can each dynamically adjust their LSP BW. Recall that the primary goal of PCE is to logically centralize the traffic-engineering decisions. It’s all about having a central holistic view of network traffic loads in order to fully optimize the entire network. So, this discussion was about whether it makes sense to then allow each network node to adjust its LSP bandwidth, using auto-BW mechanisms, in a distributed way. Doesn’t this sound counter to the goal of PCE? So, is it a centralized or is it distributed? I hope you see why I thought this entire discussion was somewhat amusing.
I2RS was also very well represented and generated lots of interesting dialogue. I believe this was only the second time this WG met, so the activity here is very early in its definition. The architecture team started off discussing the high-level architecture and a policy framework. All this discussion is centered on the Routing Information Bases (RIBs) of a network node; for example, are multiple RIBs needed? What mechanisms are needed to inject state into the RIB(s)? What mechanisms are needed to extract state from the RIB(s)?
The I2RS activity focuses on the southbound problem space; in other words, the interfaces and protocols needed from the controller or client down to the network node. It does not focus on the northbound interfaces or applications that live on top of the controller.
There was also a question about whether I2RS protocol extensions are being developed in private; or outside the knowledge of this WG. There was a request to encourage people to share those discussions and potential experiments with the general WG to spur discussion.
An interesting I2RS service chaining use case was discussed that is being co-authored by Brocade Principal Architect, Ramki Krishnan.
I dropped into the FORCES session to see how this WG is progressing. This WG has been around for many years but it never really gained much attention in terms of implementation or deployment. Now that SDN is here to stay, my sense is that this WG is trying to re-emerge to become relevant to that conversation.
To that end they have a new charter and some of the terminology being used in this WG is more aligned with the SDN problem space. For example, there are drafts that discuss Virtual Control Element (VCE) nodes and Virtual Forwarding Element (VFE) nodes.
The OPSA WG continued the discussion around a draft presented by Brocade Principal Architect, Ramki Krishnan, on mechanisms for optimal LAG/ECMP link utilization. This capability is important in not only service provider networks but also in research and education networks. Researchers are often the sources for very large IP flow dissemination and the ability to properly load-balance these large flows on LAG bundles is becoming increasingly important. I also presented this draft, on behalf of Ramki, at the recent ESnet Site Coordinators Committee (ESCC) Meeting at the Lawrence Berkeley National Labs and it was very well received by this group.
Regarding the various Routing WGs, here is a short recap.
An interesting presentation in the GROW WG was on a use case of using route reflectors for traffic steering. This idea is not new but it does provide an additional data point on how service providers desire enhanced capabilities to influence traffic patterns in their networks.
Similar to the last IETF, there was more discussion in the IDR WG about the ability to distribute link-state and TE information northbound, using extensions to BGP. This type of capability would allow higher layer applications to make more intelligent traffic-engineering decisions. Kind of sounds SDN-like, doesn’t it?
So, that wraps up this short update on the IETF 87 SP related activities. Please let me know if you have any comments or questions.
As every service provider already knows, performance and reliability are always high priorities for enterprise IT. Add cloud computing to the mix, its importance multiplies. In this week’s blog, I continue discussing the WaveLength Market Analytics study on enterprise infrastructures service needs. Specifically, I address the market opportunity for server load balancing services in both terms of size and demand, as well as service packaging, and pricing.
Like the market for IPv6 translation services, the market for server load balancing (SLB)-as-a-service is very large. When asked about their top three IT management priorities, the graph above shows that more than half of enterprise IT survey respondents chose improving user experience, service quality, and reliability as the top IT management priority. High performance and reliability requirements drive demand for server load balancing solutions, so if half of America’s 18,000 large organization’s outsource just one application’s server load-balancing, it’s a good-sized market.
Of course, server load balancing is not a new concept. Enterprises have been using the technology for more than a decade and as the table below shows, they use a mix of server load balancing solutions. Most organizations use an internally managed solution and about half already use a service provider for some type of server load balancing. Forty five percent of both the medium and large enterprises outsource using dedicated load balancers from a service provider. Although less mature, even 38% of medium and 20% of large enterprises say they outsource to a service provider using shared load balancers. As public and hybrid cloud acceptance grows, this shared load balancing service is expected to grow along with them.
Where are enterprises on their willingness to buy a new variant, server load balancing-as-a-service? What value-added features would they be willing to buy and what would they be willing to spend on them? Among the three new infrastructure services, IPv6 translation, storage area network extension, and server load balancing, server load balancing has the highest percentage of respondents who are willing to pay for services. With 86% of large enterprises, survey respondents are more likely to be willing to pay for server load-balancing services than the other two infrastructure services discussed in my previous blogs, IPv6 translation and SAN extension.
As to the above table shows, server load balancing services offer significant opportunity for differentiated services or to sell additional value-added services. Nearly three-quarters of large enterprises are willing to pay for all SLB capabilities included in the study. A service provider can create a service for global load balancing between two data centers as a stand-alone service and about 77% of large enterprises said they’d be willing to pay for it. A service provider can also earn extra revenue by packaging device redundancy for highly available SLB-as-a-service. This value-add can entice 45% of medium and 78% of large enterprises to part with some dollars.
So how much extra will enterprises likely pay for SLB value-added features? We asked how much extra in the form of a premium over the base service fees for SLB-as-a-service for 3 specific value-adds. On average, large enterprises are willing to pay an additional 27% for IPv6 migration support and for SSL offload, and an additional 31% for device redundancy. The more budget-constrained medium-sized enterprise segment reports they are willing to pay an approximately an additional 25% for each of the three value-adds.
The enterprise IT imperative to improve user experience, service quality and reliability, along with increasing cloud apps, create a large and growing service provider opportunity. With its many value-added features, SLB-as-a-service is potentially lucrative; it is both easily differentiated and average revenue per user (ARPU) increases with each a la carte value-added feature. Service providers should invest in their SLB-as-a-service portfolio today to take those new SLB revenue streams to the bank.
Multi-Chassis Trunking (MCT) is a key Brocade technology that helps network operators build scalable and resilient networks, and we are continuing to add more enhancements to MCT that provide advanced redundancy. MCT-aware routers appear as a single logical router that is part of a single link aggregation trunk interface to connected devices. While standard LAG provides link- and module-level protection, MCT adds node-level protection, and provides sub-200 ms link failover times. It works with existing devices that connect to MCT-aware routers and does not require any changes to the existing infrastructure. Pete Moyer wrote about the multicast over MCT features we added in NetIron software release 05.4.00 in his blog earlier this year, and we recently released version 05.5.00 with support for Layer 3 dynamic routing over MCT which is what I want to write about today. Together these two enhancements give network operators the ability to deploy MCT Layer 3 active-active or active-passive redundancy at the network edge or border for IP unicast and multicast.
Layer 3 Routing Over MCT Highlights
Here’s how it works in a nutshell for a quick example with OSPF. The CCEP and CEP in the diagram are on different Layer 3 networks and the routers run OSPF as the IGP. The MCT CCEPs can be configured as active in the OSPF so that they establish an OSPF adjacency with the connected device. Or they can be configured as passive in the IGP so that the interface is advertised via OSPF, and a static route is configured on the connected device.
Layer 3 traffic that is sent to the network connected to the CEP is load-balanced over the two MCT routers by the connected device, assuming an active-active configuration. If one of the CCEPs or MCT routers fails, Layer 3 traffic will still be forwarded over the MCT Inter-Chassis Link (ICL) to the remaining MCT router. I left out some details for simplicity to illustrate the functionality; obviously you’d want redundant connections via CEPs on each MCT router in order to provide full redundancy.
For more information on Layer 3 routing with MCT, refer to the “MCT L3 Protocols” section in the MCT chapter in the Multi-Service Switching Configuration Guide. If you need more information on MCT then we have two technical white papers that you can read. “Multi-Chassis Trunking for Resilient and High-Performance Network Architectures” provides an overview of the technology, and “Implementing Multi-Chassis Trunking on Brocade NetIron Platforms” has design and deployment details.
sFlow is a very interesting technology that often gets overlooked in terms of network management, operations and performance. That’s a shame; as it can be a very powerful tool in the network operator’s tool-kit. In this brief blog, I hope to shed some light on the appealing advantages of sFlow. To start with - if you are a network operator and you are not gathering network statistics from sFlow, I hope you will carefully read this blog!
While the title of this blog says enhancements to sFlow, I’d like to focus a good portion of this piece on sFlow itself and explain why it’s not just useful, but why it should be considered a necessary component of any overall network architecture. I’ll also point out some differences between sFlow, NetFlow and IPFIX (since I frequently get asked about these when I talk about sFlow with customers).
sFlow was originally developed by InMon and has been published in Informational RFC 3176. In a nutshell, sFlow is the leading, multi-vendor, standard for monitoring high-speed switched and routed networks. Additional information can be found at sFlow.org.
sFlow relies on sampling; which enables it to scale to the highest speed interfaces, such as 100GbE. It provides very powerful statistics and this data can be aggregated into very edifying graphs. Here is a pretty cool animation describing sFlow in operation. sFlow provides enhanced network visibility & traffic analysis; can contribute relevant data to an overall network security solution; and can be used for SLA verification, accounting and billing purposes. sFlow has been implemented in network switches & routers for many years and is now often implemented in end hosts.
Here are some publicly available sFlow generated graphs from AMS-IX (the sFlow samples are taken from Brocade NetIron routers).
Here is another simple output from sFlow, showing the top talkers in a specific IP subnet.
Short Comparison of sFlow, Netflow and IPFIX
While sFlow was explicitly invented as an open standards based protocol for network monitoring, Netflow was originally developed to accelerate IP routing functionality in Cisco routers (it remains proprietary to Cisco). The technology was subsequently modified to support network monitoring functions instead of providing accelerated IP routing; however, it can exhibit performance problems on high-speed interfaces. Furthermore, sFlow can provide visibility and network statistics from L2 – L7 of the network stack, while Netflow is predominantly used for L3 – L4 (there is now limited L2 support in Netflow but there is still no MPLS support).
Another key difference between the two protocols is that sFlow is a packet sampling technology; while Netflow attempts to capture entire flows. Attempting to capture an entire flow often leads to performance problems on high-speed interfaces, which are interfaces of 10GbE and beyond.
IPFIX is an IETF standards based protocol for extracting IP flow information from routers. It was derived from Netflow (specifically, Version 9 of Netflow). IPFIX is standardized in RFC 5101, 5102, and 5103. As its name correctly implies, IPFIX remains specific to L3 of the network stack. It is not as widely implemented in networking gear as sFlow is.
sFlow and OpenFlow?
There is some recent activity around integrating sFlow with OpenFlow to provide some unique “performance aware” SDN applications. For example, take a look at this diagram:
[Diagram referenced from here.]
In this example, sFlow is used to provide the real-time network performance characteristics to the SDN application running on top of an OpenFlow controller, and OpenFlow is used to re-program the forwarding paths to more efficiently utilize the available infrastructure. Pretty slick, huh? This example uses sFlow-RT, a real-time analytics engine, in place of a normal sFlow collector.
NetIron sFlow Implementation Enhancements
Brocade devices have been implementing sFlow in hardware for many years. This hardware based implementation provides key advantages in terms of performance. The sampling rate is configurable and sFlow provides packet header information for ingress and egress interfaces. sFlow can provide visibility in the default VRF and non-default VRFs. NetIron devices support sFlow v5, which replaces the version outlined in RFC 3176.
In addition to the standard rate-based sampling capability, NetIron devices are capable of using an IPv4 or IPv6 ACL to select which traffic is to be sampled and sent to the sFlow collector. This capability provides more of a flow-based sampling option, rather than just sampling packets based on a specified rate. In addition to sampling L2 and L3 information, sFlow can be configured to sample VPN endpoint interfaces to provide MPLS visibility. Neither Netflow nor IPFIX can provide this type of visibility.
One of the new enhancements to the NetIron sFlow implementation is the ability to provide Null0 interface sampling. Service providers often use the Null0 interface to drop packets during Denial of Service (DoS) attacks. sFlow can now be configured to sample those dropped packets to provide visibility into the DoS attack. This feature is in NetIron Software Release 5.5.
The other new enhancement that I’d like to mention is the ability to now capture the MPLS tunnel name/ID when sampling on ingress interfaces. This feature is coming very soon and will provide additional visibility into MPLS-based networks.
In summary, I hope you gained some additional insight into the advantages of leveraging the network visibility that sFlow provides. One last thing I’d like to correlate to sFlow is Network Analytics. These are complementary technologies which can co-exist together in the same network, while performing different functions. Brocade continues to innovate in both of these areas and I welcome any questions or comments you may have on sFlow or Network Analytics.
As data center networks scale to support thousands of servers running a variety of different services, a new network architecture using the Border Gateway Protocol (BGP) as a data center routing protocol is gaining popularity among cloud service providers. BGP has traditionally been thought of as only usable as the protocol for large-scale Internet routing, but it can also be used as an IGP between data center network layers. The concept is pretty simple and has a number of advantages over using an IGP such as OSPF or IS-IS.
Large-Scale BGP is Simpler than Large-Scale IGP
While BGP in itself may take some heavy learning to fully grok, BGP as a data center IGP uses basic BGP functionality without the complexity of full-scale Internet routing and traffic engineering. BGP is especially suited for building really big hierarchical autonomous networks, such as the Internet. So, introducing hierarchy with EBGP and private ASNs into data center aggregation and access layers down to the top of rack behaves just like you would expect. We’re not talking about carrying full Internet routes down to the top of rack here, just IGP-scale routes, so even lightweight BGP implementations that run on 1RU top of rack routers will just work fine in this application.
The hierarchy and aggregation abilities of an IGP are certainly quite extensive, but each different OSPF area type, for example, introduces different behaviors between routers, areas and how different LSA types are propagated. There’s a lot of complexity to consider when designing large-scale IGP hierarchy, and a lot of information that is flooded and computed when the topology changes. The other advantages of BGP are the traffic engineering and troubleshooting abilities. With BGP you know exactly what prefix is sent and received to each peer, what path attributes are sent and received, and you even have the ability to modify path attributes. Using AS paths you can tell precisely where the prefix originated and how it propagated, which can be invaluable in troubleshooting routing problems.
How it Works
What you basically do is divide the network into modular building blocks made up of top of rack access routers, aggregation routers, and data center core routers. Each component uses its own private ASN, with EBGP peering between blocks to distribute routing information. The top of rack component doesn’t necessarily need to be a single rack; it could certainly be a set of racks and a BGP router.
Petr Lapukhov of Microsoft gave a great overview of the concept at a NANOG conference recently in a presentation called “Building Scalable Data Centers: BGP is the Better IGP”, which goes into a lot more background on their design goals and implementation details. If you’d like to experiment with the network design as Petr describes, the commands for the BGP features on slide 23 for the Brocade NetIron software are:
AS_PATH multipath relax: multipath multi-as (router bgp)
Allow AS in: no enforce-first-as (router bgp or neighbor)
Fast EBGP fallover: fast-external-fallover (router bgp)
Remove private AS: remove-private-as (router bgp or neighbor)
Taking it a Step Further
An alternative that takes the design even further from top of rack down into the virtual server layer for high-density multitenant applications is to also use the Brocade Vyatta vRouter. In this design, EBGP would be run from the data center core at each layer to a virtual server that routes for a set of servers in the rack. This addition gives customers a lot of flexibility in controlling their own routing, for example, if they wanted to announce their own IP address blocks to their hosting provider as part of their public cloud. Customers could also use some of the other vRouter VPN and firewall features to control access into their private cloud.
In addition to using BGP to manage routing information, you can also build an OpenFlow overlay to add application-level PBR to the network. Using the Brocade hybrid port features that enables routers to forward using both OpenFlow rules and Layer 3 routing on the same port, introducing SDN into this network as an overlay is easy. In fact, this is exactly what Internet2 is doing in production on their AL2S (Advanced Layer 2 Services) network to enable dynamically provisioned Layer 2 circuits.
So is BGP better as a data center IGP? I think the design lends itself especially well to building modular data center networks with independent and autonomous modular components that can be built all the way down to the virtual server level. Perhaps you even have different organizations running their own pieces of the network, or servers that you’d rather not invite into your OSPF or IS-IS IGP.
For more information on Brocade’s high density 10 GbE, 40 GbE and 100 GbE routing solutions, please visit the Brocade MLX Series product page.
Today, Brocade announced it strategy to bridge the physical and virtual worlds of networking to enable customers to build an “On-Demand Data Center”. For service providers, an On-Demand Data Center means getting closer to becoming the greatly sought after cloud provider by increasing business agility, reducing complexity and scaling virtualization. In this blog I will focus on the announcement of the new 40 GbE interface module we have added to the Brocade MLX Series to enhance the physical aspects of the data center core that are required as the foundation for the On-Demand Data Center.
In the core of the service provider data center, network operators need to be able to respond in real time to dynamic business needs by delivering applications and services on demand. At the same time, they must contain costs through more efficient resource utilization and simpler infrastructure design. Traditional network topologies and solutions are not designed to support increasingly virtualized environments. With the Brocade MLX 4-port 40 GbE module, in conjunction with Brocade VCS Fabric technology, you can scale the data center fabric and extend across the Layer 3 boundary between data centers. High 40 GbE density with advanced Layer 3 capabilities helps consolidate devices and links needed in the data center core. Large Link Aggregation Group (LAG) capabilities provide capacity on-demand and reduce management overhead. By consolidating devices and simplifying the network, customers can reduce capital expenditures and operational expenditures in terms of power, space, and management savings, minimizing TCO. In addition to massive scalability from the 40 GbE density, the rich feature set of the Brocade MLX 4-port 40 GbE module eliminates the need for additional edge routers by enabling Layer 3 data center interconnect with full featured support for Access Control Lists (ACLs), routing, and forwarding in the data center core
Prior to 2012, optical equipment dominated the 40 GbE market. 40 GbE is now taking off on Ethernet routers and switches, principally in data centers because it helps to bridge the bandwidth and economics gap between 10 GbE and 100 GbE for customers. The market for 40 GbE in high-end routing applications is expected to ramp up quickly, with CAGR from 2013 to 2016 expected to be 125% with a total market size in 2016 of $239M (Source: Dell’Oro, 2012). Similar to 10 GbE, business drivers will be the growth of bandwidth-intensive applications:
The image shows a primary deployment model for the 40 GbE module in the Core of the data center. The high density, wire-speed performance enables 40 GbE connection with the aggregation layer – in this case the Brocade VDX 8770 supporting the VCS Fabric. Also supporting advanced MCT, this new module enables data center cores to scale in in highly resilient and efficient manner.
The MLXe also serves as an ideal border router to interconnect the data center to the WAN – or other data centers. Here 40 GbE or 100 GbE is typically used. The new 40 GbE module is often used, especially where underlying WAN optical infrastructure does not yet support 100G.
There has been lots of recent discussion about Google and AT&T targeting to provide the city of Austin, TX with a 1-gigbit-per-second Internet serv.... While the competitive and innovative spirit should make Austin feel like one of the luckiest towns in the world, I would like to tell you about a metro service provider in Clarksville, Tennessee that already provides its residential and commercial customers with 1 Gigabit Ethernet services to the premise.
CDE Lightband is the leading municipal utility provider of electricity, digital television, Internet and voice to all of the 100 square miles located within the boundaries of Clarksville, TN. They offer their services to approximately over 64,000 customers while 892 miles of power lines and 960 miles of fiber optic cable are maintained. Most distinguishably, CDE Lightband provides a true Active Ethernet network to their customers. This means that each and every one of their residential and commercial customers has their own active Ethernet, Fiber-to-the-Premise port. The value of an Active Ethernet network is that the bandwidth on the connection is not shared, and is thus an effective way of ensuring a 1-Gbps connection to each subscriber. It is certainly a feather in the hat for a service provider of any size.
Brocade is proud to support CDE Lightband’s Active Ethernet project. By using the Brocade NetIron CES series switches, CDE Lightband can sell Gigabit Internet service and provide bandwidth throughput. In the future, CDE Lightband plans to use the 10G ports on the Brocade CES so they can grow the switches into the network as they expand their internal infrastructure.
Like all service providers, CDE Lightband’s top priority is to provide world class performance and reliability. Because of the Brocade CES series switches, CDE Lightband is able to offer their customers a unique and powerful Ethernet services (as exemplified in their Active Ethernet project) and deliver them on pace with their customers’ business and personal requirements. Brocade is very honored to be the backbone of CDE Lightband’s network!
To learn more about the Brocade and CDE Lightband partnership, please watch this video.
I recently returned from IETF 86 and would like to update the folks in this community with a brief synopsis of the event. Overall, it was a very well attended, interactive and relevant event! But I think that’s pretty much the norm these days, particularly with all the interest in SDN related technologies and use cases. I will post a separate blog in our SDN community on the SDN related IETF activities, so please go there for that update. In this blog, I will focus on IETF activities related to service providers.
I’ll start off with the discussion around IPv6 in MPLS networks. While we all know that there has been some interest and IETF standards work in the area of MPLS/IPv6, it has yet to garner much real deployment interest. Techniques for providing IPv6 over IPv4-based MPLS networks, such as 6PE and 6VPE, have solved some of the issues with IPv6 and MPLS. However, it appears the IETF community is now getting behind full IPv6 support in MPLS. This would include native IPv6 LDP and RSVP-TE support. Some folks believe that although full IPv6 MPLS networks may not be needed for another 3-5 years, the IETF community should get on board now and start officially driving this. The MPLS WG will start formally tracking progress in this area, as it’s deemed important work.
Entropy labels to improve load balancing in MPLS networks was briefly discussed and this appears to be a done deal in terms of standards (RFC 6790) and having broad community support and consensus.
TRILL over Pseudo-Wires was discussed in the PWE3 WG. This is cool stuff and appears to have some degree of consensus. This basically would allow a TRILL domain in one data center to have layer-2 connectivity to another TRILL domain in another data center.
A similar topic of VXLAN over L2 VPNs was discussed in the L2VPN WG. This would provide a layer-2 MPLS connection between VXLAN or NVGRE logical overlay networks. This is also a pretty cool use case and this appears to be a needed solution if VXLAN/NVGRE solutions become more widely deployed in data centers. A somewhat related topic was discussed on how Ethernet VPNs (E-VPNs) could be leveraged to provide a data center overlay solution. In this context, E-VPNs are based on MPLS technologies. While this solution revolves around Network Virtualization Overlays, it was discussed in the L2VPN WG due to it leveraging MPLS technologies. This Internet Draft was also discussed in the NVO3 WG.
Interesting work on MPLS forwarding compliance and performance requirements was discussed in the different MPLS WGs. This work intends to document the MPLS forwarding paradigm for the perspective of the MPLS implementer, MPLS developer and MPLS network operator. Very useful work!
In the L3VPN WG, there were quite a few IDs that overlap with the NVO3 WG and data center overlay technologies. The general support for MPLS-based solutions for data center overlay architectures appears to be gathering momentum. From a high level, this does make sense as MPLS VPN technologies provide a logical network overlay in the wide area of service provider networks. As data center overlay architectures evolve, why not leverage this work and experience? I will discuss more on this topic in my SDN community blog.
To wrap up the MPLS activities; there were a number of other MPLS-related developments and enhancements that I won’t go into detail about here. Areas such as P2MP LSPs, special purpose MPLS label allocations, OAM, and additional functionality for advertising MPLS labels into the IGP (like an enhanced “forwarding adjacency”) were all discussed and are progressing at various stages though the IETF standards process.
Another WG that generated a fair amount of activity and interest is PCE. This is also an area of IETF work that is somewhat related to the SDN solution space. This WG is focused on how to enhance traffic-engineering decisions in MPLS networks. PCE functionality would “recommend” traffic-engineered LSPs for the network but would not be responsible for the actual instantiation of those LSPs into the network. That would be done by another function; and is deemed outside the scope of the PCE WG.
The WG agreed to make the PCE MIB “read-only”. This makes sense since the MIB is not a good place to implement PCE functionality. They also discussed P2MP LSPs, Service aware LSPs and even the support of wavelength switched optical networks. They also agreed that “Stateful” PCE was indeed in scope and in the charter.
Overall, nothing really ground breaking to report on in the area of routing activity at this IETF. One topic worth a mention is the North-bound distribution of link-state and TE routing information using BGP. This area is somewhat related to the SDN solution space, as it could provide upper layer applications (such as ALTO or PCE) the knowledge of link-state topology state from the network. This would allow those applications to make more intelligent traffic-engineering decisions.
Another area of routing that is interesting to mention is having the ability to make routing decisions based upon additional link-state metric information; such as latency, jitter and loss. This seems like a very logical evolution of IP routing.
And to wrap up the routing activity; as expected, the security of inter-domain routing continues to generate lots of interest. It was interesting that immediately after the IETF, there was a paper published by Boston University on the security implications of the Resource Public Key Infrastructure (RPKI) being discussed on the SIDR mailing list. This paper seems to re- ignite some of the controversy around secure routing.
I2RS was also very well represented and generated lots of interesting dialogue and debate.
This WG is fairly new. The primary goal of this WG is to provide a real-time interface into the IP routing system. This interface will not only provide a configuration and management capability into the routing system, but will also allow the retrieval of useful information from the routing system. Quite a bit of the discussion was centered around what type of state information needs to be injected into the routing system, what type of state information should be extracted from the routing system, and interesting enough, what specifically is the “routing system”. The routing system is generally understood to be the Routing Information Base (RIB) in IP routers but there was a good amount of debate on exactly what constitutes a RIB, what information does it hold and what might the interface to this RIB look and behave like. It appears this WG may have taken a step back to re-group and get more focused before moving on to solutions too rapidly.
There were five use case drafts that were presented and discussed. So, while this WG may have taken a step back to more clearly understand and define the problem space, they are also continuing to move forward with relevant use case definitions and then onward to solutions.
So, that wraps up this short update on the IETF 86 SP related activities. I should mention before closing that the I2RS WG intends to hold an interim meeting after the ONS event in April, so if you are attending the ONS event you may want to attend the I2RS interim meeting as well.
I’d like to follow Greg’s great blog from last week with a related topic. Like his blog, this blog will be focused on router hardware (unlike my previous blogs which were NetIron software related). The topic at hand is a brief discussion of the differences and the pros/cons of FPGA and ASIC technology. I’ll also briefly touch on the advantages of each of these technologies as they apply to high-end IP routers.
FPGAs (Field Programmable Gate Arrays) are specialized chips that are programmed to perform very specific functions in hardware. An FPGA is basically a piece of programmable logic.The first FPGA was invented in 1985, so this technology has been around for quite some time. Rather than executing a function in software, the same function can be implemented in an FPGA and executed in hardware. One can think of an FPGA as “soft-hardware”, since it can be reprogrammed after manufacturing. How many of you remember the bygone days of software-based IP routers? If you do, then you should also remember how poorly the Internet performed at that time! Performance was poor in software-based routers due to the fact that a centralized CPU executed all functions, both the control/management plane functions and the data plane functions of the router. Today, all modern routers execute the data plane functions in hardware; and more frequently, some vendors are moving certain control plane functions into the router hardware as well. The Bi-Directional Forwarding (BFD) protocol is one example of this; where portions of the BFD keep-alive mechanisms are implemented in the line card of the router.
While FPGAs contain vast amounts of programming logic and millions of gates, one thing to note is that there is some programming logic in an FPGA that is not used for the “customer facing” or "mission specific" application or function. In other words, not all the logic in an FPGA is designed to be directly used by the application the FPGA is providing for the customer. There are additional gates needed to connect all the internal logic that is needed to make it programmable; so an FPGA is not fully optimized in terms of “customer facing” logic.
Now, what I find interesting is that some people will still claim that FPGAs cannot scale to the speeds that are required in the today’s Internet. However, Brocade has proven this claim to be quite false and has been shipping line-rate, high-end performance routers using FPGAs for over 10 years. As shown in the line card diagram in Greg’s blog, an FPGA in this context is really a programmable network processor.
One great advantage of an FPGA is its flexibility. By flexibility, I’m referring to the ability to rapidly implement or reprogram the logic in an FPGA for a specific feature or capability that a SP customer requires. When a networking vendor has a new feature that it wants to implement, the vendor may have the choice of deciding whether to put the feature in software or hardware. This is not always the case; for example, OSPF needs to be run in the control plane of the router and cannot be implemented in hardware. The question of whether to implement something in software or hardware basically comes down to a decision of flexibility versus scalability (and cost is always part of that decision process, as one would expect). Implementing something in software usually results in a rapid implementation timeframe, but often at the detriment to performance. As usual, there is always a trade-off to be made. However, if the vendor supports programmable network processors, they can implement the feature in hardware with no detriment to performance. While it takes more time to get the feature into an FPGA rather than implementing it in software, the time-to-market timeframe is still considerably less than doing a similar feature in an ASIC. The real advantage of this becomes evident with deployed systems in a production network. When a customer requires a feature that needs to be implemented in the forwarding plane of a router, once this feature is developed by the vendor the deployed systems in the field can be upgraded to use the new feature. This requires only a software upgrade of the system; no new hardware or line cards would be required. The routers’ software image contains code for the FPGAs, as well as the code for the control and management plane of the router.
Back to the performance question: Industry has shown that high-end FPGAs are growing in density while handling higher-speed applications and more complex designs. Furthermore, if you look at the evolution of FPGAs over the years, they follow Moore's Law just like CPUs have been doing in terms of the amount of logic that you can implement into them. Recent history has shown that FPGA development in terms of density is on an exponential growth curve.
FPGAs can also be used for developing a “snapshot” version of a final ASIC design. In this way, FPGAs can be re-programmed as needed until the final specification is done. The ASIC can then be manufactured based on the FPGA design.
While ASICs have very high density in terms of logic gates on the chip, the result of higher scalability in terms of the same power metric can give ASICs a competitive edge over an FPGA. One thing to note is that an ASIC is designed to be fully optimized in terms of gates and logic. All the internal structures are used for customer facing or mission specific applications or functions. So, while an ASIC may consume more power per unit die size than an FPGA, this power is amortized over a higher density solution; and hence, provides better power efficiency.
Compare/Contrast of FPGA-ASIC
So, FPGAs and ASICs are both specialized chips that perform complex calculations and functions at high levels of performance. FPGAs, however, can be re-programmed after fabrication, allowing the line card's feature set to be upgraded in the field after deployment. Being able to upgrade the data plane of a deployed router extends the useful lifespan of the system; which correlates to extended investment protection. Since an ASIC is not re-programmable, an ASIC-based line card cannot be upgraded in the field. This is a huge differentiator between the two technologies.
One excellent real-world example of this is when Brocade introduced support for 64 ports in a single LAG. This is industry leading scale (64 10GbE ports in a single LAG!) and since this functionality is implemented in the forwarding plane of the line card, it required reprogramming the Brocade network processor. While this type of capability is in the hardware of the router, it was implemented with a system software upgrade and no hardware needed to be replaced.
There are network scenarios or use cases where it makes more sense to have an FPGA-based product and there are use cases when it makes more sense to have an ASIC-based product. For example, a SP may determine that a high density solution is more important than a solution that provides quicker feature velocity and, thus, may choose an ASIC-based product. ASIC-based line cards are often denser in terms of numbers of ports and the cores of SP networks typically do not require high feature velocity. Most of the feature velocity in today’s SP networks is at the edge of the network (ie: at the PE router) or in the data center, where innovation is currently happening at a rapid pace. The general flexibility of an FPGA results in time-to-market advantages for feature implementation and soft-hardware bug fixes.
For smaller applications and/or lower production volumes, FPGAs may be more cost effective than an ASIC. The non-recurring engineering (NRE) cost of an ASIC can run into the millions of dollars. Conversely, in high volume applications the front-end R&D costs of an ASIC are offset by a lower cost to manufacture and produce. For example, in high-end IP core routers, ASIC-based line cards are more economical due to the lower manufacturing cost, combined with the higher port density of the line card that ASICs can provide.
As costs related to ASIC development are increasing, some recent trends may suggest that FPGAs could be a better alternative even for high volume applications that traditionally used ASICs. It is unclear whether this trend is indeed sustaining or a somewhat temporary aberration.
To summarize the primary differences between FPGA and ASIC based line cards; at the highest level it basically comes down to a scalability versus a flexibility question (again, with cost a large contributing factor). ASICs are advantageous when it comes to high port density applications. FPGAs are advantageous when it comes to feature velocity with a shortened time-to-market requirement. In high end core routers, high density ASIC-based line cards can provide higher density at a lower cost than FPGA-based line cards. So, it’s based upon the use case and network application to determine which type of technology would be favored over the other.
As usual, any questions are comments are welcome!
It’s hard to believe that Ethernet is turning 40 this year, isn’t it? Since its conception by Bob Metcalfe and the team of engineers at XEROX PARC in the 1970s, Ethernet technology has continued to evolve to meet the increasing bandwidth, media diversity, cost, and reliability demands of today’s networks. The next Ethernet evolution has officially started, and I'm excited to follow the latest developments on this new technology that will enable networks to support even higher capacities.
“Here is more rough stuff on the ALTO ALOHA network.” Memo sent by Bob Metcalfe on May 22, 1973.
I wrote about 400 GbE in my blog recently as the next likely Ethernet speed, and now it’s official. Last week at the March 2013 IEEE 802 Plenary Session, 400 GbE became an official IEEE 802.3 Study Group that will start work on developing the new standard. Though 100 GbE is only a few years old, it’s important that we start working on the next speed now, so that we have the technology shipping when there is demand from network operators to deploy higher speed Ethernet.
The 400 Gb/s Ethernet Study Group is starting with strong industry consensus this time, which will enable the standard to be developed faster than before. The 400 GbE Call-For-Interest presentation was given last week to measure the interest in starting a 400 GbE Study Group in the IEEE. Based on the hard work of the IEEE 802.3 Ethernet Bandwidth Assessment (BWA) Ad Hoc and the IEEE 802.3 Higher Speed Ethernet (HSE) Consensus Ad Hoc, there was clear consensus on the direction the industry should take on the next Ethernet speed. The straw polls and official vote on the motion to authorize the Study Group formation were all in favor with a few abstains, which showed a high degree of consensus from the individuals and companies represented. This was not so with the last Ethernet speed evolution, which was simply called the Higher Speed Study Group (HSSG) when it was formed. First, the HSSG had to analyze the market and come up with feasible higher speed solutions before even deciding on the speed. This made the standardization process much longer as the HSSG debated 40 GbE and 100 GbE, and eventually standardized both speeds for different applications. Since we are already starting the 400 Gb/s Ethernet Study Group with a clear speed objective in mind, the standardization process should be much faster. This means the Study Group could have the 400 GbE standard finished in 2016 with the first interfaces available on the market soon after.
Stay tuned for more updates as we follow the road to 400 GbE! If you happen to be in the Bay Area next week, check out the Ethernet 40th Anniversary Celebration at the Ethernet Technology Summit on Wednesday evening at 6 pm, April 3, 2013.
While considering what to write about for this blog, after my previous blog about a really cool NetIron 5.3 feature, I thought I’d stick with that trend for now and talk about another highly anticipated 5.3 feature. This one also happens to be MPLS-based and it’s often a required SP capability within an MPLS-based solution. It’s called Automatic Bandwidth Label-Switched Paths, or Auto-BW LSPs for short.
The Good News
As we know, RSVP-TE based networks are capable of considerable optimizations in terms of bandwidth reservations and traffic engineering. Operators can “plumb” their networks more intelligently, by reserving LSP bandwidth onto specific paths within their network for certain traffic types or overlay services. This makes their networks run more efficiently and with better performance. Operators and network managers like this, as they are getting the most out of the network. In other words, they are “getting their monies worth”.
The Not So Good News
While bandwidth reservations and traffic engineering provide great capabilities in MPLS networks, oftentimes the configured bandwidth reservations turn out to be less than optimal. In other words, it’s great for an operator to be able to say “for this LSP between these two endpoints I want to reserve 2.5 Gbps of bandwidth” and then make that happen in the network. The operator knows that the topology can support the 2.5 Gbps of capacity due to capacity planning exercises or from offline MPLS-based TE tools. Cool. (btw: It may be desirable to integrate sFlow data into the capacity planning capability or maybe even into an offline MPLS-based TE tool, but that’s a topic for a future blog.)
But what if there is a sustained increase in traffic, well above the reserved 2.5 Gbps, for that service? How are those surges handled by the LSP? Or what if the actual sustained traffic load is only 1.5 Gbps? In that case, no other LSP may be able to reserve the “extra” 1 Gbps of capacity since it is already reserved for that specific LSP. Now the operators’ network is plumbed in a less than optimal fashion. They are no longer “getting their monies worth” out of the network.
The (now) Gooder News
Here is where Auto-BW LSPs come onto the scene to save the day and make the operator a hero (again).
Auto-BW LSPs can solve both problems mentioned; handling a sustained surge in traffic, above what was previously planned for & being able to use “extra” capacity that is actually available but because it’s allocated to an LSP, it may not be available to be reserved by other LSPs.
Overview of Auto-BW
In its simplest definition: auto-bandwidth is an RSVP feature which allows an LSP to automatically and dynamically adjust its reserved bandwidth over time (ie: without operator intervention). The bandwidth adjustment uses the ‘make-before-break’ adaptive signaling method so that there is no interruption to traffic flow.
The new bandwidth reservation is determined by sampling the actual traffic flowing through the LSP. If the traffic flowing through the LSP is lower than the configured or current bandwidth of the LSP, the “extra” bandwidth is being reserved needlessly. Conversely, if the actual traffic flowing through the LSP is higher than the configured or current bandwidth of the LSP, it can potentially cause congestion or packet loss. With Auto-BW, the LSP bandwidth can be set to some arbitrary value (even zero) during initial setup time, and it will be periodically adjusted over time based on the actual bandwidth requirement. Sounds neat, huh? Here’s how it works…
First, determine what the desired sample-interval and adjustment-interval should be set at. The traffic rate is repeatedly sampled at each sample-interval. The default sampling interval is 5 minutes. The sampled traffic rates are accumulated over the adjustment-interval period, which has a default of 24 hours. The bandwidth of the LSP is then adjusted to the highest sampled traffic rate amongst the set of samples taken over the adjustment-interval. Note that the highest sampled traffic rate could be higher or lower than the current LSP bandwidth.
That’s basically it in a nutshell, but there are other knobs available to tweak for further control (as expected, operators want more knobs to tweak).
In order to reduce the number of readjustment events (ie: too many LSPs constantly re-sizing), we allow the operator to configure an adjustment-threshold. For example, if the adjustment-threshold is set to 25%, the bandwidth adjustment will only be triggered if the difference between the current bandwidth and the highest sampled bandwidth is more than 25% of the current bandwidth.
As mentioned, the adjustment-interval is typically set pretty high, at around 24 hours. But a high value can lead to a situation where the bandwidth requirement becomes suddenly high but the LSP waits for the remaining adjustment-interval period before increasing the bandwidth. In order to avoid this, we allow the operator to configure an overflow-limit. For example,if this value is set to 3, the LSP bandwidth readjustment will be triggered as soon as the adjustment-threshold is crossed in 3 consecutive samples.
The feature will also allow the operator to set a max-bandwidthandamin-bandwidthvalue to constrain the re-sizing of an LSP to within some reasonably determined bounds.
It is also possible to simply gather statistics based on the configured parameters, without actually adjusting the bandwidth of an LSP. This option involves setting the desired mode of operation to either monitor-only or monitor-and-signal.
The Auto-BW feature also provides a template-based configuration capability, where the operator can create a template of auto-bandwidth parameter values and apply the templates on whichever path of an LSP that needs the same configuration or across multiple LSPs.
This example below shows three adjustment-intervals on the horizontal axis and traffic load of the LSP on the vertical axis. After each adjustment-interval, the LSP bandwidth is automatically adjusted based upon the sampled traffic rate. The diagram also shows where the adjustment-threshold is set and exceeded by the actual traffic rate, which then results in the bandwidth adjustment. The red line is the bandwidth of the LSP, after being adjusted at each adjustment-interval.
In the example above, each adjustment-interval has three sample-intervals. The following graphic shows the relationship between the sample-interval and the adjustment-interval.
Auto-BW Solves Real Problems
Here is one simple but real-life scenario where Auto-BW can prevent packet loss. Consider the topology below.
In this topology there are two LSPs between PE1 and PE3; each with 400 Mbps of reserved bandwidth and each with actual traffic loads approaching the 400 Mbps reservations. The entire topology consists of 1 GbE links so both of these LSPs can share any of the links since their combined bandwidth reservation is 800 Mbps. Constrained Shortest-Path First (CSPF) calculations put both LSPs onto the PE1-PE2-PE3 path.
However, over time LSP2's actual traffic load grows in size to 650 Mbps. Now the combined traffic load of both LSPs exceeds the capacity of a 1 GbE link and packet loss is now happening on the PE1-PE2-PE3 path. RSVP, specifically the CSPF algorithm, cannot take the additional “actual” traffic load into account so both LSPs remain on the same path. This is not good. The reason for this is the Traffic-Engineering Database (TED) that CSPF uses to determine paths in the network is not updated by actual traffic loads on links or in LSPs. This is just how RSVP-TE works.
When Auto-BW is enabled, both LSPs are sampled to determine their actual traffic loads. After the adjustment-interval, LSP2 is re-sized to 650 Mbps. Now both LSPs can no longer share the same path as the CSPF algorithm will compute that the combined bandwidth of the LSPs now exceeds a single 1 GbE link. So, the result is that CSPF will look into the TED for a new path in the network from PE1 to PE3 that meets the bandwidth requirement of LSP2 and will traffic engineer that LSP onto the PE1-PE4-PE5-PE3 path.
The operator is now a hero (again) because the network is back to working at its maximum efficiency and performance levels.
As usual, any questions are comments are welcome! Also, if there are future topics related to MPLS that you would like to see a blog about, please post them in the comments and we will see what we can do.
I’d like to continue a previous discussion about Brocades Multi-Chassis Trunking (MCT) technology. Please see the earlier blog: MCT with VPLS. The MCT w/VPLS capability was part of NetIron Software Release 5.3. In NetIron Software Release 5.4, we added a powerful enhancement to provide Multicast over MCT.
A diagram of this capability is shown below. In the diagram, there are two MLXe routers who are MCT peers. They have multicast receivers downstream and multicast sources upstream.
The diagram shows that the MCT Cluster Client Edge Ports (CCEPs) now have the ability to support the Internet Group Management Protocol (IGMP) and the Protocol Independent Multicast (PIM) protocol. As you recall, the CCEPs are the MCT customer facing edge ports. The diagram shows multiple host receivers behind two layer-2 switches, who are the MCT clients, and the host receivers are sending IGMP join requests toward the network. IGMP is used by hosts to establish and manage their multicast group membership. The MCT client layer-2 switches are directly connected to the CCEPs of the MCT cluster. Each layer-2 switch is doing standard link aggregation (LAG) to connect to both of the MLXe routers. As with all MCT configurations, the client layer-2 switches are unaware that they are connected to two MLXe routers; this is the active/active redundancy that MCT provides.
Both of the MCT peers will receive IGMP join requests and will subsequently send PIM joins toward the multicast Rendezvous Point (RP) or multicast source, depending on whether PIM-SM (*, G) or PIM-SSM (S, G) is being used. So, PIM runs on the network facing interfaces of the MCT peers, including the Inter-Chassis Link (ICL). The MCT ICL is also used to synchronize the IGMP membership state between the MCT peers. The result is that both of the MCT peers will install the correct multicast membership state. The diagram shows a few of the scenarios that are possible; where sources can be somewhere inside the IP network or directly attached to either MLXe router. However, the sources and receivers can actually be reversed such that sources are behind the MCT client layer-2 switches and the host receivers are either somewhere in the IP network or directly attached to either MLXe router. All variations are supported.
A hashing algorithm determines if the active multicast outgoing interface (OIF) is a local CCEP interface or the ICL. As shown in the diagram below, multicast traffic can arrive on both MLXe routers from the source but the MLXe with the local forwarding state is the only one that forwards traffic to the host receiver.
Since both MCT peers are properly synchronized, forwarding is performed as expected on the multicast shortest-path tree.
Some of the benefits of this compelling enhancement are:
As an example of a possible failure scenario; If CCEP2 fails, the MCT peers will remain synchronized such that the redundant MCT peer immediately takes over as the active forwarding device.
So, you can see that this is a very powerful feature as it provides for an active/active redundancy capability while maintaining the optimal multicast forwarding tree under failure scenarios. This is no easy feat!
Continue to stay tuned to this page for additional NetIron enhancements, as Brocade continues to lead the industry in innovation!
This is a great opportunity for me to introduce a really cool and highly anticipated feature that is part of the Brocade NetIron 5.3 Software Release. The official release date for this software is sometime next week, but because you are part of this awesome SP community, you get a sneak peak!
While 5.3 contains many new innovative features that our SP customers have been clamoring for, I thought I’d pick one in particular and write a bit about it here. The feature is Multi-Chassis Trunking integration with Virtual Private LAN Service, or MCT w/VPLS for short.
First, a short background refresher on what problem MCT solves. (BTW: Brocade has been supporting MCT for well over a year now.)
Brocade developed MCT to provide a layer-2 “active/active” topology in the data center without the need to run a spanning-tree protocol (STP). STP has traditionally been used to prevent layer-2 forwarding loops when there are alternate paths in a layer-2 switched domain. However, STP has its issues in terms of convergence, robustness, scalability, etc. Orthogonal to STP, link aggregation (IEEE 802.3ad) is also often deployed to group or bundle multiple layer-2 links together. The advantages of link aggregation are:
So, MCT leverages standards-based link aggregation but is capable of providing this “across” two switch chassis instead of just one chassis. This is shown below.
As you can see, there are two chassis that act like a single logical switch. This is called an MCT pair or cluster. The devices on either side of the MCT logical switch believe they are connected to a single switch. Standard LAG is used between these devices and the MCT logical switch. The advantage of doing this is that now both switches in the MCT cluster are functioning at layer-2 in an “active/active” manner. Both can forward traffic and if one chassis has a failure, standard failover mechanisms for a LAG bundle take effect. In addition, there are no layer-2 loops formed by an MCT pair so no STP is needed!
Now, for a short background refresher on what VPLS provides. (BTW: Brocade has been supporting VPLS for many years now.)
VPLS provides a layer-2 service over an MPLS infrastructure. The VPLS domain emulates a layer-2 switched network by providing point-to-multipoint connectivity across the MPLS domain, allowing traffic to flow between remotely connected sites as if the sites were connected by one or more layer-2 switches. The Provider Edge (PE) devices connecting the customer sites provide functions similar to a switch, such as learning the MAC addresses of locally connected customer devices, and flooding broadcast and unknown unicast frames to other PE devices in the VPLS VPN.
MCT with VPLS
Very frequently, a customer network needs to provide layer-2 connectivity between multiple data centers-- to enable VM mobility, for instance. The MCT w/VPLS feature I’m describing provides this type of connectivity in a redundant and high-available fashion. MCT provides the “active/active” layer-2 connectivity from the server farm or access layer to the core layer of the data center. The customer then leverages VPLS on the core layer data center routers to transport the layer-2 Ethernet frames between data centers. This is shown below.
In the diagram above, the CE switch uses a standard LAG to connect to the redundant MCT cluster. The same NetIron routers that form the MCT cluster are also configured to support VPLS to connect to the backbone network. So, the connection from the CE switch in one data center to a remote CE switch in another data center is a layer-2 service. VM mobility between the data centers is now provided in a redundant end-to-end fashion.
Fast failover in the VPLS network is provided by using redundant Pseudo-Wires (PWs), based on IETF Internet Draft <draft-ietf-pwe3-redundancy-03>. As shown below, each PE router signals its own PW to the remote PE. These local PE routers determine, based on configuration, which PE signals an active PW and which PE signals a standby PW. There is also a spoke PW signaled between the PE routers. In the case of the active PW failing, the primary PE router signals to the secondary PE router to bring up its standby PW. This failover is provided in a rapid manner.
The benefits of this solution are:
So, as you can see this is a really awesome capability for SPs who need to integrate their data center infrastructure with their MPLS/VPLS backbone network. We expect this solution to become a very common data center network architecture going forward for providing inter-data center layer-2 connectivity. I should also note that this solution works with Virtual Leased Line (VLL), in addition to VPLS. And, on top of that, it integrates with Ethernet Fabrics in the data center extremely well!
Stay tuned to this forum for more blogs like this.
Last year was another exciting year for 100 GbE as we saw several new technical developments and large deployments by service provider, data center, research and HPC network operators. Here's a quick recap of the highlights for 2012. AMS-IX, our biggest 100 GbE customer and one of the biggest 100 GbE networks in the world, upgraded their 10 GbE core to a 100 GbE core with over 90 x 100 GbE ports in their backbone alone for a capacity of over 7.8 Tbps. The IEEE 802.3ba standard for 40 GbE and 100 GbE, now over 2½ years old, was added to the latest IEEE 802.3-2012 "Standard for Ethernet". 2nd generation 100 GbE projects in the IEEE P802.3bj and P802.3bm Task Forces are in progress that will lower cost and increase density. We’re now well underway to the next evolution of 100 GbE technology and even to the next speed of Ethernet, 400 GbE.
One trend that I’ve noticed among service providers is that 100 GbE peering at IXPs (Internet Exchange Points) is on the rise. We saw a lot of 100 GbE deployments primarily in core networks over the past couple of years, and now 100 GbE peering is taking off too. Several IXPs around the world, most of whom are Brocade customers, have announced the availability of 100 GbE peering ports or the intent offer them this year: AMS-IX (Amsterdam), DE-CIX (Frankfurt), JPIX (Tokyo), JPNAP (Tokyo), LINX (London), Netnod (Stockholm) and NIX.CZ (Prague). AMS-IX for example has deployed three 100 GbE customer ports already, and has six more on order that are expected to go live in the next several weeks. They will also have the first customer 2 x 100 GbE LAG, which will upgrade a 12 x 10 GbE LAG.
The motivation for 100 GbE peering is obvious: to reduce the number of 10 GbE LAGs that connect to an IXP for cheaper and simpler peering. 10 GbE LAG is a great solution but when you consider the port costs, cross connect costs, management and troubleshooting costs, etc. it does start to add up. Costs are different for every network operator as all networks are different, but in general 100 GbE starts to make sense when 10 GbE LAGs exceed six to seven links. Incidentally it also made sense to upgrade to a DS3 when a link exceeded six to seven inverse-multiplexed DS1s when I was a network engineer at MindSpring in the late 1990s, so there is some strange commonality in that number of links. AMS-IX’s 100 GbE port price for example is €9000/month, which is six times the 10 GbE port price of €1500/month.
There is another motivation for 100 GbE peering that is not so obvious too, and this demand comes from IXP resellers. IXP resellers are a relatively new development in the peering industry that enables service providers to peer remotely from anywhere in the world through a reseller port. Until recently, service providers were required to have a physical presence at an IXP in order to peer, because IXPs do not offer long haul transport services. Now IXP resellers, in partnership with an IXP, can resell peering ports remotely over their network to their customers. Remote peering capacity demand is what’s driving these 100 GbE ports. In order for a reseller to offer a high capacity service to their customer, say for example 20 Gbps or 40 Gbps, their own peering port to the IXP has to have the capacity available. Deploying a 100 GbE port to the IXP gives a reseller both the capacity and the flexibility to offer more capacity on demand, without having to constantly manage 10 GbE LAGs.
So, expect more announcements from IXPs about 100 GbE ports this year as 100 GbE peering goes mainstream in 2013.
Acknowledgements: I’d like to thank Henk Steenman, AMS-IX CTO, for his valuable insight and interesting data on 100 GbE peering.
There are some interesting developments in the Research & Education Networks (RENs) space. While there is continued interest and innovation in the REN space around OpenFlow and SDN, there are some related developments in terms of network architecture. After a brief overview of one such interesting development, I will then relate this development back to Brocade.
To start with, I’d like to describe an emerging network architecture called a “Science-DMZ”; which basically moves the high-performance computing (HPC) environment of a research & education campus network into its own DMZ. The reference architecture looks something like this:
As the diagram shows, the traditional “general purpose” campus network sits behind one or more security appliances, which are typically stateful firewalls. This DMZ, or perimeter network, protects the internal network, systems and hosts on the campus network from external security threats. The research and science HPC environment also traditionally connected into the same campus network; so it was also behind the DMZ firewall. This presented some challenges to the HPC environment in terms of data throughput (eg. TCP performance), dynamic “ad-hoc” network connectivity, and general network complexity.
The concept of a Science-DMZ emerged where the connectivity to the HPC environment is moved to its own DMZ; in other words, this environment is no longer connected behind the campus DMZ and firewalls. It now sits on a network that is purposely engineered to support the high performance HPC requirements. As the diagram shows, the science and research enviornment is now connected to a Science-DMZ switch, which in turn connects to a Science-DMZ border router. Access control lists (ACLs) in the border router are leveraged to maintain security of this HPC environment. In addition to simpler access control mechanisms, when a scientist or researcher needs to set up a logical connection to another scientist or researcher to share data, the HPC network can be directly provided that connectivity with provisioning in the border router. For network performance testing and measurement, the perfSONAR tool is included in the reference architecture.
The Science-DMZ concept emerged out of work from the engineers at Energy Sciences network (ESnet). Please take a look at their website for additional details on this architecture. As I have explained, the idea here is pretty simple: to allow the local HPC environment to have better connectivity to other research & education networks by putting it on its own DMZ. The external connectivity is often provided via the national Internet 2 backbone, or it could be provided via a regional REN backbone. To deliver this type of high performance connectivity, there are some hard requirements in terms of scale, performance and feature set of the Science-DMZ Border Router. This is where Brocade enters the conversation.
The hard requirements for this border router are:
• Must be capable of linerate 100GbE, including support for very large, long-lived flows
• Must support pervasive OpenFlow & SDN, for ease of provisioning and innovative applications
• Must support deep packet buffers to handle short data bursts
• Must support linerate ACLs to provide the security mechanisms needed, without impact to data throughput or performance
The Brocade MLXe high-performance router uniquely fits the bill of these requirements! As of software version 5.4, which started shipping in September of this year, the MLXe supports OpenFlow v1.0 in its GA release. The OpenFlow rules are pushed into hardware so the MLXe maintains its high forwarding performance; as it does with IPv4, IPv6 and MPLS forwarding. The largest MLXe chassis can scale to 32 ports of 100GbE or 768 ports of 10GbE, possesses deep packet buffers to handle bursty traffic and performs ACL functions in hardware.
In summary, the Science-DMZ architecture has emerged to solve some of the performance challenges for HPC environments and this reference architecture includes innovative features such as OpenFlow & SDN. The Brocade MLXe platform possesses the unique performance, functionality, and feature set that is required to perform the role of the Science-DMZ border router.
Stay tuned to this space for additional emerging developments in the research & education network arena.
I want to give you a quick overview and update on the industry’s progress toward 400 GbE as the next Ethernet speed. Though 100 GbE is only two years old, it’s important that we start working on the next speed now, so that we have the technology shipping when there is demand from network operators to deploy higher speed Ethernet. The Call for Interest (CFI) to start the 400 GbE Study Group that will work on defining a new Ethernet standard was just announced yesterday, and is scheduled to be held on March 18, 2013 at the next IEEE Plenary meeting.
Here’s a little history on how we chose 400 GbE as the next Ethernet speed. First, the IEEE 802.3 Ethernet Bandwidth Assessment (BWA) Ad Hoc was formed in 2011 to evaluate future Ethernet wireline bandwidth needs. The BWA gathered input from the industry so that we would have accurate bandwidth growth data and requirements for the next Ethernet speed. The full report was released in July, and found continuing growth of bandwidth demands in core and transport layers beyond 100 GbE. If you are just interested in a summary and overview of the findings, then have a look at Scott Kipp’s NANOG56 presentation from last month. Next, the IEEE 802.3 Higher Speed Ethernet Consensus (HSE) Ad Hoc first met in July to develop consensus on the next speed of Ethernet based on the BWA data. The November IEEE Plenary meeting was just held a couple of weeks ago, where the HSE Ad Hoc made progress on the draft 400 GbE CFI presentation.
Why Not TbE?
I’d really love for us to build TbE! But, in order to make TbE economically feasible the cost per bit needs to be at or below the cost of 100 GbE. This means it would make sense for us to reuse current 100 Gbs technology, which implies a TbE architecture using 40 x 25 Gbps signaling lanes. Unfortunately, reusing 25 Gbps signaling means the resulting size of the pluggable media module, and the large amount of interface signals would simply be impractical to develop. Several good presentations were given at the IEEE HSE Consensus Ad Hoc meeting in September about why we should work on 400 GbE now and defer TbE for a few years. There are a couple of alternatives to 25 Gbps signaling, such as using advanced multilevel or phase modulation signaling, but these are still immature technologies that need more development before we can get the performance and low-cost manufacturing needed for volume production. Higher signaling rates will make TbE more feasible, but this technology isn’t expected to be available for the next several years.
As the 400 GbE CFI is now scheduled for March 2013, it means we will have the 400 GbE standard in mid-2015 at the earliest. It’s likely that the first generation of 400 GbE will use 16 x 25 Gbps signaling and that the first interfaces will be available in the 2016 timeframe. The questions that still need to be answered are the physical layer specifications for reaches and media, and this is what the Study Group will start working to define first. As we had for 100 GbE, the interfaces for 400 GbE will use a pluggable media module which gives network operators the most flexibility and choice. It’s likely that the 400 GbE media module will be called CDFP, which is short for “CD (400) Form-factor Pluggable”. As 400 GbE evolves with faster signaling technology, the second generation is expected to use 8 x 50 Gbps signaling which the Optical Internetworking Forum is already beginning to define. The third generation of 400 GbE is expected to use 4 x 100 Gbps signaling which has more advanced electrical and optical signaling technology that is being worked on in labs today. These key 100 Gbps signaling technologies will also be the building blocks for TbE and aren’t expected until after 2020. Stay tuned for more updates as we follow the road to 400 GbE!
There is much discussion these days in various industry forums and conferences about whether OpenFlow will replace MPLS. I guess this comes historically from the fact that ATM replaced Frame Relay (FR), and MPLS then replaced ATM. And although FR was primarily a WAN technology, ATM (who remembers LANE?) and MPLS (ala, VPLS) was/is also deployable in the LAN.
And although I’m specifically mentioning OpenFlow, I’m really thinking more broadly in terms of SDN. So, the question then becomes: Will SDN replace MPLS at some point in the future?
My personal perception is that early in the OpenFlow evolution there were some folks who thought OpenFlow will indeed replace MPLS at some point. MPLS was deemed “too complex”, amongst other arguments against MPLS. As I’ve witnessed the evolution of MPLS over the last decade, it has indeed become more complex. Becoming “too complex”; however, is an argument I don’t think I fully support. Here is my reasoning.
At one point in time, ATM solved many problems and had many advanced features. But as the technology matured, was more widely deployed and became more feature rich, it evolved into a very complex technology. I remember going to training course after training course to become fully proficient in ATM technologies. Then when I saw how ATM was being adapted into the LAN infrastructure with LANE (yuck!), I think that was my turning point. About that same timeframe, MPLS was being developed and talked about in various industry forums and conferences. It looked simple, it looked “kind of” like ATM in terms of virtual circuits (LSPs in MPLS speak), and it looked like it was starting to gain industry support. Fast forward a decade or so and MPLS is now very widely deployed but as it has matured in terms of features and functionality, it has become more complex. And, ATM is dead. So, some folks are now saying, “we need something else, something simpler than MPLS because it has become too complex”. Enter from stage left, OpenFlow! So here we are…
SDN solutions using OpenFlow, from a high level, can provide some of the basic machinery in terms of forwarding packets as MPLS does (or IP, for that matter). Distributed network routing and signaling protocols ultimately create state to populate the forwarding information base (FIB) of a router or switch, and a centralized SDN application using OpenFlow could also populate the FIB of a router or switch. This is why, I believe, that some folks think SDN with OpenFlow could indeed replace MPLS. And here is where I disagree somewhat. Sure, it appears possible and yes there is at least one widely publicized production WAN deployment that I know of (there could be more) where the network switching devices receive their FIB state from a centralized SDN application using OpenFlow. But one must realize that to fully replace all the required features and functionality of MPLS, OpenFlow and SDN will need to evolve to offer those same features and functions. Would adding those features and functions make OpenFlow and SDN as complex in the future as MPLS is today? If so, what will the industry have gained? So, would we be just moving the complexity problem somewhere else?
I believe the industry is beginning to re-appreciate all that MPLS provides and is re-realizing how widely deployed it is. I think the industry is starting to come to terms that MPLS is here to stay, and is starting to develop ways to leverage the benefits of SDN and OpenFlow into existing MPLS networks. In other words, integrate the technologies instead of having them compete against each other. Perhaps leveraging the OpenFlow classification abilities at the edge of a network using a centralized application, while maintaining the MPLS-based distributed signaling and forwarding state in the core of the network. Or adding an SDN + OpenFlow logical network “overlay” or “slice” to an existing production network for research purposes. Or perhaps even to opportunistically override the normal forwarding decisions for specific packet flows in the network in order to “steer” those flows to some sort of analytics device or value-add services appliance. Those are a few examples, but there are many.
In February, 2011 the ONF standardized OpenFlow v1.1, where MPLS label support was added. That was a great first step. The ONF is very active in the evolution of OpenFlow and SDN. While the ONF focuses more on OpenFlow and SDN, the IETF focuses on SDN and MPLS. I see more and more industry forums and conferences where MPLS and SDN are now being discussed side by side.
So, my personal belief is OpenFlow and SDN can clearly add value and additional services into existing networks, but there is no need today to rush to replace MPLS-based networks with OpenFlow and SDN. So, I guess my answer to my own question is I believe they are “better together” and are not mutually exclusive. In other words, a hybrid network approach seems to be the most feasible and promising option to me.
We announced the new 24-port 10 GbE module for the Brocade MLXe routers at our Analyst and Technology Day last month, and I’m really excited about the 3X increase in 10 GbE density that is now available to our customers. This module brings the total 10 GbE density for the MLXe-32 up to an impressive 768 ports in a single chassis, and it also supports advanced MPLS and OpenFlow features for large-scale service provider supercore and data center core networks. It’s our first ASIC-based module for the MLXe routers, and we’re using our 4th generation silicon innovation called the MaxScale-160 to pack all the features and density onto the module. I had the opportunity to sit down with John Terry, one of our principal ASIC engineers, to learn some of the more interesting technical details about the ASIC and to get a tour of their hardware lab (alas, one of the few rooms on campus my badge won’t let me enter).
The MaxScale-160 ASIC was designed and tailored specifically for high-capacity core networks, with a total of 160 Gbps bidirectional capacity that integrates eight 10 GbE ports. A total of three ASICs are used on the module to support 24 10 GbE SFP+ ports. As always in next-generation hardware evolution, the module's density, performance, and cost were key factors in designing the ASIC. We developed the MaxScale-160 processor in order to increase port density without giving up features and to reduce the number of components on the board, which also lowered the overall cost per port. The ASIC integrates a total of 33 active components that include various packet processing and statistics FPGAs, ingress and egress ACL TCAM, packet lookup SRAM, and eight 10 GbE PHYs. You can really see the difference in the 8-port 10 GbE module (2010 era) and the 24-port 10 GbE module (2012 era). Each module also has a daughter board which has been removed for clarity in the picture below, the 8-port module is on the bottom and the 24-port modules is on top. The 24-port module has triple the number of ports as the 8-port module, but many fewer active components, which results in an overall higher MTBF and makes it easier to manufacture.
Altogether the 400 MHz MaxScale-160 ASIC uses 1.38 billion transistors, and is built on 45 nm process geometry. “Process geometry” means the smallest dimension that can be drawn into the silicon to define a transistor, which is the building block used to make logic gates. A total of 352 Mbits of embedded memory plus 29.6 Mbits of TCAM made the ASIC a challenge for our foundry vendor to manufacture, because it has some of the highest integrated TCAM content. The next version of the ASIC, called the MaxScale-400, will deliver 400 Gbps of bidirectional capacity, will have over 2 billion transistors and will use 32 nm process geometry.
The 24-port module is the most green and efficient module available for the MLX Series, because the MaxScale-160 has extremely low maximum power consumption of <45 W. We have been steadily lowering 10 GbE power consumption as we release higher density cards, and the 24-port SFP+ module uses an incredibly low 13.33 W/port. By comparison, the 8-port SFP+ module uses 30.75 W/port , the 4-port XFP module uses 56.25 W/port, and the first generation 2-port XFP module that we released in 2007 uses 75 W/port. This steady gain in power efficiency enables network operators to save on operational expenses in cooling and power, while also consolidating the number of devices in the network.
In case you missed it, I wrote about some of the router architecture challenges we’re solving in my blogs earlier this year. The MaxScale ASICs and our future generation silicon builds on these technology advances, and I can’t wait to see what our ASIC engineers will be able to do next to deliver even higher 10 GbE and 100 GbE density.
For more information on Brocade’s high density 10 GbE and 100 GbE solutions, please visit the Brocade MLX Series product page
Back in January of this year I wrote about a really cool NetIron feature called “MCT with VPLS”. That feature was released in the NetIron 5.3 software. Well, in the NetIron 5.4 software release (which is GA this week!) we have a new innovative twist on how you can use VPLS. It’s called “Routing over VPLS” and this new capability allows a VPLS endpoint to provide simultaneous layer-3 routing on NetIron products.
So, layer-3 routing (& forwarding) is now supported over a VPLS endpoint! Previously, only layer-2 forwarding was supported for VPLS. Recall that VPLS provides a multi-point layer-2 Ethernet service, but with this new feature one can now combine layer-2 and layer-3 services on the same interface that maps to a VPLS instance.
It looks like this –
In the simple diagram above there are two data centers, each with two Brocade NetIron MLXe Series chassis (for high-availability purposes). Each data center also has a Brocade VCS Ethernet fabric. The inter-DC connectivity is provided by the VPLS layer-2 Ethernet service, which runs between the four MLXe chassis. In essence, all four of the MLXe chassis’s are layer-2 adjacent to each other over the VPLS service. Each MLXe is also running the Virtual Router Redundancy Protocol (VRRP-E), which is a layer-3 gateway redundancy protocol. Each data center has its own Internet connection for external connectivity.
The way it works is this: If a server or end host is forwarding packets to another server or end host at layer-2, the packets are forwarded by the VCS Ethernet fabric if the two servers are in the same data center (eg. Intra-DC), or the packets are forwarded over the VPLS service if the two servers are in different data centers (eg. Inter-DC). In either case, the traffic is forwarded at layer-2. In the intra-DC use case, no MLXe should be in the forwarding path; the VCS Ethernet fabric handles all the forwarding. That is pretty straightforward from a layer-2 service perspective.
But what if a server or end host needs to send packets to external destinations; in other words, at layer-3? Previously, this layer-3 traffic had to be forwarded over a different layer-3 enabled interface on the MLXe or depending on the topology and design, by a different layer-3 gateway router. That was an inefficient, non-optimal way of providing both layer-2 and layer-3 services.
With NetIron software release 5.4, both layer-2 and layer-3 services can be simultaneously provided over the same VPLS interface on the MLXe. This maximizes the customers’ investment in terms of ports while also simplifying the topology and network design.
Back to the diagram: Now when a server or end host sends layer-3 packets to external destinations, the traffic entering the MLXe on its VPLS interface can be routed directly to the Internet (assuming the Internet connection is on the same MLXe, as is shown here). The diagram shows that Data Center 1 has an MLXe acting as the VRRP-E master router and it is forwarding the packets directly to the Internet. This is what this new feature provides; enabling the VPLS endpoint to provide both layer-2 and layer-3 services. Another way of thinking of this feature is “VE over VPLS”. VE refers to a Virtual Ethernet interface, which is a virtual routing interface in NetIron terminology.
But what about Data Center 2 in the diagram? Its showing an MLXe acting as a VRRP-E backup router and this router is also forwarding packets directly to its Internet connection. How does that work you might ask? Well, one thing I didn’t mention is that the “E” in the VRRP-E solution is an extended tweak that we’ve done for VRRP. This is not new in NetIron software; this has been in the code for many years. It’s when you combine this feature with the routing over VPLS feature that you gain additional benefits. Basically, an MLXe acting as a backup VRRP-E router can also forward packets to external destinations if it knows how to get there (eg. It has an active routing path there). The backup router does not need to send all its packets to the router acting as the VRRP-E master. If it did, you will notice that the traffic would have to be forwarded back to Data Center 1 where the VRRP-E master router resides, which clearly results in a highly sub-optimal traffic pattern (often referred to as a “trombone traffic” pattern). While not the same scenario as we are describing, here is one explanation of traffic trombone.
So there you have it! Routing over VPLS provides simultaneous layer-2 and layer-3 services on VPLS endpoints. And I should also mention that the configuration of this is really simple. Here is an example where VE 200 is enabled under the VPLS VEoVPLS instance 10:
vpls VEoVPLS 10
router-interface ve 200
tagged ethe 4/1
tagged ethe 2/1
This is an innovative feature and the benefit to the customer is pretty clear in terms of maximizing their investment while also simplifying their design. As usual, comments or questions are always welcome!
My first experience with Application Delivery Controllers (ADCs) was way back when the market for this kind of purpose-built hardware was just developing. We had early access to Foundry’s software image that you could load on a standard NetIron router to turn it into a server load balancer, and some of the things it could do was pretty cool. We tested SMTP and NNTP load balancing, and it worked quite well for our needs at the time. The ServerIron then became an official product, and after a humble beginning the ADX Series is now an integral part of the Brocade product portfolio. I’m amazed at how sophisticated the feature set is on this product, and at how we continue to innovate and deliver exciting new features that enable our customers to offer new services. One example that I wrote about a while ago is the OpenScript Engine, which enables service providers to write custom Perl scripts that direct application traffic. There’s something else that we’ve developed that will change the way customers can use the ADX, and that’s what I want to tell you about today.
Virtualization has been available on mainframes and servers for a long time, but hasn’t been widely implemented on ADCs yet. At least not in the way server virtualization is implemented, with a hypervisior that virtualizes the system’s software and hardware resources. This is exactly what we’re bringing to the ADX Series, and it will enable service providers to deploy a truly isolated multitenancy solution that saves CapEx and OpEx. We redesigned the software architecture around a custom hypervisor that runs multiple, fully isolated ADX instances on a single physical system, each with its own dedicated software and hardware resources. The virtualization architecture is designed to maximize the consolidation benefits of deploying multitenancy solutions, while still maintaining the isolation and flexibility that using separate ADCs provides.
Provisioning and configuration is simple and flexible too. Since the software is virtualized for each tenant, features are configured independently and can even use overlapping VLANs or IP addresses. Full feature parity across tenants, based on the hypervisor architecture, provides the ability to mix and match tenants and to enable advanced features in any combination on the same module, with no extra hardware or licensing costs. This granular level of flexibility and tenant control gives service providers the most efficient allocation of hardware resources. If you are interested in more details on this feature, please grab the ADX multitenancy white paper.
Speaking of next week, we have our annual Analyst and Technology Day coming up on Wednesday the 12th. We will be sharing our strategic vision and latest innovations with the public, and will also have several technology demonstrations. The ADX demonstrations will show multitenancy and VXLAN gateway (which we just recently demonstrated at VMworld too). Oh, I should also mention our interactive “Ask the Expert” session for Software-Defined Networking (SDN) that will be running during the Tech Day activities. You can attend remotely, so please make sure to register to get all the latest details on the technologies we are developing.
For more information on Brocade’s application delivery solutions, please visit the Brocade ADX product page.
In an effort to find innovative service offerings, we have found that service providers are exploring many alternatives which include how they expose their network assets for profit. In addition, once they do create a new service offering they are looking for ways to increase service velocity to keep ahead of the competition.
Service providers are taking a hard look at their current infrastructure and operations environment and finding that their current operations model and network architectures have some key limitations which are impacting their services from being deployed quickly.
- There is no standard way to change traffic flows to handle user mobility and flip the switch applications
- There are no software tools available that enable them to “dry run” new service options without impacting the production network
- The current back office model does not provide the flexibility needed to make dynamic network changes and create new service offerings.
- Any changes to the production network are difficult, slow, and risky
Although many strides have been taken in collapsing OSS (Operational Support Systems) systems in order to streamline service creation and service deployment, it still continues to be a complex and lengthy process. As you can see from the diagram below, there are multiple devices, multiple databases, and multiple protocols that have to be assimilated at the OSS layer to create an end-to-end service. In addition, with a Video on Demand Service like this example, there is also an unnecessary use of expensive router ports and switches that are involved in the service insertion which all impact the bottom line.
SDN solves both these problems by providing the network abstraction and the operational simplification that will not only accelerate service delivery but will do it more profitably. In some cases, the implementation of SDN with OpenFlow, and RESTful APIs will result in 50% savings under present methods.
This new back office model eliminates some of the complex processes with the use of new APIs like RESTful. These changes will result in service providers increasing the service velocity of these new services. In addition, they will also realize CapEx savings since OpenFlow can set up the paths for the flows, alleviating unnecessary routing and high end router connections.
If you want to hear more about this Cost Analysis Model and the savings result, be sure to join us on Tech Day to hear about our vision and innovation as well as joining the “Ask the Expert” session on SDN here in the Brocade Communities on September 12th and 13th
There are some interesting developments going on in service provider networks these days, to say the least. Besides all the OpenFlow and SDN hoopla, which is quite the hot topic right now, traditional SP backbone networks are undergoing their own natural evolution. A few of these trends could impact router architecture in terms of density, scalability and feature set.
Today, most SP networks rely heavily on MPLS technologies. One primary driver for using MPLS over the last decade or so is network convergence. Most SP networks are multi-service networks and MPLS provides this multi-service capability. In a MPLS-based, multi-service
SP network there are two primary roles for routers; the Provider-Edge (PE) or Label-Edge Router (LER) router at the edge of the network and the Provider (P) or Label-Switch Router (LSR) router in the core of the network.
All IP/MPLS routers, both the PE/LERs and the P/LSRs, need to run many different protocols, each with its own unique design and deployment challenge. There is an IGP (OSPF or ISIS), an EGB (by default, MP-BGP) and an MPLS signaling protocol (RSVP or LDP). An IGP is needed in any IP network for internal reachability. In an MPLS network, the IGP also provides information that is required by RSVP and LDP. BGP is required in IP transit networks for external reachability and also provides routing and signaling information in MPLS networks, particularly in MPLS VPN networks. That’s a lot of protocol state being required in all these IP/MPLS routers.
BTW: Does all this sound complicated yet? This is why highly skilled IP/MPLS network engineers and architects are in high demand!
From a forwarding plane performance perspective, it should surprise no-one that Internet traffic growth continues unabated. Over-The-Top (OTT) video and other services are fueling this growth. Core routers that scale to high density 10GbE & 100GbE interfaces, when combined with high performance switch fabric technologies, allow the forwarding capacity of SP networks to continue to scale to meet this demand.
SP WAN Super-Core Architecture
BGP provides external reachability in transit networks but do all IP/MPLS routers really require this external routing information? If all the transit routers forward packets based on an IP lookup, then the answer is clearly yes. But since the P/LSR routers in an MPLS network forward packets based on a label lookup, then why do they still need to run BGP and maintain all that external routing state?
The Internet IPv4 routing table is massive; as of the time of this writing it’s at somewhere around 430k BGP routes! The IPv6 routing table is relatively small in comparison at round 10k BGP routes, but it continues to grow as IPv6 is becoming more widely deployed. It’s difficult to predict how many IPv4 or IPv6 routes there will be in the Internet in 2 years, let alone in 5 or more years.
To address some of the challenges in density, performance and capacity, a simplified “super-core” network architecture has been developing over the recent years. The definition of “super-core” may vary slightly depending on who you ask, but in general it reflects an “inner” core network that is architected differently than the rest of the network. Brocade’s general definition of a super-core architecture is based on a few different factors; one being the high density and scalability requirements of the super-core router and the other being a reduced control plane and route table capacity requirement of the super-core router. A reduced route table capacity requirement is achieved by removing the requirement to run BGP on these super-core routers. Since LSRs in the core network only forward packets by examining the MPLS label information, they do not look into the IP packet header. So, these routers do not need to know about all the external Internet routing information that is distributed by BGP; hence, they no longer need to run BGP. This can be a beautiful thing! One definition of such a network is a “BGP-free core” network. In this type of network, BGP only runs at the edges of the network where external Internet routing information is required. Makes sense, right?
In terms of router architecture, there are always trade-offs that must be made when designing a high-end core router. Please see additional detailed information on router technology trade-offs in previous blogs; here, here, and here. As discussed in those blogs, a router vendor can push technology only so far. Design decisions must be made on which capabilities are more important to SP customers. Capabilities can vary rather dramatically depending on what role in the network the router is performing. At the edge of the network, the LER needs high edge port density and a scalable BGP implementation to carry all those external Internet routes. The LSR in the core however, has different requirements in terms of density, scalability and capacity. The core is the aggregation region of all those high speed interfaces! Today’s core routers need to be more focused on high density interfaces and high forwarding performance.
So, to accommodate this super-core architecture trend that we are seeing in the largest SP networks, higher density 10GbE and 100GbE interfaces are required. This pushes today’s technology to its limits in terms of silicon density and performance. The interesting angle to this is that since the routing table capacity of the super-core LSR is reduced by not running BGP on these routers, vendors have the choice to optimize products for higher density and performance without having to also optimize for forwarding table size.
SP WAN Public IP Offload Architecture
Another trend we are seeing in the more progressive SPs is what we are calling an “IP Offload” architecture. This is basically where “commodity” Internet transit traffic is forwarded over a simplified, higher capacity yet cost efficient network; which runs in parallel to the traditional multi-service MPLS backbone. What some providers are discovering is that while Internet transit traffic continues to grow exponentially, the revenues associated with this traffic are not keeping pace. In addition, enterprise VPN and other services are not growing as rapidly and this is where a large portion of the SPs revenues are derived. This presents a dilemma for the SP; to continue to invest in and upgrade the capacity of the multi-service MPLS network (which carries both Internet and enterprise VPN traffic) or to do something different.
One answer is to move the high growth Internet traffic to a simpler and more cost effective, higher capacity core network while maintaining the enterprise VPN and other services on the traditional MPLS network. Since the traffic that is being moved to this IP Offload network is Internet transit traffic (eg. Public facing), we can refer to this as a “Public IP Offload” strategy. The requirements for the higher capacity offload network are focused around density and performance. Again, high density 10GbE and 100GbE interfaces are required in this type of offload network, similar to the super-core network previously described. Seeing a trend yet?
SP WAN Private IP Offload Architecture
Somewhat similar to the public IP offload strategy, we are seeing an Inter-Data Center offload strategy emerge. I refer to this as a “Private IP Offload” strategy; as within a large Internet facing enterprise network, the inter-DC traffic is seeing unusually high growth rates. This is somewhat of a recent trend that is being driven by cloud networking, big data, and active-active data center architectures. DC network operators that have traditionally optimized their network for “north-south” traffic patterns are now seeing massive growth in “east-west” traffic patterns. As active-active data center architectures emerge, “east-west” often translates to “inter-DC”.
Somewhat of a similar dilemma exists here that was previously described. In this case, the high growth inter-DC traffic is often internal enterprise traffic that is not related to public, Internet facing customers where the enterprise receives a majority of its revenue. So, in this case the “commodity” inter-DC traffic utilizes the same multi-service MPLS infrastructure as the Internet facing customer traffic. A similar answer can be found by moving this high growth traffic onto a simplified, higher capacity and more cost effective infrastructure. An inter-DC backbone network can carry this traffic while the Internet facing traffic is maintained on the traditional MPLS backbone. And as one can imagine, the router requirements for this type of network are high density 10GbE and 100GbE interfaces. One attribute of this private IP offload architecture that differs from the public IP offload architecture is that since this traffic is internal to the enterprise, this backbone should not need to carry the same large routing tables as a real transit network would be required to carry.
Brocade MLXe IP/MPLS Router Platform
Brocade offers industry leading core router performance and density; combined with the “right” feature set for IP and MPLS applications. The MLXe product line is our flagship carrier-class IP/MPLS router. It performs equally well at many roles within the SP network; be it LER, LER aggregation, LSR and even DC applications. The software feature set is very comprehensive and scalable, which enables this platform to perform different roles within the SP network depending on how the MLXe is configured.
In addition to its advanced and flexible software feature set, the MLXe chassis supports exceptionally high density 10GbE and 100GbE interfaces. The forwarding performance of the largest MLXe chassis, the MLXe-32, exceeds 15Tbps! The advanced hardware architecture includes a fully distributed, non blocking CLoS switch fabric design. One particularly interesting capability of this platform is its ability to support 64x10GbE interfaces in a single Link-Aggregation (LAG) bundle. Do the math for the size of that LAG interface!
The MLXe product line meets the requirements of traditional multi-service MPLS backbones, as well as the emerging requirements of the SP super-core backbone and the public & private IP offload network that are described in this blog.
So, as I previously stated – these are some interesting times in the service provider market. Internet traffic growth continues unabated and network architecture solutions are emerging to meet this growth demand; while not exacerbating the revenue challenge that SPs are struggling with. There is a strong drive to simplify networks to reduce CAPEX and OPEX. As cloud computing solutions become widely deployed, traffic patterns internal and external to the data center are changing from being north-south focused to being east-west focused. As new network architecture solutions emerge to solve these challenges, Brocade will continue to innovate to lead the industry in terms of performance, density and capacity requirements for the largest SP networks. So, stay tuned to this space for some exciting developments in this area.
Throughout my past 5 blog posts, I have reported on enterprise demand for new infrastructure services. IPv6 translation, SAN extension, server load balancing-as-a-service, and hosted desktop services are all emerging IaaS services in which service providers can invest to generate new revenue streams. Now what happens when a service provider actually offers these new services? What about enterprise buying behaviors? What do enterprises expect? And more importantly, how should service providers sell to these businesses? Again based on research by Wavelength Market Analytics, today’s blog discusses these important subjects.
Well, to be quite frank, service providers should expect a great deal of scrutiny. Purchasing new infrastructure services is a tremendous, non-trivial purchase and large enterprises find nearly all purchasing aspects equally important. When asked about the importance of each of the seven decision-making criteria in the chart below, only 3% separates the least important (service level agreements at 79%) to the most important (larger discount for larger deals ranks at 82%). Obviously, there is little difference among all purchase criteria in the eyes of large enterprises. Medium-sized enterprises, on the other hand, are not as demanding. At 54.9%, range of service and support options and proactive alerts of system availability, production issues, scheduled downtime and pending updates are most important to medium-sized enterprises.
Investing in a broad product portfolio is another important service provider requirement. This is because nearly 75% of large and more than half of medium-sized enterprises list outsourcing their IaaS services to as few vendors as possible as a top objective. The more services an enterprise has to choose from, the more likely they are to buy from a particular service provider to minimize the number of vendors.
Finally, service providers need to market to top business management, as well as to IT management. Sixty-seven percent of large and over half of medium-sized enterprises list top business management being very involved in the IaaS services decision, so purchasing IaaS services is likely a strategic one for many enterprises. As such, service providers should expect a long sales cycle and marketing/sales need to reach many purchase influencers.
To close out this blog series, using a two-layer data center network for improved performance and open standards for flexibility are increasingly important for service providers. Why? It’s because these are increasingly important to a service provider’s customers. As the graph below shows, nearly three-quarters of large enterprises prefer providers that use a two-tier network and support open standards. Nearly 41% of medium-sized enterprises prefer IaaS and other cloud providers that use a two-layer data center network and 27% prefer providers that support open standards. Finally, messaging to enterprise customers should highlight a two-layer network and opens standards support. Since there is a high awareness of Brocade’s Virtual Cluster Switching (VCS™) as an Ethernet fabric-based data center solution, some service providers may find it beneficial to mention that their flatter data center network is based on Brocade solutions.
In previous blogs through this series, I have addressed large market opportunities about infrastructure services offering service providers significant new revenue sources. These IaaS services include IPv6 translation, SAN extension, and server load balancing-as-a-service, but what about other new promising services? In this blog, we turn to more future-oriented infrastructure services with a longer adoption cycle -- virtual desktop infrastructure (VDI) and compute-as-a-service.
As the chart below indicates, enterprises that participated in the recent study by WaveLength Market Analytics experience the range of desktop management pain points. Forty percent of large enterprises report that dealing with increasing numbers and diversity of endpoint devices is their top complaint in desktop management. The next two biggest complaints with 32.9% each are managing multiple operating systems and managing software licensing complexity.
Medium-sized enterprises have an entirely different set of concerns. Forty-seven percent of them find securing desktop from unauthorized use most troubling, following by rising support costs with 40% and managing upgrades at 38%.
There are many flavors of virtual desktop ranging from internal VDI to hosted desktop services. Enterprises can choose to do it themselves, but if service providers can create services that meet enterprise needs, enterprises would likely prefer to subscribe to a hosted solution.So the crucial question to answer is what do enterprises expect in a hosted desktop service?
As the table below shows, large enterprises expect everything and medium-sized enterprises are less demanding. With 82% of respondents, large enterprises most expect the ability to customize personal desktops. Next, large enterprises want high performance; 81% expect new user setup automatically provisions data center infrastructure to maintain even workloads and 79% expect High performance data center network and WAN for maximum desktop responsiveness. Likely because they are less aware of hosted desktop services, medium-sized enterprises find less appeal in all of the aspects included in the study. With only 52% of medium responding organization, the top feature of a hosted service is minimal boot time and the least important at 33% is a broad range of enterprise apps beyond Microsoft Office.
Let’s quickly turn to a compute-as-a-service, which is currently more commonly used for test and development than production. Again, the table below shows the same overall trend; demanding large enterprises tend to expect a lot while medium-sized appear to have completely different needs. Large enterprises want compute-as-a-service to be bundled with disaster recovery as a feature of the service, 83% expect disaster recovery and backup included free. Large enterprises also want the range of service levels; they want to be able to choose the service level according the application requirements. In other words, they don’t want to overpay, either. Eighty-one percent want this feature, they expect multiple service levels to match application requirements to budgets. A two-tier network to assure the highest application performance ranks a close third, with 78% of enterprises strongly expecting this feature.
For medium-sized enterprises, they most expect a compute-as-a-service to provide the ability to use their resources well. As the top response, nearly 58% say they expect no minimum commitment. Matching service levels to application requirements is second most expected to medium-sized organizations. Like large enterprises, medium ones find high performance the third most expected feature in a compute-as-a-service.
Compute -as-a-service and hosted desktop may not generate large revenues in the near-term, but they are certain to be large and important markets for service providers. Successfully competing in the market for compute-as-a-service and hosted desktop services require the right service packaging, as well as highly virtualized, high performance data centers specifically architected for them. Brocade’s Virtual Cluster Switching (VCS™) provides unmatched VM awareness and automation compared to traditional network architectures. These competitive fabric offerings provide a solution service providers should explore as the basis of these emerging and lucrative new cloud services.
I thought I’d take an opportunity to write about and hopefully update folks on some recent happenings in the Software Defined Networking (SDN) space as it relates to standards bodies, such as the Internet Engineering Task Force (IETF) and the Open Networking Foundation (ONF).
Refresher on SDN
Software Defined Networking is all the buzz these days, especially here in Silicon Valley. I have personally been following it for about 2 years now and the buzz and hype has grown exponentially over the last year and even more so over the last 3-6 months or so. For new readers of these blogs that desire a quick refresher on SDN, please go here.
The modern history of SDN can be traced back to the introduction of the OpenFlow protocol. For those familiar with MPLS technologies, the history of MPLS can be traced back to the 1990’s with the invention of a concept called “cut-through routing” or “IP switching”. This concept and technology then became more commonly known as “tag switching”, and then ultimately, MPLS as we know it. Who remembers Ipsilon Networks? Anyway, look at where we are now with massively deployed global MPLS networks and all the network service offerings based on MPLS technologies; from what started as a technology to improve the performance of IP routers! It makes one wonder where SDN will be in 2 years … or 5 years.
So, what began with the introduction of the simple OpenFlow protocol has now turned into an industry in and of itself. But to be clear, SDN does not require OpenFlow. As previously stated in these blogs, OpenFlow is a key enabler of SDN but is not an explicit pre-requisite. More on this later.
While there are many potential use cases for SDN, from a high-level, there are two broad use cases worth discussing for SDN: Data Center (DC) Network Virtualization and Wide Area Network (WAN) Optimization. While one could argue that there are many other relevant use cases for SDN, I would like to keep this discussion simple and lump them into those two broad categories.
DC Network Virtualization Use Case: In a grossly simplified explanation, within the DC networking environment there is a need to connect physical and virtual servers at Layer-2. These resources can reside within the same DC or they can be geo-graphically dispersed and reside in different DCs (eg. “any rack, any where”). The emerging SDN solution for this connectivity involves virtualizing the network infrastructure by creating L2-over-L3 overlay networks to connect these resources; basically, abstract logical topologies that resides on the physical networking topology, but are simple to provision, manage and scale.
The industry’s move toward cloud networking is directly related to the DC virtualization use case. Cloud networking brings new challenges to DC networks in terms of dynamic and mobile workloads, network elasticity, east/west traffic patterns and rapid network and service provisioning. However, I’d like to keep this blog simple and more narrowly focused on the DC overlay problem space.
WAN Optimization SDN Use Case: In the WAN, there is a desire to have more control over how packet forwarding is traffic-engineered. In other words, a provider could optimize how traffic is forwarded over their network infrastructure with the goal of reducing cost. This manipulation of packet forwarding would be done in a centrally managed, programmatic fashion. New network services could even be created and offered based on these new capabilities. This type of capability could be equally desirable in Campus networks. So, the WAN optimization use case involves directing or programming L2, L3, or even MPLS flows in a more efficient and centrally managed manner.
Both of these broad use cases require a SDN controller where some sort of “southbound API” implements these functions into the networking infrastructure. This southbound API is often assumed to be the OpenFlow protocol or some variant. There is also a “northbound API” from the controller which talks to the SDN layer, where the real intelligence resides within the orchestration layer.
The ONF is responsible for the standards development of the OpenFlow protocol. The latest specification that is released is OF v1.3. The ONF is actively engaged in extending the functionality of the OF protocol, so stay tuned in to their website for new specifications of OF. The evolution of OF is directly correlated to the evolution of the SDN problem space. However, the SDN problem space is now evolving very rapidly on its own and is no longer directly tied to the development timelines of the OF protocol. The ONF continues to drive awareness and architecture development around the SDN problem space, including organizing and hosting industry events such as the Open Networking Summit and the industry’s first OF interoperability Plug Fest. The ONF produced a very technical OF interoperability white paper based on the Plug Fest event that is worth a read.
Along with the standards development of the OF protocol, the ONF is also driving the standards of related protocols, such as the OpenFlow Management and Configuration Protocol (OF-Config). OF-config is a companion protocol to OF and its purpose is to allow additional configuration of OF devices; in addition to pushing forwarding state into these devices which is what OF does. For example, enabling or disabling a port on an OF switch.
There are even organized working groups within the ONF that are focused on solving specific problems related to the OF protocol or the SDN problem space in general. These working groups are each focused on a specific area of OF and/or SDN. For example, while the Extensibility WG produces standards for the OF protocol, the Hybrid WG is working on exploring and documenting the requirements for a hybrid programmable forwarding plane (HPFP). Very interesting stuff. However, while the ONF is where the OF protocol and the SDN solution space were spawned, it is no longer the only standards body driving SDN solutions.
As stated on the Internet Engineering Task Force website; “The mission of the IETF is to make the Internet work better by producing high quality, relevant technical documents that influence the way people design, use, and manage the Internet.” The end goal of a specific IETF activity is to produce a Request for Comments (RFC) standard, which defines a specific standard or protocol that is intended to be used in IP networking . An RFC facilitates an industry standard, multi-vendor and interoperable implementation of an Internet related protocol and solution. The mantra of the IETF used to be “rough consensus and running code”, but as this standards body has evolved, oftentimes there are RFCs approved that do not have a broad basis of industry support. But I digress …
The IETF has several working group efforts that relate to both of the SDN use cases I previously mentioned. It is interesting to note that some of these efforts have been underway for many years in the WAN optimization area but only fairly recently has the IETF been involved in addressing some of the challenges in the DC overlay use case. More specifically, the activity within the IETF in this particular use case has dramatically accelerated in terms of Internet Drafts (IDs) and mailing list discussion over the last 6 months or so. This is no coincidence. In terms of new industry-wide networking challenges, much effort is being put into the Data Center area of networking, versus the Campus or WAN area of networking. Most of today’s innovation in networking is happening in the DC space. What I find particularly interesting is that while new technologies and protocols (such as VXLAN or NVGRE) are being developed to solve some of the emerging DC challenges in terms of DC growth, scale and Virtual Machine (VM) mobility, there is also quite a bit of recent movement in the IETF towards adapting some existing technologies and protocols to solve some of these same problems. It will be very interesting to see how this flushes out over the next 1-2 years.
A few interesting IETF activities in the WAN optimization space are Application Layer Traffic Optimization (ALTO) and Path Computation Element (PCE). While these are not SDN solutions per say, they are intending to solve some of the same SDN WAN optimization use cases. Both are worth following in terms of the SDN problem space; however, the PCE solution appears to be more applicable to the current use case for WAN optimization. PCE is geared towards intra-domain traffic-engineering while ALTO is geared more towards inter-domain traffic-engineering.
A very interesting, and somewhat new, area of IETF activity in the DC overlay space is the Network Virtualization Overlays (NVO3) WG. This WG was started within the last 6 months or so and has many Internet Drafts, ideas and discussions around the DC overlay/SDN solution space. The mailing list is also quite active, since this is a major area of innovation in networking. There are also many discussions on how existing protocols and solutions can be re-used, such as BGP and MPLS, to solve some of the DC overlay problems. As a sample of some of the ideas in this area, take a look at this, this and this.
There are even some non-working group mailing lists being created to discuss problem statements and possible solutions in the SDN space. Examples are the Software Driven Network Protocol (SDNP) and the Interface to the Internet Routing System (IRS) lists; and of note, the IRS was just created in the last few weeks. Here is the problem statement. The IRS activity is interesting in that it appears to be as focused on extracting information and state from the network as it is in implementing state into the network. The subject of extracting information from networks, or unlocking the intelligence that is inherent in networks, is one area of SDN that I believe needs to be addressed more broadly. Most of the activity in the SDN problem space seems to be focused on pushing state into networks. However, to fully utilize the network assets there needs to be a better real-time understanding of what is happening in the network. This type of two-way information exchange makes sense to me anyway. But that’s a subject for another blog.
Lastly as far as IETF activity that relates to SDN, I’d like to point out some work that has been ongoing for many years that bears quite a bit of resemblance to the goals of OpenFlow. There is a Forwarding and Control Element Separation (FORCES) WG that has been working on standardizing interfaces, mechanisms and protocols with the goal of separating the control plane from the forwarding plane of IP routers. As you may know, purists in the OpenFlow space have a desire to move the control plane of routers to a centrally managed element or server. While FORCES isn’t envisioning the exact same result, it does bear similarities to the goals of OpenFlow purists.
It is important for me to point out that the IETF IDs that I reference here are merely a sample of what is going on within the IETF that relates to SDN; so, I am not pretending that this blog covers this activity within IETF in a comprehensive manner. But I think I have mentioned most of the relevant ones.
The Internet Research Task Force "promotes research of importance to the evolution of the Internet by creating focused, long-term Research Groups working on topics related to Internet protocols, applications, architecture and technology." This effort is related to the IETF but is research oriented rather than standards oriented. But there is an important research group in the IRTF that is worth mentioning; this is the recently proposed Software Defined Networking Research Group (SDNRG). This research group is worth following to see what types of problems they will address and what the proposed solutions to those problems will be. Here is one relevant presentation being discussed on the "southbound API".
So, as you can see the IETF is becoming quite active in the SDN problem space. SDN related standards development is no longer being driven solely by the ONF. As the IETF mailing list and working group activity is increasing exponentially in this problem space, it is evident that the IETF “brain-trust” is clearly shifting towards solving problems in this space. Some of these ideas will die and some will evolve into complete and relevant RFCs.
Lastly, industry events are also now being slanted toward SDN problems and solutions. For example, this year’s Carrier Cloud Summit featured a full day on SDN and next year’s event is being titled SDN Summit. Other notable industry events with a focus on SDN are the SDN and OpenFlow World Congress and the MPLS International Conference.
So, as you can clearly see, the scope of SDN is expanding broadly across the industry and in a rapid fashion. Both the ONF and the IETF are actively involved in creating requirements and solutions for the SDN problem space. Since IETF 84 is being held in Vancouver as I write this, my fellow Brocade colleagues who are attending may have some relevant updates to bring back. So, stay tuned here for updates or comments to this blog!
Onward to the cloud …
In this week's blog on new Brocade-enabling service provider revenue opportunities, I examine storage area network (SAN) extension services. This emerging new service is designed to overcome multi-vendor, multiprotocol, or distance limitations (of 10 miles) inherent in connecting SAN islands. Once again, I am reporting on an online study among 192 medium and large businesses conducted by WaveLength Market Analytics this past spring. Today's topics are enterprise storage and data center-related issues, willingness to buy SAN extension services, and premiums for value-added features.
First of all, the market size for SAN extension services is potentially quite large. SAN island connectivity works best when they are located no more than 10 miles away from each other. Anything outside this key metric is well known to impair storage performance. According to the research, about 92% of large enterprises disperse their data centers OUTSIDE that key 10 mile distance threshold. Medium-sized companies obviously have fewer data centers and about 52% are outside the 10 mile threshold.
Large enterprises in the study report that distance impacts performance. In fact, nearly half of large enterprises report that SAN performance degradation over distance as their top storage issue. This is significant, particularly when considering the second highest ranking storage problem - unmanageable data growth, which is a widely-acknowledged storage problem.
Medium-sized enterprises also report that they experience issues that SAN extension services solve. Nearly 38% of them say they cannot meet recovery demands that are close to real-time, like SAN-to-SAN replication, making it their top storage-related issue. As a close second with 33%, medium-sized enterprises say their biggest storage issue is bridging multi-vendor SANs is too expensive.
As a result, service providers can expect a strong demand for SAN extension services. Eighty-four percent of large enterprises and 23% of medium-sized enterprises are willing to pay for SAN extension services.
In addition, service providers can expect new revenue to come from SAN extension value-added features and services. While some features in the chart below are actually built into the definition of SAN extension services, the conclusion is that enterprises find value in SAN extension services, specific SAN extension features, and are willing to pay for them. Among large enterprise survey participants, 88% of large enterprises say they are willing to pay extra for near real-time data recovery intervals and 83% say they would pay more for a SAN extension service that offers encryption in transit. Traditionally more resource constrained, fewer medium-sized enterprise participants say they are willing to pay extra for any SAN extension service feature.
So how much extra are participating enterprises willing to pay? On average, enterprises are willing to pay around 30% extra over the monthly base SAN extension services fee. Large enterprises are willing to pay 31% extra for data in-transit encryption and 28% for device redundancy. Medium-sized enterprises are willing to pay 24% extra for data in-transit encryption and 27% more for continuous service (device and circuit redundancy).
Not only is the market for SAN extension services potentially large, it also solves the biggest enterprise storage problems, so enterprises want to buy them. Service providers should move to offer feature-rich SAN extension services now. They should also bundle them with storage and security services and make SAN extension available as stand-alone services. Service providers should act now to capitalize on this opportunity.
With less than 2% of IPv4 addresses remaining, enterprises are currently planning strategies for IPv6 translation. Where exactly are enterprises in their IPv6 translation pursuits? What solutions are they considering for deployment? How do IPv6 translation services fit into IPv6 transition? A recent online study conducted by WaveLength Market Analytics reveals some key insights that should ignite service providers into action.
Nearly a quarter of large and only 5% of medium-sized enterprises have completed their IPv6 transition. Furthermore, two-thirds of medium-sized and 48% of large enterprises have an IPv6 transition plan completed but not yet implemented. Almost 30% of both medium-sized and large enterprises either have or intend to create an IPv6 plan within the next year. It all adds up to the fact that service providers that offer IPv6 translation-as-a-Service can expect to have a significant market available in which to sell. Many enterprises in need of IPv6-as-a-Service and service providers need to move quickly to tap into this burgeoning market.
IPv6 adoption is a marathon, not a sprint. Jumping directly from an IPv4-only network to IPv6-only is not an economically viable solution. Therefore, most enterprises plan to use a mix of outsourced services and internal technology. Because of this, nearly half of medium-sized enterprises and 75% of large enterprises prefer to buy IPv6 translation-as-a-service in order to delay the costly, internal network infrastructure investment. Purchasing IPv6 translation-as-a-service is not a permanent solution, but it does buy enterprises time to eventually upgrade network infrastructure and data centers to fully-support IPv6.
Because enterprises are aware of the necessity of IPv6 translation-as-a-service, service providers need to be flexible in how services are packaged. Over eighty percent of enterprises expect IPv6 translation to be bundled with hosting or content delivery. Of that eighty percent, forty-six percent of both large and medium-sized enterprises prefer to buy IPv6 translation services with hosting services while forty percent of large and 42% of medium-sized enterprises would like it bundled with content delivery services. Contrarily, only fourteen percent of large and 13% of medium-sized enterprises prefer to purchase IPv6 translation services separately from other services.
Study results show that offering IPv6 translation services will be good for a service provider's bottom line. First of all, enterprises don't expect services to be free; 81% of large and 30% of medium-sized enterprises are willing to pay for IPv6 translation services.
Further, IPv6 translation services can attract new customers. 70% of large enterprises are willing to switch to a service provider that bundles IPv6 translation services into hosting, even if that hosting service is more expensive.
In conclusion, there is significant enterprise demand, but it won't last forever so service providers need to deploy IPv6 translation services or risk losing out to competitors.
Check back in next week, as I will discuss enterprises' views on SAN extension services.
Instead of writing about some sort of exciting network technology in my blog this time, I decided to write about a really interesting book that was just published recently. The book is called “Tubes: A Journey to the Center of the Internet” by Andrew Blum, and I highly recommend reading it if you are interested in exploring the Internet from a completely different perspective.
I like the book for a number of reasons; obviously one of them is that Brocade is mentioned frequently and quite prominently at the beginning of chapter 5, as Blum spots our routers at the core of the Internet around the world powering exchanges and service provider networks. What makes this book unique though, and what sets it apart from the other books I have on the Internet, is the approach Blum took in writing the book. Instead of focusing on the technical details, the architectures, the nuts and bolts of how things work, he eloquently describes the intangible Internet in tangible words that bring out the physicality of the places he visited. Throughout the book you’ll read many descriptive words that appeal to your senses and capture the lighting, sound, smell, temperature, size, texture, and even the physical structure of the buildings that house the machines and tubes that connect the Internet – it’s almost like you are there with him.
The second thing that stands out in this book is the detail on the people and social aspect of what makes the Internet run – it’s a lot more than just technology and this is truly a critical aspect. Blum put a lot of effort into meeting and listening to the network operators that make the Internet work. He came to our conferences (I met him at NANOG48 in Austin back in 2010), he came to our social events, he came to our facilities, he made an effort to really get to know us, and patiently listened to our stories all while not being too intrusive or annoying. The social aspect of networking is also something that is very important to Brocade, and why we’ve been long time supporters of, and contributors to network operator conferences. Though you may have met some network engineers that are a little reclusive or shy, social networking among network operators is key to the development of the Internet. If we can’t talk shop over beers about ideas with our peers, and get to personally know the people whose networks we interconnect and peer with, then it’s pretty much impossible for the Internet to evolve because it would be almost impossible to work together. I like to personally know the people that I work with on a daily basis, even if they are spread around the world, don’t you?
Tubes is an easy and fascinating book to read, so pick it up at your nearest physical or virtual book store, and enjoy reading about some of the people and places that make up the Internet. And if you attend APRICOT, EPF, GPF, NANOG, RIPE, or any of the many regional NOGs or IX meetings, then I look forward to seeing you at the next one!
To help service providers and network operators better understand the IaaS market opportunity, Brocade commissioned WaveLength Market Analytics to conduct an online study of large and medium-sized enterprises that outsource at least one IT function. Among the key research questions, Brocade wanted to know developing trends and adoption drivers in the cloud services market. We also wanted to know enterprise IT priorities, storage and data center issues, plans for IPv6, and how enterprises expect to buy new services from service providers. In this first of six blog posts on the study, I am going to talk about enterprise outsourcing and enterprise IT priorities that create new opportunities for service providers.
Let’s first examine outsourcing, which includes both managed services and cloud services. According to our research, it is already pervasive across all infrastructure functions; 70% of large enterprises outsource network management and 70% of medium-sized enterprises outsource storage management. The market is clearly transitioning to cloud services as 60% of large enterprises already deploy some type of cloud service.
Turning to IT management priorities, participating enterprises seek to improve performance, simplify management, and increase agility. Fifty-eight percent of medium-sized and 50% of large enterprise respondents say their top IT priority is to improve user experience, service quality and reliability. The second most common IT management priority with 45% of both medium and large enterprise respondents is accelerate application development time/improve agility, while the next highest response with 44% of large enterprises is mitigate risks of technology change.
So how do enterprises intend to follow their IT management priorities? Within the next year, 42% of large enterprises respondents say they plan to both increase use of infrastructure services and to deploy a two-tier data center network. Meanwhile, 52% of medium-sized enterprises plan security improvements and 48% plan to virtualize more physical servers. In sum, enterprises are deploying virtualization and outsourcing with plans to continue both trends. This is good news for service providers.
In this series of six weekly blog posts, I will share survey results. Come back each week to learn more about how enterprises are willing to buy new infrastructure services that will generate new revenue streams for service providers.