For more details, please see ourCookie Policy.

Ethernet Fabric (VDX, CNA)

What is an Ethernet Fabric Anyway?

by on ‎11-05-2010 01:41 PM (649 Views)

I’ve talked about forces of nature in the data center, what happens when they make overwhelming demands on tried and true technologies, such as classic Ethernet, (hint, they win, classic Ethernet loses) and talked in general about fabric technology in another post.  So what is an Ethernet fabric?

Compared to classic hierarchical Ethernet, Ethernet fabrics provide higher levels of performance, utilization, availability, and simplicity. They have the following characteristics:

  • Flatter.  Ethernet fabrics are self-aggregating, enabling a flatter network.
  • Intelligent.  Switches in the fabric know about each other and all connected devices.
  • Scalable.  All paths are available for high performance and high reliability.
  • Efficient.  Traffic automatically travels along the shortest path.
  • Simple.  The fabric is managed as a single logical entity.

So who needs more performance, better bandwidth utilization, an always up network, and simplicity?  Yes, everyone does.  But, if we have gotten this far with classic hierarchical Ethernet networks, why can’t we just continue using it? There is a primary reason classic hierarchical Ethernet will be replaced with Ethernet fabrics.  To borrow a well used phrase, “IT’s the Applications, Stupid!!”.

Ah, the applications.  Yes, that’s why there are networks, servers, storage and ever larger data centers to provide for their care and feeding.  If you stop and think for a second about how many applications you interact with in a day, no, really, stop and think, it’s very large and growing.  Your GPS, smart phone, laptop, desk phone, and all the front and back office processes of every business you interact with in a day are powered by applications.  For example, today I checked my bank balance online, made a payment to my cell phone service, and looked up some books at Amazon, how many applications did I touch in 5 minutes? Lots and lots, and it’s growing.

Why?  Well, the internet has not only increased the speed at which we interact with each other, it’s created an explosion in the number of applications.  You did notice that Microsoft, Google, Oracle, SAP and lots more are growing --- and doing it in a recession – didn’t you?

What may not be clear is application growth is always limited by the ability of data centers to scale cost-effectively. Sever hardware scales by Moore's Law, twice the power at the same cost every 18 months.  That's an upper limit on economic scalability.  But the rate of new application deployment is even faster than that, so one application per server wasn't economically scalable.  When that became clear to IT (and the CFO in particular), things had to change.  The agent of that change is server virtualization, a curiosity 10 years ago and a necessity today.  Now, you can run 10, 20 or more applications on one server.  And since the server hardware scales according to Moore’s Law, doubling every 18 months, so will application scalability, doubling every 18 months.

But, if we fixed the application scaling problem with servers, what’s the next area that can’t economically scale up fast enough?

How about Storage? No. that got fixed last decade when storage area networking and switched fabrics took off.

How about Networks?  Yes. The classic Ethernet network is in fact “THE network” server virtualization relies on to scale.  And, it’s not scaling.

Classic Ethernet is the next road block and it has to be change starting at the access layer, and in time, through the aggregation layer to really flatten the network for economic scalability.  The agent of change is Ethernet fabric technology that can co-exist with classic Ethernet.  I say “co-exist” because if a new technology is going to succeed, it has to co-exist with what’s already in the data center.  For example, server virtualization runs on yesterday’s hardware and operating systems, so it co-exists with them.  Similarly, Ethernet fabrics have to co-exist with classic Ethernet switches.

An Ethernet fabric is transparent to any classic Ethernet switch. You plug in an Ethernet fabric (today, it’s mostly in the access layer, and tomorrow you scale it out to absorb what used to be the aggragation layer into a single fabric ... Hugh win!!) and connect it to existing classic Ethernet switches and what happens to the network?  Well, It just works like it did before.  The Ethernet fabric is transparent.  That’s not just co-existence, its graceful co-existence. There is no need to prematurely replace existing switches (or heaven help you, server NICs) to deploy an Ethernet fabric.  Instead, you can incrementally add an Ethernet fabric when you add a new rack of servers and scale it across multiple racks as you add more servers.  That’s pretty simple.

Okay, what advantages does an Ethernet fabric have for virtual servers and cost-effective application scalability?

Ethernet fabrics include distributed intelligence in the control plane with a logical chassis in the management plane.  Distributed intelligence means all switches know about each other, and about all devices connected to them.  Further, since they share a common control plane database, network policy management, (for a long time, the bane of network administrators) is configured one time and shared by all switches in the fabric.  Now, combine that with a logical chassis, and you get simple management that scales at the rate of application growth.  Every time you add another switch to the fabric, it plugs into the logical chassis showing up like a new port card would in a traditional chassis switch.  That’s very scalable and very simple.

Another way distributed intelligence improves virtual server management is a feature called Automatic Migration of Port Profiles (AMPP).  A very compelling value of server virtualization is applications are no longer owned by a server.  They are free to move, without shutting them down.  They can move to another server when a server needs maintenance or the server is being replaced with the newest generation of processors.  Maintenance windows are now 24x7x365, and no longer restricted to a couple of hours late on Sunday night.  Ah, but the classic Ethernet network is static, particularly when it comes to network policies.  Those are configured on a specific switch port so if the virtual machine and its application moves, its traffic shows up on a different switch port, and the static policy is stuck on the physical port, and isn’t applied.  Bad things happen then.  All the goodness of application mobility gets stepped on by the static nature of classic Ethernet.

A fabric with AMPP can be as nimble as the application.  The distributed control plane database makes policies accessible to every port in the fabric.  When a switch port sees traffic from an application for the first time (it really sees the Ethernet MAC address of its virtual machine), it checks the distributed policy database for a policy matching the MAC of the virtual machine, and if there is one, applies it.  Bingo.  Network policies are virtualized and as mobile as any virtual machine.  Problem solved.

Inside an Ethernet fabric is … well a fabric of inter-switch links (ISL).  An ISL in classic Ethernet is problematic.  Spanning tree still requires one active path between switches, so a LAG, or a Link Aggregation Group, can increase bandwidth between switches by exposing a logical link (the LAG) of several physical links, but aggregating multiple physical links into a single logical LAG is time consuming to configure and doesn’t scale easily.

In an Ethernet fabric an ISL is self-configuring.  Plug a cable into two switches and the ports auto discover each other, come up as an ISL connection and automatically configure equal cost multi-pathing with link cost routing.  Traffic always takes the shortest path across the fabrics for lowest possible latency. Add another ISL between the switches and now a load balanced, self-healing trunk forms. All this happens without any need for “operator intervention”.  And, when the application consolidation ratio go up with the next generation of server technology, you can plug-in more links between switches and quickly, simply adjust the subscription rate between switches as desired.  Problem not only solved, but solved simply.

I hope this answers the question, “What is an Ethernet fabric?”.  It’s a revolution in Ethernet that lets you build a flatter, intelligent, efficient, scalable and simpler network.  And it’s the network solution that places no limits on application scalability.

by nvaidya
on ‎11-16-2010 05:10 PM

Brook, very well-written blog covering vital points.

Additionally, also want to highlight the increased network reliability factor too. Customers do need a single point of management, but without single-point(s) of failure OR a reduced domain of failure. VCS satisfies both of these requirements & provides best of both the worlds.