The concept of a separate “cable infrastructure” for IP storage is not a new one. For more than 6 years now, the iSCSI market has been doing that. For more than 15 years the backup market has been doing it. Performance NAS environments are a bit newer, but the issue is the same. The traffic patterns are not a good blend. Like mixing passenger cars and freight trains. It doesn’t work out well for either side. In point of fact most network organizations have multiple networks running.
I have asked more than 200 large datacenter customers the same question over the last two years:
Given that the Network team actually runs multiple networks.
Front side client/server network
What people think of as “the network”
IP based tape/virtual tape/deduplication network
We spent more than 7 years getting the IP based backup and restore network off the front side client server network. Yes, because the “race to sunrise” was not achievable in a mixed environment; but more importantly because RESTORE almost never happens on “your” schedule. And when you launch a RESTORE in the middle of a production day on the production client/server network, people learn your name. And in IT terms, “people learn your name = bad”
The cluster guys (obviously raised as an "only child") have ZERO interest in “shared”. Their isolation was for reasons of latency and deterministic behavior.
Go figure that the actual VM migration gets isolated from client server traffic by standard design practice.
Performance iSCSI network
Whether you ask the HP/Left Hand Network guys or the DELL/Equalogix guys the answer is the same. If ALL you want is connectivity for a couple of machines you can put it anywhere. But if you want PERFORMANCE iSCSI based storage you put it on an non-shared cable infrastructure. Do NOT mix the traffic.
Performance NAS network
This is a somewhat newer conversation but not completely. Content distribution platforms were being isolated prior to this. But the concept of the NAS platform (hybrid SSD) causing performance levels that interfere with standard client server traffic is gaining ground (such as at IKEA).
Out of Band Management network
Go find the compliance guy in the environment and tell him you’re planning on collapsing this to the same cable infrastructure as the client server traffic and see what happens. The most common mechanism for moving a code update to an Ethernet platform in the market is TFTP (Trivial File Transfer Protocol). And the security available for TFTP? Oh right, that would be none. But really its just more that a physical isolation of the management cable is just good practice. Limit the number of people with access and you limit the number of threats.
This is also a fairly controlled network segment. Partially because of the cost and partially because of the criticality of the data set being moved.
Who among you has taken all of the above networks (or the subset that is present in your company) and collapsed them to a single 10GbE cable with logical isolation rather than physical isolation while successfully keeping your internal customers happy?
Out of all the 200 datacenters I have only had ONE customer tell me that they had tried it and they were unsuccessful in getting it completely collapsed and had to back out of it. So, when we speak of collapsing networks to the servers it is far more rhetoric than reality. Yes, the concept is extremely enticing. If the server guys could get all of their connectivity with just two cables (including power) they would. But since POE+ is only 25W to the desktop connection it doesn’t work for them. But of the 6-10 cables coming off of the servers today, the majority are already Ethernet.
However the point is that both the pattern in the use case for this traffic is very different and blending them together does not achieve the best support for all the traffic types. Can you connect iSCSI devices any where you want? Sure. But if you want performance iSCSI isolation is the best practice.
So in summary I think that it's fairly clear that storage networks require a different level of service due to requirements of their traffic patterns and the impact of traffic outages. And this requirement means that you need to have them on a separate network, if you want to be able to scale them without impact to the other network flows.