Tuesday, 30 March 2010

FC vs FCoE vs 10-Gig-E vs NAS Protocols

This is really just a discussion on which storage network protocol, why and when? So this is really just my view of the state of the world & what we may expect over the next few years from the storage protocol market...

So - There is a bunch of hype and FUD going on around storage protocols, network convergence and value add that network convergence may bring... Whilst there may ultimately be some decent end-game that we *could* aim for, I am not sure we are even close to working out (a) what game we are playing (b) which teams may be in the final and (c) are we aiming for the world cup or maybe (and more possibly) some amateur local league cup - with little to no win...

Some of the discussions ongoing are very reminiscent of the iSCSI vs. FC discussions that no doubt we all got involved in (my last nasty discussion on this particular subject was 2 years ago - but still bubbles to the surface occasionally...)

Who are the teams / technologies that "maybe" in play at the moment? There are the obvious two, FC and NFS, but new to the equation (or old if you take FUD into consideration) are FCoE and 10-Gig-E... I will also throw iSCSI in for good measure, if I don’t – some one else will (and before anyone pulls me up on it - yes i know that i have mentioned one protocol that runs on top of another (NFS / iSCSI / Ethernet etc).

First thing / bug-bear... DCE (or DCB or...) and 10gig-E They are NOT the same thing - peeps (and especially vendors) please stop using them interchangeably - when you do, it just shows that you DO NOT know your topic at all!!! I am sorry to those of you that think I am stating the "bloody obvious" - but it is a mistake that often.

Right – onto a bit of discussion then…..


-Fibre Channel-


You look at most of today’s enterprises and the chances are that most mission critical environments are likely to be using fibrechannel. Its proven, guarantees delivery (and not to throw away stuff when it becomes congested) and networks are well known by the storage / OS guys - as they are probably the people that put them together... You don’t encounter (too) much over-subscription in the network (Generally)... 

I think it is also fair to say - that on the whole - these networks have been pretty damn reliable with pretty much rock solid performance and availability.... I guess the old saying "if it aint broken then don’t fix it" applies here - but hey, its damn expensive and if the truth be told can be somewhat complex to manage.. Of course you also have the additional costs of HBA's and more cable etc that needs to go into the back of cabinetry in order to support connectivity .... And then there is storage vendors that support FC - they all do it, without fail - bear this in mind as you read through this article as it is relevant.

Finally - let me add the subject of maturity into this section - FC is understood... and i mean by everyone... anyone working in storage, either in the central support function or on the outskirts of it will understand it (and more importantly the demarcation of who supports it)

Typical use cases of FC in the enterprise - well to be honest, 90% of all storage (with exception of unstructured data) is provisioned in this manner, but if you wanted to name it - items where performance is guaranteed, high consolidation environments, mission critical environments etc...


-DCE / FCoE (Data Centre Ethernet / Fibre Channel over Ethernet)-


FCoE- so as the name suggests, fibrechannel over Ethernet... it obeys all the characteristics of fibrechannel and then throws them over a DCE network...ok first point ... is DCE "ordinary Ethernet".... Datacentre Bridging / Datacentre Ethernet and whatever the heck else vendors want to call it (dependant on which one of them thought they invented the technology and term) - its not the same as Ethernet that you and I have all come to know and love... its different... so bear this in mind if you choose to go down this route - DCE / DCB is another layered skill set that you need to learn & manage.

What else then, well - as I mentioned in the fibre channel section - you have spaghetti, in the back of cabs... fibre for FC, copper or fibre for Ethernet - it gets horrid if its not managed properly, and managing costs money - in comes DEC (and hence FCoE) and the promise of CNA's (Converged Network Adapters) - the idea being able to use singular PCI cards that supply both IP connectivity and storage sounds like a good one - and i guess it is? Really? Is it? Hmmm  - hold that thought - and ask yourself the question where do I use FCoE - Top of rack, or end-to-end through the network - guess its time to add more flavor with this offering then.... (not that any of this is bad)

There are a number of views as to how DCE / FCoE will emerge in the mainstream, the vendors will tell you "it is a journey" - (how i hate that term!) This will typically start as a converged top of rack switch offering, which then breaks out to the disparate networks (Ethernet and FC) - and then use those traditional networks to go to their end points (ethernet based services or storage FC targets respectively) - ultimately the aim is thatend-to-end FCoE for storage use.

DCE / FCoE / Converged network - Top of rack


Most of the offerings today will talk about using CNA's providing traditional network and storage through a converged piece of cable to a top of rack or top of pod converged protocol switch and then break out to there seperate traffic types, i.e. FC goes out to a traditional fibre channel network and Ethernet breaks out to the corporate Ethernet function - so the promise is that you need less cable in the back of server cabs, complexity is reduced and more importantly cost... But also think about fault domains, traffic analysis within a model where storage and network is mixed, performance issues.

One other thought here - who manages the physical and logical entities of this top of rack switch. Hardware could be handled by either as long as both parties are aware of the criticality of loss of Ethernet or FC... I don’t know about you, but every time i have described the concept of SAN and the fact that guaranteed delivery is king to a networks guy - the question pops up - "why?" - Guess this is just due to diffent way that ethernet (and hence the application and protocol stack takes care of things) - Look at traditional Ethernet, it is designed to throw away packets when the network gets congested / suffers some type of hassle. FC - well its got to be delivery every time - its a fundamental difference in approach and thinking (and each of the respective teams sometimes find this hard to grasp - really just a consideration if thinking about converging teams that manage these technologies).

DCE / FCoE / Converged network - End to end


The ultimate aim of this technology really is to provide a singular network that provides all network-based services. We are really talking about never breaking out to disparate networks, IP is held within the network, and storage plugs straight into the service also. This really should be where the dream becomes possible - but again it is full of issues that need to be sorted. Demarcation of role is one (who manages this network from both a physical and logical view). Tooling in terms of intricate fault-finding / performance management is not quite there yet. More importantly - ability to plug FCoE targets (i.e. storage arrays) directly into a converged network (i.e. native FCoE) is limited. Some of the vendors have an initial offering that is there, or thereabouts (NetApp spring to mind as the first example of this - and able to offer at an enterprise level).

I have touched on operating model in the above section and also within this section - who owns this, able to have administrative control, who completes items such as zoning etc creates an interesting question.... Ultimately - this technology is not just about convergence of network, but convergence of an infrastructure type service, and to be able to service this - typically the organization would need to move from a silo based support model to cross-technology... Any org change is difficult - bear this in mind when looking at this technology

Finally - it’s probably worth giving a little insight into where adoption of this technology within storage is taking place (or the rate of adoption) and what the market is saying. I attend a number of industry forums - in 2008, if you were asking people in these forums around their possible adoption timelines of FCoE, they would have said within the 2-3 year timeline.. (Funny that - not many people seem to have it!), at recent forums (last two that i have attended - back end of 2009 and Jan 2010) adoption is now leaning to the 5 year (from now) timeline. Why is this, well back in 2008 we were probably seeing the "hype cycle" - reality set in, and now we are seeing a more pragmatic approach. Of course there are other reasons why this is also prolonged - there are more and more options - the (disruptive) emergence of 10-Gig-E and use of file share protocols


-Ethernet options-


Hey - here is something i never thought i would be writing on a storage blog..

Me, Ethernet and storage have had a pretty rocky ride together...

I have had issues with the likes of iSCSI – not because the protocol specifically has many issues (it does have some… but its not all bad) but more the way that people have thrown it into a generic network used for other “stuff” and then wondered why it all went wrong…

Back in the day, if you wanted to use iSCSI in enterprise infrastructure the correct and right way to do it, would have been to throw in dedicated IP switches with dual resilience and then hey presto – but hey, doesn’t that just become an IP based SAN?? And as soon as you dedicated those director class switches to storage, they become just as expensive as putting in director class FC switches, but without all the lossless characteristics that FC gives you – so why bother…

Then there is CIFS and NFS… those nasty unstructured data protocols that we have whizzing around our networks that we love (or is that hate?)…

Whilst unstructured data itself is a right pain in the backside due to its unmanageable approach (and I am talking about the data itself) – its got to be said that the provision of storage to hosts that wish to use CIFS / NFS filers is uber-easy…

Do you have all that nasty zoning to go through?? NO!

Do you have all that LUN masking to go through??? NO!

Do you have to worry about provisioning LUN’s to certain target FC ports??? NO!

In most cases, you simply provision a file system and present it to the network – hey presto!!! Yes, you probably have to worry about some permissions, authentication and all that jazz, but that skill set is far more readily available than that of good quality storage admins.

But its not all a ray of sunlight over here in NFS and CIFS land… why not? Well two things I guess, these being performance characteristics of the arrays that support these protocols and the network that sits between host and storage array….

Lets do away with the network question first… 10-gig-E is here… Its rocked up and its larger the life… Its fast, its easy, people understand it – its just quicker. In fact it’s really quick…. This technology will now become the de-facto standard deployment model in DC’s over the next 2-3 years, which gives some interesting options. Yes – it still a shared network, but with the performance hike and ease of administration associated with CIFS and NFS – this protocol can simply no longer be ignored as a viable way of provision disk to servers. 1-gig-e was always a problem, as it just couldn’t cope with 50% of what you could throw at it…

Then there is array performance… so this is an interesting one… Some of the current leaders in the CIFS and NFS filer space have issues… the number of disk they can stripe across, performance of the overall array is still not up there with the traditional block array vendors, but now the customer has choices… We can look at start merging technologies together to get some real performance hikes…

NAS appliance “heads” married to a “wide stripe” array connected to a 10-gig-e network has real mileage, and that is without considering some of the other economies that can be considered in these configurations (examples of these being Compression, De-dupe etc…)


- Summary -


OK – enough of my ongoing dragged out rants on this blog, time for a summary and wrap up of where I think this really leaves us…

Fibre Channel

When guaranteed delivery and performance characteristics are required, Fibre Channel is going to be king for some time. It’s where you keep the crown jewels and all that mission critical stuff. This protocol is not going anywhere, and will be around for the foreseeable future


So I was not sure to put this as a heading in its own right or within the fibre channel section… This protocol(s) still observe all the FC niceness – but just over a converged network. Whilst I do not believe that FCoE end-to-end will come mainstream in the “near-time” it will happen.

You will see FCoE (or other convergence) at the server side first to converge connectivity from host into top of rack / top of pod switch – and this will happen relatively soon (in the next 12 months or so)

My guess is that you will see this technology start to make an entry at top of rack in the next 1-2 years, and then end-to-end in the next 4-5 years. The promise of converged network cabling is just to hard to avoid, however I think the discussion of what ends up on traditional block storage and what use cases make a move to Ethernet storage (NFS) is a debate that will start raging

Traditional Network based protocols (CIFS / NFS / iSCSI etc)

As mentioned above – I think there will be a renewed emergence of these more traditional protocols but for more wide-spread use cases.

People don’t like complex provision technologies (i.e. those associated with FC as an example). With wide scale implementation of 10-Gig-Ethernet on the horizon there is an ideal play here – the use of performance orientated network-attached storage for less critical applications (those such as development, testing etc). Companies such as Atlantis Computing (with their ilio product) are making this more space even more appealing….

iSCSI – well still think its dead, to set it up properly – still needs a heap of infrastructure and there are security issues here (spoofing it IP etc). If you want to use NAS type services, then use them – don’t make it look like block  - as a protocol it does not handle disconnects that well, NAS / CIFS can! Rip block storage from under a host – it typically gets totally bent out of shape – a situation to avoid!

One more point to make – The storage array vendors really need to think about this!

The sacred ground of storage array vendors being able to just sell FC (or FCoE) arrays is coming to an end – ability to handle file protocols but also to have decent performance characteristics is going to be key – there is a market to be won or lost here– I believe that this is where you will see the more mature vendors have an approach Vs one-shot-wonder vendors (before anyone asks – I am not thinking about just wacking someone else’s appliance (or your own for that matter) as an answer – but a truly integrated array with end-to-end offerings)

With people wanting cloud services – the ability to provision easily is going to be key, and FC based protocols are not easy… People don’t want to faff with zoning and masking, nor will they want complex FC storage for all use cases – its expensive and people to provision them are expensive! Food for thought all you storage vendors out there!


- Wrap Up / Final thoughts –


FC - Where mission critical / performance / availability characteristics are required by applications – Stick with FC Protocol – This will continue to be the standard approach for some time to come! (maybe the emergence of FCoE / Convergence at the server host – but this will still be using FC protocol within the stack)

FCoE End-To-End will NOT happen any time soon as there is no compelling event to push this to happen – HOWEVER - you will see Convergence start happening at the server side (CNA's into top or rack switches or converged network ports directly on servers) – but not the complete way through the fabric (yet) market and product needs to mature… This will be at least a 3 year timeline before people adopt in anger (at least!)

NFS or CIFS (or other typical NAS protocols for that matter) – Use for Non-Critical / low availability / Development / test environments Possibility to save a HEAP OF CASH.. Look at the option of using traditional NAS protocols (CIFS / NFS) over 10-Gig-E to drive costs down – I believe you will see this approach merge back into the production space as people become more trusting of 10-gig-e and performance and also with the smarts that the NAS offerings will bring to bear! – Start using them and save some dosh! This accounts for more than 50% of most peoples SAN Block storage array usage at the moment

Final note for Storage Array Vendors...
Storage Arrays / Storage Array Vendors
Follow this protocol market and carefully… The winners will come out with some level of true integration or smarts that will push for complete end-to-end solution (to provide FC  / FCoE  / NAS in one box) and enable all of the above use cases (protocols) as an option but with one back end array. Look for smarts such as de-dupe, compression, performance layouts as part of the commodity offering (along with the standard stuff we have today)

1 comment:

  1. Told with true passion... who said storage was boring?

    What I've always said, as has Chad Sakac and others, is that there has never been more choice to use any/all of these protocols: you don't have to pick one and live/die by it, as you say at the end.

    If NFS over 10GbE can save you buckets of cash in Test/Dev, go for it. Oracle get blazing performance and reliability from NFS so why can't you in prod?

    I agree that FC is a bitch to set up, everyone gets it wrong, and IMHO the storage vendors could do a better job of making it easy. pWWNs, zoning, masking, multipathing: sheesh!

    I also agree on iSCSI - I fail to see the point if you have to buy dedicated infrastructure. I also can't see how SCSI over IP is a Good Thing? For me it's NFS or FC for storage.

    DCB ethernet with its QoS, ETS and PFC brings some serious reliability to the network BUT, here's where I agree with you once more: the tools to help us admins understand "what's normal" and troubleshoot, they just aint there. All I've seen so far from the Big4 is same-old-same-old.

    Opportunity for someone? :-)