Jim Duffy wrote an interesting article on Network World’s Cisco Connection blog called “Cisco, Arista Disaggregating?” in which he speculates that Cisco and Arista may make their network operating systems (NOS) available for use on bare metal switches.
Is there any mileage in this idea?
Old News, New Timing
The idea of the big players selling their software for use on generic hardware has been floating around pretty much since SDN hit the news and the first bare metal switches came out, with Cisco for example looking like they were pretending that SDN wasn’t a thing, and their position was secure if they continued to do what they already did. To be honest, I think Cisco is still paying the price for initially lacking a strategy, then embracing SDN in such a confusing way. Nonetheless, the idea isn’t new, but has the market moved to a position where Cisco and Arista really need to do this? And what of Juniper; are they immune to being sucked into the bare metal market?
In addition to being a good addition to awesome music of G. Love, for companies like Cisco Arista and Juniper, their “special sauce” these days is their network operating systems – IOS, eOS and Junos OS respectively, at least in terms of the Ethernet switching market. The NOS is what distinguishes their products from other people’s when it comes to selecting a switching vendor. After all, D-Link (to pick a victim) can produce 48 port gigabit switches that are stackable and have 10G uplinks. Why would anybody pay Cisco prices when they could pay D-Link prices instead? There must be a reason.
Cisco’s venerable IOS is long standing and well known. Somehow, the company has managed to keep the same fundamental configuration paradigm despite the enormous change in the technologies the switches have to handle, and to say that it is ubiquitous would understate the degree to which the interface and configuration style define what network device configuration looks like. The number of people who have cloned that interface is testament to its domination of the market, even if it isn’t necessarily actually the interface preferred by many network engineers. But an interface is just a front end to functionality and it can, and has been, cloned. So why is IOS still “better”?
Cisco IOS has typically contained features that are either proprietary, cutting edge, or are enhancements of the standards-based deployments that others offer. This is a double-edged sword, as anything proprietary means vendor lock-in (which is good for Cisco), but may be considered a non-starter for other potential purchasers. EIGRP, CDP, FabricPath, PVST+, HSRP, GLBP – these are all examples of Cisco going their own way, or creating their own protocols; they aren’t bad by any means, but they mean Cisco, Cisco and more Cisco to keep them working.
None of the features I listed above need special hardware support except perhaps FabricPath, so how hard would it be to implement the same features on non-Cisco hardware? Not hard at all, from the sounds of it. Some other features may be hardware dependent, though. For example, Cisco’s ACI architecture relies on information about micro flows obtained from the hardware, so I’m not so sure that’s something that could as easily be ported to some random Broadcom or Intel chip.
Juniper has always been proud in the past that they custom-design their ASICs to do the heavy lifting on their platform, and how they don’t use merchant silicon. That may play well in the core router market, and may be a hardware advantage that gives them additional special sauce when it comes to meeting performance targets, but in the Ethernet switching market is that necessary now? What feature(s) would Juniper want that are not in an off the shelf chip and would necessitate the time and expense required to fabricate a custom ASIC for their switching platforms? In fact, look at the Juniper QFX5100 – under the hood is, guess what, a Broadcom Trident II.
So what makes Juniper different? Ask most engineers, and they’ll tell you straight out that Juniper’s special sauce is the Junos OS. Designed with operational ease of use in mind, the Junos OS really is a pleasure to use. Easy commit and rollback processes, in depth logging capabilities and a standard (-ish) command structure across their product sets, make Junos a nerd favorite, and Juniper, like Cisco, have done their best to shoehorn every new technology into the same standard configuration paradigm without corrupting it too much (except where Ethernet switching is concerned where the pooch has been well and truly, um, groomed).
Feature-wise, does Junos bring something particularly special to the table then? Actually, not so much. Juniper is proud to be standards based; I think Juniper see’s Cisco’s protocol enhancements as an unnecessary bastardization of a standard that just makes their equipment unable to interoperate with anybody else’s. It also means that somebody who chooses the Cisco product will be unable to add Juniper to their network later on too, so that’s another good reason for Juniper to dislike Cisco’s habit of adding non-standard knobs and tweaks to protocols – it shuts them out of environments that already have Cisco in play.
For the hardware side of things, Juniper’s Rami Rahim says in a J-Net Blog Post “Making the Right Silicon Decisions in Networking” that Juniper’s current strategy is to use whatever it thinks is the right tool for the job, be that customer silicon, merchant silicon or a general purpose CPU. By a huge coincidence, yesterday Rami Rahim was announced as the new CEO of Juniper, so perhaps his view will prevail. If the QFX5100 is based around Broadcom silicon and an x86 CPU, could it be easily moved to a white box switch instead? It sure sounds like it.
It’s funny, but while I hear people waxing lyrical about IOS and Junos OS, I’ve rarely heard somebody going on about Arista’s Extensible Operating System (EOS). Arista is perhaps unique out of the three vendors here because their products are all built on merchant silicon, specifically to keep costs down and keep them up to date without having to have a a team of specialist silicon designers on board to make their own ASICs. You could argue then that Arista already does white box switching, but you have to buy their white box to make it work. Well ok, maybe it’s more gray than white, but the point stands.
The linux-based EOS is by all accounts terrific. I’ve never used it myself, but it allowed Arista to steal a lead over Cisco in the high frequency trading space by introducing a very low latency switching platform still based on merchant silicon. I wonder how much of that product relied on special design outside the Ethernet switching ASICs, and how much is purely software.
Arista has also worn the “best value 40G switch” crown for a while, but if the value is in the transceivers and optics, could I just plug those in to a white box switch instead?
So this idea that Cisco and Arista, and even Juniper might offer a version of their NOS to run on white box hardware makes sense to me given that more and more people are using bare metal for their Ethernet switching needs, and if the “big guys” don’t offer software, they’re automatically out of that market. Don’t get me wrong – there are some super players in that space already – BigSwitch and Cumulus are just two – and that’s perhaps an even bigger motivation for Cisco et al to seriously consider getting in there and keeping people in their own product family (software at least) rather than losing the business to somebody else entirely. Let people spend money on the software and feature set that they want, and let the underlying hardware be a fairly irrelevant commodity. After all, look at the Quanta LB4M I picked up recently. Good hardware, it seems, but the ODM software that comes with it is a pile of donkey droppings. If I could put IOS or Junos on there, I’d certainly be happier, although I probably wouldn’t like the cost.
Something that’s less clear with the bare metal switching market is support. Existing vendors like Cumulus and BigSwitch have to have a Hardware Compatibility List (HCL) stating which vendors’ hardware their software works on. But when it comes down to it and there’s some odd bug that arises, do you risk having two companies pointing the finger at one another? As is sometimes said in the networking world, the advantage of having a single vendor implementation is that when there’s a problem, there’s just the one person to shout at.
If we imagine that the bare metal / white box market becomes truly commoditized the same way that things like WiFi Access Points have (lots of no-name vendors selling ODM reference designs), we might see all kinds of options for hardware showing up, possibly at very steep discounts. There’s no guaranteeing the quality of the hardware surrounding the ASICs though. Using the WiFi example, you would have no idea if the company has any clue what it’s doing with its antennas, but the price is very attractive. It’s also of note that some vendors consider the reference designs to be just a starting point from which they can develop a significantly improved product; just implementing a reference design often means implementing a non-optimized product that functions, but not as well as it could.
Put all this together and you will most likely end up with the same thing we see in the consumer marketplace where you can buy equipment with the same fundamental specification, some from a brand you’ve never heard of (and might not ever hear from again) and others from brands you know and trust; and some you know and should know better than to trust. Why buy Known Brand X over Unknown Brand Y? Reputation? maybe you believe that Brand Y does more optimization, or you like their support better. But if you’re deciding between Known Brand X and Known Brand Z who both offer a pretty much identical specification no doubt based on the same reference design, what’s the differentiator? Price? Support? Delivery times? Whose HCL they appear on? Something else?
So here’s the problem: where do Cisco, Juniper and Arista make their money if people can use any old hardware on which to run their software? Cisco and Juniper, I have to assume, make much of their money through the hardware side of the business rather than through licensing software. To be fair, Juniper licensing in general is fairly straightforward and tends not to get in the way of features for the most part. Cisco licensing on the other hand can be absurdly complex and at the top end, can be very pricy. Arista, I simply don’t know.
Cisco and Juniper both make good money on hardware; you only have to order spares or upgrades to know that you are a total chump for paying what seems like an inflated price for things like SFPs, RAM, Flash Memory and so on. It’s incredibly annoying to have software that won’t accept anything but a vendor-branded SFP when you know that the branded SFP in one hand is made by the exact same OEM as the unbranded one in your other hand – presumably to the same specification – and the only difference is in the vendor code with which it identifies itself to the switch. So you end up buying the SFP with the vendor’s name on it for twice the price. Crazy. Arista’s hardware from what I’ve been told is priced very competitively, which is perhaps one reason why have done as well as they have. Arista basing their products on merchant silicon undoubtedly opened up the door to decent profit margins on a lower priced product. And there’s nothing to say that Cisco, Juniper and Arista couldn’t manufacture their own, branded, bare metal switches if they wanted, and use their scale to beat the prices down. They could make Quanta and the others appear like the D-Link of the bare metal world; but is that the market they want to be in?
The money then must be in the software. How do you make a decent living selling software when you’re used to selling hardware? I don’t know; when I worked for a company selling software, the mantra was that selling software was key to profits because you write it once, and sell it many times. Once it’s written, it’s just the cost of burning the software to CDs, so every sale is gravy. Maybe that’s a rather oversimplified view, but the point is there; there’s money to made in software.
Microsoft have battled this problem for a while. They make the software but other people make the hardware, and somehow Microsoft still makes money. Maybe Microsoft’s economy was one of scale; write it once, then resell it at a relatively low price many millions of times over. Then they got back into the hardware business – the Zune, Xbox, buying Nokia mobility… so maybe software isn’t all it’s cracked up to be. But if you get people hooked on your software and your interface at the low end, will they then buy your products when they need something bigger than a Top of Rack switch? Isn’t that the same reason why computer manufacturers like Apple tried to make sure schools and colleges had their computers? For a while I wondered if that was Cisco’s twisted logic when they bought Linksys – that perhaps if you like Cisco Linksys at home you’ll buy Cisco at work. Having used a few Linksys products, I’m fairly convinced that this could surely not have been the intent, or if it was, that somebody was seriously misguided.
The point is that unlike Microsoft selling Windows, you can’t sell many millions of copies of your NOS because there just isn’t that much hardware out there, so it would appear that there isn’t an economy of scale that would allow cheap software to pay for the work spent developing the NOS. And yet Cumulus and BigSwitch are right in that market, doing exactly what I just said sounds impossible, selling software only. If they can do it, why can’t Cisco, Arista and Juniper who also have a hardware sales stream to back them up?
Will They, Won’t They?
I’m going to wave a flag in the air as to whether each of these three vendors will make their NOS available for use on bare metal switches and probably regret it later, but my guesses are that:
- Juniper: WILL NOT
- Cisco: WILL NOT
- Arista: WILL
I think Arista has the most to gain from broadening their user base through licensing their NOS on approved white box switches. Juniper I don’t think will want the hassle of supporting multiple hardware platforms in addition to their own. Cisco I think probably should make IOS available, but I think deep down it goes against every bone in their corporate body to sell the crown jewels without the display box they come in. I do like Jim Duffy’s theory that they might do that just to drive people back to their products; that’s kind of a fun thought.
This is of course a separate argument from whether to offer products for virtualized environments; all three companies are offering products in this space. Only Cisco also sells an x86 hardware platform that it could run on, so there’s an implicit benefit to them there. Either way though, to support cloud services and NFV, they all have to play in this space and will continue to develop their offerings, targeting a seamlessly managed hybrid between the physical network products and the virtual.
I’m curious to hear your opinions based on what you’ve heard or know. Do you think we’ll see these vendors licensing their NOS for bare metal platforms? I’d be delighted to be wrong.
30 Blogs in 30 Days
This post is part of my participation in Etherealmind’s 30 Blogs in 30 Days challenge.
Great article – you hit multiple nails right on the head. Given that, two thoughts:
1) If the rumors I hear are to be believed, all of these vendors have already shipped disaggregated software or hardware to certain key customers, so IMHO the question is not will they disaggregate but rather for what fraction of customers will they disaggregate and how quickly. I agree with your priority list, but my guess is once this particular pandora’s box is open, it’s going to be hard to close.
2) In a mature economics product model (e.g., TVs, stereos, cars, etc.), companies are eventually forced to charge based loosely on internal costs, so it’s interesting to think about what components cost the most. Currently, most incumbents charge a one-time per-box fee for the hardware, give the software away for free, and charge a reoccurring amount for support. That’s fine, but that’s completely different from how their costs works and evidence that networking is (economically speaking) a fairly immature market. Networking costs are a dominated by huge upfront cost for silicon development (if they’re using custom silicon), huge upfront costs of creating the software, and relatively small marginal per-unix costs for materials and support. Obviously I’m biased here (bare metal + software is the basis of my company), but I think it’s a pretty compelling point that merchant silicon (which amortizes the silicon development cost) and disaggregating hardware and software (one time fee for box, subscription for software and support) allows vendors to more directly account for their costs, creating a more mature and rational economics model, and ultimately a better cost/performance experience for the end customer. I’m curious what you think here – any thoughts?
That said, to the point of your article, it will be interesting to see how fast the incumbent vendors pick up (read: are dragged kicking and screaming) this more mature economics model.
Thanks for the article!
Customer demand will drive this more than anything else. Your comment about an odd bug cropping up is right on the money. The market demand for bare metal right now is nascent. If/when customers get more comfortable with the idea, more products will ship.
I think (as we discussed offline earlier) that you’re right about the demand issues – it’s not as hot as it’s going to get, but I do believe it’ll keep on increasing. I think there would be a huge potential market for bare metal switches running familiar Network Operating Systems, although I have to say that with some of the features that these new software-only companies are throwing into the mix, the big players may need to think long and hard about how they approach this market to keep their differentiation as more than simply a familiar CLI.
There’s also the issue of whether or not people want to support the hybrid white box + software (as you mention). For the hyperscale datacenters there are probably enough people on staff to be able to manage it or, in some cases, write their own code to run on the boxes. Enterprises (other than the very largest) probably want something more “off the shelf”, functionally speaking. Maybe they’re a bit more scared to go build their own system.
Maybe it’s a bit like audio systems. Most people used to buy an all-in-one system from a single manufacturer. The people who knew better bought components, but it took more research to make sure everything worked properly together.
The disaggregation we’re seeing is following the history of compute industry in the 80s. Maybe not at such rapid paste since we’re in Business-to-Business zone. There’s no way customers will forgo on cost savings offered by bare-metal. For now they are deterred by healthy conservatism:
a) bare metal is too new and alien (unlike my Cisco/Juniper/Arista box that I know inside out)
b) everyone is shell-shocked by SDN hype and waiting for dust to settle down
c) 95% of workforce knows nothing but vendor networking
Apart from cost-saving there’s also the positive effect of opening up the market to more players.
Only those who were truly pushed by limitation/price of vendor stuff made a transition – first Hypers (Google, FB, Amazon), now it’s turn for big enterprise.
Cumulus was first to go for that opening by offering vendor like experience with support contract:
as long as you are using gear from HCL – it’s a single window affair, they’ll deal with HW vendor (unless I’m misunderstood their FAQ)
So Cisco is worried sick – VMware NSX, the Cumulus and it’s partnership with Dell. Now BigSwitch… ACI sounds slick, but it’s 5 years late to claim the industry leadership. Now they have to fight it out.
Hence the jump to join OCP. Can’t win em – lead em… although some say it was done to stall I think they just want to be in the loop after being late to SDN party.
I agree with you on Big C – they will not split the HW/SW untill last moment. Too much to loose and they’ll gamble on bare-metal being another technology bubble.
Juniper… Do they have much place in DC anyway? After QFabric shenanigans I kinda lost interest. With new leadership they might as well go new places…
Arista is tricky. They beat Cisco by being a better version of Cisco (if that makes sense). Definitely less stacked against bare-metal, but probably not going to do it unless there will be clear demand.
On the separate note – I wonder when this will spill out of DC and hit the Campus/LAN? This is a big market that is being transformed by influx of wireless and consumerization.
Thanks for the comments, Kanat. You raise an interesting point here:
“a) bare metal is too new and alien (unlike my Cisco/Juniper/Arista box that I know inside out)”
What if Cisco decided to enter the bare metal market? Pitch a software-less platform and use their own economies of scale to price slightly higher than the competition and rely on brand recognition to clinch the sale (nobody ever got fired for buying Cisco, as the saying has it). Not a huge stretch when you look at the Nexus 3k, right? I wonder if any of the other software vendors would ever certify that product and put it on their HCL?
Of course the existence of such a switch would likely cannibalize their existing products, so it will likely never happen. The irony would be that if you could persuade somebody to buy a “bare metal” Cisco switch and then license IOS to run on it, it’d be just like buying Cisco from the start – but their own products would use custom ASICs to provide enhancements and extra services not available in the Top-Of-Rack bare metal switch products.
My head is spinning.
“What if Cisco decided to enter the bare metal market?”
hmm… that would validate bare-metal, and Cisco is not showing a change of heart.
Say they would though – what can they really offer over merchant silicon?
Are we talking AMD vs Intel kinda deal – speeds and feeds + price race? With no standards defined on what constitutes a bare metal switch I can’t see it through.
I’d suggest another player to jump on the bare-metal wagon – HUAWEI. Chinese are not so amazing at writing their own IP code. But they can do HW well.
This could be like Android for chinese smartphones. Vendors historically struggle with them in emerging markets and with bare-metal it could strike off the last argument against purchasing chinese gear – “the communists will hack my network”. Although nowadays it’s a pretty equal opportunity game in terms of backdoors.
That’s a well considered article. However Juniper has moved into this space. They just release vMX (www.juniper.net/us/en/products-services/routing/mx-series/vmx/) so your prediction didn’t stand up for very long. 😉
Your article has given me food for thought, about the longer term implications of this move. Thanks for the article. 🙂
Thanks, David. For me, vMX falls into the NFV on x86 category rather than a NOS to run on bare metal, if I’m understanding its capabilities correctly. This isn’t such a surprise because Cisco already has the CSR1000V, Juniper also has vFirefly and I believe Arista’s EOS runs in a VM as well. In other words, virtualizing the NOS to run on x86 is not unusual by any means.
So if it’s ok, I’m going to wriggle out of my immediate prediction failure and say that vMX doesn’t count. I’ll be wrong (in my eyes) when I can buy a bare metal switch built on merchant switching silicon and install Junos. Does that sound fair? 😉
Fascinating article and well thought out. Interestingly almost year later (!) Juniper announces they are disaggregating Junos. Go figure.
*laughs* And hey, I walked all the way down the path (and I was there at Juniper NXTWORK listening to Rami Rahim say almost all the same things I said in this article), but I jumped in the wrong direction at the last moment when I made my predictions.
Interestingly, speaking to Pradeep Sindhu, his comments strongly underline the statements I mentioned about using the right hardware in the right place, not a one-size-fits-all policy. I’m going to write that up shortly, and I’ll try to link back to this article because really it explains a lot about how Juniper sees disaggregation working for them.