Arista vs Nexus, which would you rollout?

[LynK]

Hey guys,

Looking to lean on a few more experienced DC guys here. We are contemplating purchasing 9504 chassis, with the new FEX's, however there is the possibility of needing DCI. Would you rather go with arista/nexus?

pros/cons of each?

[that1guy15]

The Cisco Nexus 9Ks are pretty much the only DC switch I would still recommend if Cisco is your choice.

You have not touched really on any features and requirements needed outside of possibly needing DCI, so its kinda hard to nail down a solid choice. But I dont mind throwing my opinion in the mix.

Right now I would deploy Arista in almost all scenarios unless I was forced into Cisco or had solid reasons. They have a solid product line and dealing with code upgrades, versioning is much easier than the mess Cisco Nexus is. I like their VxLAN option for both internal fabric and DCI. CloudVision is a great tool if leveraged properly and really simplifies management. Cisco well... they can sell you ACI! or Prime :|

From what I have been playing with Arista is also way ahead in modernizing their EOS platform. From a solid single API to YANG and NetConf across the board. This helps a TON when talking automation or pulling structured data out of your device instead of the old CLI way. Cisco is still a steaming pill of shit IMO.

Telemetry and real time analytics. They are both getting there. Arista though its just gonna be a code bump and module install. Where Cisco it looks like hardware refresh to get gear supporting this stuff. I could be wrong though.

As for DCI depending on how complex you are looking to go with it, Arista has a number of options all from the same hardware. Unlike Cisco if you decide to go with advanced ISP technology ( C/DWDM, MPLS, Segment Routing) you are not gonna get much out of the Nexus from last I checked. Would have to go ASRs for DCI. But then again, Aristas Service provider technology is still pretty new and that makes me somewhat nervous. Id want some serious lab time to bake it in or give it another year or so to iron out bugs.

If you are adventures my second recommendation would be to check out whitebox and go Cumulus+Apstra or Arista+Apstra for DC fabric. But this would only be feasible in a mostly greenfield deployment with how Apstra does intent driven networking. Also this is a pretty big shift into automation and newer technologies, so there would need to be a shift in how the team runs their day to day. But damn it would be cool!

[wintermute000]

You have not touched really on any features and requirements needed outside of possibly needing DCI, so its kinda hard to nail down a solid choice. But I dont mind throwing my opinion in the mix.

Right now I would deploy Arista in almost all scenarios unless I was forced into Cisco or had solid reasons.

Agree except with caveat that there is presently no EVPN story (lacks loads of features that are a given in Juniper/Cisco), so right now, you have to be content with the 'traditional' CVX solution for a control plane.

In fact Juniper are clearly the technology leader in the EVPN space (right now). ESI blows the crap out of vPC or MLAG, their silicon has the highest numbers (outside of the latest gen of Arista software-tweaked Jericho (I think?) merchant silicon e.g. the FiB compression algos etc.), above all, JunOS stack re: automation API > everything else, they've been doing it for the longest and at service provider scale. But going to the J team is a big obstacle for most Cisco shops, whereas the Arista syntax makes it really easy to switch. And Arista do have excellent API. Not a fan of how they went eos_template with Ansible, but I digress....

Cloudvision runs RINGS around DCMN if you want a turn-key (well nothing in this SD space is really turn-key... but I digress again) automation platform.

But yes circle back to requirements, requirements and more requirements.

BTW Would you trust Apstra? We've had the dog and pony show, looks nice but very beta... only 1 use-case (leaf/spine) 1 topology.... have you run proper evaluation vs DCMN/VTS and Network Director and Cloudvision?

[burnyd]

Actually, there is a broad EVPN type 2 through type 5 play right now and it works flawlessly.

http://www.eantc.de/fileadmin/eantc/downloads/events/2011-2015/MPLSSDNNFV_2017/EANTC-MPLSSDNNFV2017-WhitePaper-Final_v2.pdf

^^ If you have the time this is an excellent read where they are not biased towards any real hardware vendor.

Nothing wrong with the Juniper automation if you have the time to go through 100000 lines of documentation and want to work with netconf if you feel that is the real future of automation.  Arista is highly embracing openconfig for newer ways of automation.  If you want to use the api its really simple and uses CLI as the transport so it makes it real for most network people.

As for eos_template its actually depricated.  You need to use eos_config .. I am assuming you are referring to jinja2 templates. Jinja is great.

As far as your last question goes I would just wait for open config to mature.

[wintermute000]

I'm well aware of Type-1 through to Type-5, I meant Arista's EVPN implementation. @ my last meeting with your guys down under, they advised me that they still hadn't gotten ARP suppression nor symmetrical-IRB (inter-VXLAN IRB via Type-5) nor IGMP snooping + a whole bunch of other caveats on how the underlay/overlay needs to be setup (e.g. only BGP allowed, can't have say OSPF underlay). Are you saying that this info is wrong? Coz I was shown a slide saying its not possible with the Arista logo on it, + roadmap for when stuff would get knocked off... NDA and all that, but I'm sure you've got the same slides. I can PM you the SEs involved if you think some wires are crossed or loop into an email chain?

Then again reading your attached doco I note specifically the Type-5 symmetric IRB working so I am going to prod your colleagues a bit... I won't mention where I got this from   or maybe this was on unreleased software (likely scenario I suppose).

Not meaning to vendor bash on you guys specifically, but as EVPN is a hot topic, I must admit I was taken a bit aback by the above info. Nothing wrong with CVX if that fits your requirements, I know a lot of people use it.

Yep I meant eos_config (the jinja2 stuff) - as much as jinja2 works great, I'm a bit sad that its at the end of the day still focusing on getting a show run, complete with having to care about indents etc. (though you could level that argument back @ YAML I guess). I must admit I haven't looked at specifically how JunOS interacts with ansible.

re: Juniper automation, I get what you mean, though have you tried pzEZ? A router/switch as a python class. Magnificent. (basically what Cisco wanted to do with onePK except dropped it like its hot). I'm also pretty sure their RESTAPI is feature complete.

At least we can all agree Cisco is #3 when it comes to automation LOL

[burnyd]

I'm well aware of Type-1 through to Type-5, I meant Arista's EVPN implementation. @ my last meeting with your guys down under, they advised me that they still hadn't gotten ARP suppression nor symmetrical-IRB (inter-VXLAN IRB via Type-5) nor IGMP snooping + a whole bunch of other caveats on how the underlay/overlay needs to be setup (e.g. only BGP allowed, can't have say OSPF underlay). Are you saying that this info is wrong? Coz I was shown a slide saying its not possible with the Arista logo on it, + roadmap for when stuff would get knocked off... NDA and all that, but I'm sure you've got the same slides. I can PM you the SEs involved if you think some wires are crossed or loop into an email chain?

I dont feel like quoting because its all messed up so response in bold.

Check that link out both connected and non connected IRB and ARP suppression are both within there. 

BGP is the only underlay allowed.  You should only BGP all the things no idea why you would want to OSPF stuff.

Keep in mind this is only on a few model switches.

Not meaning to vendor bash on you guys specifically, but as EVPN is a hot topic, I must admit I was taken a bit aback by the above info. Nothing wrong with CVX if that fits your requirements, I know a lot of people use it.


Yep, Im all on board with EVPN.  If you do not need a type 5 of vrf isolation then go cvx.  But looking at that NFV conference nobodies stuff right now is compatable with anyone elses!


Yep I meant eos_config (the jinja2 stuff) - as much as jinja2 works great, I'm a bit sad that its at the end of the day still focusing on getting a show run, complete with having to care about indents etc. (though you could level that argument back @ YAML I guess). I must admit I haven't looked at specifically how JunOS interacts with ansible.

You can run any CLI command against the switch with the eos_cli command and then you can pretty print it or even ask for certain parts of the configuration.


At least we can all agree Cisco is #3 when it comes to automation LOL

lololololol

I know where we do good and I know where we dont.  We were late to the evpn game.  But once everyone see's that fit as the DC fabric of the future and everyone can kind of talk to everyone elses stuff including type 3 IRB its going to be awesome.  Use openconfig to orchestrate everything and you will have a pretty cool automated fabric data center.

[LynK]

We are still early in this project, but we are possibly looking to extend out datacenter to a new building with dark fiber from a carrier inbetween. We have had discussions on whether these datacenters are going to be autonomous (L3), or L2 extensions. We do not want to make this ridiculously difficult if we do not have to. I have heard some significant horror stories about OTV, which is also making me look to other competitors. Price is also another big reason, I know the 9504's are cheap, but if arista offers the best option, for the best price, I find it hard pressed to go cisco.

As for juniper, I have little to no experience. We purchased a 3300 for testing, but other than that I have no experience. I cannot stand their CLI, and I do not know if I am comfortable implementing a first time vendor into our core when I have no experience.

The biggest reason I see a need for DCI would be virtualization, and vmotion. I can see a need to move vms between data centers. I am trying my best to push L3 separation, but I am not sure if our proprietary connections will allow for that.

I am also interested in arista because they have full routing features/functionality on their 7500R platform. This makes it easier to push a product knowing we will not have to purchase anything additional when we can do our internet edge/routing all on one machine.

[deanwebb]

The biggest reason I see a need for DCI would be virtualization, and vmotion. I can see a need to move vms between data centers. I am trying my best to push L3 separation, but I am not sure if our proprietary connections will allow for that.

All the DBAs and sysadmins say in unison:

PLZ LAYYER 2 ADDJASENSEE ERRYWHERE N THA DATASENTIR PLZ KTHXBAI

[burnyd]

The biggest reason I see a need for DCI would be virtualization, and vmotion. I can see a need to move vms between data centers. I am trying my best to push L3 separation, but I am not sure if our proprietary connections will allow for that.

All the DBAs and sysadmins say in unison:

PLZ LAYYER 2 ADDJASENSEE ERRYWHERE N THA DATASENTIR PLZ KTHXBAI

Do not roll out OTV.  You will pretty much lock yourself into a platform and technology for life.

I am sort of biased because I do work for Arista.  7500R platform is fantastic.  Both for buffers, speeds feeds and the typical Arista automation.  The chip itself its extremely similar to the chip in the 7500E in many ways. So the reliability is still there. Its practically a swiss army knife in all the features it supports.

But as Wintermute and I have said EVPN has been amazing.  Like if you replace anything at this point EVPN and VXLAN are the technologies you would want to go with.  Typically MPLS has been pretty expensive although we have some boxes now and other vendors have some cheep boxes but everything supports VXLAN/VXLAN routing wether its single pass or not its 100% the way to go imho.

[wintermute000]

We are still early in this project, but we are possibly looking to extend out datacenter to a new building with dark fiber from a carrier inbetween. We have had discussions on whether these datacenters are going to be autonomous (L3), or L2 extensions. We do not want to make this ridiculously difficult if we do not have to. I have heard some significant horror stories about OTV, which is also making me look to other competitors. Price is also another big reason, I know the 9504's are cheap, but if arista offers the best option, for the best price, I find it hard pressed to go cisco.

As for juniper, I have little to no experience. We purchased a 3300 for testing, but other than that I have no experience. I cannot stand their CLI, and I do not know if I am comfortable implementing a first time vendor into our core when I have no experience.

The biggest reason I see a need for DCI would be virtualization, and vmotion. I can see a need to move vms between data centers. I am trying my best to push L3 separation, but I am not sure if our proprietary connections will allow for that.

I am also interested in arista because they have full routing features/functionality on their 7500R platform. This makes it easier to push a product knowing we will not have to purchase anything additional when we can do our internet edge/routing all on one machine.

Sounds stock standard. There's very little besides inertia/politics/documentation (publicly available white papers, design guides, you know the score) to recommend Cisco except for EVPN maturity although burnyd has given his refutation (and I personally am going to hit up some official channels to get some clarity, anything not NDA happy to share).

re: DCI and OTV etc., the horror stories are from people who don't design their routing properly and don't properly think through the implications and traffic flows (FOR ALL SCENARIOS) for stretched L2, especially stateful devices in path, or trying to avoid that problem via stretching FW clusters etc (run away!). You don't sound like you have enormous scale, Arista VXLAN is rock solid and will tunnel anything including EIGRP peerings (have seen this live... not recommend, but the point is that its a great solution), flood and learn isn't as big a deal as it sounds until scaling is a problem.

However,  with Anycast GW topologies (esp. again EVPN....) and VXLAN IRB the ballgame changes, you might want to stop thinking re: traditional DCI as in L2 tunnels point to point. It also starts to get vendor/HW specific (e.g. the old Trident2 can't IRB on same pass caveat).

burynd/aspiring, I think it would be of benefit if you were able to share the old Arista VXLAN Bridging and routing slides (CHI-NOG 05, Darrin Machay) as an intro? Centralised vs Direct vs Indirect vs Underlay routing etc...  As for EVPN, this book seems good (reading it now) - Building Data Centers with VXLAN BGP EVPN: A Cisco NXOS Perspective - it is biased though (as you'd expect) but a good summary so far

[burnyd]

In all honesty the Juniper MPLS in the SDN era is probably the best reference for EVPN it has both MPLS and VXLAN for EVPN.

The world according to burnyd next year will be EVPN / Openconfig for intent based network configuration.  However, that is just me. 

IRB with VXLAN and EVPN is the real deal for sure!

All those things that Darren did on the slides were all a result of that previous nfv link I sent.  Pretty much everything is there. 

[NetworkGroover]

Hehe - this thread is funny.

[LynK]

Just got a list price quote from arista. VERY EXPENSIVE. We are going to be looking at their other options (were looking at their 7504R platform). It was going to be like ~400K list. lol.

[wintermute000]

you said it yourself - list - what's the real price?

note I have seen this before i.e. Cisco winning on pants-drop pricing. That, and CIO level conversations. lawsuits don't help.

out of curiosity why are you looking @ a 6RU chassis switch? Can the 1-2RU options in leaf-spine not handle your requirements? I know the Jericho switches are the new hotness but can a 'traditional' 1/2RU fixed leaf-spine scale-out topology handle your requirements.

burnyd, that MPLS in the SDN era book is possibly the greatest technical textbook I ever read. Really wish I had the time (and real-life XP and opportunities) to go into the SP space but alas I'll never get the required prod time on core SP infra to fully get into it. MPLS is fascinating

[burnyd]

Yep they will in your words do pants dropping prices just to compete.

That Juniper book is amazing.  Even at a company like Arista you would be really shocked at some of the large SP's what they are doing now a days.  Things like small pockets of segment routing, UCMP just really whacky neato stuff.

But anyways, thats list price!