Where should your switches handle L3?

[deanwebb]

Architecture / design question... where is the right place to have the switches deal with the L3 stuff at a site?

Obviously, if it's a tiny site, everything is collapsed, so we leave that aside. Let's consider a campus, with around 5000 users in several buildings.

I see a few options to guide the design:

L3 in the core only

L3 in the distribution switches in each building's main comm room

L3 in the distribution switch in each wiring closet, 2 per floor

Consider generally even distribution of users on the floors of each building.

Where do you put the L3 functions and why?

[Otanx]

I like pushing L3 as far down as I can for a campus. I was joking with a coworker I wanted to do /31 DHCP scopes, and every workstation is on their own. The main reason is failure isolation. Broadcast storm, arp spoofing, IP address conflicts/stealing, etc. Are all limited. Also I get to use OSPF instead of HSRP/VRRP for redundancy if every switch is L3. ECMP works better.

Of course not all products want to work this way. A lot of stuff wants L2 adjacency to work. You then need to figure out how to handle those items. Campus VXLAN is becoming a thing because of that. Then you get to do L3 with a L2 overlay just like in the DC. I think it is just a band-aid. We need to push vendors to get rid of the L2 adjacency instead of finding ways to make it work. I think an overlay in the campus is just adding complexity for no good reason.

Another downside is L3 to access usually is more expensive. You also may run into other issues you didn't expect. One I have hit was two security domains down to the access layer. Cat3K needs IPServices to do VRF Lite which looking at CDW pricing doubles the price of the switch. If you did L2 a VLAN gives you the separation, and you put the gateway on the firewall, or at the core where you have vrf-lite. The other option is deploy another switch for the other security domain. Then you are doubling your fiber runs, optics, etc.

This is all for wired as well. Wireless is different. Especially if you want roaming. I don't do wireless, but my first design would still be doing L3 for wired, APs connected, and route back to a controller. Then tunnel the wireless traffic to the controller.

-Otanx

[deanwebb]

Roaming is pretty much gonna happen with wireless. And, yes, it's s totally different beast.

[Nerm]

In the end it is a case by case and business need decision. If however I had my way I agree I would push L3 as close to the end users as possible.

The campus project (~6k users across 18 buildings) I just finished in China we did L3 at distribution points in each campus building and then IGP back to core. The IGP we had to use wasn't what I would prefer but we were constrained by some prior business decisions and just had to make it work. In the end it worked out pretty well even with all the "China" difficulties we had to deal with.

[icecream-guy]

I was going to half jokingly say L3 on your border firewall with one big giant flat IPv6 Network.

[wintermute000]

L3 > L2 if it fits requirements/budget every single time. Everything is better - control, convergence, stability, visiblity.

Unfortunately I have yet to meet the customer who doesn't have a requirement for a VLAN to span multiple switches. I've never seen a L3 to the edge deployment IRL.

I have seen L3 at distro plenty, however, this is a 2010 era design. With modern capacity only the largest campus sites or some ridiculous physical / cabling layout needs a distribution layer. Look at the capacity figures on a Cat9500 and tell me that's not enough for your HQ (and they are campus users, not multi-tenanted servers or storage).

This is why everyone is pushing fabric solution where any port in the fabric can be in any "VLAN".