Stacked switch CPU load balancing

config t · August 17, 2015, 05:33:01 AM

Is there a way to force stacked switches (2 3560's in this case) to load balance IP forwarding across their CPU's? I'm dealing with a traffic bottleneck caused by high CPU utilization on a legacy 3560 installed to act as a core switch.

Mostly I'm asking as a gee-whiz at this point.. I've advised them to upgrade that thing post haste.

wintermute000 · August 17, 2015, 07:28:30 AM

are you logging ACLs or something like that. normal packet forwarding is CEF and doesn't really stress the CPU, look for anything that punts to CPU or rogue processes
oldie but goodie

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html#wp1026038

Reggle · August 17, 2015, 10:38:14 AM

Don't count on it. Practically everything is done with the stack master CPU.

icecream-guy · August 17, 2015, 10:54:48 AM

3560's can be stacked ? I'll believe it when I see it.

oh, 3560-X? I get it now.

NetworkGroover · August 17, 2015, 10:56:13 AM

One of a few reasons some folks don't like stacked switches... but I don't work in the campus space much at all anymore so my info may be severely outdated.

LynK · August 17, 2015, 03:46:37 PM

Quote from: ristau5741 on August 17, 2015, 10:54:48 AM
3560's can be stacked ? I'll believe it when I see it.

oh, 3560-X? I get it now.

Silly cisco.. taking a garbage switch and making it feasible just by adding an X...

srg · August 17, 2015, 10:13:01 PM

Uhm.. 3560X doesn't stack. And packet forwarding isn't done in the CPU, unless you're doing it really really wrong.

config t · August 18, 2015, 02:51:06 AM

Quote from: wintermute000 on August 17, 2015, 07:28:30 AM
are you logging ACLs or something like that. normal packet forwarding is CEF and doesn't really stress the CPU, look for anything that punts to CPU or rogue processes
oldie but goodie

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html#wp1026038

I'm going to be chasing this line of thinking as soon as I finish this post, thank you. It's a weird issue that has popped up over the last few weeks. The network in question hasn't grown, so it has been hard to wrap my mind around.

Quote from: ristau5741 on August 17, 2015, 10:54:48 AM
3560's can be stacked ? I'll believe it when I see it.

oh, 3560-X? I get it now.

Now I feel silly.. and I had just read last week that the only real difference between 3650 and 3750 is stacking. I swear sometimes I have the memory of a goldfish.

Top 3 sorted for reference.. going to have to dig deep on this one.

Code Select


CPU utilization for five seconds: 88%/23%; one minute: 70%; five minutes: 72%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 191  2554873458   5050195     505981 25.95% 26.73% 27.99%   0 IP Input
   4   172639165   7687804         22456 13.69%  1.62%  1.14%   0 Check heaps
 172   312627154 309262764      1010  5.09%  3.07%  3.17%   0 Port-Security

icecream-guy · August 18, 2015, 06:57:01 AM

Quote from: srg on August 17, 2015, 10:13:01 PM
Uhm.. 3560X doesn't stack.

I thought I read that yesterday, must have misread that one.

config t · August 18, 2015, 11:58:00 PM

Well we figured it out.. someone had a scanner going all

Right from the start I asked if anyone was scanning.. but of course the answer was no. The answer is always no until you catch them red-handed. Sometimes I feel like half my job is being a lawyer and the other half is being a babysitter.

Reggle · August 19, 2015, 02:23:35 AM

Quote from: config t on August 18, 2015, 11:58:00 PMSometimes I feel like half my job is being a lawyer and the other half is being a babysitter.

You're not alone in that one. You'd expect people to cooperate towards a solution, yet not telling the truth seems to be preferred often.

NetworkGroover · August 19, 2015, 12:31:18 PM

Quote from: Reggle on August 19, 2015, 02:23:35 AM
Quote from: config t on August 18, 2015, 11:58:00 PMSometimes I feel like half my job is being a lawyer and the other half is being a babysitter.
You're not alone in that one. You'd expect people to cooperate towards a solution, yet not telling the truth seems to be preferred often.

UuUUUGGGGHHHH I hate that! It's so frustrating!!!! The worst case I remember was when I was working at Websense Tech Support and working with this lady who was DEATHLY afraid of losing her job. I made sure everything was good to go on the proxy, and when I started asking questions about the network (the part she's responsible for), she became extremely evasive. Being remote support, I'm kind of at the mercy of what remote sessions I can set up and what information I'm fed by who I'm working with. So several days drag on where people's Internet access has slowed to a crawl, I've showed her through various data points that it's not the proxy. (People always blame the network, unless they have a proxy, then they blame that instead first, always) Long story short, I was finally able to force her to give me some switch output and her utilization was through the roof on several core switches. On future support cases with that customer, I was working with someone else, so I think she did in fact lose her job. ;(

config t · August 20, 2015, 01:52:38 AM

Quote from: AspiringNetworker on August 19, 2015, 12:31:18 PM
People always blame the network, unless they have a proxy, then they blame that instead first, always

Quote from: Reggle on August 19, 2015, 02:23:35 AM
You're not alone in that one. You'd expect people to cooperate towards a solution, yet not telling the truth seems to be preferred often.

This.

At least I learned some cool stuff, I feel like I read a book in Elder Scrolls and leveled up my troubleshooting skill. Cisco has amazing documentation.

I'm still mildly disappointed we can't load balance across CPU's. I bet they did that on purpose to sell more o' them sexy chassis.