hey guys,
I have reached out to TAC referring this issue, but curious to see if anyone else has come across this as well. We are running a 3 switch 2960x stack which consists of:
* 1 52 WS-C2960X-48FPD-L 15.0(2)EX5 C2960X-UNIVERSALK9-M
2 52 WS-C2960X-48FPS-L 15.0(2)EX5 C2960X-UNIVERSALK9-M
3 52 WS-C2960X-48FPD-L 15.0(2)EX5 C2960X-UNIVERSALK9-M
We are seeing high CPU usage relating to this process:
CPU utilization for five seconds: 40%/1%; one minute: 40%; five minutes: 40%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
163 110518151 10468050 10557 22.55% 22.50% 22.25% 0 Hulc LED Process
169 9068 6569 1380 0.59% 0.09% 0.16% 1 SSH Process
I shut down the ports that are not in use (which will be) and the process goes down significantly.
Thoughts?
Did a search on "hulc LED process" and there were LOTS of hits asking about the high CPU on that process, going back a few years. Cisco claims it's fixed in 12.2(55), but people still see it. Shutting down ports, especially SFPs, is a workaround to getting the CPU utilization down, but opening up the ports sends it right back up again.
What IOS are the switches running?
Bug "CSCtg86211" Don't know how helpful it would be for your situation.
https://tools.cisco.com/bugsearch/
well known bug with switches running 15.x code.
we fight it every few months, Cisco's recommendation is to shut down all ports not in use and reload when CPU peaks over 75%.
@seittit
thanks man.... i thought as much. Everything I was reading seemed to have applied to S-series switches.
I plan on turning off the extra ports when fully finished migrating, i went in and downed about 30-40 ports, and instantly saw a 10% decrease in CPU.
@dean
I did read that article that you read, I turned off the extra SFPs, and it went down about 5-6%
We've been seeing this with 2960S and 3750X stacks for a couple years now. This issue is one of the reasons we've decided where I work to not stack 3750X's more than 5. Even at 5 sometimes they're just plain sluggish on the command line.