Switch SNMP Connectivity Issue

config t · February 05, 2024, 05:49:16 PM

On one of the enclaves I support there are around 190 switches and routers managed by my NAC tool. The switch objects are configured with profiles so the settings are the same for each group. Several of the switches are getting SNMP cred fail errors and the answer doesn't seem to be obvious (to me).

-the snmp service account and snmp group/view is configured the same across switches
-snmpwalk from the connecting appliance says cred fail
-all of the switches are on the same network with connectivity to managing appliances, so pretty sure its not fw related
-switches are same model/version (C9300)

Can't figure out what I am missing. I think we already looked for an inbound/outbound acl. MIB? I would think MIB is the same on a particular version.

I hate to say it could be something as dumb as a wrong password. Nobody can seem to agree on what the SNMP password is for this particular account. Terrible PAM practices should be frowned upon.

Otanx · February 06, 2024, 09:37:02 AM

If nobody can agree on the password is then that is probably your issue. Using SNMPv3 there are five items you need to make sure all match.

- Username: kind of obvious
- Password: kind of obvious
- Encryption Key: This is different than the password. The password authenticates the user, the key encrypts the data.
- hashing algo: This is how the password will be hashed. MD5, SHA, etc. Not all systems support the same algorithm so you need to pick one that all the systems can do. If it only says SHA it is probably SHA1, and not SHA2/SHA256.
- encryption algo: The is the encryption that will be used. 3DES, AES, etc. Same as hashing. Not all support the same ones. Also if the choice is just "AES" it is probably AES128 not AES256.

If you are getting credential failure then the encryption algo and key are probably OK. It has to decrypt the packet to authenticate. Then check hashing, password, and username. If you can get a pcap wireshark can decrypt SNMP you just have to supply the info under the protocol preferences.

-Otanx

deanwebb · February 06, 2024, 09:45:27 AM

I'm assuming from the context of the post that this is SNMPv3.

SNMPv3 cred fail means that there can be an issue with the auth, the priv, the appliance not being in the SNMP server group, the appliance IP address not being in the ACL that controls SNMP communications, or the SNMP setup limiting MIBs and one that the appliance requires is missing from the permitted MIB list.

Most common mistakes are the auth or priv mismatched/mis-set. Next most common are not adding IPs to server lists/ACLs. And if these guys actually defined MIBs to use, I'd try an access-all for the appliance just for troubleshooting and if that works, then the MIB list is in need of at least one more entry.

EDIT: Otanx ninja'd me!

config t · February 06, 2024, 09:41:18 PM

Yeah, v3.

All of the settings you guys mentioned are correct. I will know about the password and encryption key by tomorrow. The guy has been out of office for several days so it has been a backburner issue.

I'm pretty interested in learning to use the SNMP decrypt function in wireshark now.

icecream-guy · February 07, 2024, 08:53:45 AM

try using SNMP get commands, SNMP walking is super intensive on device CPU

try it with no auth/no priv to test basic connectivity:
snmpget -v 3 -u <username> -l noAuthNoPriv -a SHA -A <Password> -x AES -X <Password> <IP Address> system.sysDescr.0

once you get that working, verify authorization works:
snmpget -v 3 -u <username> -l authNoPriv -a SHA -A <Password> -x AES -X <Password> <IP Address> system.sysDescr.0

one that works, verify auth and priv work:
snmpget -v 3 -u <username> -l authPriv -a SHA -A <Password> -x AES -X <Password> <IP Address> system.sysDescr.0

this will help identify where the issue resides.

config t · February 07, 2024, 05:23:19 PM

Ok so snmpget is the new thing I didn't know I needed until you told me what it is.

I played with it for a bit today and had some errors about password length that confused me for a little while. Turns out Linux (CentOS in this case) doesn't like special characters in passwords so you have to escape them to make it work. "\"<password>""

Definitely a wrong password, can't auth.

deanwebb · February 08, 2024, 08:48:25 AM

Oh yeah, no special characters in passwords is a thing with SNMPv3 on that CentOS. Bit me more than once at Forescout.

config t · February 08, 2024, 08:58:03 PM

Did you run into unpredictable behavior? It works on most of the switches but there are a substantial amount that it doesn't. Again though I can't be confident in their PAM.. because it doesn't exist.

Otanx · February 09, 2024, 09:05:05 AM

Most systems won't like some special characters. For linux try to stay away from any quotes " ' ` slashes /\ ampersand & dollar sign $. All of those have special meaning and have to be escaped to work, but it is just easier to not use em at all.

-Otanx

icecream-guy · February 09, 2024, 03:08:27 PM

Quote from: Otanx on February 09, 2024, 09:05:05 AMMost systems won't like some special characters. For linux try to stay away from any quotes " ' ` slashes /\ ampersand & dollar sign $. All of those have special meaning and have to be escaped to work, but it is just ~~easier~~ more secure to ~~not~~ use em at all.

-Otanx

sorry had to do that..

deanwebb · February 10, 2024, 08:31:50 AM

"We have a password that no machine account can possibly use!"
"You're welcome!"