NTP authentication, and SNMPv3,

[icecream-guy]

Seems with the ongoing GAO audit here at work, we are moving to implements NTP authentication, and SNMPv3, we are facing challenges.

for one, some of out tools don't support NTP authentication nor SNMPv3 e.g. stealthwatch, Riverbed
ASA firewalls. will support both, but seems like SNMPv2c and SNMPv3 not from the same ip, more testing and TAC case opening soon near you
NAC Forescout, no support for NTP Auth.  and still testing SNMPv3
APC UPS'  neither supported

Have you gone this route? and what were your findings ?  specifically with the network gears/tools is of interest.

[deanwebb]

I'm on it with the question about when we support NTP Auth. SNMPv3 works, but the underlying OS of the device better also support it... for both Cisco and Juniper gear, the general rule is the later the release, the better the support.

[Otanx]

NTP Auth is a shit show. That is still on our to do list because of compatibility issues. Note that NTP Auth is only a STIG item for network gear. Windows, and Linux systems do not have a requirement for NTP Auth so your server guys will not care. I don't know if it is fixed, but when we started this if you used a Cisco IOS device as your NTP server and configured auth then your up streams also had to use auth with the same key. Also using VMs for NTP Servers will cause issues (even without auth).

SNMPv3. Was actually easy once you get into it. On ASA traps will source from the interface closest to the destination. There is no "source interface" command. Same for polling. You need to query the closest interface to the server. If you try to poll another interface the ASA will silently drop it. However, if your request comes in on a IPSec tunnel terminated on the ASA you can poll any interface except the interface the IPSec tunnel terminates on, but the snmp-server host entry needs to be configured for that interface. This is a pain when DNS is configured for one interface, but the monitoring server comes in from another. We have host entries on the monitoring server for the ASAs to fix this for now. This "nearest interface" issue with ASAs also applies to ping, and SSH.

Items we found that don't do SNMPv3 were usually older gear, and "IoT" stuff like PDUs, temperature monitors, etc. The old Cisco FWSM only does v2c. Also some items that do v3 only will support des/md5 instead of aes128/sha. We had pretty good luck with pinging vendors to update their code. Especially when we pointed to government requirements. I think we finally got rid of the last item that didn't do v3 with AES.

If you are already monitoring with v2c you need to remember that v3 will increase the load on your monitoring server. It may not matter, but with a few thousand items we noticed a big change in load. If your monitoring server is on Linux make sure you have the right snmp libraries loaded. One supports multi-threading, and one does not. If you are halfway current you should be fine, but if it is an older server you may want to double check.

The big operational issue we had when we did SNMPv3 was doing password changes. You have your monitoring server, and then the device has the username/password configured. So during password changes you will break authentication until you update both sides. To fix this we put a date stamp in the username. So instead of the username r1-snmp-sw for "router1 snmp solarwinds" we set r1-snmp-sw-20180912. Then when we need to change the password we create a new user with a new date stamp, and push it to the device. Then change it on the monitoring server. Verify everything works, and remove the old username. Side benefit is I can tell if I have to change passwords just by looking at the username.

-Otanx

[Dieselboy]

We have host entries on the monitoring server for the ASAs to fix this for now. This "nearest interface" issue with ASAs also applies to ping, and SSH.

DNS not being up causes havoc with virtualisation platform like Red Hat Virtualisation and Openstack. I've mitigated this by using host entries on the nodes so that they are not reliant on our DNS servers being up. Even if they have 2 DNS servers configured and the primary goes down, admin portal grinds to a halt. The Openstack tripleO deployment I used had created a ton of host entries automatically for itself during install. In my experience it's best not to rely too heavily on DNS for operational stuff or monitoring because it can be unpredictable.
I have just begun a Business Continuity Plan (phase 1 at the moment) and I'm thinking ahead to how it can actually be accomplished. I am really hoping I dont have to F around with DNS to make it work  I'd prefer to use GRE tunnels to stretch a L2 VLAN over an IPSEC VPN (to the DR site) because it would be easier and less painful I think.

[wintermute000]

you can't stretch L2 over GRE. You CAN run MPLS over GRE over IPSEC if you're game ROFL. L2TPv3 also works, but I'm not sure over IPSEC.

[Otanx]

I really hate host entries. Too many times a host entry was left behind, and nobody knew about it. Then something changes, and hours are spent troubleshooting. If DNS is broken then fix DNS. In my case I need to either use DNS Views, or setup a monitoring host name that resolves the IP. The host entry was a quick fix till I can get a real fix in place (I know, temp fixes always become permanent). I have seen a write up on using host files controlled by Ansible. That was an interesting idea, and I could see how that works really well.

We may be doing MPLS over GRE over IPSec in the future. I might be able to just do MPLS over WAN MACSec. Need to get the lab up and running again.

-Otanx

[Dieselboy]

you can't stretch L2 over GRE. You CAN run MPLS over GRE over IPSEC if you're game ROFL. L2TPv3 also works, but I'm not sure over IPSEC.

I researched this about 12 months ago and found an ancient Cisco config guide on it from like 2001 or something. Can't find that right now but given some links below as reference. BTW Openstack implementation uses GRE to link layer 2 domains (default is vxlan though, but I have configured GRE as my network doesnt support vxlan and would be broadcasted). For example, you have VM1 running on compute hardware 1 and VM2 in the same L2 domain running on a different piece of hardware. This hardware might be in another location but the L2 domains need to be reachable as if they were directly connected using a switch. This is where the GRE comes in. In practice, it's working great but the Hosts are in the same chassis and has 40gbps bandwidth between them. Not sure about setting up hardware in a remote location at the moment but that may be on the cards in the future.

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/cether/configuration/xe-16/ce-xe-16-book/ce-L2-EoGRE.pdf
https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/interface/configuration/xe-3s/ir-xe-3s-book/ir-eogre.html
https://mpreath.wordpress.com/2014/10/10/layer-2-over-gre/
https://docs.openstack.org/mitaka/networking-guide/intro-overlay-protocols.html
https://assafmuller.com/2013/10/14/gre-tunnels-in-openstack-neutron/

[deanwebb]

Stretching VLANs in general... this is a database developer request, right?