Evolution in Attacks Against Cisco IOS Software Platforms - AKA owned 4ever

[AnthonyC]

Love it. /s

http://tools.cisco.com/security/center/viewAlert.x?alertId=40411

https://www.schneier.com/blog/archives/2015/08/nasty_cisco_att.html

[deanwebb]

 

But, yeah, get your TACACS+ set up for router/switch access and keep the comm room doors securely locked. Never let anyone into the comm room without an escort and, most importantly, don't hire anyone posing as a network engineer for purposes of engaging in acts of industrial espionage.

[Otanx]

I keep getting questions on this from customers, and I am still surprised that this is such a surprise to people. It has been a theoretical threat for years. The only difference is they actually are seeing it in the wild now. The mitigation/best practices document had some interesting information. Mainly how to md5 the running image. Of course if the box is already running compromised code it could return any value it wants for the hash. The problem is that there is no way to validate the code running without ROMMON running on the box. Cisco probably can using some debugging interfaces to get at the storage directly, but for an end user the box can not be validated as clean because you can not trust the output. Some boxes that store everything on removable flash may be able to be verified on a separate box, but most switches and routers would not. The only option you have if you suspect the box is to replace it.

-Otanx

[deanwebb]

And this also means putting those IOS devices under scrutiny, to see if they're sending/receiving traffic that shouldn't be sent/received.

[Reggle]

Good point with the md5 hash being compromised as well on the box, Otanx.
I have a central Linux SCP server here where I upload the images and do a hash verification, that solves it.

[Otanx]

That solves you installing an invalid image. The issue isn't admins doing this. The issue is attackers are getting the admins login info, and loading their own firmware on the device from their own source. Basically a rootkit for your router. Then if you change the admin password they can still get back in.

So how do you know if your device is owned? Check the hash, maybe the attackers didn't think about faking the /md5. I would consider that a more advanced detection evasion technique. Of course writing your own ROMMON is already pretty advanced. If it comes up with a bad md5 you know something is wrong, but if the md5 is good it does not mean you are OK.

Monitor and log everything. Did Joe login to the router when he was on vacation? Might want to suspect the box. Especially if the first thing done is disable logging on the box (you are logging commands right?) Of course now you don't know what was done because logging is turned off. Another thing to look for are unexpected reboots. Remember that to load the new compromised ROMMON the box needs a reboot. However, you can't replace every switch/router that has rebooted. Root cause analysis for reboots will help. Also as deanwebb said look at traffic flows. Was there a file copy to the box from a random IP? Having netflow, or something similar setup is the key here. Don't forget that a SCP file copy may show up as a SSH connection. However interactive SSH connections don't transfer multi-megabytes in minutes. If you have all three of these (unexplained admin login, file copy, reboot) you are probably owned.

So now what do you do? You can call Cisco, and see if they are interested. At the very least you could probably get a SmartNet replacement for the box. You may never know if you were really owned, or just paranoid, but you have a new box. You can call one of the cyber forensics companies/teams, someone like Tripwire, Mandant, Verizon, etc. These guys don't come cheap, but right now you might be able to work a deal to get a discount as they would be interested in getting their hands on the ROMMON to reverse. Finally you can contact law enforcement. They are probably going to want more proof than "My router rebooted" to start investigating, but they have some serious tools and skill sets to research the attack.

To avoid all that mitigate the risk. Cisco did not say how the attackers got the username/password that was used. I would guess some stupid login account existed like cisco/cisco, spear phishing, or password reuse from a compromised site (If you used the same password on your router that you used for your ashley-madison account you may want to change it). So don't open unknown attachments, don't reuse passwords. Then lock down access to the devices with ACLs, etc. Use two factor. Really just the basics that everyone should be doing already.

-Otanx

[AnthonyC]

Good point with the md5 hash being compromised as well on the box, Otanx.
I have a central Linux SCP server here where I upload the images and do a hash verification, that solves it.

Although that's a big improvement it doesn't quite solve it unfortunately.  Working hash collision attack has been demonstrated for almost a decade.  Granted the ability to provide an identical hash on a corrupted image is not trivial but neither is the ability to write ROMMON.

http://www.technologyreview.com/view/411199/the-nostradamus-attack/

You can even write your own HelloWorld/GoodbyeWorld and compile it to see it in action.
https://www.win.tue.nl/hashclash/SoftIntCodeSign/
http://www.mathstat.dal.ca/~selinger/md5collision/

And yes news of hacking ROMMON has been out for years (even featured in Blackhat 09); the question is how often are we seeing this in the wild?  Cisco should put some effort in their advisory on how to detect it, even if these are logging/monitoring best practice it'd help the community as a whole.