Cisco AnyConnect high system load, many Toast alerts "reconnecting"

[Dieselboy]

I have an issue where users are getting spammed with anyconnect messages which are difficult and disruptive to clear.

Environment

I am using the following systems
-   HA Cisco FTDv running purely as a AnyConnect SSL/TLS VPN server
-   Windows 10 clients
-   Cisco AnyConnect 4.10 VPN client software

Diagram
[ FTDv VPN server (IPv4 ONLY) ] -------> (( internet )) <----- [ Anyconnect client on Windows 10 at users Home (Dual Stacked IPv4/IPv6) ]

In terms of connectivity, this is important to note:

IPv4 connectivity is the sole IP used everywhere at the VPN server side, between FTDv and the public internet cloud

IPv6/IPv4 dual-stack client Windows 10 devices due to working from home and their chosen ISP (including mine) provides dual-stacked internet.


Problem description
A few people have reported this mass spam of "reconnecting" being alerted by cisco anyconnect client. I noticed this myself when waking my computer screen and logging in; but actually the laptop had gone to sleep and anyconnect reconnected due to the profile set. In my case I received 18 messages in the notification panel as well as pop ups on the screen, plus additional messages from anyconnect and other applications. The experience was that I was unable to use my computer for what felt like an eternity and it was impossible for me to close these notifications due to the high load.

Other people have noticed this when first initially connecting to the VPN.


Troubleshooting

-   DART logs taken via Cisco-provided powershell script
-   Cisco initially advised that the issue related to some other software on the computer due to multiple log messages: "Description: An established connection was aborted by the software in your host machine."

Log debug:
-   Logs show "Description: SOCKETTRANSPORT_ERROR_WRITE"
-   Even though we don't use IPv6 for the VPN connection or client addressing we also see "Private IPv6 Address: FE80:0000:0000:0000:4D5C:84AD:5344:6FFD/126"
-   Additionally:
o   Description: ROUTETABLE_ERROR_GETBESTROUTE_FAILED
o   Failed to add route to DNS server FE80:0:0:0:D2DB:B7FF:FEA9:1852 via interface FE80:0:0:0:AD7C:8530:699E:A3D3 (index 11)
-   And:
o   Description: ROUTETABLE_ERROR_GETBESTROUTE_FAILED
o   Unable to find matching route to FE80:0:0:0:D2DB:B7FF:FEA9:1852 (preferred interface index: 11)

Note: fe80 is the ipv6 equivalent of 169.254.x.x and is a link-local IP address

So what seems to be happening is this:
-   Windows 10 computer joins a dual stack network and receives ipv4 and ipv6 addresses on it's wifi interface
-   Windows OS additionally self-assigns link-local addressing on all it's enabled interfaces
-   Computer joins anyconnect VPN and this enables the previously disabled/unused anyconnect virtual network adapter
-   Windows OS assigns link-local addresses (multiple!) to the anyconnect adapter
-   Anyconnect needs to identify which network adapter on the system is associated with the VPN connection and chooses to identify this adapter by one of the IPv6 addresses
-   At some point the IPv6 address used as the identifier gets cleared out and removed, replaced with a different FE80:: address
-   Anyconnect breaks as it's unable to locate the adapter because the identifier is no longer present on the interface


Note:
-   When running "netsh interface ipv6 show addresses" at the windows command prompt I often see 2x FE80:: addresses assigned to the anyconnect virtual adapter
-   The lifetime of FE80 addresses are "infinite" which means they are permanent and unchanging. The RFC [1] states that addresses will be assigned when the adapter gets enabled
-   In my case, my computer went to sleep which I can understand but worth to note "assume" will disable the network adapter. Returning from sleep, then, will enable the adapter and consequently receive new addresses – however, attempting to reproduce this specific issue scenario yesterday (to first confirm the issue and then to confirm my proposed resolution resolves the issue) I did not get any FE80 change at all on both of the assigned FE80 addresses.

Solution

I'm still working on it as I have been unable to test an effective solution, but suggestions so far are:

-   Config the xml profile "ip protocol support" to only ipv4
o   I tested this yesterday but still the anyconnect virtual adapter still received FE80:: addresses
-   Cisco suggested running the client bypass config on the VPN server but the description of this command states that it allows the anyconnect client to transmit data in clear text to the VPN server so without more context I am avoiding this at the moment. The default is not enabled for this command and therefore as well as the documentation specifically stating this, that the anyconnect client will drop the traffic instead. I cant see how this would resolve the issue for me either.
-   This is in no way an option for me due to the large org. and many users but I did confirm and obviously, turning off IPv6 within the anyconnect virtual adapter (simply uncheck the box for ipv6) results in the adapter no longer receiving FE80:: addresses and so should essentially mimic the equivalent experience as a ipv4 only client network where this issue is not present. I could perhaps implement this via GPO but I feel it's rather messy


Did anyone of you see this yet? We're a bit behind over here so, ipv6 is perhaps being rolled out to users homes first before being rolled out to offices and businesses.



[1] RFC 4862: IPv6 Stateless Address Autoconfiguration (rfc-editor.org)

[Dieselboy]

Thinking more about this and again checking logs, there are specific log messages stating:

- unable to add fe80: route to DNS server (my home uses the isp router as a dns server)
- route table issues: "ROUTETABLE_ERROR_UNACCOUNTED_ROUTE_CHANGE_ENTRY" and:

Invoked Function: CRouteHandlerCommon::FindMatchingRoute
Return Code: -22806495 (0xFEA40021)
Description: ROUTETABLE_ERROR_GETBESTROUTE_FAILED
Unable to find matching route to FE80:0:0:0:D2DB:B7FF:FEA9:1852 (preferred interface index: 11)

Invoked Function: CRouteHandlerCommon::addHostRoute
Return Code: -22806495 (0xFEA40021)
Description: ROUTETABLE_ERROR_GETBESTROUTE_FAILED
Failed to add route to DNS server FE80:0:0:0:D2DB:B7FF:FEA9:1852 via interface FE80:0:0:0:AD7C:8530:699E:A3D3 (index 11)

I was wondering:
- why anyconnect wants to update the ipv6 route table
- how the next hop is determined when every interface is in the same ipv6 subnet due to every interface having FE80 addresses.

So because of the multiple L3 interfaces in same subnets issue, when targeting an FE80 address, supposed to also use a zone identifier to reflect which outbound interface to send the packet. Anyconnect does have this logged correctly: "(preferred interface index: 11)" the "A3D3" relates to the wifi interface mac address which is how the system was accessing the internet, so it's the correct interface IP address for IPv6. Being in the same subnet though, there is no need to add a route - however I tested this out on my home PC and adding a static route in command prompt for the fe80 our via the network interface was successful.


I am uncertain of the exact issue. I've asked cisco for more detail and we're waiting for the backend team at the moment.

[deanwebb]

This is why I like cloud reverse proxy solutions better. I'm using Cyolo for access to my work lab and it does a great job. Caveat is that some applications have to have code re-done so that they're compliant with a reverse proxy, but I can get around that by doing a reverse proxy to an RDP host and then using the RDP host to access the application.