Times like these, I wish it *was* the firewall...

deanwebb · March 17, 2017, 06:11:39 PM

We have two guest wireless networks, legacy and the next-gen (NG) one.

Legacy is working with no issues. Fast registration, fast browsing. NG is having slow registration (10+ minutes for page to appear) and slow browsing.

Both have the same WLC handling their SSIDs, as well as the same IPS, Firewall, router. No load balancer in the path, all switchports handling the traffic are up/up, no duplex mismatches, no dropped packets or CRC errors, it all looks clean.

Because of the slowness on the NG guest wireless, I can see errors in the captures showing clients rejecting TLSv1.2 sessions because they can't validate the certificate presented by the web server handling registration or they can't reach the CA responsible for that cert. Once a client gets past the slowness and downloads the page, it can validate the cert and go merrily forward.

One wrinkle in this is that laptops can register quickly on the NG guest wireless, but then have slow browsing. Mobile devices get slow everything.

Total bandwidth on the NG wireless is low, around 8Mbps or so. It's not rolled out in general, this is just one building with ~400 people.

We do have the same solution in a test run in our Europe location, so I can check with them on Monday if they're also having issues.

If it was just slowness getting registered, I'd be all over that like a donkey on a waffle. But it's also slowness browsing in general. And mobile devices are the only ones having The Slow when they register.

Same devices having poor performance on NG guest work just fine on legacy guest and vice-versa.

Any ideas from you guys on other things to look for? We may have already checked, but we may not...

Otanx · March 17, 2017, 10:16:46 PM

DNS always check DNS. I don't know why that would be mobile devices and not laptops, but almost every time I have a slowness issue it is because some DNS lookup is not working.

-Otanx

SimonV · March 20, 2017, 03:35:50 AM

Also check if your NGFW is not blocking anything cert-related, like OCSP. We had one issue with that when we started our PAs, which impacted web performance.

wintermute000 · March 20, 2017, 06:36:08 AM

well you haven't mentioned anything re: the actual wireless setup other than its on the same WLCs? Could be misconfigured, rate-limited, etc

also, please do the needful and supply a packet capture

Even if its on the endpoint at least you can see if there are TCP funny buggers or it actually is getting super late responses etc.

deanwebb · March 20, 2017, 11:05:58 AM

Quote from: Otanx on March 17, 2017, 10:16:46 PM
DNS always check DNS. I don't know why that would be mobile devices and not laptops, but almost every time I have a slowness issue it is because some DNS lookup is not working.

-Otanx

Checking this thread after a troubleshooting call this AM, and I think this is the path we're on. Go to the IP address, page starts to load right away and slows down for elements that are called by FQDN. Go to the URL for the same IP address, everything takes forever to start.

@wintermute: Yep, we're going to get a capture of the guest wireless SSID traffic, especially since we're able to show that every other guest wireless environment in the world is running fine.

I'm guessing either the reverse DNS FQDN got changed or some device is sitting on the SSID blaring out stuff on TCP/UDP 53, or there's a device calling in a DNS reflection attack.

deanwebb · March 20, 2017, 11:17:23 AM

Follow up: Just before we start to get the capture, the issue clears up.

That makes me put more emphasis on a device connected to the SSID that was either malfunctioning or victim of malware. With it off the SSID, no problems.

icecream-guy · March 20, 2017, 12:21:36 PM

Quote from: deanwebb on March 20, 2017, 11:17:23 AM
Follow up: Just before we start to get the capture, the issue clears up.

That makes me put more emphasis on a device connected to the SSID that was either malfunctioning or victim of malware. With it off the SSID, no problems.

mmm, insider job, someone in your inner circle that knew you were going to do a packet capture and took something off the network so they wouldn't be found out. mmmm, look at everybody suspiciously.

deanwebb · March 20, 2017, 12:57:09 PM

I'm in security, so I already got that ground covered.

Otanx · March 20, 2017, 03:46:46 PM

I am going to guess that the allow-query line (or similar) did not list the new wifi subnet as being allowed to do look-ups. Probably only on one server too. Somebody fixed it, and didn't say anything. We saw this all the time at an old job. The DNS admin would only allow subnets that were in use instead of 10.0.0.0/8. So if a new vlan got stood up DNS would not work till someone remembered to go have the DNS guy add the new subnet.

-Otanx

wintermute000 · March 20, 2017, 03:48:24 PM

Surely that should flat out not work, not make it slow

Otanx · March 20, 2017, 05:43:27 PM

If it was just one server that was mis-configured then it would timeout, and try the second DNS server which has the correct config.

-Otanx

deanwebb · March 21, 2017, 12:24:55 PM

External DNS support is handled by marketing. I keep forgetting this, and then I email them, expecting a technical response. And now the guy that said the other guest wireless was fine is walking back on that and saying maybe it wasn't fine, how about we put that new IPS into layer 2 fallback mode and see what happens?

If he hadn't said legacy guest was fine, I wouldn't be going down this marketing rabbit hole. I expect end users to lie, but fellow engineers? That cuts deep, bro. That cuts deep.

It also leads to frustration on my part...

Otanx · March 21, 2017, 07:14:51 PM

Quote from: deanwebb on March 21, 2017, 12:24:55 PM
External DNS support is handled by marketing.

I am scratching my head over that. Not just because you said it, but this is the second time this month I have heard this. Is this common?

-Otanx

icecream-guy · March 22, 2017, 06:58:42 AM

Quote from: Otanx on March 21, 2017, 07:14:51 PM
Quote from: deanwebb on March 21, 2017, 12:24:55 PM
External DNS support is handled by marketing.

I am scratching my head over that. Not just because you said it, but this is the second time this month I have heard this. Is this common?

-Otanx

Marketing supporting DNS.

1. Marketing trying to understand DNSSEC
2. Marketing trying to implement DNSSEC
3. Marketing trying to manage DNSSEC
4. Marketing troubleshooting DNSSEC

deanwebb · March 22, 2017, 07:46:07 AM

Our internal DNS guys are focused on maintaining internal naming conventions, DHCP reservations, stuff like that. Proper tech dudes.

External DNS... that deals with our domain names and brand projection... better give that to marketing!

It's true. I try to reserve a host name in internal DNS and if it doesn't follow our naming convention, forget about it. External DNS? If the host name isn't something horrible like "Wekillpuppies" or "URMOM" or "upyours", they don't care what we name it because those other names reflect poorly upon the brand identity. That flexibility in naming convention is balanced out by a total and utter lack of troubleshooting.