Apple Thunderbolt Displays causing Spanning-Tree Loop Issues

[mmcgurty]

So we have had two incidents now with Apple Thunderbolt displays causing an issue that appear to be a spanning-tree loop on our network.  We will get notices from our NAGIOS XI monitoring system that our two main 6509's at the site (distribution layer switch) has CPU at 99%.  You cannot SSH into them they are so burdened.  We will then start hitting each IDF switch in the building looking for the offending port.   We typically find it with the following command:  sh int | i is up|rate

This is what we will see:

SWITCH#sho int g4/39
GigabitEthernet4/39 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet Port, address is 0026.0b3f.1a56 (bia 0026.0b3f.1a56)
  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 172/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, link type is auto, media type is 10/100/1000-TX
  input flow-control is on, output flow-control is on
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:56, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1165
  Queueing strategy: Class-based queueing
  Output queue: 0/40 (size/max)
  5 minute input rate 67511000 bits/sec, 70912 packets/sec
  5 minute output rate 12000 bits/sec, 13 packets/sec
     419856990 packets input, 104992222534 bytes, 0 no buffer
     Received 115445450 broadcasts (115114558 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     484547557 packets output, 282248309092 bytes, 0 underruns

This is how our port on the switch is configured:

interface GigabitEthernet4/39
switchport access vlan 501
switchport mode access
switchport voice vlan 405
switchport port-security maximum 3
storm-control broadcast level 50.00
storm-control action shutdown
qos trust device cisco-phone
macro description AccessEdgeQoS
spanning-tree portfast
spanning-tree bpduguard enable
service-policy input CISCOPHONE-POLICY
service-policy output 1P7Q1T

From what we can tell, the user has an Apple Macbook Pro in all cases we have identified this issue.  They are plugging the Apple Macbook Pro into the Apple Thunderbolt which is acting like a dock of sorts with its own Ethernet port on it.  This Ethernet port is connected to the Cisco IP phone.  The Cisco IP phone connects to our wall jacks which goes to either a North or South IDF switch which are WS-C4506-E switches with SUP8-E supervisor cards.  We know from our MDM solution that both Apple Macbook Pro's are running OSX 10.10.5 and are from the same timeframe (Mid-2015).  We know we have BPDUGuard enabled but it looks like they are sending unicast traffic which would never be caught by this.  We are thinking we are going to have to start rolling out something like:  storm-control unicast level pps 65 (K, thousand)

Has anyone else experienced this issue?

[mmcgurty]

I just found this article:

https://discussions.apple.com/thread/6443650?start=0&tstart=0

This explains our issue exactly.  We have Apple Enterprise Support, so we are likely going to open a case with them and possibly Cisco TAC to see if there is any resolution on either side.

[deanwebb]

Simple solution: use NAC to ban all Apple stuff on the LAN.

[mmcgurty]

What is interesting is that the Apple Thunderbolt displays don't seem to use a standard MAC address from what we have seen, so my guess is that they wouldn't get blocked by NAC.  Plus, we are a very Apple heavy environment at the moment so I think some would take issue with that.

We could just deploy mGig AP's and linecards and newer Cisco 3800 series AP's and just make them run all wifi and this wouldn't be an issue.  But as you probably know, Apple doesn't wireless very well either.

[wintermute000]

Seen similar before with targus docking stations. Hairs breath away from the customer RMAing a pair of spanking new 6800s!!!

[Dieselboy]

But as you probably know, Apple doesn't wireless very well either.

I'm not aware about Apple wirelss - are you referring to bonjour problems?

This isn't good about the spanning tree issue. I've not noticed this in my env. but I only have 2 3k nexus switches and our access switches are vPC. Also, I'm steering away from Thunderbolt adapters for networking because they need the thunderbolt port for HDMI screens. They can use the usb port for networking and have a slower connection since this is fine. Does the USB network adapters exhibit the same issues?

Please can you keep us updated? We are also mostly Apple. If you need extra weight to get Cisco to do something about it I will reproduce the issue and raise a TAC here in Australia. I have raised many TAC cases before only to have Cisco tell me they wont do certain things because it's just me raising the case. Since I have a similar set up I am happy to raise a TAC if needed.

[SimonV]

Have you already run a wireshark capture on those access ports? Can you see how the packets look like? I have had a similar issue with Intel network cards, and googling the header details lead me to this article

A broadcast storm would seem odd from the sounds of it, as there is no physical loop. Unless the Macbooks are bridging WLAN and LAN interfaces? Also seen that happen 

edit: sorry, just read that article on apple forums 

[Nerm]

I haven't seen that specific Apple issue but I have dealt with my fair share of wireless network related issues with Apple devices.

Have you already run a wireshark capture on those access ports? Can you see how the packets look like? I have had a similar issue with Intel network cards, and googling the header details lead me to this article

A broadcast storm would seem odd from the sounds of it, as there is no physical loop. Unless the Macbooks are bridging WLAN and LAN interfaces? Also seen that happen 

edit: sorry, just read that article on apple forums 

We are currently dealing with that Intel NIC issue in our environment.

[Dieselboy]

Can someone pls share a link or something I can reference with regards to the wifi issues and Apple devices?

I've been having wifi issues with all users, and we mainly use Apple Mac laptops (Mac air).I wasn't aware of this specific issue related to Apples.

[Nerm]

Can someone pls share a link or something I can reference with regards to the wifi issues and Apple devices?

I've been having wifi issues with all users, and we mainly use Apple Mac laptops (Mac air).I wasn't aware of this specific issue related to Apples.

I don't have any specific tech documented links for you, but the problem I mostly experienced was with iOS 9 and "WiFi Assist". For some reason it has a tendency to prefer cellular connections over WiFi connections even if the WiFi connection is better (which in most cases it is going to be).

[NetworkGroover]

Has anyone else experienced this issue?

This blows my mind.

[Dieselboy]

I don't have any specific tech documented links for you, but the problem I mostly experienced was with iOS 9 and "WiFi Assist". For some reason it has a tendency to prefer cellular connections over WiFi connections even if the WiFi connection is better (which in most cases it is going to be).

Oh.. My Samsung has something similar. If my Samsung Galaxy S5 detects bad wifi (I might have moved outside or something where wifi is going to be bad / expected to be bad) then it will start using the 4G connection. But if I then walk back inside it can refuse to use wifi for an annoying amount of time.

My Apple issues are related to delays with the wifi adapter sending traffic to the AP. The wifi adapter does not send traffic in a timely manner.