Central WLC - remote FlexConnect APs with flexconnect group for software broken

[Dieselboy]

I have a central WLC in Australia running code 8.1.131
I have 2 1602 AP's in flexconnect mode in Sri Lanka.
I have configured the 2 remote AP's in a flexconnect group so that software upgrades are faster, as there will be only one download from the WLC and not 2.

One of the APs here has just started doing this: when it boots up from cold (POE unplug, plug back in) the AP does not come up. The light flashes orange, and looking at the WLC I can see that it has contacted the controller with a very old software image that was in use over a year ago (7.x). Because of this, it is downloading code on each boot. Additionally, it doesn't appear to be getting the code from the master AP in the flexconnect group.

Any ideas what could cause this? Or is this TAC time again?

[SimonV]

Once it completes, is it functional?

Have you read this field notice? https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63916.html

[Dieselboy]

Thanks for the link, I'm running 8.1.130 for around 2 or 3 months or so now. Just had this issue last week and thought it was a one-off as I was re-locating the APs onto the ceiling.

Yes once it completes it is functional. Until the AP loses power again. At this point when the AP receives power it boots up and contacts the WLC for 8.1.130 download again as well as the backup download. Within the WLC it shows the AP using 7.5.x image to contact the controller and download 8.1.130.

No idea why it loads 7.5.x image. I guess the flash might be broken and there might be some kind of backup boot / recovery partition which is able to load and get to a controller that way.

[SimonV]

I would try to connect a console cable to it and see what's going on. You could try a "debug capwap enable" on the WLC, sometimes you will find some event that leads you to the problem. Local console is best though, if it isn't 6 meters above the ground

[Dieselboy]

I didn't do that because the tac engineer said he would come back to me and I have a billion things to do
I have a bluetooth console cable so console is fine too.. but just need to get the thing set up again and haven't had a chance yet.

I just SSH'd to it and done this:

Code Select Expand


show boot
BOOT path-list:      flash:/ap1g2-rcvk9w8-mx/ap1g2-rcvk9w8-mx
Config file:         flash:/config.txt

#dir flash://ap1g2-rcvk9w8-mx/ap1g2-rcvk9w8-mx
Directory of flash:/ap1g2-rcvk9w8-mx/ap1g2-rcvk9w8-mx

   17  -rwx     6724019   Mar 1 1993 00:03:23 +00:00  ap1g2-rcvk9w8-mx

more flash:/ap1g2-rcvk9w8-mx/info
version_suffix: rcvk9w8-
version_directory: ap1g2-rcvk9w8-mx
image_name: ap1g2-rcvk9w8-mx
ios_image_file_size: 6728192
total_image_file_size: 6728192
image_feature: WIRELESS LAN|LWAPP|RECOVERY
image_family: AP1G2
system_type: 0x9d
ws_management_version: 7.5.1.73
info_end:


So looks like the primary version isn't being installed properly. Wondering if I can erase flash and do the download again or something.

I had a thought - I think the flexconnect group is a new feature, hence why it's not working when the AP boots up 7.5.x above. If I do the erase flash, since it's now running the right version, it should get the image from the master AP.

TAC said it's a warranty replacement initially.

[icecream-guy]

memory issue? on why the AP won't load the updated  code.

[Dieselboy]

Mem looks ok while it's working (see below). But here is what happens:

AP running ok
Ap loses power and then powers on
AP loads 7.5 code and contacts WLC running 8.1.130
AP semi-joins WLC and downloads 8.1.130
AP boots up using 8.1.130 and then downloads 8.0 as a backup
AP now running again

While the AP is running, I cannot find code 8.1.130 in the flash. I can only find 7.5. Hence when the AP boots up again it loads 7.5.
I don't know how the AP is able to download 8.1.130 and boot from it, without getting rid of 7.5 or even not having 8.1.130 in the flash. I just got my bluetooth console cable working again with windows 10 (grrr) so.. > actually they just finished the meeting so I can work on this now. Going to erase the flash and see if I can sort it out.

Code Select Expand


1602AP-1#sh mem
                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor    32E20A4   139573084    82331796    57241288    17658184    17526732
      I/O    E000000    33554432    26780704     6773728     6723212     6720008
      I/O      14000      966656      155456      811200      808312      808264


[Dieselboy]

So this was fixed.. consoled in to it and run debug capwap console cli  - the command was hidden.

on the flash was 7.5 and 8.0 but no primary 8.1 version. So I really don't know how it was booting up 8.1 without it being on the flash.
Erased 7.5 and done an archive download-sw of the 8.1.130 version we're using. done reboot tests and power off tests afterwards and it's fixed.

Not really sure what had happened since I didn't want to do a full recreate (because recovering took 1.5 hours)  and I could see the flash had 7.5 image when it shouldn't of had. No RMA needed.