In this post we will look at CIMC upgrade process of WLC HA pair. If you have not deployed WLC in HA -SSO (High Availability with Stateful Switch Over) it is high time to think about it. If you have standalone WLC deployment , then you require to get longer outage window (1 hr+ ) to perform this sort of maintenance work. Refer my previous post if you are upgrading CIMC of a standalone WLC.
I have upgraded pair of 8540 & 5520 recently, here is some of my learnings worth to note. At the time of this writing, HUU v4.0(2h) is the latest firmware version. It is important you to read UCS-C series release notes before doing this work.
Good news is that you can do CIMC upgrade of WLC-SSO pair without having an outage.
I had my 8540 CIMC versions 2.0(6d) & 2.0(8d). Sometime you will see CPU missing alerts and overall status “faulty” with old versions of CIMC. I had this behavior with my 8540s. When I logged a case, first response from TAC was to reseat 2nd CPU 🙂 , I had to check with few of my friends how many CPU they can see in their 8540s.
As I did not hear any of them got 2 CPUs in their 8540, I kept asking question from TAC. Then they gave me a bugID CSCux20012 (note There are 477 support cases 🙂 ) as explanation & suggesting later version of CIMC would fix it.
First of all, make sure you got WLC’s CIMC port connected to the network & you can access that IP address via https. If you haven’t configured it at all default admin/password combination should work in general. If that does not work for you & you haven’t set it up earlier try “cisco1234“. You know why when you read below field notice.
FN64093 – UCS-C series default password incorrect for units shipped 17 Nov 2015 – 6 Jan 2016
It is always recommended to configure CIMC IP address, that can easily remember which WLC you connect to.
(WLC1) >show interface summary
Interface Name Port Vlan Id IP Address Type Ap Mgr Guest
-------------------------------- ---- -------- --------------- ------- ------ -----
management LAG 1000 x.x.x.200 Static Yes No
redundancy-management LAG 1000 x.x.x.201 Static No No
(WLC1) >imm summary User ID.......................................... admin DHCP............................................. Disabled IP Address....................................... y.y.y.201 Subnet Mask...................................... 255.255.254.0 Gateway.......................................... y.y.y.250
I configured my CIMC IP address y.y.y.201 in WLC1 which is one unit in HA-pair. In that way I know .201 is that single WLC (irrespective of you connect via “redundancy-management” interface or “CIMC interface”. Other Unit got .202 in respective subnets for redundancy management & CIMC.
Once you got your CIMC accessible, one of the most useful features to enable is “SOL- Serial Over LAN” . In this way, you can get your WLC’s console access remotely. Unless you like to be your DC & physically connect to console port (2nd port of 4), this is the best way to do it. Once you enable this feature, your physical serial port will be disabled (as COM0 is mapped to physical port). I tried to used COM1 for SOL, that did not work for me.
In older version of CIMC, you can go to “Server -> Remote Presence -> Serial over LAN -> enable“. Remember to use COM0 & 9600bps, if you get it working with COM1, let me know. Note that by default they use port number 2400 and if you SSH to your CIMC IP address using port 2400, you will get WLC console access.
In later versions of CIMC, you can go to “Compute -> Remote Management -> Serial over LAN“. I modified SSH port number to 8540.
You can refer “Console access to 8540/5520 via CIMC” cisco document for more details.
Prior to the upgrade, you can start a ping to 5 different IP addresses (WLC management, redundancy management of WLC1 & WLC2, CIMC address of WLC1 & WLC2). Once you boot standby WLC with HUU (pls refer previous post of detail instruction), you will see that WLC redundant management IP address is unreachable on your ping monitors. Once you boot from kVM mapped DVD, it will take around 15 min to copy firmware files/tools and discover components of your C series server.
Once you click “Update All” and confirm you would like to proceed, it will start upgrading CIMC,BIOS,LOM & RAID firmware. I would say it will take around 30 min for component upgrades to complete.
Once upgrade finishes, you can click “Exit” & confirm. Then your server will reboot a couple of times (if you monitor your ping to CIMC IP address, you will notice it will go down a couple of times.
Roughly around 20-25 min later, you will see your WLC redundancy management start reachable. If you have your SOL configured. you will see activity on your WLC console while you waiting patiently.
Once, you check redundancy status using “show redundancy summary” & ensure everything is normal, you can failover traffic to this Unit (redundancy force-switchover CLI command on the active unit) & follow the same procedure on next Unit. So in this way, you can perform this CIMC upgrade on a WLC – HA pair without having much trouble.
(WLC1) >show redundancy summary Redundancy Mode = SSO ENABLED Local State = ACTIVE Peer State = STANDBY HOT Unit = Secondary (Inherited AP License Count = 3000) Unit ID = 70:E4:22:x:x:x Redundancy State = SSO Mobility MAC = 5C:83:8F:x:x:x Redundancy Port = UP BulkSync Status = Complete Average Redundancy Peer Reachability Latency = 117 Micro Seconds Average Management Gateway Reachability Latency = 2153 Micro Seconds
I hope this post is useful if you are performing this task on your WLC HA pair.
Update – 25 Apr 2020
There is a high severity vulnerability published that could impact these servers. So it is recommended to upgrade the CIMC version to 4.1(1f) to address this vulnerability.
Field Notice: FN – 70545 – SSD Will Fail at 40,000 Power-On Hours – BIOS/Firmware Upgrade Recommended
CSCvt55829-SSDs will experience data loss at 40k power on hours
You can check your SSD power-on hours using CIMC GUI (Storage -> Physical Driver Info) as shown below. Here is one of 8540 WLC info, You can see it got 38888 power-on hours and product ID listed as Samsung.
By information published (see below post responses), it looks like SanDisk SSD having this issue not other vendor SSD.
https://community.cisco.com/t5/other-wireless-mobility-subjects/wlc-affected-of-ssd-bug/m-p/4073041
So these WLCs may not have that issue. Anyway for other vulnerabilities fixed in 4.1.1f (refer this release note for more detail) still may worth upgrading your WLC CIMC.
References
1. WLC – High Availability (SSO) Deployment Guide
RELATED POSTS
mrn-cciew 
This is the quality of what Cisco documentation is meant to be (but is not the case). Sadly, it is left to people like you to knock something up so people will avoid/minimize making the mistake of reading Cisco’s poor state of documentation.
Good write-up, Rasika.
Hi Leo, Thank you for appreciation. Without people like you sharing info, I could not know some of this info. I am glad you are a close friend I can trust with info you providing. Based on your info I challenged TAC with their initial response.
Rasika
Hello ,
We are using AIR-CT8540-K9 in HA mode OS version 8.3.133.7. CIMC version is C240M4.2.0.8b.0.080620151546 Release Date: 08/06/2015,we are planning to upgrade WLC software to 8.5.151.0, is it mandatory to upgrade CIMC tool before WLC OS upgrade ? what is the use of CIMC tool ?
No, it is not mandatory to upgrade CIMC.
HTH
Rasika
i would highly recommand you to upgrade your CIMC.
in Version 2 you have Java and Flash.
In Version 4 you get HTML5 which is fully chrome compatible.
Thank you very much for the guide. i used it now on several clusters and all worked like a charm.
Thank you Janis, Glad to hear it helped you..
Rasika
Thank you for your excellent instructions.
Just a quick question regarding the the bug CSCvo33873, that you may or maynot know the answer to.
We have a pair of WLC 8540’s in an SSO deployment. In the event that this bug causes the primary unit to reboot “without being able to access FlexFlash, with the result that the manufacturing certificates are unavailable, and thus SSH and HTTPS connections will fail, and access points will be unable to join”, would the APs just join secondary unit? Would pulling the plug on the primary unit force the APs to join the secondary unit?
thanks
Ciaran
Hello, thank you for the step by step. I am working on doing this across all our controller and wasnt sure about how exactly does the secondary get upgraded in this. In our environment most of our appliances dont have cimc configured at all and the only way I can figure out to configure with minimal downtime is using switchover feature so that standby until becomes primary and i can access imm config. if there is a better way do share..I work for a hospital and anything to reduce downtime is greatly appreciated
Hi Munira, Thank you for the kind words. Always start with your secondary box in HA scenario. So no downtime while you do CIMC on secondary. Once upgraded, do a failover and then repeat it on new secondary box.
If CIMC not configured, do it first. Also 8540/5520 allow you extend serial over LAN, so you can extend console session over LAN, it is quite useful.
Click to access 212170-Console-access-to-5520-8540-WLC-via-CIMC.pdf
HTH
Rasika
Really apprecaite this post. This is very helpful as Cisco docs do not begin to explain any of this.
Do you have any advice in terms of rollback of this upgrade should there be any issues? Where I work, we have to present the risk with mitigating + backout actions.
Make sure you have console access to WLC when you do it. Physically on site or Serial over LAN connection.
In HA scenario, always do it on secondary, so it is no impact even something happen. You can roll back to as long as you have CIMC iso image of current version with you. (same process)
Stick to those steps and don’t try to go & modify any other settings (label,name,etc)
Let me know how it goes, you should be good
HTH
Rasika
Really helpful post !
Thank you !
Hi,
how did you assign IP address to CIMC interface of the secondary WLC?
thanks,
Hi,
This part was not clear to me as well!
@Rasika,
Can you please let us know how you manage to assign IP an address to the CIMC of secondary WLC? Incase we have the sso wlc is already inplace.
Thanks for the helpful post.
Regards,
Siron
Hi Siron, Easiest way is just to reserve DHCP address for the MAC address of those CIMC interfaces
Rasika
I did like Rasika explains, using a temp DHCP scope and helper, with only 2 IPs in range, then when the 2 CIMC interfaces got the IPs, I went to their https admin page and set the static IPs. Finally I removed the DHCP scope and helper. It worked like a charm.
Good to hear that
Thank you for the article, very helpful!
Can you tell me how I can find out the mac addresses of the CIMC interfaces?
“imm summary” doesn’t show anything useful here.
I would simply check the MAC address table of the switch that connects CIMC port
Rasika
Hello, thanks for this great how-to.
I’m planning to upgrade our outdated CIMC on 2 WLCs 5520 in HA SSO, it’s currently on version: 2.0(10b).
Can I work with standalone image of “UCS C220 M4 Rack Server Software” for Cisco UCS Host Upgrade Utility ? Or I need to work with an HA version ?
I know it’s a silly question, but when you have finished the upgrade on the primary WLC after doing redundancy “force-switchover” to standby switch, you also need to do again “force-switchover” to revert to primary ? Don’t you ?
Yes, there are only single image available for HUU. You use same for standalone WLC or HA pair.
Once you failover first time, then you have to upgrade new standby (previously active WLC) CIMC upgrade. If you want to 100% sure failover works, you can to failover 2nd time, so your original active WLC remain the active unit after that point
HTH
Rasika
Thanks Rasika.
I’ve read that when you do a switchover the previous active reboots, is that right ?
Yes, that is the normal behavior.
Thanks Rasika for your quick answer !
Great post Nayarasi, very helpful stuff.
Just confirming, you first upgraded standby unit (secondary) and waited for it to join HA. Did it join HA without any issue? like for HA aireos code needs to match and not cimc code, just confirming.
Then you failed over APs and users to standby and proceeded with upgrade on the primary wlc?
Thanks,
Hi Khawar, that is right. You do not want to match CIMC firmware to establish HA. Once you do it on secondary Unit, you failover to it and then work on previously active unit
Rasika
Thanks for the prompt reply.
Do you think it is a good idea to upgrade standby (secondary) and let it join HA, wait for a week or so, check how stable it is, and then upgrade active (primary)? We have good number of controllers so i am thinking to divide the work as well, this way i don’t have to upgrade all controllers on the same day.
No issue with that approach as well.
My preference is take one WLC pair at a time and do the upgrade of active/standby one at a time.
From client perspective, they will not see any outage, so it is less risk overall.
HTH
Rasika
Hi Rasika and thanks for this great documentation.
I will have direct console access to the WLC. Does that affect using KVM console? I will also be doing a direct connection from the CIMC port (configure and accessible now) to my laptop . My question is can I still use KVM console….even when I have direct console access because it seems that is how the ISO can be mounted and where to get the prompt to change boot option.
Hi Abi, As long as you have access to CIMC, it should work. I have not tried it myself when doing CIMC upgrades
Rasika
Thanks Rasika for the prompt response. Am wondering if anyone has been able to upgarde from 2.x to 4.x….Rasika’s example suggest so but Cisco docs stated that upgrade needs to be done from 2.x >>3.x and then 4.x….. Not sure or if documentation needs to be updated.
Just completed the Firmware upgrade. Thanks Rasika for the guidance. Moved from 2.x to 4.x. But Had to move from 2.x to 3.x and then 4.x according to Cisco. 2.x to 3.x was smooth. 3.x to 4.x was also ok . Just like Rasika mentioned…I would say a lot of patience is required. After mapping the image when moving from 3.x to 4.x…it took like extra 20min more to discover the component than it did in 2.x to 3.x. All in all it was all good.
Very good to hear all went well… Thank you for the update Abi..
Rasika
Many thanks Rasika, I updated my HA 8540s straight from 2.0.13i to 4.0.2r. It went very well thanks to your documentation. I used MaxthonPortable to get past the Adobe Flash issue with the earlier CIMC.
Hi Davide, Good to hear all went well in CIMC upgrade. Also thank you for providing workaround you used to get Adobe Flash to access 2.x CIMC GUI
Rasika
Hi Rasika, When the CIMC reboots does this affect wifi clients or not?
Ross
In a HA environment it should not have outage
HTH
Rasika
Just wanted to say thank you so much, Upgraded a 5520 HA Pair to avoid that nasty certificate bug. The serial over lan came in handy too. It took about 1.5 hours. I was on CMIC 2.0 had to use a portable Firefox with flash to get in, once I loaded the java console it was fine.
Hi John,
I’m glad to hear that you successfully upgraded the CIMC, and I’m pleased that my blog post was helpful. I’ve received a lot of positive feedback on this post.
Rasika