In this post we will look at CIMC upgrade process of WLC HA pair. If you have not deployed WLC in HA -SSO (High Availability with Stateful Switch Over) it is high time to think about it. If you have standalone WLC deployment , then you require to get longer outage window (1 hr+ ) to perform this sort of maintenance work. Refer my previous post if you are upgrading CIMC of a standalone WLC.

I have upgraded pair of 8540 & 5520 recently, here is some of my learnings worth to note. At the time of this writing, HUU v4.0(2h) is the latest firmware version. It is important you to  read UCS-C series release notes  before doing this work.

Good news is that you can do CIMC upgrade of WLC-SSO pair without having an outage.

I had my 8540 CIMC versions 2.0(6d) & 2.0(8d). Sometime you will see CPU missing alerts and overall status “faulty” with old versions of CIMC. I had this behavior with my 8540s. When I logged a case, first response from TAC was to reseat 2nd CPU 🙂 , I had to check with few of my friends how many CPU they can see in their 8540s.

As I did not hear any of them got 2 CPUs in their 8540, I kept asking question from TAC. Then they gave me a bugID CSCux20012 (note There are 477 support cases 🙂 ) as explanation & suggesting later version of CIMC would fix it.

First of all, make sure you got WLC’s CIMC port connected to the network & you can access that IP address via https. If you haven’t configured it at all default admin/password combination should work in general. If that does not work for you & you haven’t set it up earlier try “cisco1234“. You know why when you read below field notice.

FN64093 – UCS-C series default password incorrect for units shipped 17 Nov 2015 – 6 Jan 2016

It is always recommended to configure CIMC IP address, that can easily remember which WLC you connect to.

(WLC1) >show interface summary

Interface Name Port Vlan Id IP Address Type Ap Mgr Guest
-------------------------------- ---- -------- --------------- ------- ------ ----- 
management LAG 1000 x.x.x.200 Static Yes No 
redundancy-management LAG 1000 x.x.x.201 Static No No

(WLC1) >imm summary

User ID.......................................... admin
DHCP............................................. Disabled
IP Address....................................... y.y.y.201
Subnet Mask...................................... 255.255.254.0
Gateway.......................................... y.y.y.250

I configured my CIMC IP address y.y.y.201 in WLC1 which is one unit in HA-pair. In that way I know .201 is that single WLC (irrespective of you connect via “redundancy-management” interface or “CIMC interface”. Other Unit got .202 in respective subnets for redundancy management & CIMC.

Once you got your CIMC accessible, one of the most useful features to enable is “SOL- Serial Over LAN” . In this way, you can get your WLC’s console access remotely. Unless you like to be your DC & physically connect to console port (2nd port of 4), this is the best way to do it. Once you enable this feature, your physical serial port will be disabled (as COM0 is mapped to physical port). I tried to used COM1 for SOL, that did not work for me.

In older version of CIMC, you can go to “Server -> Remote Presence -> Serial over LAN -> enable“. Remember to use COM0 & 9600bps, if you get it working with COM1, let me know. Note that by default they use port number 2400 and if you SSH to your CIMC IP address using port 2400, you will get WLC console access.

In later versions of CIMC, you can go to “Compute -> Remote Management -> Serial over LAN“. I modified SSH port number to 8540.

You can refer “Console access to 8540/5520 via CIMC” cisco document for more details.

Prior to the upgrade, you can start a ping to 5 different IP addresses (WLC management, redundancy management of WLC1 & WLC2, CIMC address of WLC1 & WLC2). Once you boot standby WLC with HUU (pls refer previous post of detail instruction), you will see that WLC redundant management IP address is unreachable on your ping monitors. Once you boot from kVM mapped DVD, it will take around 15 min to copy firmware files/tools and discover components of your C series server.

Once you click “Update All” and confirm you would like to proceed, it will start upgrading CIMC,BIOS,LOM & RAID firmware. I would say it will take around 30 min for component upgrades to complete.

Once upgrade finishes, you can click “Exit” & confirm. Then your server will reboot a couple of times (if you monitor your ping to CIMC IP address, you will notice it will go down a couple of times.

Roughly around 20-25 min later, you will see your WLC redundancy management start reachable. If you have your SOL configured. you will see activity on your WLC console while you waiting patiently.

Once, you check redundancy status using “show redundancy summary” & ensure everything is normal, you can failover traffic to this Unit (redundancy force-switchover CLI command on the active unit)  & follow the same procedure on next Unit. So in this way, you can perform this CIMC upgrade on a WLC – HA pair without having much trouble.

(WLC1) >show redundancy summary 
Redundancy Mode = SSO ENABLED 
Local State = ACTIVE 
Peer State = STANDBY HOT 
Unit = Secondary (Inherited AP License Count = 3000)
Unit ID = 70:E4:22:x:x:x
Redundancy State = SSO
Mobility MAC = 5C:83:8F:x:x:x
Redundancy Port = UP
BulkSync Status = Complete
Average Redundancy Peer Reachability Latency = 117 Micro Seconds
Average Management Gateway Reachability Latency = 2153 Micro Seconds

I hope this post is useful if you are performing this task on your WLC HA pair.

Update – 25 Apr 2020
There is a high severity vulnerability published that could impact these servers. So it is recommended to upgrade the CIMC version to 4.1(1f) to address this vulnerability.

Field Notice: FN – 70545 – SSD Will Fail at 40,000 Power-On Hours – BIOS/Firmware Upgrade Recommended
CSCvt55829-SSDs will experience data loss at 40k power on hours

You can check your SSD power-on hours using CIMC GUI (Storage -> Physical Driver Info) as shown below. Here is one of 8540 WLC info, You can see it got 38888 power-on hours and product ID listed as Samsung.

By information published (see below post responses), it looks like SanDisk SSD having this issue not other vendor SSD.

https://community.cisco.com/t5/other-wireless-mobility-subjects/wlc-affected-of-ssd-bug/m-p/4073041

So these WLCs may not have that issue. Anyway for other vulnerabilities fixed in 4.1.1f (refer this release note for more detail)  still may worth upgrading your WLC CIMC.

 

References
1. WLC – High Availability (SSO) Deployment Guide

RELATED POSTS

1. CIMC Upgrade – Standalone 8540/5520