Tags

, , , , ,

If you are familiar with Cisco UCS servers, you may be already familiar with CIMC upgrade procedure. If not this post may help you to upgrade CIMC of your Cisco 8540/5520  WLCs.

What is CIMC ?
The Cisco Integrated Management Controller (CIMC) is the management service for the C-Series servers. CIMC runs within the serve.

CIMC is a separate management module that is built into the motherboard. CIMC has its own ARM-based processor which runs the CIMC software. It is shipped with a running version of the firmware. Users can update CIMC firmware through the Firmware Update Management page. You need not worry about installing the initial CIMC firmware.

Why do you require a CIMC upgrade ?
If you read these security advisories on CIMC, you understand why it is require to upgrade CIMC of those C series servers.

Specific to WLC, there are few critical bugs fixed in later version of CIMC. So it is a good idea to keep your WLC’s CIMC upgraded

CSCvo33873
Symptom:
After a wireless LAN controller reloads, no access points are able to join.
SSH and HTTPS connections to the controller fail.
If you access the WLC via the console, and issue the command "show certificate all" - no certificates are seen.

Conditions:
5520 or 8540 WLC that has just reloaded.
The WLC was manufactured after 9-Sep-2015 (when a manufacturing change that was supposed to have fixed this problem was implemented.)

The WLC has 8.2MR2, 8.3 or above installed (which contains the CSCuy67885 fix, that was supposed to have fixed the problem.)

The WLC does NOT however have CIMC HUU 3.0.4d (or above) installed.

Workaround:
Connect CIMC. Enable CIMC connectivity on the controller, using "imm" commands. Browse to the CIMC interface. Find the FlexFlash, and manually enable HyperVisor in the "Enable/Disable Virtual Disk(s)" action. See:
http://www.cisco.com/c/en/us/td/docs/wireless/controller/technotes/troubleshooting/trb-guide-wlc-5520-8540.html#pgfId-1309804

Then perform a full HUU install of CIMC 3.0.4d (or whatever the latest supported CIMC is, for the 5520/8540.)

Cisco 8540 and 5520 series controllers are based on Cisco  UCS C series servers. Specifically Cisco 8540 is C240M4 series  and Cisco 5520 controller is C220M4 hardware. It is recommended to use UCS Host Upgrade Utility (HUU) for this work. You can download that firmware from Cisco download page. Since this post based on standalone 8540 (not SSO) controller, I downloaded ucs-c240m4-huu-4.0.2h.iso  which is the latest. (for 5520, you require ucs-c220m4-huu-4.0.2h.iso image)

Screen Shot 2019-09-06 at 1.44.43 PM

You should have your WLC CIMC port connected to network with IP address configured on that interface. CIMC is the first port (see below, note it is 5520 WLC in this photo) of 4 ports available.

CIMC.jpg

You can use “imm summary” CLI command to verify CIMC interface IP address configuration.

(8540-WLC) >imm summary

This will take some time... 
Please be patient!
User ID.......................................... admin
DHCP............................................. Enabled
IP Address....................................... x.x.32.100
Subnet Mask...................................... 255.255.254.0
Gateway.......................................... x.x.33.250

If it is not configured you can configure it statically or let it get IP via DHCP

(8540-WLC) >imm ?
address IMM Static IP configuration
dhcp Enable | Disable | Fallback DHCP.
restart Saves settings and Restarts IMM Module.
summary Displays IMM Parameters.
username Configures Login Username for IMM.

(8540-WLC) >imm address x.x.32.100 255.255.254.0 x.x.33.250
(8540-WLC) >imm username admin password <cimc_password>

You can access CIMC interface IP using https:

Screen Shot 2019-09-06 at 12.23.33 PM

You can “Launch KVM Console” to open it. Make sure you security setting allow it if you using Mac OS

Screen Shot 2019-09-06 at 12.23.59 PM

You have to simply follow instruction and keep continue until you get KVM console page.

Screen Shot 2019-09-06 at 12.25.26 PMScreen Shot 2019-09-06 at 12.25.51 PM

You can go to “Virtual Media -> Activate Virtual Devices” as shown below

Screen Shot 2019-09-06 at 12.28.07 PM

Once you activate Virtual Devices, you can map ISO image to CD/DVD

Screen Shot 2019-09-06 at 12.28.42 PM

Note: Image below shows c220M4 (when I capture screenshot incorrectly select that image 🙂 )  , you should select c240m4 iso image for 8540

Screen Shot 2019-09-06 at 12.29.56 PM

Then you can go  to “Power” options and click “reset system” or “Power Cycle System”

Screen Shot 2019-09-06 at 12.30.14 PM

Once server is booting up, You should hit F6 to change boot options. We need server to boot from KVM mapped DVD.

Screen Shot 2019-09-06 at 12.31.33 PM

This will allow you to select boot from KVM map DVD image.

Screen Shot 2019-09-06 at 12.32.25 PM

You will see it is boot with HUU image and prompt you to agree (be patience, it will take time)

Screen Shot 2019-09-06 at 2.11.49 PM

It will take 15-20 min to copy all required files and prompt you the update options. You should go with update All unless you specifically want one component upgrade.

Screen Shot 2019-09-06 at 2.25.36 PM

Each component will be upgraded and it will take 30-45 min to finish these component upgrade process.

Screen Shot 2019-09-06 at 2.52.10 PM

Once upgrade finished, you can exist from that window and power cycle

Screen Shot 2019-09-06 at 2.52.27 PM

Screen Shot 2019-09-06 at 2.53.23 PM

It will take 15-20min to server to properly boot after CIMC upgrade. So overall you will have little more than 1hr outage while this work completed. Make sure you take necessary outage window arranged, if you do this upgrade to production 8540/5520 WLCs

You can also use “show imm chassis <>” command to verify  those upgraded BIOS information

(8540-WLC) >show imm chassis ?

bios Fetch Chassis BIOS information
current Fetch Chassis Current information
fan Fetch Chassis FAN information
fan-profile Fetch Chassis FAN power profile
mac Fetch Chassis MAC information
memory Fetch Chassis Memory information
power-supply Fetch Chassis Power Supply information
sol-info Fetch Serial Over Lan information
temperature Fetch Chassis Temperature information

(8540-WLC) >show imm chassis bios 
BIOS Information
Vendor: Cisco Systems, Inc.
Version: C240M4.4.0.2d.0.0627191030
Release Date: 06/27/2019

I have to do this upgrade on 8540-HA pair  5520-HA pair soon. I will post the process once I do that task.

Update – 25 Apr 2020
There is a high severity vulnerability published that could impact these servers. So it is recommended to upgrade the CIMC version to 4.1(1f) to address this vulnerability.

Field Notice: FN – 70545 – SSD Will Fail at 40,000 Power-On Hours – BIOS/Firmware Upgrade Recommended
CSCvt55829-SSDs will experience data loss at 40k power on hours

You can check your SSD power-on hours using CIMC GUI (Storage -> Physical Driver Info) as shown below. Here is one of 8540 WLC info, You can see it got 38888 power-on hours and product ID listed as Samsung.

8540-SSD

By information published (see below post responses), it looks like SanDisk SSD has this issue, not other vendor SSD.

https://community.cisco.com/t5/other-wireless-mobility-subjects/wlc-affected-of-ssd-bug/m-p/4073041

So these WLCs may not have that issue. Anyway, other vulnerabilities fixed in 4.1.1f (refer to this release note for more detail)  still may be worth upgrading your WLC CIMC.

Here is another CSC post if you are having trouble getting upgrade your CIMC
https://community.cisco.com/t5/wireless/5520-cimc-update-fails-no-flash-access-to-old-cimc-r2-0/td-p/4658768

References
1. 8540/5520 Console access via CIMC
2. 8540/5520 Troubleshooting Guide
3. Firmware Upgrade on UCS servers through Host Upgrade Utility (HUU)

RELATED POSTS

  1. CIMC Upgrade on WLC-HA