Tags
If you are familiar with Cisco UCS servers, you may be already familiar with CIMC upgrade procedure. If not this post may help you to upgrade CIMC of your Cisco 8540/5520 WLCs.
What is CIMC ?
The Cisco Integrated Management Controller (CIMC) is the management service for the C-Series servers. CIMC runs within the serve.
CIMC is a separate management module that is built into the motherboard. CIMC has its own ARM-based processor which runs the CIMC software. It is shipped with a running version of the firmware. Users can update CIMC firmware through the Firmware Update Management page. You need not worry about installing the initial CIMC firmware.
Why do you require a CIMC upgrade ?
If you read these security advisories on CIMC, you understand why it is require to upgrade CIMC of those C series servers.
Specific to WLC, there are few critical bugs fixed in later version of CIMC. So it is a good idea to keep your WLC’s CIMC upgraded
CSCvo33873 Symptom: After a wireless LAN controller reloads, no access points are able to join. SSH and HTTPS connections to the controller fail. If you access the WLC via the console, and issue the command "show certificate all" - no certificates are seen. Conditions: 5520 or 8540 WLC that has just reloaded. The WLC was manufactured after 9-Sep-2015 (when a manufacturing change that was supposed to have fixed this problem was implemented.) The WLC has 8.2MR2, 8.3 or above installed (which contains the CSCuy67885 fix, that was supposed to have fixed the problem.) The WLC does NOT however have CIMC HUU 3.0.4d (or above) installed. Workaround: Connect CIMC. Enable CIMC connectivity on the controller, using "imm" commands. Browse to the CIMC interface. Find the FlexFlash, and manually enable HyperVisor in the "Enable/Disable Virtual Disk(s)" action. See: http://www.cisco.com/c/en/us/td/docs/wireless/controller/technotes/troubleshooting/trb-guide-wlc-5520-8540.html#pgfId-1309804 Then perform a full HUU install of CIMC 3.0.4d (or whatever the latest supported CIMC is, for the 5520/8540.)
Cisco 8540 and 5520 series controllers are based on Cisco UCS C series servers. Specifically Cisco 8540 is C240M4 series and Cisco 5520 controller is C220M4 hardware. It is recommended to use UCS Host Upgrade Utility (HUU) for this work. You can download that firmware from Cisco download page. Since this post based on standalone 8540 (not SSO) controller, I downloaded ucs-c240m4-huu-4.0.2h.iso which is the latest. (for 5520, you require ucs-c220m4-huu-4.0.2h.iso image)
You should have your WLC CIMC port connected to network with IP address configured on that interface. CIMC is the first port (see below, note it is 5520 WLC in this photo) of 4 ports available.
You can use “imm summary” CLI command to verify CIMC interface IP address configuration.
(8540-WLC) >imm summary This will take some time... Please be patient! User ID.......................................... admin DHCP............................................. Enabled IP Address....................................... x.x.32.100 Subnet Mask...................................... 255.255.254.0 Gateway.......................................... x.x.33.250
If it is not configured you can configure it statically or let it get IP via DHCP
(8540-WLC) >imm ? address IMM Static IP configuration dhcp Enable | Disable | Fallback DHCP. restart Saves settings and Restarts IMM Module. summary Displays IMM Parameters. username Configures Login Username for IMM. (8540-WLC) >imm address x.x.32.100 255.255.254.0 x.x.33.250 (8540-WLC) >imm username admin password <cimc_password>
You can access CIMC interface IP using https:
You can “Launch KVM Console” to open it. Make sure you security setting allow it if you using Mac OS
You have to simply follow instruction and keep continue until you get KVM console page.
You can go to “Virtual Media -> Activate Virtual Devices” as shown below
Once you activate Virtual Devices, you can map ISO image to CD/DVD
Note: Image below shows c220M4 (when I capture screenshot incorrectly select that image 🙂 ) , you should select c240m4 iso image for 8540
Then you can go to “Power” options and click “reset system” or “Power Cycle System”
Once server is booting up, You should hit F6 to change boot options. We need server to boot from KVM mapped DVD.
This will allow you to select boot from KVM map DVD image.
You will see it is boot with HUU image and prompt you to agree (be patience, it will take time)
It will take 15-20 min to copy all required files and prompt you the update options. You should go with update All unless you specifically want one component upgrade.
Each component will be upgraded and it will take 30-45 min to finish these component upgrade process.
Once upgrade finished, you can exist from that window and power cycle
It will take 15-20min to server to properly boot after CIMC upgrade. So overall you will have little more than 1hr outage while this work completed. Make sure you take necessary outage window arranged, if you do this upgrade to production 8540/5520 WLCs
You can also use “show imm chassis <>” command to verify those upgraded BIOS information
(8540-WLC) >show imm chassis ? bios Fetch Chassis BIOS information current Fetch Chassis Current information fan Fetch Chassis FAN information fan-profile Fetch Chassis FAN power profile mac Fetch Chassis MAC information memory Fetch Chassis Memory information power-supply Fetch Chassis Power Supply information sol-info Fetch Serial Over Lan information temperature Fetch Chassis Temperature information (8540-WLC) >show imm chassis bios BIOS Information Vendor: Cisco Systems, Inc. Version: C240M4.4.0.2d.0.0627191030 Release Date: 06/27/2019
I have to do this upgrade on 8540-HA pair 5520-HA pair soon. I will post the process once I do that task.
Update – 25 Apr 2020
There is a high severity vulnerability published that could impact these servers. So it is recommended to upgrade the CIMC version to 4.1(1f) to address this vulnerability.
Field Notice: FN – 70545 – SSD Will Fail at 40,000 Power-On Hours – BIOS/Firmware Upgrade Recommended
CSCvt55829-SSDs will experience data loss at 40k power on hours
You can check your SSD power-on hours using CIMC GUI (Storage -> Physical Driver Info) as shown below. Here is one of 8540 WLC info, You can see it got 38888 power-on hours and product ID listed as Samsung.
By information published (see below post responses), it looks like SanDisk SSD has this issue, not other vendor SSD.
https://community.cisco.com/t5/other-wireless-mobility-subjects/wlc-affected-of-ssd-bug/m-p/4073041
So these WLCs may not have that issue. Anyway, other vulnerabilities fixed in 4.1.1f (refer to this release note for more detail) still may be worth upgrading your WLC CIMC.
Here is another CSC post if you are having trouble getting upgrade your CIMC
https://community.cisco.com/t5/wireless/5520-cimc-update-fails-no-flash-access-to-old-cimc-r2-0/td-p/4658768
References
1. 8540/5520 Console access via CIMC
2. 8540/5520 Troubleshooting Guide
3. Firmware Upgrade on UCS servers through Host Upgrade Utility (HUU)
RELATED POSTS
Did you encounter this bug on a reload? Or was it there out of the box? I’ve run across this multiple times for a 5520 out of the box but never on a reload. Maybe Cisco’s bug report throws together the first boot and reload?
Hi Jordan,
No I haven’t encounter that bug. Did that upgrade as keep up with best practices.
HTH
Rasika
Hi Nayarasi, I also have to do a CIMC upgrade on an HA 5520 pair. Have you done your HA pair upgrade yet? Can you please share the procedures.
Thank you very much.
Michael Duong
HI Michael, Yes I did
https://mrncciew.com/2019/09/21/cimc-upgrade-wlc-in-ha/
Rasika
Rasika, did I tell you how awesome this post is?
Updating tomorrow…will keep you posted if anything missing.
Pls do let me know if anything you find in the process. Hope all goes well.
Rasika
Thanks Nayarasi,
I guess the same procedure will apply to upgrading CIMC code on Prime and CMX (running on UCS C-Series servers)?
Hi Khawar,
If it is same C-series, process should be same.
I tried CMX and no issues.
Prime,DNA-C, ISE I would be cautious & verify with TAC, if that upgrade does not cause any issue with those appliances.
I noticed certain field notices (eg shown below)
https://www.cisco.com/c/en/us/support/docs/field-notices/704/fn70456.html?emailclick=CNSemail
HTH
Rasika
Thanks,
Keeping in mind the flash going out of support in 2020, it would be must to go to non-flash CIMC codes, isn’t?
That’s a good point Khawar
Rasika
Hi Rasika,
did you configure ‘imm username’ ? Because when I configure this an try to log into the CIMC, this username / pwd i just configured do not work; I have to use the default “admin” and “password” and there you can change the password.
I could not remember, which method I used (modify admin user password once login first time or use that CLI command)
Thanks for the feedback
Rasika
Hi Riska,
Dang, all I get from my cimc, laptop connected directly.. after logon is that darn barber pole.. it is version 2 code. I can get access via CLI but I sure don’t have a clue after that.
I guess I have to say welcome back.. a bit belatedly .. or is it i’m back. I moved on during that time you were in trasit to down-under and you were a bit busy to blog.
Hi David,
You mean, you cannot get GUI access to CIMC IP ?
Regarding coming back, I am slowly picking up and will do some WiFi6 related post within next few months…
Regards
Rasika
Hi Riska,
Yes. in CIMC GUI, I get the logon page, and can enter credentials. After which the machine seems to reachout and load something.. that never completes. Have an enterprise wide outage window (today) and will power cycle the 5520’s then. Expect the CIMC’s to recover.
Hope you get CIMC GUI back once your power cycle it. Keep us updated
Rasika
All’s well. twas something about giving FF permission to run flash.. LOL. However I discovered bug CSCvf57867 single IMM/CIMC IP addr configured for both active and standby wlc. I moved the active sys (via cimc) to a new IP hoping the standby would wake up on one of the previous ip’s. It did not. Since this be a HA pare I guess I have to disable sso and reboot.. I can always unpatch the cimc’s and forget about it.
I spoke too soon. Standby woke up on the original ‘active’ assignment. I have everything back to ‘normal’. Now for the upgrade..
david
Thanks for the update. Hope it goes well from there onward
I remember I had that same CIMC ip issue. Whenever I failover, that unit get same CIMC IP.
Rasika
The release notes speak specifically to 3.0(4d) being certified for use with the WLC controllers. I took that to mean that no other version, including later versions is supported or should be used.
Hi Jeff, As we know all these documents does not get updated as we think. Therefore it may be simply not updating that doc (or you may be right). There is no way to know it 100%.
I have upgraded 5520/8540 CIMC with latest & did not experience any issues with it.
Rasika
I did find evidence that newer WLC’s ship with new CIMC code so I’ve sent the wireless BU a request to update the documentation.
I updated two 8540’s from 2.0.10c to 4.0.2h, and both completed, but on one of the two, the CIMC can no longer get the inventory for the Power Supplies. That same controller is now hitting a published WLC code bug related to the 2nd PS showing “OK” Off, “OK” on. A CIMC reboot did not solve it.
What Cisco wireless controllers support CIMC or have a CIMC port in them
8540/5520 are UCS based hardware, they got CIMC port.
Also CMX 3365 appliances too
Rasika
Hi Rasika,
Great post!!!
Side question
Which aireos code you are running on your controllers? And how stable it is?
HI Khawar, we are still with 8.5.140.0 as we got some 3602 in our network. It was quiet stable and haven’t upgrade since almost a year now
Rasika
thanks a lot. upgrading some remote wlc 5520 from 8.3 to 8.5.151 on sunday. CIMC is verion 2.x. As i can not connect locally to wlc i want minimize risks and let cimc as is on v2.x and just upgrade wlc to 8.5.151. Bad idea?
No, it is a good idea. Leave CIMC as it is in current version and do it later once you can it locally.
You should extend serial connection over LAN and in that way you can access console remotely. That is how I do it CIMC
Rasika
thanks. lets cross fingers upgrade of wlc runs fine tomorrow. apreachiate your superb work here. cisco should hire you to help them do easy to use howTos.
😂👌
Nayarasi, thank you for your excellent blog, and this post in particular was of great help to me. Interestingly, Cisco TAC pointed me towards your post as a technical reference in regards to the problems I was encountering, and it seems as if you know more about the products behavior than the official TAC does 🙂
Hi Jess, Thank you very much for the complement. I know it was a hard time to find quality documentation on that specific topic, hence documented what I have to go through and that became a really good document how to upgrade CIMC
KIT
Rasika
Hi Rasika, very usefull post.
Just a question, is it the same procedure to update CIMC if both WLC are in SSO mode ?
Thank you,
Hi Fred,
Yes, I have done a seperate post to cover that.
https://mrncciew.com/2019/09/21/cimc-upgrade-wlc-in-ha/
Rasika
Thank you so much for your answer.
It will be realy helpfull.
Great job.
Fred.
— Issue —
cimc upgrade not enough memory
— Workaround —
https://community.cisco.com/t5/unified-computing-system/cimc-supervisor-firmware-upgrade-times-out-on-c220-m3-servers/td-p/3693636
Thank you for sharing the reference link for given issue.
Rasika
Any idea how to do this now? It wants flash…..I need a time machine!
Hi John, I think you need to have an older version of a machine that supports flash.
Also, look at below CSC post, which may help you too
https://community.cisco.com/t5/wireless/5520-cimc-update-fails-no-flash-access-to-old-cimc-r2-0/td-p/4658768
HTH
Rasika
Hi Rasika-
I know this is an old post, question for you.
The Cisco release notes for 8.5.xxx on the 5520 recommend an upgrade to 3.x version and then to a 4.x version of CIMC. Did you upgrade from 2.x straight to 4.x or did you perform the intermediate 3.x and then to 4.x?
Hi Andrew, I did upgrade directly to 4.x from 2.x. No issues with that & if you read other comments you can see they also done in that way & no problem experienced.
Rasika
Thank you Rasika, the upgrade for both WLCs went off without a hitch! no issues encountered other than trying to find an old browser with flash support.
Good to hear that Andrew, Thank you for the update
Rasika
Hello Rasika, Hope all is well with you and your family.
Is there a procedure to upgrade the CIMC using the CLI? I have a pair of 8540 – HA running 2.0(10b) and this requires flash…I have tried a few plug in/containers on Firefox and Chrome but the CIMC never successfully loads. I am able to log into the CLI and GUI only the GUI never loads after login.
Thank you for any advice you may have…And for this awesome site you have built for us all.