Sunday, March 13, 2016

Nexus 5548UP Firmware Upgrade with Dual-Homes FEXes

We have dual-homed our FEX for about 2 years now. Until recently, I have been told by lots of Cisco SEs that Cisco do not recommend dual-homed FEX, for various reason, including complexity.
We dual-home our fexes for a very specific reason, and I would still continue to do it for our 1GE FEX. Anyway, it's not the point of this post.

I have always been so worried about firmware upgrade the Nexus since the dual-home FEX is like an unknown area for me. Also, I know that our Nexus Environment do not qualify for ISSU since we have other switches downstream from the Nexus, that means we have to do DISRUPTIVE upgrade.

Today, I just went through the upgrade, from 6.0(2)N2(4) to 7.0(7)N1(1).

Before the upgrade, I found 2 articles from Cisco. One show how to do firmware upgrade on Dual-home FEX situation on 4.x, and the 7.x document that don't even mention about dual-home fex.



The #1 concern for me is that Disruptive Upgrade will reboot all FEX at once. The instruction is to upgrade Primary switch, FEX will  get new firmware, but stay operation and not-yet-upgraded after Switch1 upgrades. After Secondary switch upgrades, all FEX will reboot at once and get new firmware. I don't like that at all. But I spotted on the early document that after Primary Switch is upgrade, the FEX will be "unrecognized" by Primary, but still fully operational under Secondary switch, which means I can manual reload the FEX, and when it comes back, it will be "online" with Switch-1, and "offline" with Switch-2 until Switch-2 upgrade. I couldn't get TAC to confirm my theory, so I bid the bullet and went through the upgrade. It went well, really well with no hiccup.

Below was the procedure, step by step.
1. Upgrade Primary vPC switch using "install all" command. Do not use "force" switch.
2. Once Primary is upgrade, it will also "upload" the new firmware to all FEXes, but the FEX will not be upgraded.
3. Wait 10 minutes after the Primary fully-operate after reboot, do "show fex". You will see that the Primary Switch now see all FEX with "AA .... not matching".
4. Connect to Switch-2 console, do "show fex" and all FEX are online.
5. Under Switch-2 console, do "reload fex xxx", one fex at a time. Switch-2 will state that FEX is down.
6. Connect to Switch-1 console, wait for roughly 3 to 4 minutes, the reloaded FEX will start communicate with Switch-1 (which is now in newer firmware). Wait until Switch-1 reports FEX is online, then repeat step [5 to 6] for the rest of the FEX.
7. After all FEX are upgraded and "online" with Switch-1, upgrade Switch-2 with "install all...." command.

That should wrap up the firmware upgrade process.