NSX 2.5.0 to NSX 2.5.1 fails with error “restart service install-upgrade” on the NSX Manager.

I`ve recently ran through a problem when trying to upgrade NSX-T from version 2.5.0 to 2.5.1. When using the Upgrade function within the UI i was getting the following error:

This page is only available on the NSX Manager where Upgrade Coordinator is running. To configure the service, run the command “restart service install-upgrade” on the NSX Manager.

White checking the status of the service the service seemed to be running with no issues. I also checked the release notes for a couple of releases back and i found a similar issue in the release notes for the 2.3 release https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.3/rn/VMware-NSX-T-Data-Center-23-Release-Notes.html

Due to my install being a home lab i could not contact support. If you are experiencing this issue i would strongly advise to contact support before continuing further. VMware support contact information can be found here: https://www.vmware.com/group/vmware/get-help/

White reading the NSX-T 2.5.1 Upgrade guide from vmware documentation at page 22 i stumbled on instructions to upgrade the CSM. The instructions reference a .nub file but with no instructions on how to retrieve it. Based on whats available on the vmware download portal i was able to find a .mub file.

In order to bypass the error i was experiencing i downloaded the 2.5.1 version of the .mub file from vmware download portal.

After downloading the .mub file i used an unrachiver in my case 7-zip trying to extract an archive from the .mub file. Ive found that the .mub file included a .tar.gz archive and a .sig file. After extracting the tar.gz archive i was presented with a number of folders that included the VMware-NSX-unified-appliance-<version>.nub file i was looking for.

The file should be under Manager\nub. Once extracted it should be uploaded to /image/vmware/nsx/file-store/ on the nsx manager server

Verify the upgrade bundle by running: verify upgrade-bundle VMware-NSX-unified-appliance-<version> as the admin user. The output in my case was

verify upgrade-bundle VMware-NSX-unified-appliance-2.5.1.0.0.15314292
Checking upgrade bundle /var/vmware/nsx/file-store/VMware-NSX-unified-appliance-2.5.1.0.0.15314292.nub contents
Verifying bundle VMware-NSX-unified-appliance-2.5.1.0.0.15314292.bundle with signature VMware-NSX-unified-appliance-2.5.1.0.0.15314292.bundle.sig
Moving bundle to /image/VMware-NSX-unified-appliance-2.5.1.0.0.15314292.bundle
Extracting bundle payload
Successfully verified upgrade bundle
Bundle manifest:
appliance_type: ‘nsx-unified-appliance’
version: ‘2.5.1.0.0.15314292’
os_image_path: ‘files/nsx-root.squashfs’
os_image_md5_path: ‘files/nsx-root.squashfs.md5’
Current upgrade info:
{
“info”: “”,
“body”: {
“meta”: {
“from_version”: “2.5.0.0.0.14390405”,
“old_config_dev”: “/dev/mapper/nsx-config”,
“to_version”: “2.5.1.0.0.15314292”,
“new_config_dev”: “/dev/mapper/nsx-config__bak”,
“old_os_dev”: “/dev/sda2”,
“bundle_path”: “/image/VMware-NSX-unified-appliance-2.5.1.0.0.15314292”,
“new_os_dev”: “/dev/sda3”
},
“history”: []
},
“state”: 1,
“state_text”: “CMD_SUCCESS”
}

The next step was to upgrade using the bundle:

start upgrade-bundle VMware-NSX-unified-appliance-2.5.1.0.0.15314292 playbook VMware-NSX-manager-2.5.1.0.0.15314292-playbook

Node Upgrade is in progress. Please do not make any changes, until
the upgrade operation is complete.

2020-04-20 01:03:25,418 – Validating playbook /var/vmware/nsx/file-store/VMware-NSX-manager-2.5.1.0.0.15314292-playbook.yml
2020-04-20 01:03:25,492 – Running “unregister_ccp” (step 1 of 13)
2020-04-20 01:03:30,930 – Running “shutdown_manager” (step 2 of 13)
2020-04-20 01:05:18,077 – Running “install_os” (step 3 of 13)
2020-04-20 01:06:14,179 – Running “migrate_manager_config” (step 4 of 13)
2020-04-20 01:06:17,657 – Running “switch_os” (step 5 of 13)
2020-04-20 01:06:30,330 –

System will now reboot (step 6 of 13)
{
“info”: “”,
“body”: null,
“state”: 1,
“state_text”: “CMD_SUCCESS”
}