Deploying vRA 8.8 using the Easy Installer

In this guide i will go over the deployment of vRA 8.8 using the Easy Installer. The reason i like the Easy Installer is because it deploys everything that i need. vIDM, Lifecycle Manager as well as vRA.

To get started we need to download the installer from here.

Once downloaded mount the iso as a drive using your favorite tool or extract it as a archive using something like 7-zip

Once mounted go to the CDROM in the mac directory if on mac or win32 if you are on windows. In here run the installer file

We will be presented with the following window:

The migration option allows us to migrate from from vRSLCM 2.1 to 8.x, it can migrate Datacenter and vCenters, all existing environments, DNS, SNMP, NTP, My VMware, proxy details, migration of vIDM installation as well as Blackstone Content endpoints. For the purpose of this guide we will process with the Install option

Once we click Install we will be presented with an Introduction of what components will be included part on the installation. In my case i can see vRealize Automation 8.8, Lifecycle Manager 8.8, Identity Manager 3.3.6.

Click Next, Review, accept the agreement and click next

Provide the vCenter details and click next. In my case i used the administrator account. A detailed list of permissions needed for deployment can be found here

Sections 4, 5, 6 and 7 are straight forward so i will skip them

In section 8 select a password that will be used across all products for the local usernames. ex for LCM root and local admin, vIDM admin, sshuser, root etc

Select a VM name for the Lifecycle Manager appliance and its ip credentials. If you are expecting a large repository in lifecycle manager we can add additional disk under the Increase Disk Size in GB section.

The FIPS Mode compliance option enforces FIPS compliance, however keep in mind that with the FIPS mode enabled there are limitation on what the product can do. This can be disabled later and re enabled as needed.

Complete the fields and click next

The next portion is vIDM. In case there is an existing vIDM appliance in the environment we can also import and existing vIDM appliance. In my case i will deploy a new one.

One important option under vIDM is the Sync Group Members to the Directory When Adding Group. When enabled, members of the groups are synced when groups are added from Active Directory. When this is disabled, group names are synced to the directory, but members of the group are not synced until the group is entitled to an application or the group name is added to an access policy. Note: Post deployment this value cannot be changed from vRealize Suite Lifecycle Manager. To update this field post deployment, navigate to VMware Identity Manager

My configuration page looks like this:

The next section is the vRA Configuration. In here we have a couple of options. We can perform a standard 1 node deployment or a cluster deployment which includes 3 appliances. The FIPS Compliance mode enables FIPS compliance. Unlike LCM this mode cannot be disabled after the deployment. This disables a number of options in vRA from an LCM perspective. Please make sure that its only enabled if required.

The advanced configuration at the bottom of the page includes the option to change the internal Kubernetes cluster and Service ip range. This is useful if the default range is already in use on the internal network. We want to make sure we pick an ip range thats not used somewhere else in order to avoid routing issues. Once complete click next

The next page gives us a summary of our deployment and we can click submit to start the deployment.

Next we are presented with the installation process. We can follow it along, in my environment the full deployment took about 1.5 hrs

After the install is complete we are presented to links for the different services

Next i would recommend LCM certificate management found here

ccp-backups folder missing in NSX-T backup

If you like me tried to to cleanup the backups in NSX-T and ran into error Cleanup script works only in folders, that contains subfolders "cluster-node-backups", "ccp-backups" and "inventory-summary" this post is for you.

I was trying to cleanup the backups before going to the next major release of nsx and i kept getting an error running the nsx_backup_cleaner.py script.

It would seem that the ccp-backups folder has been removed from the backup job so it simply doesn’t exist. VMware did fix the script with the 3.2 release.

If you have anything preventing you from getting to 3.2 here is the updated script

#!/usr/bin/env python
# ***************************************************************************
# Copyright 2020-2021 VMware, Inc.  All rights reserved. VMware Confidential.
# ***************************************************************************
# The purpose of this script is to remove old NSX backup files. Typically, this script
# will be placed on the SFTP server where the NSX Manager is uploading backup files,
# and included into a scheduler, for example cron.  Before running this script, you
# should update the BACKUP_ROOT variable.  This script works on Linux and Windows with
# both Python 2 and Python 3.
#
# On Linux SFTP server:
# You can add this script in the crontab to automatically run this script once daily
# Edit the anacron at /etc/cron.d or use crontab -e and add following line to execute the script at 10am everyday
# 00 10 * * * /sbin/nsx_backup_cleaner.py
#
# On Windows SFTP server:
# schtasks /Create /SC DAILY /TN PythonTask /TR "PATH_TO_PYTHON_EXE PATH_TO_PYTHON_SCRIPT"
# or you can add the same in TaskScheduler



from stat import S_ISREG, ST_ATIME, ST_CTIME, ST_MODE, S_ISDIR, S_IWUSR
import os, sys, time, datetime, shutil, getopt

def delete_files(delete_path_list, count):
    deleted_files = []

    for file in delete_path_list:
        for root, dirs, files in os.walk(file):
            for fname in files:
                full_path = os.path.join(root, fname)
                os.chmod(full_path, S_IWUSR)
        if count > 0:
            deleted_files.append(file)
            if os.path.isdir(file):
                shutil.rmtree(file)
            else:
                os.remove(file)
            count = count - 1
    return deleted_files


def delete_old_backup_enteries(folder, keep_days, min_count):
    keep_files = []
    for elem in os.listdir(folder):
        paths_sorted = []
        entries1 = (os.path.join(folder, elem, fn) for fn in os.listdir(os.path.join(folder, elem)))
        entries2 = ((os.stat(path), path) for path in entries1)
        entries3 = ((stat[ST_CTIME], path) for stat, path in entries2)
        for cdate, path in sorted(entries3):
            paths_sorted.append(path)

        if (len(paths_sorted) <= min_count):
            for file in paths_sorted:
                keep_files.append(file)
            continue

        delete_path_list = []
        for path in paths_sorted:
            file_create_time = os.path.getmtime(path)
            time_now = time.time()
            if ((time_now - file_create_time) > (keep_days * 24 * 60 * 60)):
                delete_path_list.append(path)

        deleted_files = delete_files(delete_path_list, min(len(delete_path_list), len(paths_sorted) - min_count))
        for file in deleted_files:
            paths_sorted.remove(file)

        for file in paths_sorted:
            keep_files.append(file)

    print(("Keeping the following backup files for folder %s" % folder))
    for file in keep_files:
        print(file)

def usage():
    print("""\
    Usage: nsx_backup_cleaner.py -d backup_dir [-k 1] [-l 5] [-h]
           Or
           nsx_backup_cleaner.py --dir backup_dir [--retention-period 1] [--min-count 5] [--help]

           Required
               -d/--dir: Backup root directory
               -k/--retention-period: Number of days need to retain a backup file
           Optional
               -l/--min-count: Minimum number of backup files to be kept, default value is 100
               -h/--help: Display help message
           """)
def main():
    BACKUP_ROOT = None

    BACKUPS_KEEP_DAYS = None
    # Minimum allowed: 100
    BACKUPS_MINCOUNT = 100

    try:
        opts, args = getopt.getopt(sys.argv[1:], "hd:k:l:", ["dir=", "retention-period=", "min-count=", "help"])
    except getopt.GetoptError as err:
    # print help information and exit:
        print ((str(err))) # will print something like "option -a not recognized"
        usage()
        sys.exit()

    for opt, arg in opts:
        if opt in ("-h", "--help"):
            usage()
            sys.exit()
        elif opt in ("-d", "--dir"):
            BACKUP_ROOT = arg
        elif opt in("-k", "--retention-period"):
            BACKUPS_KEEP_DAYS = int(arg)
        elif opt in ("-l", "--min-count"):
            BACKUPS_MINCOUNT = int(arg)
        else:
            usage()
            sys.exit()

    if (BACKUP_ROOT == None):
        print("Missing Backup Root")
        usage()
        sys.exit()

    if (BACKUPS_KEEP_DAYS == None):
        print("Missing Backup Retention Period in number of days")
        usage()
        sys.exit()

    if (not os.path.isdir(BACKUP_ROOT)):
        print("Wrong backup root directory")
        usage()
        sys.exit()

    backup_dirs = os.listdir(BACKUP_ROOT)
    if (all(elem in ["cluster-node-backups", "inventory-summary"] for elem in backup_dirs)):
        for elem in backup_dirs:
            if (elem in ["cluster-node-backups"]):
                delete_old_backup_enteries(os.path.join(BACKUP_ROOT, 'cluster-node-backups'), BACKUPS_KEEP_DAYS, BACKUPS_MINCOUNT)
            if (elem in ["inventory-summary"]):
                delete_old_backup_enteries(os.path.join(BACKUP_ROOT, 'inventory-summary'), BACKUPS_KEEP_DAYS, BACKUPS_MINCOUNT)
    else:
        print ("Cleanup script works only in folders, that contains subfolders \"cluster-node-backups\" and \"inventory-summary\"")


if __name__ == "__main__":
    main()

Additionally for the script to work make sure there are no other folders within the backup folder. the backup folder should only have the cluster-node-backups and inventory-summary directories

vCLS not starting with Failed – Feature ‘cpuid.mwait’ was absent, but must be present error

With the release of vCenter 7 Update 1, VMware introduced the vCLS (vSphere Clustering Service). More information can be found here.

Looking at the error details it looks like it is looking for a feature called cpuid.mwait

Reviewing the VMX file it seems like EVC is enabled on the vCLS VMs. I didnt want to enable EVC on the whole cluster so i wanted to do it only on the specific VMs.

Doing some research i found that the VMs need to be at version 14. After upgrading the VM i was able to disable EVC on the specific VMs by following these steps:

In the vSphere Client, navigate to the virtual machine

Under the Actions -> Compatibility -> Upgrade VM compatibility

We can disable EVC on per VM level on version 14 and above, so in my case i chose ESXi 6.7U2 and later

Next go to the Configure Tab

Pick VMware EVC and click on Edit

Click on Yes

Click on Disable EVC and Click OK

The next time it tries to power on the VM it should go through.

Once the first VM starts up it will most likely deploy a few additional ones, follow the same steps as above again on the new VMs

Skip-Level Upgrading VCF 4.x to 4.4.1 Step by Step

With the release of of VCF 4.4.1 i wanted to get my lab upgraded. The release blog can be found here and the release notes are here

In order to get to 4.4.1.0 in the past we had to upgrade to each version incrementally. As per the documentation found here we can now upgrade from any release past 4.1.0.1 directly to 4.4.1 skipping the releases in between. We can do so by going to Repository -> Bundle management -> Download now.

Alternatively if we are on 4.4.1.0 it can be found under Lifecycle Management -> Bundle Management -> Download now

We can also download it directly from the domain by going to Inventory -> Workload Domains -> Select the domain -> Update/Patches -> Select the VCF version -> Download now

The next step is to upgrade VCF by going to Inventory -> Workload Domains -> Select the workload domain -> Update/Patches -> Select the cloud foundation version were upgrading to and click on Update Now for the VMware Cloud Foundation Update 4.4.1.0

Next we are taken to the Upgrade page where we can follow the upgrade for each one of the components

Once the upgrade is complete we can click Finish to be returned back to the main screen

Because we are changing the SDDC-Manager versions i would strongly recommend to clear cache and log back in before going forward.

Next step is to download and install the configuration drift package. We can find it under Lifecycle management -> Bundle management -> Download now

We can also download it directly from the domain by going to Inventory -> Workload Domains -> Select the domain -> Update/Patches -> Select the VCF version -> Download now

Once the download is complete go to Inventory -> Workload Domains -> Select the domain -> Update/Patches -> Select the VCF version and click on update now

We can view the upgrade process by clicking on view status

Next step is to upgrade NSX-T installation to NSX-T 3.1.3.7.4. The release notes can be found here. We can go to inventory -> Workload Domains -> Select the workload domain -> Update/Patches -> Under Available updates click on the drop down and select Cloud Foundation 4.4.1.0 -> Download now.

Once the download is complete we can proceed with updating the NSX components by clicking on the update now

Make the proper selection and click next

Make the proper selection and click next

Make the proper selection and click next

Review the options and click on Finish

The upgrade will go thought upgrading the NSX edges. We can view the upgrade status by clicking on view status

Once the edges are upgraded we an go back to Inventory -> Workload Domains -> Select the workload domain -> Update/Patches -> Under Available updates click on Update Now

Review the selection and click next

Review the host clusters and click next

Review the upgrade options and click next

Review the selection and click finish

We can view the status of the upgrade by selection view status

Once the upgrade is complete we can proceed with the vCenter Upgrade. VCF 4.4.1 comes with vCenter Server 7.0 Updated 3d. The release notes can be found here. We can go to inventory -> Workload Domains -> Select the workload domain -> Update/Patches -> Under Available updates click on the drop down and select Cloud Foundation 4.4.1 -> Download now.

Once the download is complete we can click on Update now

We can follow the status of the upgrade by clicking on the view status tab

Here we can see the different components that are getting upgraded

Once the upgrade is complete we are taken back to the previous page where we can see that the ESXi servers are next. The release notes can be found here. Click on Download Now

Once the download is complete we can click on Update now

If we have multiple clusters we can enable Cluster-level selection and select the specific cluster(s) we want to upgrade.

We can also enable sequential cluster upgrade as well as quick boot

We get to review the options once again before we click finish to to submit the task

Once submitted we can view the status by clicking on View Status

And with that we are finished with the workload domain. We can follow the same steps for the other domains

Don’t forget to clean up the download bundles by following the steps from my other blog here

Upgrading vROPS (vRealize Operations Manager) to 8.6.3 via vRSLCM

In this post i will go over upgrading my 8.x vROPS appliance to 8.6.3 using vRSLCM (vRealize Suite Lifecycle Manager). As a pre requirement we do need to have vRSLCM (vRealize Lifecycle Manager) upgraded to 8.8 Instructions can be found here. The upgrade already includes the latest Product Support Pack so an update to the Product Support Pack is not required.

To get started we can go to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> Binary Mapping. (If you haven’t added your My VMware credentials you will need to do that first by going to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> My VMware)

Click on Add Binaries under Product Binaries

Select My VMware and click on Discover

We can see a list of binaries that have been discovered. Make sure we select the upgrade package not the install package. We can select what we need and click on Add

This will create a request and start downloading the package. To view the progress we can click on the Click Here hyperlink

Click on the in Progress button to view the details

We now have to wait for the download to complete

After the download is complete we can go to Environments -> View Details on the environment that includes vROPS

Click on Upgrade

An Inventory sync is recommended if the environment has changed since LCM performed the last sync. We trigger the sync from the UI or click on Proceed to continue

Select product Version 8.6.3 and click Next. We can also review the compatibility matrix to make sure the environment is compatible.

Run the Upgrade Assessment tool to make sure the currently used dashboards, reports, metrics etc are still compatible with the new version

Once the report has finished running we can either Download or view the report. Once everything has been reviewing we can click on the I have viewed the report and agree to proceed box and click next to proceed to the next step.

A new feature that was added was the capability to automatically create a snapshot prior to the upgrade and remove it after the upgrade. On this screen we also have the ability to chose if we want to keep the snapshots post upgrade for validation testing for example. Click next

Run the Precheck to make sure there are no errors or issues.

Once the check is complete we can review the checks that were performed and we can continue by clicking Next.

Review the upgrade details and click on Next and the Submit. We are taken to the progress screen where we can follow the progress.

The system will get rebooted and once its back up we will be on 8.6.3

Since we are doing a major upgrade i strongly recommend to clean the cache before using the new vROPS version.

Upgrading vRLI (vRealize Log Insight) to 8.6.2 via vRSLCM

In this post i will go over upgrading my 8.x vRLI appliance to 8.6.2 using vRSLCM (vRealize Suite Lifecycle Manager). As a pre requirement we do need to have vRSLCM (vRealize Lifecycle Manager) upgraded to 8.6.2 or 8.7. Instructions can be found here. The upgrade already includes the latest Product Support Pack so an update to the Product Support Pack is not required.

To get started we can go to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> Binary Mapping. (If you haven’t added your My VMware credentials you will need to do that first by going to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> My VMware)

Click on Add Binaries under Product Binaries

Select My VMware and click on Discover

We can see a list of binaries that have been discovered. Make sure we select the upgrade package not the install package. We can select what we need and click on Add

This will create a request and start downloading the package. To view the progress we can click on the Click Here hyperlink

Click on the in Progress button to view the details

We now have to wait for the download to complete

After the download is complete we can go to Environments -> View Details on the environment that includes vRLI

Click on Upgrade

An Inventory sync is recommended if the environment has changed since LCM performed the last sync. We trigger the sync from the UI or click on Proceed to continue

Select product Version 8.8 and click Next. We can also review the compatibility matrix to make sure the environment is compatible.

A new feature that was added was the capability to automatically create a snapshot prior to the upgrade and remove it after the upgrade. On this screen we also have the ability to chose if we want to keep the snapshots post upgrade for validation testing for example. Click next

Run the Precheck to make sure there are no errors or issues.

Once the check is complete we can review the checks that were performed and we can continue by clicking Next.

Review the upgrade details and click on Next then Finish. We are taken to the progress screen where we can follow the progress.

The system will get rebooted and once its back up we will be on 8.8

Since we are doing a major upgrade i strongly recommend to clean the cache before using the new vRLI version.

Upgrading vRA (vRealize Automation) to 8.8

In this post i will go over upgrading my 8.x vRA appliance to 8.8. As a pre requirement we do need to have vRSLCM (vRealize Lifecycle Manager) upgraded to 8.78 Instructions can be found here. The upgrade already includes the latest Product Support Pack so an update to the Product Support Pack is not required.

To get started we can go to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> Binary Mapping. (If you haven’t added your My VMware credentials you will need to do that first by going to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> My VMware)

Click on Add Binaries under Product Binaries

Select My VMware and click on Discover

We can see a list of binaries that have been discovered. We can select what we need and click on Add

This will create a request and start downloading the package. To view the progress we can click on the Click Here hyperlink

Click on the in Progress button to view the details

We now have to wait for the download to complete

After the download is complete we can go to Environments -> View Details on the environment that includes vRA

Click on Upgrade

An Inventory sync is recommended if the environment has changed since LCM performed the last sync. We trigger the sync from the UI or click on Proceed to continue

Select product Version 8.8.0 and click Next. We can also review the compatibility matrix to make sure the environment is compatible.

Anew feature that was added was the capability to automatically create a snapshot prior to the upgrade and remove it after the upgrade. On this screen we also have the ability to chose if we want to keep the snapshots post upgrade for validation testing for example. Click next

Run the Precheck to make sure there are no errors

The next screen remind us of hardware requirements for vRA and vIDM which can be checked here for vRA and vIDM which can be checked here. As far as i can tell they haven’t changed since the 8.3 release. Check the I took care of the manual steps above and am ready to proceed check mark and click on Run Precheck

Once the check is complete we can review the checks that were performed and we can continue by clicking Next.

Review the upgrade details and click on Next. We are taken to the progress screen where we can follow the progress.

The system will get rebooted and once its back up we will be on 8.8

Since we are doing a major upgrade i strongly recommend to clean the cache before using the new vRA version.

SSC 8.8 urllib3 (1.25.11) or chardet (4.0.0) doesn’t match a supported version

I recently upgraded my LCM deployed SSC server to 8.8. If you need a guide to go through the upgrade you can find my other post here.

After the upgrade was completed i was noticing strange behavior in the SSC UI so i checked the status the of the services. Here are the errors i found and how i fixed them

The first step was to check the status of the service

systemctl status salt-master

The return was this

* salt-master.service - The Salt Master Server
   Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2022-05-16 03:32:35 UTC; 6s ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltproject.io/en/latest/contents.html
 Main PID: 4577 (salt-master)
    Tasks: 39 (limit: 9830)
   Memory: 335.0M
   CGroup: /system.slice/salt-master.service
           |-4577 /bin/python3 /usr/bin/salt-master
           |-4581 /bin/python3 /usr/bin/salt-master
           |-4589 /bin/python3 /usr/bin/salt-master
           |-4593 /bin/python3 /usr/bin/salt-master
           |-4602 /bin/python3 /usr/bin/salt-master
           |-4606 /bin/python3 /usr/bin/salt-master
           |-4608 /bin/python3 /usr/bin/salt-master
           |-4609 /bin/python3 /usr/bin/salt-master
           |-4616 /bin/python3 /usr/bin/salt-master
           |-4697 /bin/python3 /usr/bin/salt-master
           |-4699 /bin/python3 /usr/bin/salt-master
           |-4703 /bin/python3 /usr/bin/salt-master
           |-4711 /bin/python3 /usr/bin/salt-master
           |-4712 /bin/python3 /usr/bin/salt-master
           |-4713 /bin/python3 /usr/bin/salt-master
           |-4714 /bin/python3 /usr/bin/salt-master
           |-4715 /bin/python3 /usr/bin/salt-master
           `-4717 /bin/python3 /usr/bin/salt-master

May 16 03:32:34 ssc-01a.corp.local systemd[1]: Starting The Salt Master Server...
May 16 03:32:35 ssc-01a.corp.local salt-master[4577]: [WARNING ] /usr/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.11) or chardet (4.0.0) doesn't match a supported version!
May 16 03:32:35 ssc-01a.corp.local salt-master[4577]:   RequestsDependencyWarning)
May 16 03:32:35 ssc-01a.corp.local systemd[1]: Started The Salt Master Server.

The way i got around the error was by running

pip3 install --upgrade requests

Alternatively the official documentation is here talks about extracting the .whl file from the my vmware customer connect portal here. The file we are looking for is vRA_SaltStack_Config-8.8.0.7-1_Installer.tar.gz

Once Downloaded we are looking for SSEAPE-8.8.0.7-py2.py3-none-any.whl found under sse-installer/salt/sse/eapi_plugin/files

The file needs to be uploaded on the node having the issue and we would run

sudo pip3 install SSEAPE-8.8.0.7-py2.py3-none-any.whl --prefix /usr

Finally we can restart the saltstack service and verify that its running without errors:

* salt-master.service - The Salt Master Server
   Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2022-05-15 20:02:56 UTC; 51s ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltproject.io/en/latest/contents.html
 Main PID: 31309 (salt-master)
    Tasks: 39 (limit: 9830)
   Memory: 330.0M
   CGroup: /system.slice/salt-master.service
           |-31309 /bin/python3 /usr/bin/salt-master
           |-31315 /bin/python3 /usr/bin/salt-master
           |-31320 /bin/python3 /usr/bin/salt-master
           |-31323 /bin/python3 /usr/bin/salt-master
           |-31325 /bin/python3 /usr/bin/salt-master
           |-31326 /bin/python3 /usr/bin/salt-master
           |-31327 /bin/python3 /usr/bin/salt-master
           |-31328 /bin/python3 /usr/bin/salt-master
           |-31330 /bin/python3 /usr/bin/salt-master
           |-31397 /bin/python3 /usr/bin/salt-master
           |-31398 /bin/python3 /usr/bin/salt-master
           |-31400 /bin/python3 /usr/bin/salt-master
           |-31411 /bin/python3 /usr/bin/salt-master
           |-31412 /bin/python3 /usr/bin/salt-master
           |-31413 /bin/python3 /usr/bin/salt-master
           |-31414 /bin/python3 /usr/bin/salt-master
           |-31415 /bin/python3 /usr/bin/salt-master
           `-31416 /bin/python3 /usr/bin/salt-master

May 15 20:02:54 ssc-01a.corp.local systemd[1]: Starting The Salt Master Server...
May 15 20:02:56 ssc-01a.corp.local systemd[1]: Started The Salt Master Server.

If the status page a return similar to this

sseapi_rpc_queue: could not connect to SSE server

Follow my other guide here

SSC 8.8 sseapi_rpc_queue: could not connect to SSE server

I recently upgraded my LCM deployed SSC server to 8.8. If you need a guide to go through the upgrade you can find my other post here.

After the upgrade was completed i was noticing strange behavior in the SSC UI so i checked the status the of the services. Here are the errors i found and how i fixed them

The first step was to check the status of the service

systemctl status salt-master

The return was this

root@ssc-01a [ ~ ]# systemctl status salt-master
* salt-master.service - The Salt Master Server
   Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2022-05-15 16:39:26 UTC; 8min ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltproject.io/en/latest/contents.html
 Main PID: 3035 (salt-master)
    Tasks: 39 (limit: 9830)
   Memory: 357.0M
   CGroup: /system.slice/salt-master.service
           |-3035 /bin/python3 /usr/bin/salt-master
           |-3041 /bin/python3 /usr/bin/salt-master
           |-3110 /bin/python3 /usr/bin/salt-master
           |-3115 /bin/python3 /usr/bin/salt-master
           |-3119 /bin/python3 /usr/bin/salt-master
           |-3122 /bin/python3 /usr/bin/salt-master
           |-3123 /bin/python3 /usr/bin/salt-master
           |-3124 /bin/python3 /usr/bin/salt-master
           |-3125 /bin/python3 /usr/bin/salt-master
           |-3203 /bin/python3 /usr/bin/salt-master
           |-3204 /bin/python3 /usr/bin/salt-master
           |-3206 /bin/python3 /usr/bin/salt-master
           |-3214 /bin/python3 /usr/bin/salt-master
           |-3216 /bin/python3 /usr/bin/salt-master
           |-3219 /bin/python3 /usr/bin/salt-master
           |-3220 /bin/python3 /usr/bin/salt-master
           |-3221 /bin/python3 /usr/bin/salt-master
           `-4871 /bin/python3 /usr/bin/salt-master

May 15 16:39:26 ssc-01a.corp.local systemd[1]: Started The Salt Master Server.
May 15 16:39:27 ssc-01a.corp.local salt-master[3035]: [ERROR   ] sseapi_rpc_queue: could not connect to SSE server: [Errno 111] Connection refused
May 15 16:39:27 ssc-01a.corp.local salt-master[3035]: [ERROR   ] sseapi_event_queue: could not connect to SSE server: [Errno 111] Connection refused
May 15 16:39:27 ssc-01a.corp.local salt-master[3035]: [ERROR   ] Failed to get the salt environments: [Errno 111] Connection refused
May 15 16:39:28 ssc-01a.corp.local salt-master[3035]: [ERROR   ] Failed to retrieve commands from SSE: [Errno 111] Connection refused
May 15 16:39:32 ssc-01a.corp.local salt-master[3035]: [ERROR   ] sseapi_rpc_queue: could not connect to SSE server: [Errno 111] Connection refused
May 15 16:39:32 ssc-01a.corp.local salt-master[3035]: [ERROR   ] sseapi_event_queue: could not connect to SSE server: [Errno 111] Connection refused
May 15 16:39:37 ssc-01a.corp.local salt-master[3035]: [ERROR   ] sseapi_rpc_queue: could not connect to SSE server: [Errno 111] Connection refused
May 15 16:39:38 ssc-01a.corp.local salt-master[3035]: [ERROR   ] Failed to retrieve commands from SSE: [Errno 111] Connection refused
May 15 16:39:38 ssc-01a.corp.local salt-master[3035]: [ERROR   ] sseapi_event_queue: could not connect to SSE server: [Errno 111] Connection refused

The way i got around the error was by editing the /etc/salt/master.d/raas.conf. The file seems to be missing a key auth parameter. The engines section should look like this

engines:
  - sseapi: {}
  - eventqueue: {}
  - rpcqueue: {}
  - jobcompletion: {}
  - keyauth: {}

After restarting the salt master i was able to verify that the error was gone. To restart the service i ran

systemctl restart salt-master

To verify the status i ran

systemctl restart salt-master
* salt-master.service - The Salt Master Server
   Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2022-05-15 20:02:56 UTC; 51s ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltproject.io/en/latest/contents.html
 Main PID: 31309 (salt-master)
    Tasks: 39 (limit: 9830)
   Memory: 330.0M
   CGroup: /system.slice/salt-master.service
           |-31309 /bin/python3 /usr/bin/salt-master
           |-31315 /bin/python3 /usr/bin/salt-master
           |-31320 /bin/python3 /usr/bin/salt-master
           |-31323 /bin/python3 /usr/bin/salt-master
           |-31325 /bin/python3 /usr/bin/salt-master
           |-31326 /bin/python3 /usr/bin/salt-master
           |-31327 /bin/python3 /usr/bin/salt-master
           |-31328 /bin/python3 /usr/bin/salt-master
           |-31330 /bin/python3 /usr/bin/salt-master
           |-31397 /bin/python3 /usr/bin/salt-master
           |-31398 /bin/python3 /usr/bin/salt-master
           |-31400 /bin/python3 /usr/bin/salt-master
           |-31411 /bin/python3 /usr/bin/salt-master
           |-31412 /bin/python3 /usr/bin/salt-master
           |-31413 /bin/python3 /usr/bin/salt-master
           |-31414 /bin/python3 /usr/bin/salt-master
           |-31415 /bin/python3 /usr/bin/salt-master
           `-31416 /bin/python3 /usr/bin/salt-master

May 15 20:02:54 ssc-01a.corp.local systemd[1]: Starting The Salt Master Server...
May 15 20:02:56 ssc-01a.corp.local systemd[1]: Started The Salt Master Server.

VMware documentation also talks about the procedure above in the Upgrade the Master Plugin documentation found here

If the status page a return similar to this

 [py.warnings      :110 ][WARNING ][5488] /usr/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.11) or chardet (4.0.0) doesn't match a supported version!
  RequestsDependencyWarning)

Follow my other guide here

Upgrading SSC (SaltStack Config) to 8.8

In this post i will go over upgrading my 8.x SSC appliance to 8.8. As a pre requirement we do need to have vRSLCM (vRealize Lifecycle Manager) upgraded to 8.8. Instructions can be found here. The upgrade already includes the latest Product Support Pack so an update to the Product Support Pack is not required.

To get started we can go to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> Binary Mapping. (If you haven’t added your My VMware credentials you will need to do that first by going to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> My VMware)

Click on Add Binaries under Product Binaries

Select My VMware and click on Discover

We can see a list of binaries that have been discovered. We can select what we need and click on Add

This will create a request and start downloading the package. To view the progress we can click on the Click Here hyperlink

Click on the in Progress button to view the details

We now have to wait for the download to complete

After the download is complete we can go to Environments -> View Details on the environment that includes SSC

Click on Upgrade

An Inventory sync is recommended if the environment has changed since LCM performed the last sync. We trigger the sync from the UI or click on Proceed to continue

Select product Version 8.8.0 and click Next. We can also review the compatibility matrix to make sure the environment is compatible.

We can automatically create and delete a snapshot part of the upgrade process

Run the Precheck to make sure there are no errors

Once the check is complete, click on Next. Review the upgrade details and click on Next. We are taken to the progress screen where we can follow the progress.

The system will get rebooted and once its back up we will be on 8.8