Shutting down a vSAN Cluster

I have the need to completely shut down some of my vSAN clusters for various clusters and ive been having a hard time finding the proper procedure. As of 2/16/2021 VMware released guidance here

Here are the steps i took to do it on my end. If you have the vCLS service enabled follow my other instructions here prior to starting the rest of this guide.

Disable cluster member updates from vCenter on each ESXi host in the cluster by running

esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListUpdates

After the above is completed run the below only on 1 of the ESXi hosts. Take note of the host

python /usr/lib/vmware/vsan/bin/reboot_helper.py prepare

Place all ESXi hosts in maintenance mode with NoAction

esxcli system maintenanceMode set -e true -m noAction

Perform the necessary maintenance. Once the hosts are back up we will run the above in reverse

Remove the maintence mode on all ESXi hosts by running

esxcli system maintenanceMode set -e false

Run the below command on the same host it was originally run

python /usr/lib/vmware/vsan/bin/reboot_helper.py recover

vSAN availability can be checked by running

esxcli vsan cluster get

Enable cluster member updates for vCenter

esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListUpdates

Upgrading VCF 4.1.0.0 to 4.2.0.0 Step by Step

With the release of of VCF 4.2 i wanted to get my lab upgraded. The release blog can be found here and the release notes are here

In order to get to 4.2.0.0 we have to upgrade to 4.1.0.1. We can do so by going to Repository -> Bundle management -> Download now

The next step is to upgrade VCF by going to Inventory -> Workload Domains -> Select the workload domain -> Update/Patches -> Update Now for the VMware Cloud Foundation Update 4.1.0.1. The release notes can be found here

Next we are taken to the Upgrade page where we can follow the upgrade for each one of the components

Once the upgrade is complete we can click Finish to be returned back to the main screen

Because we are changing the SDDC-Manager versions i would strongly recommend to clear cache and log back in before going forward.

Next is the 4.2.0.0 update. Repository -> Bundle management -> Download now. In my case i already had it downloaded so the next step is to apply the upgrade by going to Inventory -> Workload Domains -> Select the workload domain -> Update/Patches -> Update Now for the VMware Cloud Foundation Update 4.2.0.0. The release notes can be found here

Once the upgrade starts we can follow its progress

Once the upgrade is completed we can click finish and go to the next step

Again i would recommend clearing the cache since we changed sddc-manager versions.

Once the upgrade is complete we are taken back to the previous page where we can see that the ESXi servers are next. The release notes can be found here. Click on Download Now

Once the download is complete we can click on Update now

If we have multiple clusters we can enable Cluster-level selection and select the specific cluster(s) we want to upgrade.

We can also enable sequential cluster upgrade as well as quick boot

We get to review the options once again before we click finish to to submit the task

Once submitted we can view the status by clicking on View Status

And with that we are finished with the workload domain. We can get back to the Update/Patches page

The next update is the configuration drift bundle. We can go to inventory -> Workload Domains -> Select the workload domain -> Update/Patches -> Download now. You will notice a new drop down that allows us to pick the Cloud Foundation version.

Once the download is complete click on update now

Once the upgrade started i got redirected to the Update status page.

Considering the update is only 219 MB the upgrade went through pretty quickly. Once its completed we can click finish to get back to the main sddc manager page

Next step is to upgrade NSX-T installation to NSX-T 3.1.0. The release notes can be found here. We can go to inventory -> Workload Domains -> Select the workload domain -> Update/Patches -> Download now.

Once the download is complete click on Update Now

We can view the status and the steps by clicking on View Status.

Once the upgrade is complete we are redirected back to the available updates page showing that the vCenter server is next

Click on Download now and wait for the download to complete. Once the download is complete click on update now

We can view the task by clicking on View Status

Next are the the additional domains that we might have where we can follow the same instructions as above. The process will be allot quicker because the upgrades are already downloaded

If no additional upgrades are needed we can clean up the downloads by following the instructions on my other post here

vRA cloud API getting started

I wanted to keep track of what needs to be done prior to actually being able to query API on vRA cloud. Ive been having a hard time finding the documentation i needed in the past

First step is to get an API token for the specific username. We can do this by going to My account under User Settings in vRA Cloud

Go to API Tokens and click on Generate a new api token

Give it a name and select what it will have access to and click on Generate

Once we have the api token generated we can use it to generate an access token by running

curl --location --request POST 'https://console.cloud.vmware.com/csp/gateway/am/api/auth/api-tokens/authorize' --header 'Content-Type: application/x-www-form-urlencoded' --data-urlencode 'refresh_token=api token generated earlier'

Now we can use the output from “access_token”. There`s a number of Cloud Assembly examples here

Full guide available here

vIDM upgrade to 3.3.4 no networking detected

If you are like me and you tried to perform an upgrade of vIDM 3.3.x to 3.3.4 you were most likely greeted by no network connectivity after upgrade with the following screen:

If you made a backup of the network configuration this is where we would restore it.

If not we can perform it manually by running

/opt/vmware/share/vami/vami_config_net

Press 6 and go through the screens to configure the ip

Press 2 for default gateway

Press 4 for the DNS

Press 0 to show the current configuration

If you notice in my configuration the dns server didnt take. In order to fix it i recreated the /etc/resolv.conf.

Remove the resolv.conf running

rm -f /etc/resolv.conf

Create a symlink for resolv.conf

ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

In my case vIDM was still not responding so i had to reboot the server. After the reboot everything started working properly

vIDM 3.3.4 online upgrade

I wanted to go through the upgrade of vIDM 3.3.4 outside of lifecycle manager. I wanted to do it using the online upgrade via the console. Official documentation can be found here. First step is to log on to the console via the root credentials and upgrade the upgrade manager to 3.3.4

/usr/local/horizon/update/updatemgr.hzn updateinstaller

Set the upgrade version that you are upgrading from in my case was 3.3.2 so i ran

/usr/local/horizon/update/configureupdate.hzn manifest --set-version 3.3.2.0

Because we are switching operating systems to Photon OS 3 we need to download the dual boot update package from here. Look for VMware Identity Manager Service Virtual Appliance Dual Boot Update and transfer it to the vIDM server

And finally run the update

/usr/local/horizon/update/updatemgr.hzn update

Specify the path to the dualboot file. It would look something like this:

If everything went through properly you would be greeted with something like this

All we need to do now is to reboot.

If you are greeted by the following screen follow my other post here

Upgrading vRA (vRealize Automation) to 8.3

In this post i will go over upgrading my 8.2 vRA appliance to 8.3. As a pre requirement we do need to have vRSLCM (vRealize Lifecycle Manager) upgraded to 8.3. Instructions can be found here

To get started we can go to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> Binary Mapping. (If you havent added your My VMware credentials you will need to do that first by going to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> My VMware)

Click on Add Binaries under Product Binaries

Select My VMware and click on Discover

We can see a list of binaries that have been discovered. We can select what we need and click on Add

This will create a request and start downloading the package. To view the progress we can click on the Click Here hyperlink

Click on the in Progress button to view the details

We now have to wait for the download to complete

After the download is complete we can go to Environments -> View Details on the environment that includes vRA

Click on Upgrade

An Inventory sync is recommended if the environment has changed since LCM performed the last sync. We trigger the sync from the UI or click on Proceed to continue

Select product Version 8.3.0 and click Next. We can also review the compatibility matrix to make sure the environment is compatible.

Run the Precheck to make sure there are no errors

Once the check is complete, click on Next. Review the upgrade details and click on Next. We are taken to the progress screen where we can follow the progress.

The system will get rebooted and once its back up we will be on 8.3

Upgrading vRSLCM (vRealize Lifecycle Manager) to 8.3

In this guide i will go over the steps of getting an existing 8.x vRSLCM appliance upgraded to the latest 8.3 release

The first step is to lo in to vRealize Suite Lifecycle Manager under the Lifecycle Operations section

Go to settings -> System Upgrade

Click on Check for Upgrade

We can see that the check found a new version available for 8.3

Click on Upgrade

This will fire up the upgrade process and start upgrading packages. The system will automatically reboot on 8.3 once completed. We can check the version by going to Settings -> System Details

Deploying SaltStack Config via Lifecycle Manager in a VRA environment

Assuming that you already downloaded the SaltStack Config binaries in this guide we will go over the deployment of SaltStack Config in an existing VRA 8.3 environment via Lifecycle Manager 8.3

Log on to Lifecycle Manager -> Go to environments, on the vRA environment click on the Menu and go to Add Product

Select the checkbox next to vRealize Automation SaltStack Config and click Next

Review the EULA, click on the agree box on the bottom of the page and click on next

Click on Select and select a vrealize or vra advanced\enterprise. Clieck on the validate button and click next

Select an existing certificate or click on the + sign and generate a new certificate. In my case i didnt have a certificate created so im going to create a new one. If you already have a certificate this step can be skipped.

Complete the fields and click on generate

Now the list is populated with the certificate we just created. Click Next

Select the infrastructure details and click on Next

Verify the network details and click next

If the tenant list is empty a sync of vra is required

To do this we can go to Environments. Dont worry the progress will be saved

View details

Click on submit

Once the sync is complete we can go back to requests

Under requests click on Pending Action

Complete the details and click next

Click on Run precheck in order to make sure everything is in a working order

If everything looks good click on Next

Verify the summary and click on Submit

LCM will go through everything needed to. As we can see theres allot of steps that otherwise would need to perform manually

Get a cup of coffee and wait. In my case it took about 13 minutes for the installation to finish

Deploying vRA 8.3 using the Easy Installer

In this guide i will go over the deployment of vRA 8.3 using the Easy Installer. The reason i like the Easy Installer is because it deploys everything that i need. vIDM, Lifecycle Manager as well as vRA.

To get started we need to download the installer from here.

Once downloaded mount the iso as a drive using your favorite tool or extract it as a archive using something like 7-zip

Once mounted go to the CDROM in the mac directory if on mac or win32 if you are on windows. In here run the installer file

We will be presented with the following window:

The migration option allows us to migrate from from vRSLCM 2.1 to 8.3, it can migrate Datacenter and vCenters, all existing environments, DNS, SNMP, NTP, My VMware, proxy details, migration of vIDM installation as well as Blackstone Content endpoints. For the purpose of this guide we will process with the Install option

Once we click Install we will be presented with an Introduction of what components will be included part on the installation. In my case i can see vRealize Automation 8.3, Lifecycle Manager 8.3, Identity Manager 3.3.4.

Click Next, Review, accept the agreement and click next

Provide the vCenter details and click next. In my case i used the administrator account. A detailed list of permissions needed for deployment can be found here

Sections 4, 5, 6 and 7 are straight foward so i will skip them

In section 8 select a password that will be used across all products for the local usernames. ex for LCM root and local admin, vIDM admin, sshuser, root etc

Select a VM name for the Lifecycle Manager appliance and its ip credentials. If you are expecting a large repository in lifecycle manager we can add additional disk under the Increase Disk Size in GB section.

The FIPS Mode compliance option enforces FIPS compliance, however keep in mind that with the FIPS mode enabled there are limitation on what the product can do. This can be disabled later and re enabled as needed.

Complete the fields and click next

The next portion is vIDM. In case there is an existing vIDM appliance in the environment we can also import and existing vIDM appliance. In my case i will deploy a new one.

One important option under vIDM is the Sync Group Members to the Directory When Adding Group. When enabled, members of the groups are synced when groups are added from Active Directory. When this is disabled, group names are synced to the directory, but members of the group are not synced until the group is entitled to an application or the group name is added to an access policy. Note: Post deployment this value cannot be changed from vRealize Suite Lifecycle Manager. To update this field post deployment, navigate to VMware Identity Manager

My configuration page looks like this:

The next section is the vRA Configuration. In here we have a couple of options. We can perform a standard 1 node deployment or a cluster deployment which includes 3 appliances. The FIPS Compliance mode enables FIPS compliance. Unlike LCM this mode cannot be disabled after the deployment. This disables a number of options in vRA from an LCM perspective. Please make sure that its only enabled if required.

The advanced configuration at the bottom of the page includes the option to change the internal Kubernetes cluster and Service ip range. This is useful if the default range is already in use on the internal network. We want to make sure we pick an ip range thats not used somewhere else in order to avoid conflicts. Once complete click next

The next page gives us a summary of our deployment and we can click submit to start the deployment.

Next we are presented with the installation process. We can follow it along, in my environment the full deployment took about 1.5 hrs

After the install is complete we are presented to links for the different services

Next i would recommend LCM certificate management found here

Creating a customized Snapshot as a Day 2 Action in vRA 8.2

I wanted to give my users the capability to create a snapshot but limit them to only 1 snapshot and have a predefined description from the snapshot so i can know that it was taken from vRA.

For the purpose of this i will be using a customer workflow in Orchestrator and Resource Actions and in Cloud Assembly.

Since Orchestrator comes with a number of pre defined workflows in always like to jump start by find something thats similar with what im trying to do. For the purpose of this we will be using the Create Snapshot Workflow.

To get started we can go to Orchestrator -> Workflows -> search for Create a snapshot. Once found we can click on Actions -> Duplicate

We can see that a Copy has been created. We will modify this so it can fit the custom specifications required.

Click on Open on the Copy:

Change the name to something more meaningful like VM Snapshot

Click on Variables, select all the variables and click Delete

Click on New, give it a title like errMachineHasSnapshot and a value thatw ill displayed if a VM has an existing snapshot like The virtual machine already has a snaphost. Please delete this snapshot before trying this action again. Click Create

Click on the Schema tab and delete the existing items by clicking on the red x on the top right corner

Drag and drop from the left a Decision, throw exception and a workflow element in to the canvas

I named the first step VM Has Snapshot?. Under inputs i added vm. It would look like the screenshot below

Under the javascript we will be using this code

    var snapshots = System.getModule("com.vmware.library.vc.vm.snapshot").getAllSnapshotsOfVM(vm) ;  
    if (snapshots == null || snapshots.length == 0) {  
      // No snapshots found; proceed with creating a new one  
      return true;  
    } else {  
      // Found at least one snapshot; end the workflow  
      return false;  
    } 

The error handling Exception Handling we will be using the previously created errMachineHasSnapshot

The workflow element should look like this after selecting the Workflow Create a snapshot

The next step is to fix the inputs. We will be removing the Choose the VM tab by clicking on the x

We will be replacing the first item with vm and the display type

It should look like this:

In the end we should have the following:

Next go to Actions and click on New Action

Give it a Name and a Module name

Under Script type in

var allVms = VcPlugin.getAllVirtualMachines();
for (var I in allVms) {
    if (allVms[I].name === name) {
        return allVms[I];
    }
}
return null;

Under the Return type enter VC:VirtualMachine. Under the Inputs type in name and click Create on the bottom left

Once were done we can save it and move on to Cloud assembly to create the resource action

Go to Cloud Assembly -> Design -> Resource Actions click on new resource action

In the name field type something like CustomSnapshot, Display name VM Snapshot, give a description and toggle the activate switch to on

Under Resource type click on add pick Cloud.vSphere.Machine

Under Workflow click Add and pick VM Snapshot

In the Property Binding pick getVmByName action and under string enter ${properties.resourceName}

On the bottom click on edit request parameters

click on vm and change the Label to Reason and Display type to DropDown

Under Values Type in the options that you want to have shown Separated by , ex: Patching,New Release

Under Constraints make it as required by Picking Yes in the required field

The other fields and options can be modified as needed

Once everything is saved we can test out the Day 2 action by going to an existing deployment and trying to create a snapshot. We can see in the menu an additional option all the way to the bottom:

When choosing the option we can see the custom form that we filled in earlier