I recently ran through an issue where i wanted to automatically delete snapshots after 5 days instead of the default 7 days that comes out of box in automation central. I wanted to change delete old snapshot restriction from 7 days to 5 days.
It seems like the restriction comes from the Reclaim Settings. In order to change it we can go to Optimize -> Reclaim -> Settings
Change the Snapshots setting to older than
Going to Automation Central i was able to confirm that the minimum is now 5 days
I recently ran into a problem where i needed to route the subnet that is being used for the docker routing on the vROPS Cloud Proxy appliance. The network used is 172.17.0.0/16. For this we will try to route a portion of that subnet through my ethernet interface
To test the configuration out without adding a persistent route i used the following command:
ip route add 172.17.11.0/24 via 172.16.11.1 dev eth0 Verify that everything is running properly, if anything went wrong the route should get reverted on the next reboot. If the changes are successful we can add the route as persistent by editing the network configuration using vi:
vi /etc/systemd/network/10-eth0.networkNext add something similar to the Route option
[Route]
Destination=172.17.11.0/24
Gateway=172.16.11.1Please note that the above is not official guidance from VMware. If you need support please reach out to technical support.
In this guide we will go over Deploying the vROPS cloud proxy for cloud as well as on premise. The official VMware documentation can be found here
To get started log in to your vROPS instance. If its cloud it would be something similar to https://www.mgmt.cloud.vmware.com/vrops-cloud/ui/index.action For on premise it would be https://vrops_url/ui/login.action
Once you are logged on We can go to Data Sources -> Cloud Proxies and press on New. example:
Here we can see the download cloud proxy OVA option as well as a copy button. We can also see the OTK key. Keep a note of this as we will need it during the deployment. The first step is do get the proxy deployed. by either downloading the OVA or by copying the url. In my case i will copy the URL.
For reference you should be at this screen:
Click on the clipboard icon to copy the path to the ova
Next we will go to our vCenter to get it deployed. Go to one of the hosts or clusters, then go to actions -> Deploy OVF Template… example:
If you are deploying the cloud proxy for vROPS cloud the URL will look similar to this:
If its for on premise it would look similar to this:
Click on next and accept the certificate thumbprint
Select a name and location where the deployment should go and click on Next
Select a compute resource and click on Next
Review the details of the deployment and click on next
Accept the licensing agreement and click on next
Select a size and click next. If the environment is larger than 8k VMs you would want to deploy the Standard size. The sizing guide can be found here
Select a storage device and click on Next
Select a network and click on Next
Here is where we would add that OTK key from earlier. Paste in the OTK key. Give the VM a friendly name (this name will be what`s displayed in vROPS cloud proxies page. The network Proxy Settings are only applicable if you need to use a proxy to get out to the internet. The rest of the fields should be pretty self explanatory
Verify everything is correct and click on finish
Once the deployment is complete power on the machine. At this time we need to wait for a couple of minutes before it appears under the cloud proxies. It took about 20 minutes in my environment before i was able to see it in the vROPS cloud console
We can check the console while we wait for everything to be provisioned
Once the deployment is complete the console would look similar to this:
We can also see the proxy coming online in the vROPS cloud proxies menu
Once complete the proxy will show as online
The full vROPS cloud documentation can be found here
The full vROPS on premise documentation can be found here
You can request a trial from here
In this article i will go over one of the workaround instructions to address CVE-2021-44228 and CVE-2021-45046 in vRealize Operations 7.x. I have tested the workaround on vROPS 7.5 as its still shipped with VCF 3.x and i haven’t yet seen documentation on a workaround for this version. If you are looking for instructions for version 8.x consult kb article 87076. This has been tested on December 21 2021. Please check the official documentation or open a ticket for production usage.
Create a snapshot of the vROPS components to make sure we have something to revert to in case anything were to go wrong.
Log into the vROPS instance admin UI typically https://ip_address/admin and take the cluster offline. This applies to all nodes including but not limited to Analytic, Primary, Replica, Data, Remote Collectors and Witness nodes.
Give a reason and press ok
Verify the cluster is offline before continuing
Log in via ssh to a temporary path ex /tmp. Because vROPS 7.5 doesn`t come with the newer OpenSSL modules we need to find other means to get the files to the server without using a direct download method like wget.
In my case in put the code below in a file called vrops-log4j-fix.sh in my /tmp directory
#!/bin/bash
file=/tmp/impacted_jars.txt
echo "Searching for impacted .jar files. Please wait..."
find /usr/lib -type f -name "*.jar" -exec sh -c "zipinfo -1 '{}' | grep "org/apache/logging/log4j/core/lookup/JndiLookup.class" && echo {}" \; | grep "/usr/lib" > $file
line_qty=$(wc -l < $file)
if [ $line_qty -ne 0 ]; then
echo "Found $line_qty impacted .jar files"
else
echo "No impacted .jar files found"
exit 0
fi
echo "Starting to patch impacted .jar files"
while IFS= read -r line;
do
echo "patching -> $line"
own_user=$(stat -c '%U' "$line")
own_group=$(stat -c '%G' "$line")
zip -q -d "$line" org/apache/logging/log4j/core/lookup/JndiLookup.class
if [ $? -ne 0 ]; then echo "ERROR: Fail to Patch $line"; fi
chown $own_user:$own_group "$line"
done < $file
rm -f $fileMake the file executable by running chmod +x vrops-log4j-fix.sh
And then run the script by running ./vrops-log4j-fix.sh
The system will go through and find impacted .jar files and try to patch them. If successful we should end up with something like this
Next we will do the same with cp-log4j-fix.sh file
#!/bin/bash
#set -x
FAILURE="0"
WRAPPER_FILES="/usr/lib/vmware-vcops/user/conf/collector/wrapper.conf"
for f in $WRAPPER_FILES
do
last_idx=""
if [[ -f $f ]]; then
echo "********************************"
echo "Updating file: $f"
let last_idx=$(egrep "^wrapper.java.additional.[[:digit:]]+=" $f | cut -d= -f1 | awk -F '.' '{print $4}' | sort -n | tail -1)
if [[ -z $last_idx ]]; then
echo -e "ERROR: Failed to get JVM additional index"
let FAILURE="1"
continue
fi
((last_idx++))
echo -e "\n#Fixing Apache Log4j2 Remote Code Execution Vulnerability\nwrapper.java.additional.$last_idx=-Dlog4j2.formatMsgNoLookups=true" >> $f
if [[ $? != 0 ]]; then
echo -e "ERROR: Failed to update file: $f\n"
let FAILURE="1"
else
echo -e "Sucessfully updated file: $f\n"
fi
else
echo -e "ERROR: file is not found: $f\n"
let FAILURE="1"
fi
done
CASA_JVM="/usr/lib/vmware-casa/casa-webapp/bin/setenv.sh"
echo "********************************"
echo "Updating file: $CASA_JVM"
echo 'JAVA_OPTS="$JAVA_OPTS -Dlog4j2.formatMsgNoLookups=true"' >> $CASA_JVM
if [[ $? != 0 ]]; then
echo -e "ERROR: Failed to update file: $CASA_JVM\n"
let FAILURE="1"
else
echo -e "Sucessfully updated file: $CASA_JVM\n"
fi
if [[ "X$FAILURE" == "X1" ]]; then
exit 1
fi
exit 0and lastly the data-rc-witness-log4j-fix.sh file
#!/bin/bash
#set -x
FAILURE="0"
WRAPPER_FILES="/usr/lib/vmware-vcops/user/conf/analytics/wrapper.conf
/usr/lib/vmware-vcops/user/conf/collector/wrapper.conf
/usr/lib/vmware-vcops/user/conf/gemfire/wrapper.conf
/usr/lib/vmware-vcops/user/conf/tomcat-enterprise/wrapper.conf"
for f in $WRAPPER_FILES
do
last_idx=""
if [[ -f $f ]]; then
echo "********************************"
echo "Updating file: $f"
let last_idx=$(egrep "^wrapper.java.additional.[[:digit:]]+=" $f | cut -d= -f1 | awk -F '.' '{print $4}' | sort -n | tail -1)
if [[ -z $last_idx ]]; then
echo -e "ERROR: Failed to get JVM additional index"
let FAILURE="1"
continue
fi
((last_idx++))
echo -e "\n#Fixing Apache Log4j2 Remote Code Execution Vulnerability\nwrapper.java.additional.$last_idx=-Dlog4j2.formatMsgNoLookups=true" >> $f
if [[ $? != 0 ]]; then
echo -e "ERROR: Failed to update file: $f\n"
let FAILURE="1"
else
echo -e "Sucessfully updated file: $f\n"
fi
else
echo -e "ERROR: file is not found: $f\n"
let FAILURE="1"
fi
done
CATALINA_FILES="/usr/lib/vmware-casa/casa-webapp/bin/setenv.sh
/usr/lib/vmware-vcops/tomcat-web-app/bin/setenv.sh"
for f in $CATALINA_FILES
do
if [[ -f $f ]]; then
echo "********************************"
echo "Updating file: $f"
echo 'JAVA_OPTS="$JAVA_OPTS -Dlog4j2.formatMsgNoLookups=true"' >> $f
if [[ $? != 0 ]]; then
echo -e "ERROR: Failed to update file: $f\n"
let FAILURE="1"
else
echo -e "Sucessfully updated file: $f\n"
fi
else
echo -e "ERROR: file is not found: $f\n"
let FAILURE="1"
fi
done
if [[ "X$FAILURE" == "X1" ]]; then
exit 1
fi
exit 0It would look similar to this in the end
To verify that CVE-2021-44228 was applied run the following
ps axf | grep --color log4j2.formatMsgNoLookups | grep -v grepRunning ./vrops-log4j-fix.sh will also verify that there are no .jar files that need to be patched
Next bring the instance back online in the admin console
I recently ran through an routing issue where the Kubernetes IP range in vRO 8.6 was used somewhere else on the network. I didn’t want to redeploy the appliance so i went through the below to get it updated
First i identified what the ip range in use is by running
vracli network k8s-subnets
Lets check the status of the pods to make sure everything is running as it should. If any of the pods experience issues the change wont go through and it will cause additional issues
kubectl get pods -n preludeThe expectation is that under the Ready tab we have something similar
NAME READY STATUS RESTARTS AGE
docker-registry-695f9b8b45-d8gqr 1/1 Running 0 53m
postgres-0 1/1 Running 0 53m
proxy-service-5d8f64b54-lmxg5 1/1 Running 0 54m
vco-app-78499d8cbd-4mcnk 3/3 Running 0 54mTo set a new internal Kubernetes ip range i ran
vracli network k8s-subnets --cluster-cidr 192.168.0.0/22 --service-cidr 192.168.4.0/22Then in order to apply i changes i ran
vracli upgrade execI was prompted with a question
The services will be shut down while upgrade is in progress. Confirm you want to continue with the upgrade operation.[Y/n]By pressing Y the system went ahead and reconfigured\redeployed the pods on the proper network
And lastly i wanted to check the status of the pods to make sure they all came back
kubectl get pods -n preludeIn this guide we will go over the vIDM 3.3.5 HA configuration. The official documentation can be found here
Im going to assume that the load balancer configuration is already completed, the vIDM appliance has a the required certificate in the LCM inventory. Please read the official documentation for the full requirements.
We will be using the scale out feature in Lifecycle Manager. To do so we can navigate to Lifecycle Operations -> Environments -> globalenvironment -> View Details -> Click on Add Components
It is recommended that an inventory sync is performed prior to starting the process. It can be triggered by pressing on Trigger Inventory sync button. In my case i don`t need one as i did it earlier so ill just click Proceed
Network configuration should be populated. Verify the config and click next
Towards the bottom of the Components page there will be a components section. Click on the Plus sign next to it and select VMware Identity Manager Secondary Node. Perform this task task 2 times so we can have 3 vidm nodes.
Complete the required fields Like network configuration and Cluster Virtual IP
On the next page run the precheck in order to execute the data validation
Verify the Manual Validation as described in the Pop Up Window and click on Run Precheck
Once all the check are complete click on next, Verify the Summary and click on Submit
This will take us to the Request Details Page where we can follow the steps taken
Once the additional nodes are installed validate that everything is working as expected.
In this guide we will go over Creating a Credential for MongoDB in vROPS for the True Visibility Management pack. The original documentation was posted here however it hasn’t yet been updated to vROPS 8.6 and the new vROPS cloud.
Click the Add button on the top left. In the Manage Credential window that appears, select whether or not authentication is required for your MongoDB environment, then enter the information indicated below:
If authentication is required we would just select Authentication required under Credential Kind
During VMworld 2021 we announced vRealize Operations 8.6 which now provides allot of new features and capabilities. You can see an Whats New overview here as well as a feature walk through here.
I this blog we will be focusing specifically on the Administration menu and how it maps to the new 8.6 release. I covered the UI Navigation on my blog other here.
As you might be aware the Administration has now been redesigned to make it easier to navigate and find items. Lets jump in to it.
The first item on the list is Data Sources. To make it easier to get started the Data Sources can now be found on the home page as we get started. The accounts are now grouped together by account type ex: vCenter servers, AWS, Azure and so on. The Other Accounts sub-menu has been consolidated under the same menu. Because we combined the Cloud Accounts with Other Accounts we renamed the menu as Integrations.
The repository is now another tab under the Data Sources -> Integrations -> Repository
The Inventory menu has been moved under the Environment -> Inventory
The Policies have been moved to Configure -> Policies
The Access Control and Authentication Sources can now be found under the Administration menu
Custom Profiles has been renamed to Custom VM Profiles and it can be found under Configure -> Custom VM Profile
End Point Operations functionality is now included part of the telegraf agent via the cloud proxy and this menu item has been retired as its no longer in use.
The Group Types can now be found with the Custom Groups under Environment -> Custom Groups -> Group Types
Icons is considered an Administration item and it can be found under Administration -> Icons
Maintenance Schedules are considered a Non Administration Option that an non vROPS admin should be able to configure so its been relocated to the Configure sub menu
Configuration Files are locate under Configure -> Configuration Files
The object Relationships has been removed from the UI however it can be accessed by going directly to https://vrops_instance/ui/index.action#configure/object-relationships. The URL is the same for the SaaS as well as on premise
Optimizations Schedules has been moved under the Workload Optimization. It can be found by navigating to Optimize -> Workload Placement -> Optimization Schedules
Super Metrics are under Configure -> Super Metrics
Cost Settings are found under Configure -> Cost Settings
As described earlier in this post the Integrations have been combined with Other accounts and Other Accounts under Data Sources -> Integrations
Certificates, Cluster Management, Collector Groups and Collection Status have all been moved under Administration
The credentials have been consolidated under Data Sources -> Integrations -> Credentials
Global Settings, Licensing, Log Forwarding, Content Management can all be found under Administration
Outbound Settings are found now under Configure -> Alerts -> Outbound Settings
The rest of the items like Audit, Recent Tasks, Dynamic Thresholds, Logs, Redescribe, Cost Reference Database and Support Bundles can all be found under the administration menu
For more information visit us at https://www.vmware.com/products/vrealize-operations.html
During VMworld 2021 we announced vRealize Operations 8.6 which provides a lot of new features and capabilities. You can see an Whats New overview here as well as a feature walk through here.
With the new UI release some of the content might not be where were used to find it in the previous versions. In this post we will navigate through some of the changes and point to “Where is it now”. I also covered the UI overview on my blog here
Lets jump in to it. As shown in the screenshot below in the old UI we used to have navigation items up top as well as on the side. Many of the menu options across the top were duplicated in the left-hand navigation menu and were consolidated in this release.
As we can see in the new UI we still have the 4 pillars that were used too however the top menu bar was removed in order to be more consistent with the rest of the products in our SaaS platform. The sidebar is also collapsible allowing us to use the space for other purposes.
The Home button can be easily be found as the first item in the menu
The Dashboards are now conveniently located under the Visualize menu. This allows us now to go directly to dashboards, views as well as reports.
The Alerts tab has been relocated under Troubleshooting. We can now easily navigate between different troubleshooting options like Alerts, the troubleshooting Workbench and log analysis all under the same menu.
The environment tab can be easily be spotted on the side menu. Once we click on it we will notice a number of additional items like the new Object Browser which allows us to navigate through the environment more easily as well as the older Inventory browser. Some of the other options that are included are the new Business Applications, the previously Applications, Custom Groups, Custom Datacenters as well as Cloud Zones
Lastly the Administration menu. The administration menu has been broken into 2 different menus. The first one is under Configure where we can make configuration changes for our monitoring options things like defining Policies, Alerts configuration, Super Metrics, Application Discovery, Application Monitoring, Cost Settings, Custom VM Profiles, Configuration Files as well as Maintenance Schedules
The second administration menu goes specifically in to overall product administration like Access Control, Certificates, Licensing, Collector Groups, Content Management, Orphaned Content, Global Settings, Icons, Audit, Recent Tasks, Dynamic Thresholds and Cost Calculation
Now that we have the top menu out of the way lets take a look at where the side Menu items are in the new UI.
The quick Start menu item is now our default Home page
The operations overview is now conveniently placed as another tab on the home screen
Workload optimization is now conveniently placed with other optimization items under the optimize menu and it has been renamed to Workload Placement
From a rightsizing perspective we can see that rightsizing is right under Workload Placement in the Optimize sub menu and it has been renamed to Rightsize
The Recommended Actions menu item has been deprecated however i did create the menu item as a dashboard and it can be downloaded from the Sample Exchange on code.vmware.com or by clicking here
The Optimize Capacity Overview page can be found under Capacity in the Optimize Menu
The Reclaim menu can be found under the Optimize Menu
What-If Analysis has been renamed as Capacity and it can be found under the Plan Menu
We can find the Troubleshooting Workbench under the Troubleshoot menu
The Virtual Machine menu was pointing to the VM Contention Dashboard which can now be found under Visualize -> All -> Performance -> Consumer -> VM Contention. The Dashboard can also be found by performing a search in the dashboard menu
The vSAN menu was pointing to the vSAN Contention Dashboard which can now be found under Visualize -> All -> Performance -> Provider -> vSAN Contention. The Dashboard can also be found by performing a search in the dashboard menu
Using Logs can now be found under Troubleshoot -> Log Analysis
The compliance menu has been moved under Optimize -> Compliance
Discover Services has been renamed to Application Discovery and it can be found under Configure -> Application Discovery
Monitor Applications was renamed to Application Monitoring and it can be found under the Configure menu
Automation Central can still be found as a separate item on the side menu
For more information visit us at https://www.vmware.com/products/vrealize-operations.html
During one of the recent upgrades in my VCF environment i accidentally upgraded the ESXi 7.x build number to a later build that was not in the validated design which caused all kinds of issues and errors in my sddc manager. In order to address this i wanted to see i could actually downgrade the packages to a lower build number. It goes without saying that this procedure should not be done in a production environment and you should contact VMware customer support for an supported downgrade method. We will be working in the ESXi cli on this guide
The following command brings all of the version currently available in the depot
esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xmlI was specifically looking for build 18426014 so i leveraged grep to get the proper version
esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | grep 18426014The command above returned 2 versions for me. One with VMware tools and one without
ESXi-7.0U2c-18426014-standard VMware, Inc. PartnerSupported 2021-08-24T00:00:00 2021-08-24T00:00:00
ESXi-7.0U2c-18426014-no-tools VMware, Inc. PartnerSupported 2021-08-24T00:00:00 2021-08-04T11:40:25We are specifically interested in the first part of each line
ESXi-7.0U2c-18426014-standard and ESXi-7.0U2c-18426014-no-toolsFirst i tried to install it using
esxcli software profile update -p ESXi-7.0U2c-18426014-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xmlHowever that only works if were actually upgrading to a later build. In order to downgrade i found an option –allow-downgrades that allows us to install a lower build. In the end i ended up with
esxcli software profile update -p ESXi-7.0U2c-18426014-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml --allow-downgradesThis actually installed the lower versions of the packages associated with that build number and removed the later packages. The output in my case was
Update Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: VMware_bootbank_clusterstore_7.0.2-0.20.18426014, VMware_bootbank_cpu-microcode_7.0.2-0.20.18426014, VMware_bootbank_crx_7.0.2-0.20.18426014, VMware_bootbank_esx-base_7.0.2-0.20.18426014, VMware_bootbank_esx-dvfilter-generic-fastpath_7.0.2-0.20.18426014, VMware_bootbank_esx-update_7.0.2-0.20.18426014, VMware_bootbank_esx-xserver_7.0.2-0.20.18426014, VMware_bootbank_gc_7.0.2-0.20.18426014, VMware_bootbank_loadesx_7.0.2-0.20.18426014, VMware_bootbank_native-misc-drivers_7.0.2-0.20.18426014, VMware_bootbank_vdfs_7.0.2-0.20.18426014, VMware_bootbank_vsan_7.0.2-0.20.18426014, VMware_bootbank_vsanhealth_7.0.2-0.20.18426014
VIBs Removed: VMware_bootbank_clusterstore_7.0.2-0.25.18538813, VMware_bootbank_cpu-microcode_7.0.2-0.25.18538813, VMware_bootbank_crx_7.0.2-0.25.18538813, VMware_bootbank_esx-base_7.0.2-0.25.18538813, VMware_bootbank_esx-dvfilter-generic-fastpath_7.0.2-0.25.18538813, VMware_bootbank_esx-update_7.0.2-0.25.18538813, VMware_bootbank_esx-xserver_7.0.2-0.25.18538813, VMware_bootbank_gc_7.0.2-0.25.18538813, VMware_bootbank_loadesx_7.0.2-0.25.18538813, VMware_bootbank_native-misc-drivers_7.0.2-0.25.18538813, VMware_bootbank_vdfs_7.0.2-0.25.18538813, VMware_bootbank_vsan_7.0.2-0.25.18538813, VMware_bootbank_vsanhealth_7.0.2-0.25.18538813
VIBs Skipped: VMW_bootbank_atlantic_1.0.3.0-8vmw.702.0.0.17867351, VMW_bootbank_bnxtnet_216.0.50.0-34vmw.702.0.20.18426014, VMW_bootbank_bnxtroce_216.0.58.0-20vmw.702.0.20.18426014, VMW_bootbank_brcmfcoe_12.0.1500.1-2vmw.702.0.0.17867351, VMW_bootbank_brcmnvmefc_12.8.298.1-1vmw.702.0.0.17867351, VMW_bootbank_elxiscsi_12.0.1200.0-8vmw.702.0.0.17867351, VMW_bootbank_elxnet_12.0.1250.0-5vmw.702.0.0.17867351, VMW_bootbank_i40enu_1.8.1.137-1vmw.702.0.20.18426014, VMW_bootbank_iavmd_2.0.0.1152-1vmw.702.0.0.17867351, VMW_bootbank_icen_1.0.0.10-1vmw.702.0.0.17867351, VMW_bootbank_igbn_1.4.11.2-1vmw.702.0.0.17867351, VMW_bootbank_irdman_1.3.1.19-1vmw.702.0.0.17867351, VMW_bootbank_iser_1.1.0.1-1vmw.702.0.0.17867351, VMW_bootbank_ixgben_1.7.1.35-1vmw.702.0.0.17867351, VMW_bootbank_lpfc_12.8.298.3-2vmw.702.0.20.18426014, VMW_bootbank_lpnic_11.4.62.0-1vmw.702.0.0.17867351, VMW_bootbank_lsi-mr3_7.716.03.00-1vmw.702.0.0.17867351, VMW_bootbank_lsi-msgpt2_20.00.06.00-3vmw.702.0.0.17867351, VMW_bootbank_lsi-msgpt35_17.00.02.00-1vmw.702.0.0.17867351, VMW_bootbank_lsi-msgpt3_17.00.10.00-2vmw.702.0.0.17867351, VMW_bootbank_mtip32xx-native_3.9.8-1vmw.702.0.0.17867351, VMW_bootbank_ne1000_0.8.4-11vmw.702.0.0.17867351, VMW_bootbank_nenic_1.0.33.0-1vmw.702.0.0.17867351, VMW_bootbank_nfnic_4.0.0.63-1vmw.702.0.0.17867351, VMW_bootbank_nhpsa_70.0051.0.100-2vmw.702.0.0.17867351, VMW_bootbank_nmlx4-core_3.19.16.8-2vmw.702.0.0.17867351, VMW_bootbank_nmlx4-en_3.19.16.8-2vmw.702.0.0.17867351, VMW_bootbank_nmlx4-rdma_3.19.16.8-2vmw.702.0.0.17867351, VMW_bootbank_nmlx5-core_4.19.16.10-1vmw.702.0.0.17867351, VMW_bootbank_nmlx5-rdma_4.19.16.10-1vmw.702.0.0.17867351, VMW_bootbank_ntg3_4.1.5.0-0vmw.702.0.0.17867351, VMW_bootbank_nvme-pcie_1.2.3.11-1vmw.702.0.0.17867351, VMW_bootbank_nvmerdma_1.0.2.1-1vmw.702.0.0.17867351, VMW_bootbank_nvmxnet3-ens_2.0.0.22-1vmw.702.0.0.17867351, VMW_bootbank_nvmxnet3_2.0.0.30-1vmw.702.0.0.17867351, VMW_bootbank_pvscsi_0.1-2vmw.702.0.0.17867351, VMW_bootbank_qcnic_1.0.15.0-11vmw.702.0.0.17867351, VMW_bootbank_qedentv_3.40.5.53-20vmw.702.0.20.18426014, VMW_bootbank_qedrntv_3.40.5.53-17vmw.702.0.20.18426014, VMW_bootbank_qfle3_1.0.67.0-14vmw.702.0.0.17867351, VMW_bootbank_qfle3f_1.0.51.0-19vmw.702.0.0.17867351, VMW_bootbank_qfle3i_1.0.15.0-12vmw.702.0.0.17867351, VMW_bootbank_qflge_1.1.0.11-1vmw.702.0.0.17867351, VMW_bootbank_rste_2.0.2.0088-7vmw.702.0.0.17867351, VMW_bootbank_sfvmk_2.4.0.2010-4vmw.702.0.0.17867351, VMW_bootbank_smartpqi_70.4000.0.100-6vmw.702.0.0.17867351, VMW_bootbank_vmkata_0.1-1vmw.702.0.0.17867351, VMW_bootbank_vmkfcoe_1.0.0.2-1vmw.702.0.0.17867351, VMW_bootbank_vmkusb_0.1-4vmw.702.0.20.18426014, VMW_bootbank_vmw-ahci_2.0.9-1vmw.702.0.0.17867351, VMware_bootbank_elx-esx-libelxima.so_12.0.1200.0-4vmw.702.0.0.17867351, VMware_bootbank_esx-ui_1.34.8-17417756, VMware_bootbank_lsuv2-hpv2-hpsa-plugin_1.0.0-3vmw.702.0.0.17867351, VMware_bootbank_lsuv2-intelv2-nvme-vmd-plugin_2.0.0-2vmw.702.0.0.17867351, VMware_bootbank_lsuv2-lsiv2-drivers-plugin_1.0.0-5vmw.702.0.0.17867351, VMware_bootbank_lsuv2-nvme-pcie-plugin_1.0.0-1vmw.702.0.0.17867351, VMware_bootbank_lsuv2-oem-dell-plugin_1.0.0-1vmw.702.0.0.17867351, VMware_bootbank_lsuv2-oem-hp-plugin_1.0.0-1vmw.702.0.0.17867351, VMware_bootbank_lsuv2-oem-lenovo-plugin_1.0.0-1vmw.702.0.0.17867351, VMware_bootbank_lsuv2-smartpqiv2-plugin_1.0.0-6vmw.702.0.0.17867351, VMware_bootbank_qlnativefc_4.1.14.0-5vmw.702.0.0.17867351, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.42-1vmw.702.0.0.17867351, VMware_locker_tools-light_11.2.6.17901274-18295176And with that all i had to do was reboot the server and it came back at the version i needed.
