vSAN TRIM/UNMAP Functionality

With my new role, I am once again drinking from the firehose. The result of which has primarily seen me go heads-down on all things vSAN related, including a course on the new features included in the vSphere 6.7 U1 release.

The latest vSAN release included in vSphere 6.7 U1, includes new functionality to support TRIM/UNMAP guest OS commands to reclaim unused disk space within a vSAN datastore. Previous versions of vSAN disregarded ATA TRIM and SCSI UNMAP commands issued by a virtual machine guest OS. The added support in vSphere 6.7 U1 helps with reclaiming unused disk space that are marked as such by a guest OS.

There are a few requirements for leveraging the new functionality.

  1. VMs must be thin-provisioned.
  2. Windows OS VMs must be at least HW version 9.
  3. Linux OS VMs must be at least HW version 13.
  4. The vSAN datastore must be On-Disk format version 7.
  5. VMs must be powered cycled after TRIM/UNMAP is enabled on the cluster or ESXi host. A reboot is not sufficient.

When enabled, the TRIM/UNMAP functionality will begin to reclaim space object on each vSAN disk group. If the vSAN cluster is leveraging deduplication, the work behind-the-scenes will potentially impact performance as there are more operations occurring transparent to the consumer. The performance latency values for TRIM/UNMAP commands can be viewed on each ESXi host in the /usr/lib/vmware/vsan/bin/vsanTraceReader log file.

To enable the TRIM/UNMAP functionality on a vSAN cluster or ESXi host, the following commands should be executed.

ESXi Host

To enable the functionality:
$ esxcfg-advcfg -s 1 /VSAN/GuestUnmap

To verify the functionality:
$ esxcfg-advcfg -g /VSAN/GuestUnmap

To disable the functionality:
$ esxcfg-advcfg -s 0 /VSAN/GuestUnmap

vSAN Cluster

Using the RVC, to enable the functionality:
> vsan.unmap_support -e ~CLUSTER_NAME

To verify the functionality:
> vsan.unmap_support ~CLUSTER_NAME

To disable the functionality:
> vsan.unmap_support -d ~CLUSTER_NAME

Once enabled on all the hosts within the vSAN cluster, and the VMs have all undergone a power cycle operation, there is one more thing to consider. There are two methods for the TRIM/UNMAP functionality to actually reclaim unused space as reported by the guest OS — Passive and Active.

Passive

  • For a Microsoft Windows Server 2012+ operating systems, it is enabled by default and reclaim operations are performed automatically.
  • For Linux operating systems, it is not enabled by default unless the filesystem has been mounted with the discard parameter.

Active

  • For a Microsoft Windows Server operating system, the Optimize Drive Utility must be leveraged.
  • For a Linux operating system, the fstrim command must be leveraged.

Enjoy!

Claim vSAN Capacity Disks for VCF 3.0

The latest release of VMware Cloud Foundation (VCF 3.0) removed the host imaging functionality. As past of the laundry list of pre-requisites for preparing an environment for VCF, one necessary step in an All-Flash vSAN environment is to mark the appropriate capacity disks.

During a POC deployment last week of VCF 3.0, this pre-requisite became evident and required a quick solution for marking the disks without having to glean all of the information manually. The following method is a quick way to identify which disks should be used for capacity and correctly allocating them as such for vSAN to claim during the VCF deployment workflows for either the Management or Workload Domain.

On the first ESXi node, we need to execute the following command to determine the capacity disk size. This command can be omitted on all remaining ESXi nodes as you prep them for VCF.

$ esxcli storage core device list
naa.58ce38ee20455a75
   Display Name: Local TOSHIBA Disk (naa.58ce38ee20455a75)
   Has Settable Display Name: true
   Size: 3662830
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/naa.58ce38ee20455a75
   Vendor: TOSHIBA
   Model: PX05SRB384Y
   Revision: AS0C
   SCSI Level: 6
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: true
   Is Removable: false
   Is SSD: true
   Is VVOL PE: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters:
   VAAI Status: unknown
   Other UIDs: vml.020000000058ce38ee20455a75505830355352
   Is Shared Clusterwide: false
   Is Local SAS Device: true
   Is SAS: true
   Is USB: false
   Is Boot USB Device: false
   Is Boot Device: false
   Device Max Queue Depth: 254
   No of outstanding IOs with competing worlds: 32
   Drive Type: physical
   RAID Level: NA
   Number of Physical Drives: 1
   Protection Enabled: false
   PI Activated: false
   PI Type: 0
   PI Protection Mask: NO PROTECTION
   Supported Guard Types: NO GUARD SUPPORT
   DIX Enabled: false
   DIX Guard Type: NO GUARD SUPPORT
   Emulated DIX/DIF Enabled: false

The above output is an example of a vSAN SSD capacity disk. The only bit of information we need to automate the rest of the work is the size of the disk. Once you have the known size, substitute the value into the first grep command and execute the following CLI script on each node.

$ esxcli storage core device list | grep -B 3 -e "Size: 3662830" | grep ^naa > /tmp/capacitydisks; for i in `cat /tmp/capacitydisks`; do esxcli vsan storage tag add -d $i -t capacityFlash;  vdq -q -d $i; done

As each disk is marked as eligible for vSAN, the script will output that information for the user.

That’s it!

If you’d like to read more about the VCF 3.0 release, please check out the DataReload blog post.

Deploy vCenter Server using Ansible

Fresh off an amazing week at VMworld, I got right back into the lab to finish up a few things to complete the SDDC deployment roles I have been working on the past few months. I wanted to get this particular role published prior to VMworld, but alas the time flew by too quickly!

One of the most critical components of the SDDC is the vCenter Server, and deploying it through the OVA provided in the ISO by VMware can be challenging if you want to automate it. The ISO provides the ovftool, which can be leveraged to perform a command-line installation of the vCenter Server appliance. A team of consultants inside VMware published an Ansible role a bit ago to help them automate their SDDC installations, which was the basis for the role I have here.

The original role can be found on GitHub here.

The use-case for the above role did not match what I was trying to do, or what I think most customers would be deploying within their own production environments. So I forked the code, and re-wrote it to deploy either a VCSA with embedded PSC, standalone VCSA, and/or an external PSC appliance.

I removed many of the templates for in-band and out-of-band deployments the Chaperone project used for their configurations, and aligned the new role to match up with a typical vCenter Server deployment.

How the role works

The Ansible role vcsa-deploy is essentially a wrapper for ovftool. The role takes a specific set of variables based on the deployment configuration you’ve chosen — VCSA with embedded PSC, standalone VCSA, and/or an external PSC appliance. From there it uses the corresponding template to generate the proper set of command-line parameters ovftool leverages for the deployment, writes the newly created task to a file, and executes it.

The role also expects the vCenter Server ISO to be accessible, with the location being defined by the repo_dir and vcsa_iso variables respectively. I also modified the role to leverage the ovftool binary that is included inside the vCenter Server ISO — this makes it more portable to other environments that may not be leveraging the virtualelephant/ubuntu-ansible Docker container.

But you are right?

The role can be downloaded from GitHub as part of the vsphere-sddc repository under the Virtual Elephant space. There is also a playbook that can be leveraged to perform the deployment of a vCenter Server Appliance to your environment within the repository as well.

vcenter-sddc-deploy.yml

  1 # Licensed under the Apache License, Version 2.0 (the "License");
  2 # you may not use this file except in compliance with the License.
  3 #
  4 # You may obtain a copy of the License at
  5 #   http://www.apache.org/licenses/LICENSE-2.0
  6 #
  7 # Unless required by applicable law or agreed to in writing, software
  8 # distributed under the License is distributed on an "AS IS" BASIS,
  9 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 10 # See the License for the specific language governing permissions and
 11 # limitations under the License.
 12 #
 13 # Author: Chris Mutchler (chris@virtualelephant.com)
 14 #
 15 # Description:
 16 #   Playbook will deploy and configure a vCenter Server
 17 #
 18 # Note:
 19 #   Use of the virtualelephant/ubuntu-ansible Docker container will include
 20 #   all of necessary Ansible modules and libraries necessary to execute the
 21 #   playbook.
 22 ---
 23 - hosts: all
 24   connection: local
 25   gather_facts: false
 26 
 27   vars:
 28     repo_dir: '/opt/repo'
 29     vcsa_iso: 'VMware-VCSA-all-6.7.0-9451876.iso'
 30     vcsa_task_directory: '/opt/ansible/roles/vcsa-deploy/tasks'
 31 
 32     ovftool: '/mnt/vcsa/ovftool/lin64/ovftool'
 33     vcsa_ova: 'vcsa/VMware-vCenter-Server-Appliance-6.7.0.14000-9451876_OVF10.ova'
 34     mount_dir_path: '/mnt'
 35 
 36     appliance_type: 'embedded'
 37 
 38     net_addr_family: 'ipv4'
 39     network_ip_scheme: 'static'
 40     disk_mode: 'thin'
 41     ssh_enable: true
 42 
 43     vcenter_appliance_name: 'vcenter'
 44     vcenter_appliance_size: 'medium'
 45 
 46     target_esxi_username: '{{ vault_esxi_username }}'
 47     target_esxi_password: '{{ vault_esxi_password }}'
 48     target_esx_datastore: 'local-t410-3TB'
 49     target_esx_portgroup: 'Management'
 50 
 51     time_sync_tools: false
 52 
 53     vcenter_password: '{{ vault_vcenter_password }}'
 54     vcenter_fqdn: 'vcenter.local.domain'
 55     vcenter_ip_address: '192.168.0.25'
 56     vcenter_netmask: '255.255.0.0'
 57     vcenter_gateway: '192.168.0.1'
 58     vcenter_net_prefix: '16'
 59 
 60     dns_servers: '192.168.0.1,192.168.0.2'
 61     ntp_servers: '132.163.96.1,132.163.97.1'
 62 
 63     sso_password: '{{ vault_vcenter_password }}'
 64     sso_site_name: 'Default-Site'
 65     sso_domain_name: 'vsphere.local'
 66 
 67   roles:
 68     - vcsa-deploy

The inclusion of the role completes the foundational parts of deploying a complete VMware vSphere SDDC with ESXi, vCenter Server and NSX-v. I hope to add functionality to the role for deploying a highly-available vCenter Server cluster in the future.

Until then, I hope this helps you find success with your Ansible automation efforts. Enjoy!

Next Chapter in my VMware Journey

I joined VMware in June of 2015 as a member of the internal private cloud architecture team. At the time, it was my first full-time role as an architect and I was extremely excited to both have the opportunity to improve my skills as an architect and join the VMware family. During my tenure on the team, I have been able to grow my skills immensely, work with amazing team members who mentored, coached, and taught me how to be a better ‘practicing’ architect.

In May of 2017, I earned my VCDX certification – a major accomplishment both personally and professionally. Since earning that certification, I have become a VCDX panelist and begun handling the VCDX Online Workshops to help other candidates be successful within the program. Through the program I’ve met even more great architects within VMware, and unbeknownst to me, I met my future boss Joe Silvagi.

I am excited to announce that starting September 1, 2018, I am moving internally to the Customer Success Architecture team, focused on VMware HCI technologies (vSAN, VCF, and VVD). I am looking forward to being able to leverage the operational experience I have gained over the past 20+ years to directly help customers be successful utilizing the VMware SDDC and HCI technologies within their environments.

I want to take a brief moment and thank several people — Brian Smith, Tom Ralph, Simon Long, John Tompkins and Lyubo Lyubenov — whom I’ve had the pleasure of working closely with the past 3 years at VMware. Without them I would not have been ready to take this next step in my career.

While I will continue to maintain the Virtual Elephant site, you’ll likely notice a shift in the content to be more focused on these technologies. I look forward to sharing my experiences with all of you.

Ansible Roles for VMware SDDC Deployments

I’ve been excited for weeks now as I prepped for VMworld 2018 in Las Vegas and anticipating being able to talk more about leveraging Ansible to deploy and operate a VMware SDDC environment. As you can tell from my recent posts, I am heavily involved in automation using Ansible, both within my side projects and as a practicing architect at VMware. As an internal team, we are working hard to improve and enhance several of the upstream Ansible modules, and I hope to be able to share those externally in the future as they are contributed back.

In my spare time, I’ve been working the lab to provide a set of Ansible roles that anyone can leverage to configure ESXi hosts, deploy and configure a vCenter Server Appliance (VCSA), and deploy and configure NSX-v within their environments. I am happy to announce the initial release of these roles in the Virtual Elephant GitHub space.

The following roles have been published within the space and are operational:

  • esxi-adv-settings – Configure advanced ESXi settings on an ESXi node
  • esxi-host-config – Configure DNS, hostname and NTP settings on an ESXi node
  • esxi-services – Configure ESXi services on an ESXi node
  • esxi-vmk-interfaces – Create/delete VMkernel interfaces on an ESXi node
  • nsxv-cluster-prep – Prepare vCenter cluster for NSX-v
  • nsxv-controllers – Create/delete NSX-v controllers
  • nsxv-license – Assign NSX-v license
  • nsxv-logical-switch – Create/delete NSX-v logical switch
  • nsxv-manager-config – Configure NSX-v Manager
  • nsxv-manager-deploy – Deploy NSX-v Manager OVA to vCenter Server
  • nsxv-manager-roles – Configure NSX-v Manager user roles
  • nsxv-transport-zone – Create/delete NSX-v transport zone
  • vcenter-add-hosts – Add or remove ESXi nodes to vCenter Server
  • vcenter-cluster – Create/delete/modify vCenter cluster
  • vcenter-datacenter – Create/delete vCenter datacenter object
  • vcenter-maintenance-mode – Manage the maintenance mode state of an ESXi node
  • vcenter-networking – Create/delete DVS
  • vcenter-portgroups – Create/delete port groups

Wherever possible, each role has been written to allow the creation, deletion or modification of said objects within the SDDC environment.

In addition to the roles themselves, there are published playbooks that execute the roles in a specific order, based on dependencies, to perform the actual deployment of the SDDC environment.

  • esxi_sddc_configure.yml – Configure ESXi nodes
  • nsxv_sddc_deploy – Deploy and configure NSX-v Manager and controllers

If you are looking to start leveraging Ansible to deploy and manage your VMware SDDC environments, these roles are a great starting point. Reach out to me over Twitter, or come find me this afternoon in the VMware {code} Theatre at 1:00PM.

Enjoy!