Backup vCenter and NSX to AWS S3

As I go deeper into the Ansible rabbit-hole, I am beginning to look for ways to manage upgrade operations through Ansible playbooks. As part of that journey, I wanted to begin backing up my VCSA and NSX-v VM appliances using their built-in methods prior to executing playbooks to perform the upgrades. Both appliances allow FTP, SFTP or SCP connections through their management interfaces for backing up the configuration data — all that is needed is an endpoint.

I wondered if it would be possible to backup these items to S3 using my AWS account. A quick search through my AWS portal showed me that I could use the AWS Storage Gateway, setup a S3 bucket for backups and mount the partition on a Linux VM for the vSphere appliances to use as an endpoint. With minimal effort, I was able to configure both appliances to backup to the local Linux VM and see that data replicated into S3 in a matter of minutes.

Fortunately, AWS has outstanding documentation for deploying the Storage Gateway within a vSphere environment (here). Once the Storage Gateway is deployed, the S3 bucket is created and the file share is created you can mount it on a Linux VM.

linux-vm$ mount -t nfs -o nolock 10.180.138.20:/usa1-2-lab-backups /opt
linux-vm$ mkdir -p /opt/vcsa
linux-vm$ mkdir -p /opt/nsxv

I created a separate backup location on the NFS mount point to the Storage Gateway — one for the VCSA and one for the NSX-v. At this point, it just a matter of configuring the two appliances to use the endpoint.

For the VCSA, log into port 5480 over HTTPS and select the Backup option on the left-hand menu.

The above screenshot shows how to configure the backup schedule and then you can perform a backup job using those details manually.

Similiary, the NSX-v Manager has a Backup and Restore are inside its management interface where you can configure the endpoint. NSX-v only supports FTP or SFTP today, but using SFTP I was able to use the endpoint.

Once the backup location is configured, you can execute a backup job through the admin interface.

From there it was just a matter of verifying the data was being sent and replicated to the S3 bucket I created in AWS.

That is all there is to it! Backups of the appliance data to an AWS S3 bucket using the Storage Gateway is nice and easy. Now I can begin working on the Ansible playbooks to upgrade the VCSA through the API, knowing the data is backed up to the cloud!

Enjoy!

Deploying an SDDC with Ansible

The small effort I started at the end of last year using Ansible to deploy NSX components has snowballed a bit and found its way into a project at work. As we are working to deploy a new HCI architecture internally, one of the efforts we are embarking on is a fully automated, infrastructure-as-code architecture design. There are several components that are working in conjunction with one another to be able to accomplish that task, but the part I am going to talk about today is automation through Ansible.

As many of you have seen, I’ve recently been automating NSX component delivery and configuration using the open source VMware NSX Ansible modules. I’ve been fortunate enough to put my meager coding skills to work and enhance those models this year — adding new capabilities exposed through the API for NSX Edge configuration. In addition to the NSX Ansible modules, there are a multitude of upstream Ansible modules for VMware components. The first step was evaluating what the current upstream modules were capable of performing and putting together a small demo for my colleagues to observe both the power of Ansible and the ease of use.

My initial impressions of Ansible is that it is probably the most user-friendly of the configuration management/automation tools currently available. And for the VMware SDDC components, it appears to be rather robust. I have identified a few holes, but nothing insurmountable — the great thing is if something is exposed via an API, creating an Ansible module to leverage said API is rather simplistic.

The Ansible playbooks are a first step, I really want to convert most of them into Ansible roles. I’ve started committing the code in my Github space. You can download the playbooks and start using them if you’d like.

https://github.com/virtualelephant/vsphere-sddc

I currently have playbooks for creating a datacenter, cluster, adding hosts, configuring several advanced settings on each ESXi host, creating a DVS with port groups and performing a few other configuration tasks. The bit that I want to work out next is deployment of the vCenter server through Ansible. It’s currently a work in progress, but it has been a fun effort thus far.

Enjoy!

NSX Ansible Updates

It has been a hectic few months for me. I relocated my family to Colorado last month, and as a result all of my side-projects were put on the back-burner. During the time I was away, I was selected for the fourth year in a row as a vExpert! I am grateful to be a part of this awesome community! I strive to make the work I do and submit here worthwhile and informative for others.

Now that I am back into the swing of things, I was able to jump back into improving the NSX-v Ansible module. Recently a member of the community opened an issue regarding the implementation method I had used for Edge NAT rules. Sure enough, they were correct in that the method I was using was really an append and not a creation.

Note: I wrote and tested the code against a pre-release version of NSX-v 6.4.1. There are no documented differences between the two API calls used in the nsx_edge_nat.py Ansible module in NSX 6.4.x or 6.3.x

When I looked at the NSX API, I realized there were two methods for adding NAT rules to an NSX Edge:

PUT /api/4.0/edges/{edgeId}/nat/config

URI Parameters:
edgeId (required) Specify the ID of the edge in edgeId.
Description:
Configure NAT rules for an Edge.

If you use this method to add new NAT rules, you must include all existing rules in the request body. Any rules that are omitted will be deleted.

And also:

POST /api/4.0/edges/{edgeId}/nat/config/rules

URI Parameters:
edgeId (required)
Specify the ID of the edge in edgeId.

Query Parameters:
aboveRuleId (optional)
Specified rule ID. If no NAT rules exist, you can specify rule ID 0.

Description:
Add a NAT rule above a specific rule in the NAT rules table (using aboveRuleId query parameter) or append NAT rules to the bottom.

The original code was using the second method, which meant each time an Ansible playbook was run, it would return an OK status because it was adding the rules — even if they already existed.

I decided to dive into the issue and spent a few more hours than I anticipated improving the code. It is now possible to use either method — one to create a full set of rules (deleting any existing rules) or appending new rules to the existing ruleset.

In order to create multiple rules, I modified how the Ansible playbook is interpreted. The example is in the readme.md file, but I want to highlight it here:

  1 ---
  2 - hosts: localhost
  3   connection: local
  4   gather_facts: False
  5   vars_files:
  6     - nsxanswer.yml
  7     - envanswer.yml
  8 
  9   tasks:
 10   - name: Create NAT rules
 11     nsx_edge_nat:
 12       nsxmanager_spec: '{{ nsxmanager_spec }}'
 13       mode: 'create'
 14       name: '{{ edge_name }}'
 15       rules:
 16         dnat0: { description: 'Ansible created HTTP NAT rule',
 17             loggingEnabled: 'true',
 18             rule_type: 'dnat',
 19             nat_enabled: 'true',
 20             dnatMatchSourceAddress: 'any',
 21             dnatMatchSourcePort: 'any',
 22             vnic: '0',
 23             protocol: 'tcp',
 24             originalAddress: '10.180.138.131',
 25             originalPort: '80',
 26             translatedAddress: '192.168.0.2',
 27             translatedPort: '80'
 28           }
 29         dnat1: { description: 'Ansible created HTTPS NAT rule',
 30             loggingEnabled: 'true',
 31             rule_type: 'dnat',
 32             vnic: '0',
 33             nat_enabled: 'true',
 34             dnatMatchSourceAddress: 'any',
 35             dnatMatchSourcePort: 'any',
 36             protocol: 'tcp',
 37             originalAddress: '10.180.138.131',
 38             originalPort: '443',
 39             translatedAddress: '192.168.0.2',
 40             translatedPort: '443'
 41           }

Please note the identifiers, dnat0 and dnat1, are merely that — identifiers for your playbook. They do not influence the API call made to the NSX Manager.

A new function was required in the Ansible module to allow for multiple rules to be appended to one another to make a single API call that would add each rule. The data structure used to create this dictionary of lists was rather convoluted since Python struggles to convert these sort of thing to XML properly. With some help from a few people in the NSBU, I was able to get it working.

def create_init_nat_rules:

 55 def create_init_nat_rules(client_session, module):
 56     """
 57     Create single dictionary with all of the NAT rules, both SNAT and DNAT, to be used
 58     in a single API call. Should be used when wiping out ALL existing rules or when
 59     a new NSX Edge is created.
 60     :return: return dictionary with the full NAT rules list
 61     """
 62     nat_rules = module.params['rules']
 63     params_check_nat_rules(module)
 64 
 65     nat_rules_info = {}
 66     nat_rules_info['natRule'] = []
 67 
 68     for rule_key, nat_rule in nat_rules.items():
 69         rules_index = rule_key[-1:]
 70         rule_type = nat_rule['rule_type']
 71         if rule_type == 'snat':
 72             nat_rules_info['natRule'].append(
 73                                     {'action': rule_type, 'vnic': nat_rule['vnic'], 'originalAddress': nat_rule['originalAddress'],
 74                                      'translatedAddress': nat_rule['translatedAddress'], 'loggingEnabled': nat_rule['loggingEnabled'],
 75                                      'enabled': nat_rule['nat_enabled'], 'protocol': nat_rule['protocol'], 'originalPort': nat_rule['originalPort'],
 76                                      'translatedPort': nat_rule['translatedPort'], 'snatMatchDestinationAddress': nat_rule['snatMatchDestinationAddress'],
 77                                      'snatMatchDestinationPort': nat_rule['snatMatchDestinationPort'], 'description': nat_rule['description']
 78                                     }
 79                                   )
 80         elif rule_type == 'dnat':
 81             nat_rules_info['natRule'].append(
 82                                     {'action': rule_type, 'vnic': nat_rule['vnic'], 'originalAddress': nat_rule['originalAddress'],
 83                                      'translatedAddress': nat_rule['translatedAddress'], 'loggingEnabled': nat_rule['loggingEnabled'],
 84                                      'enabled': nat_rule['nat_enabled'], 'protocol': nat_rule['protocol'], 'originalPort': nat_rule['originalPort'],
 85                                      'translatedPort': nat_rule['translatedPort'], 'dnatMatchSourceAddress': nat_rule['dnatMatchSourceAddress'],
 86                                      'dnatMatchSourcePort': nat_rule['dnatMatchSourcePort'], 'description': nat_rule['description']
 87                                     }
 88                                   )
 89 
 90         if nat_rule['protocol'] == 'icmp':
 91             nat_rules_info['natRule']['icmpType'] = nat_rule['icmpType']
 92 
 93     return nat_rules_info

I also took the opportunity to clean up some of the excessively long lines of code to make it more clearly readable. The result is a new working playbook for initial Edge NAT rule creation and the ability to add new rules later. There are a few items that remain where I would like to see some improvements — mainly I would like to add logic in to the code that, when performing an append, it will check to see if the rule already exists and skip it.

In the meantime, the code has been checked into GitHub and the Docker image I use for running Ansible has been updated.

Enjoy!

Ansible NSX module for creating NAT rules

After working on the code last weekend and testing the functionality this past week, I am proud to announce the ability to create NAT rules on an NSX Edge is possible through Ansible! The ability to create SNAT and DNAT rules on the NSX Edge was a necessity for the Infrastructure-as-Code project, as each environment deployed uses it’s own micro-segmented network. I am doing that so that each environment can be stood up multiple times within the same vSphere environment and be solely dependent upon itself.

The current Ansible module allows for the creation of both SNAT and DNAT rules. I used the NSX API Guide to be able to determine which variables are acceptable to be passed to either types of NAT rules and included each one in the function. As such, there are no features missing from the module today.

The module can be downloaded from the GitHub virtualelephant/nsxansible repo.

To use the module, I have created an example Ansible playbook (also available on GitHub):

test_edge_nat.yml

  1 ---
  2 - hosts: localhost
  3   connection: local
  4   gather_facts: False
  5   vars_files:
  6     - nsxanswer.yml
  7 
  8   tasks:
  9   - name: Create SSH DNAT rule
 10     nsx_edge_nat:
 11       nsxmanager_spec: '{{ nsxmanager_spec }}'
 12       mode: 'create'
 13       name: '{{ edge_name }}'
 14       rule_type: 'dnat'
 15       vnic: '0'
 16       protocol: 'tcp'
 17       originalAddress: '10.0.0.1'
 18       originalPort: '22'
 19       translatedAddress: '192.168.0.2'
 20       translatedPort: '22'
 21 
 22   - name: Create default outbound SNAT rule
 23     nsx_edge_nat:
 24       nsxmanager_spec: '{{ nsxmanager_spec }}'
 25       mode: 'create'
 26       name: '{{ edge_name }}'
 27       rule_type: 'snat'
 28       vnic: '0'
 29       protocol: 'any'
 30       originalAddress: '192.168.0.0/20'
 31       originalPort: 'any'
 32       translatedAddress: '10.0.0.1'
 33       translatedPort: 'any'

Update Jan 30th:

The module was updated a few days ago to include the ability to delete a NAT rule from an Edge. The functionality allows the consumer to write a playbook with the following information to delete an individual rule.

  1 ---
  2 - hosts: localhost
  3   connection: local
  4   gather_facts: False
  5   vars_files:
  6     - nsxanswer.yml
  7     - envanswer.yml
  8 
  9   tasks:
 10   - name: Delete HTTP NAT rule
 11     nsx_edge_nat:
 12       nsxmanager_spec: '{{ nsxmanager_spec }}'
 13       mode: 'delete'
 14       name: '{{ edge_name }}'
 15       ruleId: '196622'

Let me know if you are using the NSX Ansible modules and what other functionality you would like to see added.

Enjoy!

Enhanced NSX Modules for Ansible

The published NSX modules from VMware lack certain functionality that I’ve needed as I worked on the Infrastructure-as-Code project over the holiday break. A few of the things I need to be able to do include:

  • Enable Edge firewall and add/delete rules
  • Enable DHCP and add IP pools
  • Search for DVS VXLAN port group
  • Associate vNIC with default gateway
  • Create Edge SNAT/DNAT rules

As I investigated methods for accomplishing these tasks, I found another VMware repository of Python scripts that had some of the functionality. The library is designed as a command-line tool, but I was able to take several of the code blocks and modify them for use within an Ansible playbook. In order to track the changes that I’ve made to the Ansible modules, I’ve forked the vmware/nsxansible repo into virtualelephant/nsxansible on Github. After a bit of work over the holiday, I’ve managed to add functionality for all but the SNAT/DNAT rule creation.

In addition to writing the Python modules, I have modified the Docker ubuntu-ansible container I spoke about previously to include my forked branch of the vmware/nsxansible modules.

Creating DHCP Pools

The module nsx_edge_dhcp.py allows an Ansible playbook to create a new IP pool and enable the DHCP service on a previously deployed NSX Edge. The playbook can currently support all of the basic IP pool options, as seen in the following image:

The playbook can contain the following code:

  1 ---
  2 - hosts: localhost
  3   connection: local
  4   gather_facts: False
  5   vars_files:
  6     - nsxanswer.yml
  7     - envanswer.yml
  8 
  9   tasks:
 10   - name: Create DHCP pool on NSX Edge
 11     nsx_edge_dhcp:
 12       nsxmanager_spec: "{{ nsxmanager_spec }}"
 13       name: '{{ edge_name }}'
 14       mode: 'create_pool'
 15       ip_range: '{{ ip_range }}'
 16       subnet: '{{ netmask }}'
 17       default_gateway: '{{ gateway }}'
 18       domain_name: '{{ domain }}'
 19       dns_server_1: '{{ dns1_ip }}'
 20       dns_server_2: '{{ dns2_ip }}'
 21       lease_time: '{{ lease_time }}'
 22       next_server: '{{ tftp_server }}'
 23       bootfile: '{{ bootfile }}'

A future enhancement to the module will allow for the DHCP Options variables to be updated as well. This is key for the project so that the scope points to the TFTP server where Core OS is installed from.

Update: The ability to add a TFTP next-server and specify the filename for downloading has been added and is contained in the Github repo virtualelephant/nsxansible.

Edge Firewall Rules

Fortunately, another author on Github already wrote this module — they even submitted a pull request to have it included in the vmware/nsxansible repo last year, but since it had yet to be included, I forked it into my own repo for use.

The nsx_edge_firewall.py module allows you to modify the default rule and create new rules on a NSX Edge device.

The Ansible playbook contains the following to create the default firewall policy:

  1 ---
  2 - hosts: localhost
  3   connection: local
  4   gather_facts: False
  5   vars_files:
  6     - nsxanswer.yml
  7     - envanswer.yml
  8 
  9   tasks:
 10   - name: Set default firewall rule policy
 11     nsx_edge_firewall:
 12       nsxmanager_spec: "{{ nsxmanager_spec }}"
 13       mode: 'set_default_action'
 14       edge_name: '{{ edge_name }}'
 15       default_action: 'accept'

Specify vNIC for Default Gateway

The original nsx_edge_router.py module included code to create the default gateway, however it did not allow you to modify the MTU or specify which vNIC should be associated with the default gateway. The forked nsx_edge_router.py version in the VirtualElephant Github repo includes the necessary code to specify both of those options.

150 def config_def_gw(client_session, esg_id, dfgw, vnic, mtu, dfgw_adminDistance):
151     if not mtu:
152         mtu = '1500'
153     rtg_cfg = client_session.read('routingConfigStatic', uri_parameters={'edgeId': esg_id})['body']
154     if dfgw:
155         try:
156             rtg_cfg['staticRouting']['defaultRoute'] = {'gatewayAddress': dfgw, 'vnic': vnic, 'mtu': mtu, 'adminDistance': dfgw_adminDistance}
157         except KeyError:
158             rtg_cfg['staticRouting']['defaultRoute'] = {'gatewayAddress': dfgw, 'vnic': vnic, 'adminDistance': dfgw_adminDistance, 'mtu': mtu}
159     else:
160         rtg_cfg['staticRouting']['defaultRoute'] = None
161 
162     cfg_result = client_session.update('routingConfigStatic', uri_parameters={'edgeId': esg_id},
163                                        request_body_dict=rtg_cfg)
164     if cfg_result['status'] == 204:
165         return True
166     else:
167         return False

The Ansible playbook is then able to include the following bits to create the default gateway with the preferred settings:

 42   - name: NSX Edge creation
 43     nsx_edge_router:
 44       nsxmanager_spec: "{{ nsxmanager_spec }}"
 45       state: present
 46       name: "{{ edge_name }}"
 47       description: "{{ description }}"
 48       resourcepool_moid: "{{ gather_moids_cl.object_id }}"
 49       datastore_moid: "{{ gather_moids_ds.object_id }}"
 50       datacenter_moid: "{{ gather_moids_cl.datacenter_moid }}"
 51       interfaces:
 52         vnic0: {ip: "{{ ext_ip }}", prefix_len: 26, logical_switch: "{{ uplink }}", name: 'uplink0', iftype: 'uplink', fence_param: 'ethernet0.filter1.param1=1'}
 53         vnic1: {ip: '192.168.0.1', prefix_len: 20, logical_switch: "{{ switch_name }}", name: 'int0', iftype: 'internal', fence_param: 'ethernet0.filter1.param1=1'}
 54       default_gateway: "{{ default_route }}"
 55       default_gateway_vnic: '0'
 56       mtu: '9000'
 57       remote_access: 'true'
 58       username: 'admin'
 59       password: "{{ nsx_pass }}"
 60       firewall: 'true'
 61       ha_enabled: 'true'
 62     register: create_esg
 63     tags: esg_create

When specifying the vNIC to use for the default gateway, the value is not the name the Ansible playbook gives the vNIC — uplink0 — but rather the vNIC number within the Edge — which will be 0 if you are using my playbook.

Once I have the SNAT/DNAT functionality added, I will write another blog post and progress on the Infrastructure-as-Code project will be nearly complete.

Enjoy!