VMware OpenStack Heat Resource Plugin Code Review

The post a week ago Monday announced the release of the Heat resource plugin for VMware OpenStack and VMware Big Data Extensions. As I was working through the final issues before the plugin was released and the initial blog post announcing it, I realized it may be helpful to describe how the OpenStack Heat resource plugin works. The initial code for the plugin was rather simplistic and based pretty closely off of the Python scripts I wrote to test the Big Data Extensions REST API. Once testing of the resource plugin began it became evident far more code was going to be necessary to be successful. This blog post will describe the workflows inside the resource plugin, the challenges that I encountered and where improvements could be made in the future. My ultimate hope for the resource plugin is for VMware to take the framework and develop a supported OpenStack Heat resource plugin for everyone to consume.

Basic Workflow

The basic workflow for the resource plugin is not as simplistic as I had hoped. Due to the integration of VMware OpenStack with VMware NSX-v, there were several additional API calls required before the Big Data Extension instances would be accepted into the fold and allowed to communicate over the network.

bde-resource-plugin-workflow

You can see from the figure above the number of API calls required to create a new cluster and allow it to communicate with the Big Data Extensions management server.

Properties

The resource plugin requires several variables in order to operate properly. These variables can then be specified within the JSON Heat template. The following variables are setup:

 48     properties_schema = {
 49         BDE_ENDPOINT: properties.Schema(
 50             properties.Schema.STRING,
 51             required=True,
 52             default='bde.localdomain'
 53         ),
 54         VCM_SERVER: properties.Schema(
 55             properties.Schema.STRING,
 56             required=True,
 57             default='vcenter.localdomain'
 58         ),
 59         USERNAME: properties.Schema(
 60             properties.Schema.STRING,
 61             required=True,
 62             default='administrator@vsphere.local'
 63         ),
 64         PASSWORD: properties.Schema(
 65             properties.Schema.STRING,
 66             required=True,
 67             default='password'
 68         ),
 69         CLUSTER_NAME: properties.Schema(
 70             properties.Schema.STRING,
 71             required=True
 72         ),
 73         CLUSTER_TYPE: properties.Schema(
 74             properties.Schema.STRING,
 75             required=True
 76         ),
 77         NETWORK: properties.Schema(
 78             properties.Schema.STRING,
 79             required=True
 80         ),
 81         CLUSTER_PASSWORD: properties.Schema(
 82             properties.Schema.STRING,
 83             required=False
 84         ),
 85         CLUSTER_RP: properties.Schema(
 86             properties.Schema.STRING,
 87             required=True,
 88             default='openstackRP'
 89         ),
 90         VIO_CONFIG: properties.Schema(
 91             properties.Schema.STRING,
 92             required=True,
 93             default='/usr/local/bin/etc/vio.config'
 94         ),
 95         BDE_CONFIG: properties.Schema(
 96             properties.Schema.STRING,
 97             required=False,
 98             default='/usr/local/bin/etc/bde.config'
 99         ),
100         SECURITY_GROUP: properties.Schema(
101             properties.Schema.STRING,
102             required=False,
103             default='9d3ecec8-e0e3-4088-8c71-8c35cd67dd8b'
104         ),
105         SUBNET: properties.Schema(
106             properties.Schema.STRING,
107             required=True
108         )
109     }

NSX-v Integration

The integration with NSX-v was a multi-step process and honestly took me nearly a full week of testing to sort out. I simply could not have worked through all of these steps were it not for the awesome VIO and BDE engineering teams at VMware who offered guidance, suggestions and screenshots on where NSX-v was hooked into the instances (or needed to be). I am hoping to see them at VMworld this year so that I can buy them dinner for all the help they provided.

Before we go any farther, let me state writing code in Python is not yet a strength for me. In fact, this project was really my first significant foray in using Python — so forgive me if the code is horrid looking.

Identifying the vDS Portgroup

The first piece that had to be identified was the UUID of the vDS Portgroup and having the resource plugin make that Portgroup available to Big Data Extensions.

289         # determine actual NSX portgroup created
290         # hack - regex in Python is not a strength
291         mob_string = '/mob/?moid=datacenter-2'
292         curl_cmd = 'curl -k -u ' + bde_user + ':' + bde_pass + ' ' + prefix + vcm_server + mob_string
293         grep_cmd = " | grep -oP '(?<=\(vxw).*(?=" + network + "\))' | grep -oE '[^\(]+$'"
294         awk_cmd = " | awk '{print $0 \"" + network + "\"}'"
295         full_cmd = curl_cmd + grep_cmd + awk_cmd
296 
297         p = subprocess.Popen(full_cmd, stdout=subprocess.PIPE, shell=True)
298         (net_uid, err) = p.communicate()
299 
300         # Check to see if network_id is as we expect it
301         if 'vxw' in net_uid:
302             network_id = net_uid
303         else:
304             network_id = "vxw" + net_uid
305 
306         network_id = network_id.rstrip('\n')
307 
308         # Authenticate in a requests.session to the BDE server
309         curr = self._open_connection()
310 
311         # Should check to see if network already exists as available network
312         # This logs a big fat error message in /opt/serengeti/logs/serengeti.log
313         # when the network doesn't exist.
314         header = {'content-type': 'application/json'}
315         api_call = '/serengeti/api/network/' + network
316         url = prefix + bde_server + port + api_call
317         r = curr.get(url, headers=header, verify=False)
318 
319         # Add new network to BDE as an available network if check fails
320         payload = {"name" : network, "portGroup" : network_id, "isDhcp" : "true"}
321         api_call = '/serengeti/api/networks'
322         url = prefix + bde_server + port + api_call
323         r = curr.post(url, data=json.dumps(payload), headers=header, verify=False)
324         logger.info(_("VirtualElephant::VMware::BDE - Network creation status code %s") % r.json)

As the code shows, determining the actual name of the vDS Portgroup required taking the UUID — that was passed into the resource plugin as a reference from the JSON file — and parsing HTML output from a call to the MOB. Depending on how many Portgroups existed within the vCenter, it also meant having to deal with additional HTML code that hid the complete list of Portgroups from view unless a user clicked a hyperlink. There is probably a better way of grep’ing and awk’ing in Python, but old habits from decades of writing Perl code are shining through there. You may also notice the MOB link is hard-coded (line 291), this has to be changed sometime soon and is not a sustainable method for using the resource plugin in an enterprise environment.

When writing the plugin, I had hoped this was all that would be needed in order for the BDE nodes to be able to tie into the NSX portgroup and be capable of passing traffic. Unfortunately, that was not the case and I had to also figure out how to add the MAC addresses of every node, through Neutron, into NSX-v.

Creating Neutron ports for NSX-v

The method is called after the BDE create cluster command is sent to the management server. The method is defined in the code on line 139:

139     def _create_nsx_ports(self):

After getting the necessary OpenStack Neutron credential information, the method starts by gathering the node list from BDE in order to match the node name in BDE and the corresponding name in vCenter. This is necessary because BDE relies on vCenter to track the MAC addresses of the nodes created.

192         # Get the node names for the cluster from BDE
193         curr = self._open_connection()
194         header = {'content-type': 'application/json'}
195         api_call = '/serengeti/api/cluster/' + cluster_name
196         url = prefix + bde_server + port + api_call
197         r = curr.get(url, headers=header, verify=False)
198         raw_json = json.loads(r.text)
199         cluster_data = raw_json["nodeGroups"]

I then chose to iterate through the cluster_data variable and work through each one serially to gather the MAC addresses.

201         # Open connect to the vSphere API
202         si = SmartConnect(host=vcm_server, user=admin_user, pwd=admin_pass, port=443)
203         search_index = si.content.searchIndex
204         root_folder = si.content.rootFolder
205         for ng in cluster_data:
206             nodes = ng["instances"]
207             for node in nodes:
208                 logger.info(_("VirtualElephant::VMware::BDE - Creating NSX port for %s") % node.get("name"))
209                 vm_name = node.get("name")
210                 vm_moId = node.get("moId")
211                 port_name = vm_name + "-port0"
212 
213                 # moId is not in format we need to match
214                 (x,y,z) = vm_moId.split(":")
215                 vm_moId = "'vim." + y + ":" + z + "'"
216 
217                 # Go through each DC one at a time, in case there are multiple in vCenter
218                 for dc in root_folder.childEntity:
219                     content = si.content
220                     objView = content.viewManager.CreateContainerView(dc, [vim.VirtualMachine], True)
221                     vm_list = objView.view
222                     objView.Destroy()
223 
224                     for instance in vm_list:
225                         # convert object to string so we can search
226                         i = str(instance.summary.vm)
227                         if vm_moId in i:
228                             # Matched the VM in BDE and vCenter
229                             logger.info(_("VirtualElephant::VMware::BDE - Match found for BDE node %s") % instance)

Once the match is found, then we get the MAC address:

230                             for device in instance.config.hardware.device:
231                                 if isinstance(device, vim.vm.device.VirtualEthernetCard):
232                                     mac_address = str(device.macAddress)
233                                     logger.info(_("VirtualElephant::VMware::BDE - Found MAC address %s") % mac_address)

Now it is possible for the MAC address to be used to create the Neutron port, adding it in properly to NSX-v, enabling BDE nodes to be able to access the network properly.

250                 # Create a new port through Neutron
251                 neutron = client.Client('2.0',
252                                         username=os_username,
253                                         password=os_password,
254                                         auth_url=os_auth_url,
255                                         tenant_name=os_tenant_name,
256                                         endpoint_url=os_url,
257                                         token=os_token)
258                 port_info = {
259                                 "port": {
260                                         "admin_state_up": True,
261                                         "device_id": vm_name,
262                                         "name": port_name,
263                                         "mac_address": mac_address,
264                                         "network_id": network_id
265                                 }
266                             }
267                 logger.info(_("VirtualElephant::VMware::BDE - Neutron port string %s") % port_info)
268 
269                 response = neutron.create_port(body=port_info)
270                 logger.info(_("VirtualElephant::VMware::BDE - NSX port creation response - %s") % response)

Conclusion

There remain some timing issues with creating the Neutron ports and getting the BDE nodes to be able to get a DHCP IP address. Depending on ‘quickly’ the virtual machines are reconfigured and being loading the OS, the Neutron ports will enable the network traffic. If the OS is already to the point where it is trying to get a DHCP address, then the virtual machine has to be reset/rebooted to get the address. The issue here is how do you know which stage the VM is in the BDE deployment process? This is the issue I continue to work through and I have not found a resolution yet.

If you have ideas, please reach out and let me know. I am hoping to solve it through the resource plugin, but ultimately I believe a more elegant solution will involve BDE to handle the NSX-v updates.