VMware Big Data Extensions Python REST API Scripts


As part of the work I am doing to get VMware Big Data Extensions to integrate with OpenStack Heat, I had to write several Python scripts to test the REST API that BDE offers. I currently have scripts to create clusters and also start|stop|delete clusters within a vSphere environment.

They are functional and were a good introduction to Python — I’ve been a PERL coder since the mid-1990s — though I am sure there are improvements that could still be made. I have put the scripts up in the Virtual Elephant GitHub repository.

I will make disclaimer that the createRestAPI.py script will not work out of the box for non-Hadoop cluster types, unless you modify several of the jar files BDE uses. I have been able to work with another engineer who assisted me in adding some of functionality that was missing in order to create Mesos clusters.

There will be an upcoming post that will explain how to expand the current REST API to allow for all cluster types that are setup within BDE can be deployed through the API. Adding this functionality has been an important step my integration efforts.

The testing scripts can be downloaded here.

The code can be reviewed after the break.

createRestAPI.py:

  1 #!/usr/bin/python
  2 
  3 #    Testing module for REST API code in BDE
  4 #
  5 #    Chris Mutchler - chris@virtualelephant.com
  6 #    http://www.VirtualElephant.com
  7 #
  8 #    Licensed under the Apache License, Version 2.0 (the "License"); you may
  9 #    not use this file except in compliance with the License. You may obtain
 10 #    a copy of the License at
 11 #
 12 #         http://www.apache.org/licenses/LICENSE-2.0
 13 #
 14 #    Unless required by applicable law or agreed to in writing, software
 15 #    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 16 #    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 17 #    License for the specific language governing permissions and limitations
 18 #    under the License.
 19 
 20 import logging, json, requests
 21 requests.packages.urllib3.disable_warnings()
 22 
 23 # Setup logging
 24 logger = logging.getLogger(__name__)
 25 logger.setLevel(logging.INFO)
 26 logger.addHandler(logging.StreamHandler())
 27 
 28 
 29 # Big Data Extensions Endpoint
 30 bde_endpoint = 'bde.localdomain'
 31 username = 'administrator@vsphere.local'
 32 password = 'password'
 33 
 34 # Make initial session authenticated session
 35 header = {'content-type': 'application/x-www-form-urlencoded'}
 36 prefix = "https://"
 37 port = ":8443"
 38 auth_string = "/serengeti/j_spring_security_check"
 39 creds = 'j_username=' + username + '&j_password=' + password
 40 url = prefix + bde_endpoint + port + auth_string
 41 
 42 s = requests.session()
 43 r = s.post(url, creds, headers=header, verify=False)
 44 
 45 #DEBUG
 46 print url
 47 print r.json
 48 #/DEBUG
 49 
 50 # Variables that will be passed through Heat
 51 clusterType = "mesos"
 52 clusterName = "mesos_api_01"
 53 
 54 # Setup necessary bits for creating a new cluster
 55 header = {'content-type': 'application/json'}
 56 
 57 payload = {"name": clusterName, "distro": clusterType, "networkConfig": { "MGT_NETWORK": ["defaultNetwork"]}}
 58 
 59 api_call = '/serengeti/api/clusters'
 60 url = prefix + bde_endpoint + port + api_call
 61 r = s.post(url, data=json.dumps(payload), headers=header, verify=False)
 62 
 63 #DEBUG
 64 print
 65 print url
 66 print r.json
 67 print
 68 print r.headers
 69 print r.text
 70 #/DEBUG

stopRestAPI.py:

  1 #!/usr/bin/python
  2 
  3 #    Testing module for REST API code in BDE
  4 #
  5 #    Chris Mutchler - chris@virtualelephant.com
  6 #    http://www.VirtualElephant.com
  7 #
  8 #    Licensed under the Apache License, Version 2.0 (the "License"); you may
  9 #    not use this file except in compliance with the License. You may obtain
 10 #    a copy of the License at
 11 #
 12 #         http://www.apache.org/licenses/LICENSE-2.0
 13 #
 14 #    Unless required by applicable law or agreed to in writing, software
 15 #    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 16 #    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 17 #    License for the specific language governing permissions and limitations
 18 #    under the License.
 19 
 20 import logging, json, requests
 21 requests.packages.urllib3.disable_warnings()
 22 
 23 # Setup logging
 24 logger = logging.getLogger(__name__)
 25 logger.setLevel(logging.INFO)
 26 logger.addHandler(logging.StreamHandler())
 27 
 28 
 29 # Big Data Extensions Endpoint
 30 bde_endpoint = 'bde.localdomain'
 31 username = 'administrator@vsphere.local'
 32 password = 'password'
 33 
 34 # Make initial session authenticated session
 35 header = {'content-type': 'application/x-www-form-urlencoded'}
 36 prefix = "https://"
 37 port = ":8443"
 38 auth_string = "/serengeti/j_spring_security_check"
 39 creds = 'j_username=' + username + '&j_password=' + password
 40 url = prefix + bde_endpoint + port + auth_string
 41 
 42 s = requests.session()
 43 r = s.post(url, creds, headers=header, verify=False)
 44 
 45 #DEBUG
 46 print url
 47 print r.json
 48 #/DEBUG
 49 
 50 # Variables that will be passed through Heat
 51 clusterType = "mesos"
 52 clusterName = "mesos_api_01"
 53 clusterState = "stop"
 54 
 55 # Setup necessary bits for creating a new cluster
 56 header = {'content-type': 'application/json'}
 57 api_call = '/serengeti/api/cluster/' + clusterName + '?state=' + clusterState
 58 url = prefix + bde_endpoint + port + api_call
 59 r = s.put(url, headers=header, verify=False)
 60 
 61 #DEBUG
 62 print
 63 print url
 64 print r.json
 65 print
 66 print r.headers
 67 print r.text
 68 #/DEBUG

startRestAPI.py:

  1 #!/usr/bin/python
  2 
  3 #    Testing module for REST API code in BDE
  4 #
  5 #    Chris Mutchler - chris@virtualelephant.com
  6 #    http://www.VirtualElephant.com
  7 #
  8 #    Licensed under the Apache License, Version 2.0 (the "License"); you may
  9 #    not use this file except in compliance with the License. You may obtain
 10 #    a copy of the License at
 11 #
 12 #         http://www.apache.org/licenses/LICENSE-2.0
 13 #
 14 #    Unless required by applicable law or agreed to in writing, software
 15 #    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 16 #    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 17 #    License for the specific language governing permissions and limitations
 18 #    under the License.
 19 
 20 import logging, json, requests
 21 requests.packages.urllib3.disable_warnings()
 22 
 23 # Setup logging
 24 logger = logging.getLogger(__name__)
 25 logger.setLevel(logging.INFO)
 26 logger.addHandler(logging.StreamHandler())
 27 
 28 
 29 # Big Data Extensions Endpoint
 30 bde_endpoint = 'bde.localdomain'
 31 username = 'administrator@vsphere.local'
 32 password = 'password'
 33 
 34 # Make initial session authenticated session
 35 header = {'content-type': 'application/x-www-form-urlencoded'}
 36 prefix = "https://"
 37 port = ":8443"
 38 auth_string = "/serengeti/j_spring_security_check"
 39 creds = 'j_username=' + username + '&j_password=' + password
 40 url = prefix + bde_endpoint + port + auth_string
 41 
 42 s = requests.session()
 43 r = s.post(url, creds, headers=header, verify=False)
 44 
 45 #DEBUG
 46 print url
 47 print r.json
 48 #/DEBUG
 49 
 50 # Variables that will be passed through Heat
 51 clusterType = "mesos"
 52 clusterName = "mesos_api_01"
 53 clusterState = "start"
 54 
 55 # Setup necessary bits for creating a new cluster
 56 header = {'content-type': 'application/json'}
 57 api_call = '/serengeti/api/cluster/' + clusterName + '?state=' + clusterState
 58 url = prefix + bde_endpoint + port + api_call
 59 r = s.put(url, headers=header, verify=False)
 60 
 61 #DEBUG
 62 print
 63 print url
 64 print r.json
 65 print
 66 print r.headers
 67 print r.text
 68 #/DEBUG

deleteRestAPI.py:

  1 #!/usr/bin/python
  2 
  3 #    Testing module for REST API code in BDE
  4 #
  5 #    Chris Mutchler - chris@virtualelephant.com
  6 #    http://www.VirtualElephant.com
  7 #
  8 #    Licensed under the Apache License, Version 2.0 (the "License"); you may
  9 #    not use this file except in compliance with the License. You may obtain
 10 #    a copy of the License at
 11 #
 12 #         http://www.apache.org/licenses/LICENSE-2.0
 13 #
 14 #    Unless required by applicable law or agreed to in writing, software
 15 #    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 16 #    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 17 #    License for the specific language governing permissions and limitations
 18 #    under the License.
 19 
 20 import logging, json, requests
 21 requests.packages.urllib3.disable_warnings()
 22 
 23 # Setup logging
 24 logger = logging.getLogger(__name__)
 25 logger.setLevel(logging.INFO)
 26 logger.addHandler(logging.StreamHandler())
 27 
 28 
 29 # Big Data Extensions Endpoint
 30 bde_endpoint = 'bde.localdomain'
 31 username = 'administrator@vsphere.local'
 32 password = 'password'
 33 
 34 # Make initial session authenticated session
 35 header = {'content-type': 'application/x-www-form-urlencoded'}
 36 prefix = "https://"
 37 port = ":8443"
 38 auth_string = "/serengeti/j_spring_security_check"
 39 creds = 'j_username=' + username + '&j_password=' + password
 40 url = prefix + bde_endpoint + port + auth_string
 41 
 42 s = requests.session()
 43 r = s.post(url, creds, headers=header, verify=False)
 44 
 45 #DEBUG
 46 print url
 47 print r.json
 48 #/DEBUG
 49 
 50 # Variables that will be passed through Heat
 51 clusterType = "mesos"
 52 clusterName = "mesos_api_01"
 53 clusterState = "delete"
 54 
 55 # Setup necessary bits for creating a new cluster
 56 header = {'content-type': 'application/json'}
 57 api_call = '/serengeti/api/cluster/' + clusterName
 58 url = prefix + bde_endpoint + port + api_call
 59 r = s.delete(url, headers=header, verify=False)
 60 
 61 #DEBUG
 62 print
 63 print url
 64 print r.json
 65 print
 66 print r.headers
 67 print r.text
 68 #/DEBUG