As part of the work I am doing to get VMware Big Data Extensions to integrate with OpenStack Heat, I had to write several Python scripts to test the REST API that BDE offers. I currently have scripts to create clusters and also start|stop|delete clusters within a vSphere environment.
They are functional and were a good introduction to Python — I’ve been a PERL coder since the mid-1990s — though I am sure there are improvements that could still be made. I have put the scripts up in the Virtual Elephant GitHub repository.
I will make disclaimer that the createRestAPI.py script will not work out of the box for non-Hadoop cluster types, unless you modify several of the jar files BDE uses. I have been able to work with another engineer who assisted me in adding some of functionality that was missing in order to create Mesos clusters.
There will be an upcoming post that will explain how to expand the current REST API to allow for all cluster types that are setup within BDE can be deployed through the API. Adding this functionality has been an important step my integration efforts.
The testing scripts can be downloaded here.
The code can be reviewed after the break.
createRestAPI.py:
1 #!/usr/bin/python 2 3 # Testing module for REST API code in BDE 4 # 5 # Chris Mutchler - [email protected] 6 # http://www.VirtualElephant.com 7 # 8 # Licensed under the Apache License, Version 2.0 (the "License"); you may 9 # not use this file except in compliance with the License. You may obtain 10 # a copy of the License at 11 # 12 # http://www.apache.org/licenses/LICENSE-2.0 13 # 14 # Unless required by applicable law or agreed to in writing, software 15 # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 16 # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 17 # License for the specific language governing permissions and limitations 18 # under the License. 19 20 import logging, json, requests 21 requests.packages.urllib3.disable_warnings() 22 23 # Setup logging 24 logger = logging.getLogger(__name__) 25 logger.setLevel(logging.INFO) 26 logger.addHandler(logging.StreamHandler()) 27 28 29 # Big Data Extensions Endpoint 30 bde_endpoint = 'bde.localdomain' 31 username = '[email protected]' 32 password = 'password' 33 34 # Make initial session authenticated session 35 header = {'content-type': 'application/x-www-form-urlencoded'} 36 prefix = "https://" 37 port = ":8443" 38 auth_string = "/serengeti/j_spring_security_check" 39 creds = 'j_username=' + username + '&j_password=' + password 40 url = prefix + bde_endpoint + port + auth_string 41 42 s = requests.session() 43 r = s.post(url, creds, headers=header, verify=False) 44 45 #DEBUG 46 print url 47 print r.json 48 #/DEBUG 49 50 # Variables that will be passed through Heat 51 clusterType = "mesos" 52 clusterName = "mesos_api_01" 53 54 # Setup necessary bits for creating a new cluster 55 header = {'content-type': 'application/json'} 56 57 payload = {"name": clusterName, "distro": clusterType, "networkConfig": { "MGT_NETWORK": ["defaultNetwork"]}} 58 59 api_call = '/serengeti/api/clusters' 60 url = prefix + bde_endpoint + port + api_call 61 r = s.post(url, data=json.dumps(payload), headers=header, verify=False) 62 63 #DEBUG 64 print 65 print url 66 print r.json 67 print 68 print r.headers 69 print r.text 70 #/DEBUG
stopRestAPI.py:
1 #!/usr/bin/python 2 3 # Testing module for REST API code in BDE 4 # 5 # Chris Mutchler - [email protected] 6 # http://www.VirtualElephant.com 7 # 8 # Licensed under the Apache License, Version 2.0 (the "License"); you may 9 # not use this file except in compliance with the License. You may obtain 10 # a copy of the License at 11 # 12 # http://www.apache.org/licenses/LICENSE-2.0 13 # 14 # Unless required by applicable law or agreed to in writing, software 15 # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 16 # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 17 # License for the specific language governing permissions and limitations 18 # under the License. 19 20 import logging, json, requests 21 requests.packages.urllib3.disable_warnings() 22 23 # Setup logging 24 logger = logging.getLogger(__name__) 25 logger.setLevel(logging.INFO) 26 logger.addHandler(logging.StreamHandler()) 27 28 29 # Big Data Extensions Endpoint 30 bde_endpoint = 'bde.localdomain' 31 username = '[email protected]' 32 password = 'password' 33 34 # Make initial session authenticated session 35 header = {'content-type': 'application/x-www-form-urlencoded'} 36 prefix = "https://" 37 port = ":8443" 38 auth_string = "/serengeti/j_spring_security_check" 39 creds = 'j_username=' + username + '&j_password=' + password 40 url = prefix + bde_endpoint + port + auth_string 41 42 s = requests.session() 43 r = s.post(url, creds, headers=header, verify=False) 44 45 #DEBUG 46 print url 47 print r.json 48 #/DEBUG 49 50 # Variables that will be passed through Heat 51 clusterType = "mesos" 52 clusterName = "mesos_api_01" 53 clusterState = "stop" 54 55 # Setup necessary bits for creating a new cluster 56 header = {'content-type': 'application/json'} 57 api_call = '/serengeti/api/cluster/' + clusterName + '?state=' + clusterState 58 url = prefix + bde_endpoint + port + api_call 59 r = s.put(url, headers=header, verify=False) 60 61 #DEBUG 62 print 63 print url 64 print r.json 65 print 66 print r.headers 67 print r.text 68 #/DEBUG
startRestAPI.py:
1 #!/usr/bin/python 2 3 # Testing module for REST API code in BDE 4 # 5 # Chris Mutchler - [email protected] 6 # http://www.VirtualElephant.com 7 # 8 # Licensed under the Apache License, Version 2.0 (the "License"); you may 9 # not use this file except in compliance with the License. You may obtain 10 # a copy of the License at 11 # 12 # http://www.apache.org/licenses/LICENSE-2.0 13 # 14 # Unless required by applicable law or agreed to in writing, software 15 # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 16 # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 17 # License for the specific language governing permissions and limitations 18 # under the License. 19 20 import logging, json, requests 21 requests.packages.urllib3.disable_warnings() 22 23 # Setup logging 24 logger = logging.getLogger(__name__) 25 logger.setLevel(logging.INFO) 26 logger.addHandler(logging.StreamHandler()) 27 28 29 # Big Data Extensions Endpoint 30 bde_endpoint = 'bde.localdomain' 31 username = '[email protected]' 32 password = 'password' 33 34 # Make initial session authenticated session 35 header = {'content-type': 'application/x-www-form-urlencoded'} 36 prefix = "https://" 37 port = ":8443" 38 auth_string = "/serengeti/j_spring_security_check" 39 creds = 'j_username=' + username + '&j_password=' + password 40 url = prefix + bde_endpoint + port + auth_string 41 42 s = requests.session() 43 r = s.post(url, creds, headers=header, verify=False) 44 45 #DEBUG 46 print url 47 print r.json 48 #/DEBUG 49 50 # Variables that will be passed through Heat 51 clusterType = "mesos" 52 clusterName = "mesos_api_01" 53 clusterState = "start" 54 55 # Setup necessary bits for creating a new cluster 56 header = {'content-type': 'application/json'} 57 api_call = '/serengeti/api/cluster/' + clusterName + '?state=' + clusterState 58 url = prefix + bde_endpoint + port + api_call 59 r = s.put(url, headers=header, verify=False) 60 61 #DEBUG 62 print 63 print url 64 print r.json 65 print 66 print r.headers 67 print r.text 68 #/DEBUG
deleteRestAPI.py:
1 #!/usr/bin/python 2 3 # Testing module for REST API code in BDE 4 # 5 # Chris Mutchler - [email protected] 6 # http://www.VirtualElephant.com 7 # 8 # Licensed under the Apache License, Version 2.0 (the "License"); you may 9 # not use this file except in compliance with the License. You may obtain 10 # a copy of the License at 11 # 12 # http://www.apache.org/licenses/LICENSE-2.0 13 # 14 # Unless required by applicable law or agreed to in writing, software 15 # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 16 # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 17 # License for the specific language governing permissions and limitations 18 # under the License. 19 20 import logging, json, requests 21 requests.packages.urllib3.disable_warnings() 22 23 # Setup logging 24 logger = logging.getLogger(__name__) 25 logger.setLevel(logging.INFO) 26 logger.addHandler(logging.StreamHandler()) 27 28 29 # Big Data Extensions Endpoint 30 bde_endpoint = 'bde.localdomain' 31 username = '[email protected]' 32 password = 'password' 33 34 # Make initial session authenticated session 35 header = {'content-type': 'application/x-www-form-urlencoded'} 36 prefix = "https://" 37 port = ":8443" 38 auth_string = "/serengeti/j_spring_security_check" 39 creds = 'j_username=' + username + '&j_password=' + password 40 url = prefix + bde_endpoint + port + auth_string 41 42 s = requests.session() 43 r = s.post(url, creds, headers=header, verify=False) 44 45 #DEBUG 46 print url 47 print r.json 48 #/DEBUG 49 50 # Variables that will be passed through Heat 51 clusterType = "mesos" 52 clusterName = "mesos_api_01" 53 clusterState = "delete" 54 55 # Setup necessary bits for creating a new cluster 56 header = {'content-type': 'application/json'} 57 api_call = '/serengeti/api/cluster/' + clusterName 58 url = prefix + bde_endpoint + port + api_call 59 r = s.delete(url, headers=header, verify=False) 60 61 #DEBUG 62 print 63 print url 64 print r.json 65 print 66 print r.headers 67 print r.text 68 #/DEBUG
