The post includes the final pieces necessary to get a Mesosphere stack deployed through Big Data Extensions within a VMware environment. I’ve included the Chef cookbooks and commands required for tying all of the pieces together for a cluster deployment. The wonderful thing about the framework is the extensibility — once I had Mesos deploying, it became very clear how simple it is to extend the framework even further — look for future posts.
The idea that you can now turn a large cluster of VMs into a single Mesos cluster for use by a product, engineering team or operations team opens up an entirely new world within our environments. This is a very exciting place to be investing time.
Chef Roles
Big Data Extensions uses role definitions within the framework, so the first step was to create a new role for Mesos. If you remember from Part 2, we defined the role in the JSON file and called it ‘mesos’.
The role files can be found in /opt/serengeti/chef/roles. I created the roles for both mesos_master and mesos_worker through the command line interface:
# knife role create mesos_master
{
"name": "mesos_master",
"description": "",
"json_class": "Chef::Role",
"default_attributes": {
},
"override_attributes": {
},
"chef_type": "role",
"run_list": [
"recipe[mesos]",
"recipe[mesos::master]"
],
"env_run_lists": {
}
}
# knife role create mesos_worker
{
"name": "mesos_worker",
"description": "",
"json_class": "Chef::Role",
"default_attributes": {
},
"override_attributes": {
},
"chef_type": "role",
"run_list": [
"recipe[mesos]",
"recipe[mesos::worker]"
],
"env_run_lists": {
}
}
Chef Cookbooks
I broke the cookbooks into four parts — one common recipe (mesos), one installer recipe and one each for the master and worker nodes. The common recipe calls the installer recipe that sets up the Mesosphere repository on the local node and installs the Apache Mesos package (along with any missing dependencies). The other important piece within the recipe contains the block of the code that provides the ‘spark of life’ for the cluster — it recognizes which nodes have registered themselves as the master nodes with Zookeeper and properly configures the /etc/mesos/zk file on the nodes.
Mesos/recipes/default.rb
1 # 2 # Cookbook Name:: mesos 3 # Recipe:: default 4 # 5 6 # Licensed under the Apache License, Version 2.0 (the "License"); 7 # you may not use this file except in compliance with the License. 8 # You may obtain a copy of the License at 9 # 10 # http://www.apache.org/licenses/LICENSE-2.0 11 12 # Unless required by applicable law or agreed to in writing, software 13 # distributed under the License is distributed on an "AS IS" BASIS, 14 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 15 # See the License for the specific language governing permissions and 16 # limitations under the License. 17 # 18 19 include_recipe "java::sun" 20 include_recipe "hadoop_common::pre_run" 21 include_recipe "hadoop_common::mount_disks" 22 include_recipe "hadoop_cluster::update_attributes" 23 24 set_bootstrap_action(ACTION_INSTALL_PACKAGE, 'mesos', true) 25 include_recipe "mesos::install_from_package" 26 27 clear_bootstrap_action
Mesos/recipes/install_from_package.rb
1 #
2 # Copyright (c) 2012-2014 VMware, Inc. All Rights Reserved.
3 # Licensed under the Apache License, Version 2.0 (the "License");
4 # you may not use this file except in compliance with the License.
5 # You may obtain a copy of the License at
6 #
7 # http://www.apache.org/licenses/LICENSE-2.0
8
9 # Unless required by applicable law or agreed to in writing, software
10 # distributed under the License is distributed on an "AS IS" BASIS,
11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
14 #
15
16 include_recipe "hadoop_common::add_repo"
17 include_recipe "hadoop_cluster::update_attributes"
18
19 #
20 # Setup Mesosphere repo
21 #
22 remote_file "/root/mesosphere-el-repo-6-2.noarch.rpm" do
23 source "http://repos.mesosphere.io/el/6/noarch/RPMS/mesosphere-el-repo-6-2.noarch.rpm"
24 action :create
25 end
26
27 rpm_package "mesosphere-el-repo-6-2.noarch.rpm" do
28 source "/root/mesosphere-el-repo-6-2.noarch.rpm"
29 action :install
30 end
31
32 %w( mesos ).each do |pkg|
33 yum_package pkg do
34 action :install
35 end
36 end
37
38 #
39 # Write Zookeeper node information to /etc/mesos/zk
40 #
41
42 run_in_ruby_block "Discover zookeeper nodes" do
43 set_action(HadoopCluster::ACTION_WAIT_FOR_SERVICE, node[:zookeeper][:zookeeper_service_name])
44 zookeeper_count = all_nodes_count({"role" => "zookeeper"})
45 zookeeper_nodes = all_providers_for_service(node[:zookeeper][:zookeeper_service_name], true, zookeeper_count)
46 zookeeper_ips = all_provider_private_ips(node[:zookeeper][:zookeeper_service_name], true, zookeeper_count)
47
48 conf_prefix = "zk://"
49 conf_suffix = "/mesos/"
50 conf_line = conf_prefix
51
52 zookeeper_ips.each do |zk|
53 #$stderr.puts zk
54 conf_line = conf_line + zk + ":2181,"
55 end
56
57 conf_line = conf_line.chop
58 conf_line = conf_line + conf_suffix
59
60 fd = IO.sysopen "/etc/mesos/zk", "w"
61 ios = IO.new(fd, "w")
62 ios.puts conf_line
63 ios.close
64 clear_action
65 end
Mesos/recipes/master.rb
1 # 2 # Cookbook Name:: mesos 3 # Recipe:: master 4 # 5 6 # Licensed under the Apache License, Version 2.0 (the "License"); 7 # you may not use this file except in compliance with the License. 8 # You may obtain a copy of the License at 9 # 10 # http://www.apache.org/licenses/LICENSE-2.0 11 12 # Unless required by applicable law or agreed to in writing, software 13 # distributed under the License is distributed on an "AS IS" BASIS, 14 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 15 # See the License for the specific language governing permissions and 16 # limitations under the License. 17 # 18 19 template '/etc/default/mesos' do 20 source 'mesos.erb' 21 variables config: node[:mesos][:common] 22 action :create 23 end 24 25 remote_file "Copy mesos-init-wrapper file" do 26 path "/usr/bin/mesos-init-wrapper" 27 source "file:///var/chef/cache/cookbooks/mesos/templates/default/mesos-init-wrapper" 28 end 29 30 # 31 # Start mesos-master service 32 # 33 34 execute "Starting mesos-master" do 35 command "nohup /usr/bin/mesos-init-wrapper master &" 36 end 37 38 # 39 # Install and setup Marathon & Chronos 40 # 41 42 %w( marathon chronos ).each do |pkg| 43 yum_package pkg do 44 action :install 45 end 46 end 47 48 execute "Starting Marathon" do 49 command "nohup /usr/local/bin/marathon &" 50 end 51 52 execute "Starting Chronos" do 53 command "nohup /usr/local/bin/chronos &" 54 end
Mesos/recipes/worker.rb
1 # 2 # Cookbook Name:: mesos 3 # Recipe:: worker 4 # 5 6 # Licensed under the Apache License, Version 2.0 (the "License"); 7 # you may not use this file except in compliance with the License. 8 # You may obtain a copy of the License at 9 # 10 # http://www.apache.org/licenses/LICENSE-2.0 11 12 # Unless required by applicable law or agreed to in writing, software 13 # distributed under the License is distributed on an "AS IS" BASIS, 14 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 15 # See the License for the specific language governing permissions and 16 # limitations under the License. 17 # 18 19 template '/etc/default/mesos' do 20 source 'mesos.erb' 21 variables config: node[:mesos][:common] 22 action :create 23 end 24 25 remote_file "Copy mesos-init-wrapper file" do 26 path "/usr/bin/mesos-init-wrapper" 27 source "file:///var/chef/cache/cookbooks/mesos/templates/default/mesos-init-wrapper" 28 end 29 30 # 31 # Start mesos-slave service 32 # 33 34 execute "Starting mesos-slave" do 35 command "nohup /usr/bin/mesos-init-wrapper slave &" 36 end 37 38 # 39 # Install worker-specific packages 40 # 41 42 %w( chronos ).each do |pkg| 43 yum_package pkg do 44 action :install 45 end 46 end 47 48 execute "Starting Chronos" do 49 command "nohup /usr/local/bin/chronos &" 50 end
There are several other files within the Mesos cookbook directory structure, including templates and attributes that are required for the deployment to work. All of the necessary files will be available in a day or so on the site and on GitHub.
Final steps
The last few things you will need to do include uploading all the cookbooks to the Chef server, restarting tomcat on the BDE management server to see the spec.json file and creating the cluster. For my deployments, I generally use the BDE CLI which is installed on the management server:
# java -jar /opt/serengeti/cli/serengeti-cli-2.1.0.jar ================================================= * _____ _ _ ** / ____| ___ _ __ ___ _ __ __ _ ___| |_(_) ** \____ \ / _ \ '__/ _ \ '_ \ / _` |/ _ \ __| | ** ____) | __/ | | __/ | | | (_| | __/ |_| | ** |_____/ \___|_| \___|_| |_|\__, |\___|\__|_| ** |___/ ** * ================================================= Version: 2.1.0 Welcome to Serengeti CLI serengeti>connect --host 10.55.204.80:8443 Enter the username: [email protected] Enter the password: ************ Connected serengeti>cluster list serengeti>cluster create --name mesosphere_test --specFile /opt/serengeti/www/specs/Mesos/spec.json --networkName defaultNetwork --password yes Hint: Password are from 8 to 128 characters, and can include alphanumeric characters ([0-9, a-z, A-Z]) and the following special characters: _, @, #, $, %, ^, &, * Enter the password: ********** Confirm the password: **********
Upon completion of the deployment, you will have a working Mesosphere cluster with Apache Mesos, Mesosphere Marathon and Chronos.
Mesos Cluster within BDE UI
Mesos Web UI
Mesosphere Marathon Web UI
Check back in a couple days for links to all the code so that you can try this out yourself!
Acknowlegements: I want to say thank you to Andrew Nelson (@vmwnelson) for letting me bounce ideas off of him constantly. To Ken Sipe (@kensipe) for getting me excited about Mesos during his presentation to us at Adobe. To Frans Van Rooyen (@jfvanrooyen) for helping me daily and constantly challenging me.
Finally, there is a lot of great open source code out there for Mesos and the BDE team has done a great job of making all of the Serengeti code available on GitHub for use. Go check it out!
