Apache Mesos Clusters – Part 3

The post includes the final pieces necessary to get a Mesosphere stack deployed through Big Data Extensions within a VMware environment. I’ve included the Chef cookbooks and commands required for tying all of the pieces together for a cluster deployment. The wonderful thing about the framework is the extensibility — once I had Mesos deploying, it became very clear how simple it is to extend the framework even further — look for future posts.

The idea that you can now turn a large cluster of VMs into a single Mesos cluster for use by a product, engineering team or operations team opens up an entirely new world within our environments. This is a very exciting place to be investing time.

Chef Roles

Big Data Extensions uses role definitions within the framework, so the first step was to create a new role for Mesos. If you remember from Part 2, we defined the role in the JSON file and called it ‘mesos’.

The role files can be found in /opt/serengeti/chef/roles. I created the roles for both mesos_master and mesos_worker through the command line interface:

# knife role create mesos_master
{
  "name": "mesos_master",
  "description": "",
  "json_class": "Chef::Role",
  "default_attributes": {
  },
  "override_attributes": {
  },
  "chef_type": "role",
  "run_list": [
    "recipe[mesos]",
    "recipe[mesos::master]"
  ],
  "env_run_lists": {
  }
}
# knife role create mesos_worker
{
  "name": "mesos_worker",
  "description": "",
  "json_class": "Chef::Role",
  "default_attributes": {
  },
  "override_attributes": {
  },
  "chef_type": "role",
  "run_list": [
    "recipe[mesos]",
    "recipe[mesos::worker]"
  ],
  "env_run_lists": {
  }
}
Chef Cookbooks

I broke the cookbooks into four parts — one common recipe (mesos), one installer recipe and one each for the master and worker nodes. The common recipe calls the installer recipe that sets up the Mesosphere repository on the local node and installs the Apache Mesos package (along with any missing dependencies). The other important piece within the recipe contains the block of the code that provides the ‘spark of life’ for the cluster — it recognizes which nodes have registered themselves as the master nodes with Zookeeper and properly configures the /etc/mesos/zk file on the nodes.

Mesos/recipes/default.rb
  1 #
  2 # Cookbook Name:: mesos
  3 # Recipe::        default
  4 #
  5 
  6 # Licensed under the Apache License, Version 2.0 (the "License");
  7 #   you may not use this file except in compliance with the License.
  8 #   You may obtain a copy of the License at
  9 #
 10 #     http://www.apache.org/licenses/LICENSE-2.0
 11  
 12 #   Unless required by applicable law or agreed to in writing, software
 13 #   distributed under the License is distributed on an "AS IS" BASIS,
 14 #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 15 #   See the License for the specific language governing permissions and
 16 #   limitations under the License.
 17 #
 18 
 19 include_recipe "java::sun"
 20 include_recipe "hadoop_common::pre_run"
 21 include_recipe "hadoop_common::mount_disks"
 22 include_recipe "hadoop_cluster::update_attributes"
 23 
 24 set_bootstrap_action(ACTION_INSTALL_PACKAGE, 'mesos', true)
 25 include_recipe "mesos::install_from_package"
 26 
 27 clear_bootstrap_action
Mesos/recipes/install_from_package.rb
  1 #
  2 #   Copyright (c) 2012-2014 VMware, Inc. All Rights Reserved.
  3 #   Licensed under the Apache License, Version 2.0 (the "License");
  4 #   you may not use this file except in compliance with the License.
  5 #   You may obtain a copy of the License at
  6 #
  7 #     http://www.apache.org/licenses/LICENSE-2.0 
  8 
  9 #   Unless required by applicable law or agreed to in writing, software
 10 #   distributed under the License is distributed on an "AS IS" BASIS,
 11 #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 12 #   See the License for the specific language governing permissions and
 13 #   limitations under the License.
 14 #
 15 
 16 include_recipe "hadoop_common::add_repo"
 17 include_recipe "hadoop_cluster::update_attributes"
 18 
 19 #
 20 # Setup Mesosphere repo
 21 #
 22 remote_file "/root/mesosphere-el-repo-6-2.noarch.rpm" do
 23   source "http://repos.mesosphere.io/el/6/noarch/RPMS/mesosphere-el-repo-6-2.noarch.rpm"
 24   action :create
 25 end
 26 
 27 rpm_package "mesosphere-el-repo-6-2.noarch.rpm" do
 28   source "/root/mesosphere-el-repo-6-2.noarch.rpm"
 29   action :install
 30 end
 31 
 32 %w( mesos ).each do |pkg|
 33   yum_package pkg do
 34     action :install
 35   end
 36 end
 37 
 38 #
 39 # Write Zookeeper node information to /etc/mesos/zk
 40 #
 41 
 42 run_in_ruby_block "Discover zookeeper nodes" do
 43   set_action(HadoopCluster::ACTION_WAIT_FOR_SERVICE, node[:zookeeper][:zookeeper_service_name])
 44   zookeeper_count = all_nodes_count({"role" => "zookeeper"})
 45   zookeeper_nodes = all_providers_for_service(node[:zookeeper][:zookeeper_service_name], true, zookeeper_count)
 46   zookeeper_ips = all_provider_private_ips(node[:zookeeper][:zookeeper_service_name], true, zookeeper_count)
 47 
 48   conf_prefix = "zk://"
 49   conf_suffix = "/mesos/"
 50   conf_line = conf_prefix
 51 
 52   zookeeper_ips.each do |zk|
 53     #$stderr.puts zk
 54     conf_line = conf_line + zk + ":2181,"
 55   end
 56 
 57   conf_line = conf_line.chop
 58   conf_line = conf_line + conf_suffix
 59 
 60   fd = IO.sysopen "/etc/mesos/zk", "w"
 61   ios = IO.new(fd, "w")
 62   ios.puts conf_line
 63   ios.close
 64   clear_action
 65 end
Mesos/recipes/master.rb
  1 #
  2 # Cookbook Name:: mesos
  3 # Recipe::        master
  4 #
  5 
  6 # Licensed under the Apache License, Version 2.0 (the "License");
  7 #   you may not use this file except in compliance with the License.
  8 #   You may obtain a copy of the License at
  9 #
 10 #     http://www.apache.org/licenses/LICENSE-2.0
 11 
 12 #   Unless required by applicable law or agreed to in writing, software
 13 #   distributed under the License is distributed on an "AS IS" BASIS,
 14 #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 15 #   See the License for the specific language governing permissions and
 16 #   limitations under the License.
 17 #
 18 
 19 template '/etc/default/mesos' do
 20   source 'mesos.erb'
 21   variables config: node[:mesos][:common]
 22   action :create
 23 end
 24 
 25 remote_file "Copy mesos-init-wrapper file" do
 26   path "/usr/bin/mesos-init-wrapper"
 27   source "file:///var/chef/cache/cookbooks/mesos/templates/default/mesos-init-wrapper"
 28 end
 29 
 30 #
 31 # Start mesos-master service
 32 #
 33 
 34 execute "Starting mesos-master" do
 35   command "nohup /usr/bin/mesos-init-wrapper master &"
 36 end
 37 
 38 #
 39 # Install and setup Marathon & Chronos
 40 # 
 41 
 42 %w( marathon chronos ).each do |pkg|
 43   yum_package pkg do
 44     action :install
 45   end
 46 end
 47 
 48 execute "Starting Marathon" do
 49   command "nohup /usr/local/bin/marathon &"
 50 end
 51 
 52 execute "Starting Chronos" do
 53   command "nohup /usr/local/bin/chronos &"
 54 end
Mesos/recipes/worker.rb
  1 #
  2 # Cookbook Name:: mesos
  3 # Recipe::        worker
  4 #
  5 
  6 # Licensed under the Apache License, Version 2.0 (the "License");
  7 #   you may not use this file except in compliance with the License.
  8 #   You may obtain a copy of the License at
  9 #
 10 #     http://www.apache.org/licenses/LICENSE-2.0
 11 
 12 #   Unless required by applicable law or agreed to in writing, software
 13 #   distributed under the License is distributed on an "AS IS" BASIS,
 14 #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 15 #   See the License for the specific language governing permissions and
 16 #   limitations under the License.
 17 #
 18 
 19 template '/etc/default/mesos' do
 20   source 'mesos.erb'
 21   variables config: node[:mesos][:common]
 22   action :create
 23 end
 24 
 25 remote_file "Copy mesos-init-wrapper file" do
 26   path "/usr/bin/mesos-init-wrapper"
 27   source "file:///var/chef/cache/cookbooks/mesos/templates/default/mesos-init-wrapper"
 28 end
 29 
 30 #
 31 # Start mesos-slave service
 32 #
 33 
 34 execute "Starting mesos-slave" do
 35   command "nohup /usr/bin/mesos-init-wrapper slave &"
 36 end
 37 
 38 #
 39 # Install worker-specific packages
 40 #
 41 
 42 %w( chronos ).each do |pkg|
 43   yum_package pkg do
 44     action :install
 45   end
 46 end
 47 
 48 execute "Starting Chronos" do
 49   command "nohup /usr/local/bin/chronos &"
 50 end

There are several other files within the Mesos cookbook directory structure, including templates and attributes that are required for the deployment to work. All of the necessary files will be available in a day or so on the site and on GitHub.

Final steps

The last few things you will need to do include uploading all the cookbooks to the Chef server, restarting tomcat on the BDE management server to see the spec.json file and creating the cluster. For my deployments, I generally use the BDE CLI which is installed on the management server:

# java -jar /opt/serengeti/cli/serengeti-cli-2.1.0.jar 
=================================================
*   _____                                 _   _  ** / ____|  ___ _ __ ___ _ __   __ _  ___| |_(_) ** \____ \ / _ \ '__/ _ \ '_ \ / _` |/ _ \ __| | **  ____) |  __/ | |  __/ | | | (_| |  __/ |_| | ** |_____/ \___|_|  \___|_| |_|\__, |\___|\__|_| **                             |___/             **                                               *
=================================================
Version: 2.1.0
Welcome to Serengeti CLI
serengeti>connect --host 10.55.204.80:8443
Enter the username: administrator@vsphere.local
Enter the password: ************
Connected
serengeti>cluster list
serengeti>cluster create --name mesosphere_test --specFile /opt/serengeti/www/specs/Mesos/spec.json --networkName defaultNetwork --password yes
Hint: Password are from 8 to 128 characters, and can include alphanumeric characters ([0-9, a-z, A-Z]) and the following special characters: _, @, #, $, %, ^, &, *
Enter the password: **********
Confirm the password: **********

Upon completion of the deployment, you will have a working Mesosphere cluster with Apache Mesos, Mesosphere Marathon and Chronos.

Mesos Cluster within BDE UI

Mesos-BDE-UI

Mesos Web UI

Mesos-Web-UI

Mesosphere Marathon Web UI

Marathon-UI-1

Marathon-UI-2

Check back in a couple days for links to all the code so that you can try this out yourself!

Acknowlegements: I want to say thank you to Andrew Nelson (@vmwnelson) for letting me bounce ideas off of him constantly. To Ken Sipe (@kensipe) for getting me excited about Mesos during his presentation to us at Adobe. To Frans Van Rooyen (@jfvanrooyen) for helping me daily and constantly challenging me.

Finally, there is a lot of great open source code out there for Mesos and the BDE team has done a great job of making all of the Serengeti code available on GitHub for use. Go check it out!