Tag: Mesosphere


Apache Mesos Chef Recipes

The recent update to v2.3.1 of VMware Big Data Extensions required a few updates on the management server to enable the proper deployment of Apache Mesos, Mesosphere Marathon and Docker. In order to have BDE deploy the latest code, the Apache Mesos Chef recipes needed a few updates. I prefer to use the Mesosphere CentOS repo for installing the packages, as it allows for keeping the clusters up-to-date with the proper packages. In order to do so, the following file should be created in /opt/serengeti/www/yum.


name=Mesosphere Packages for EL 6 - $basearch

name=Mesosphere Packages for EL 6 - noarch

name=Mesosphere Packages for EL 6 - $basearch - Source

name=Cloudera's Distribution for Hadoop, Version 4
gpgkey =  http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera
gpgcheck = 1

The file adds an extra repo for Zookeeper that Cloudier provides. These repos were necessary in order for the updated Chef recipes to properly install those packages. The remaining changes are strictly within the Chef recipes on the BDE management server.


 69 when 'rhel', 'centos', 'amazon', 'scientific'
 70   %w( unzip libcurl ).each do |pkg|
 71     yum_package pkg do
 72       action :install
 73     end
 74   end
 76   template "/etc/pki/rpm-gpg/RPM-GPG-KEY-mesosphere" do
 77     source "RPM-GPG-KEY-mesosphere.erb"
 78   end
 80   package 'mesos'
 81   package 'chronos' if node.role?('mesos_chronos')
 82   package 'marathon' if node.role?('mesos_marathon')
 83 end

100 if distro == 'debian'
101   bash 'reload-configuration-debian' do
102     user 'root'
103     code <<-EOH
104     update-rc.d -f mesos-master remove
105     EOH
106     not_if { ::File.exist? '/usr/sbin/mesos-master' }
107   end
108 else
109   bash 'reload-configuration' do
110     user 'root'
111     code <<-EOH
112     initctl reload-configuration
113     EOH
114     not_if { ::File.exist? '/usr/sbin/mesos-master' }
115   end
116 end


A colleague pointed out the addition for the RPM-GPG-KEY-mesosphere.erb file was not as clear as it should have been. Make sure you add lines 76-78 AND copy the GPG key into /opt/serengeti/chef/cookbooks/mesos/templates/default with a .erb file extension.


Several of the template files have an old path for executables that need to be updated. The Chronos configuration file needs the following change.

  9 exec /usr/bin/chronos


  6 exec /usr/bin/mesos-init-wrapper slave


  9 exec /usr/bin/marathon

The final step was to update the Chef server so that the changes will take effect by executing the knife cookbook upload -a command. When all of the outlined steps have been completed, the Apache Mesos Chef recipes on the Big Data Extensions management server will be all set to deploy the latest code for Apache Mesos, Mesosphere Marathon, Chronos and Docker on CentOS.

There is a series of posts coming to show how to tie VMware NSX together with Apache Mesos, Apache Hadoop and VMware Big Data Extensions.


Read More

The VMware Big Data Extensions v2.2 release included the cookbooks for Apache Mesos and Kubernetes from the Fling released this past spring. However, those cookbooks are not exposed when you deploy the new version. Fortunately, unlocking them only takes a few minutes before they can be made available! I will cover exactly what is needed in order to begin using these Cloud Native App cluster deployments below.

If you jump onto your v2.2 management server and look in the /opt/serengeti/chef/cookbooks directory, you will see all of the Cloud Native App additions.


A quick look to be sure the Chef roles are still defined tells us that they are.



They even did us the favor of including the JSON spec files in the /opt/serengeti/www/specs/Ironfan directory.


The missing pieces are the entries in the /opt/serengeti/www/specs/map and /opt/serengeti/www/distros/manifest files. Those are rather easy to copy out of the VMware Fling itself or re-create manually. If you want to edit the files yourself, here is what needs to be added to the files.


  "vendor" : "Kubernetes",
  "version" : "^(\\d)+(\\.\\w+)*",
  "type" : "Basic Kubernetes Cluster",
  "appManager" : "Default",
  "path" : "Ironfan/kubernetes/basic/spec.json"
  "vendor" : "Mesos",
  "version" : "^(\\d)+(\\.\\w+)*",
  "type" : "Basic Mesos Cluster",
  "appManager" : "Default",
  "path" : "Ironfan/mesos/basic/spec.json"


  "name" : "kubernetes",
  "vendor" : "KUBERNETES",
  "version" : "0.5.4",
  "packages" : [
      "tarball": "kubernetes/kubernetes-0.5.4.tar.gz",
      "roles": [
  "name" : "mesos",
  "vendor" : "MESOS",
  "version" : "0.21.0",
  "packages" : [
      "package_repos": [ ""],
      "roles" : [

The repos built into the Fling are not present (unfortunately) on the management server. This was the only tedious portion of the entire process. The easiest method is to grab the files out of an existing BDE Fling management server and copy them into the new one. The other option is find the latest RPMs on the Internet and add them to the management server manually. In either case, you’ll need to run the CentOS syntax for creating the repository.

Create local repo for Apache Mesos

# su - serengeti
$ cd /opt/serengeti/www/yum
$ vim mesos.repo
name=Apache Mesos

$ mkdir -p repos/mesos/current/RPMS
$ cd repos/mesos/current

The Fling included the following files:
- bigtop-utils-
- chronos-2.3.0-0.1.20141121000021.x86_64.rpm
- docker-io-1.3.1-2.el6.x86_64.rpm
- marathon-0.7.5-1.0.x86_64.rpm
- mesos-0.21.0-1.0.centos65.x86_64.rpm
- subversion-1.6.11-10.el6_5.x86_64.rpm
- zookeeper-
- zookeeper-server-

$ createrepo .

A restart of Tomcat is all that is needed and then you will be able to start deploying Apache Mesos and Kubernetes clusters through BDE v2.2.

If you want to take advantage of the Instant Clone functionality, you will need to be running vSphere 6.0 and BDE v2.2. There are also a couple adjustments to the /opt/serengeti/conf/serengeti.properties files that will be need to be made. I will be going over those in a future post discussing how to use the Photon OS as the template for BDE to deploy.

Read More

Ever since the VMware Photon Technical Preview was made available, I have wanted to install Apache Mesos on a node. The Photon OS is a very minimal Linux installation and so my initial attempts to work through the process led me down the old-fashioned rabbit hole of manually compiling packages. It was very reminiscent of the later 1990’s and using Debian installed via floppy disk. I finally found the time to work my way down and back out of the rabbit hole and have been able to successfully get Mesos to install and run on Photon! This is a good first step towards building a Photon template to be used inside Big Data Extensions for deploying Cloud Native Apps with Mesos|Marathon|Chronos or Kubernetes.

The lab environment I used is running vSphere 5.5 and consists of a small set of nested ESXi hypervisors. I am not going to cover install Photon on a VM, but just be sure you have one that was installed with the full OS — not the minimal installation. After the Photon VM is configured to communicate with the Internet, you can follow these instructions to get Apache Mesos installed. The end result of the guide will be a working Mesos node running on Photon that can launch Docker containers.

Apache Mesos on Photon

Be sure to follow it in the order listed as the ordering of the packages is important. Also have a Photon VM with at least 3GB of Memory allocated to it for the compile processes.


Missing prerequisites
APR Library
# wget http://apache.claz.org//apr/apr-1.5.2.tar.gz
# tar zxvf apr-1.5.2.tar.gz
# cd apr-1.5.2
# ./configure —prefix=/usr/local/lib/apr
# make
# make test
# make install

APR-UTIL Library
# wget http://apache.claz.org//apr/apr-util-1.5.4.tar.gz
# tar zxvf apr-util-1.5.4.tar.gz
# cd apr-util-1.5.4
# ./configure —prefix=/usr/local/lib/apr —with-apr=/usr/local/lib/apr
# make
# make install

# wget http://apache.osuosl.org/subversion/subversion-1.8.13.tar.gz
# tar zxvf subversion-1.8.13.tar.gz
# cd subversion-1.8.13
# ./configure —prefix=/usr/local/lib/subversion —with-apr=/usr/local/lib/apr —with-apr-util=/usr/local/lib/apr
# make
# make install

OpenJDK Java
Download the Java JDK source tarball from the Oracle website (http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.tar.gz)
# tar zxvf jdk-7u79-linux-x64.tar.gz
# mv jdk1.7.0_79 /usr/local/java
# echo JAVA_HOME=/usr/local/java >> /etc/environment
# source /etc/environment

Apache Maven Library
# wget http://apache.mirrors.ionfish.org//ant/binaries/apache-ant-1.9.5-bin.tar.gz
# tar zxvf apache-ant-1.9.5-bin.tar.gz
# mv apache-ant-1.9.5 /usr/local
# ln -s /usr/local/apache-ant-1.9.5 /usr/local/apache-ant
# wget http://apache.cs.utah.edu/maven/maven-3/3.3.3/source/apache-maven-3.3.3-src.tar.gz
# tar zxvf apache-maven-3.3.3-src.tar.gz
# cd apache-maven-3.3.3
# /usr/local/apache-ant/bin/ant -Dmaven.home=“/usr/local/maven-3.3.3"
# echo MAVEN_HOME=/usr/local/maven-3.3.3 >> /etc/environment
# export /etc/environment

Install Apache Mesos
# wget http://www.apache.org/dist/mesos/0.22.1/mesos-0.22.1.tar.gz
# tar zxvf mesos-0.22.1.tar.gz
# cd mesos-0.22.1
# mkdir build
# cd build
# ../configure --prefix=/usr/local/mesos —with-apr=/usr/local/lib/apr —with-svn=/usr/local/lib/subversion
# make
# make check
# make install
After Apache Mesos is installed, you can start both the master and slave processes on the node to run a quick test.
# /usr/local/mesos/bin/mesos-master.sh —ip= —work_dir=/var/lib/mesos
# /usr/local/mesos/bin/mesos-slave.sh —master=
Afterwards, open a web browser and point to to and you will see the Apache Mesos interface. The next step will be to deploy multiple Photon nodes and configure them to be a part of a single cluster. I will likely orchestrate all of this through Chef next and incorporate it into the Big Data Extensions framework.
There is always more to do, but I am glad to have gotten this working and sharing it with everyone. Reach out to me on Twitter if you have questions.

Read More

Apache Mesos was initially created in order to support multiple frameworks within a single cluster. As I have used Mesos over the past six months it has worked amazing for segmenting multiple frameworks from one another, while consolidating the CPU and memory resources provided to the cluster into a single resource pool. Recently I began to ponder how to accommodate for multiple frameworks requiring differing storage I/O profiles. When you consider how different workloads are within an enterprise environment, being able to support storage I/O profiles within Apache Mesos is an important feature. Unfortunately a cursory Google search did not result in anything definitive around this subject.

After a bit of research and going through the white paper again, I discovered the functionality already exists within Apache Mesos and Mesosphere Marathon — it is just a matter of implementing it. The Placement Constraints feature within Mesosphere Marathon can be used to launch frameworks to Mesos slaves with specific storage profiles. An attribute on each Mesos slave node can then be tailored to specify what type of backend storage profile is available. The attribute can be specified in a file on each Mesos slave in the /etc/mesos-slave/attributes/ directory.

For example, let’s say you have a Hadoop framework that requires a high I/O storage profile and a NGiNX framework that can get by on slightly less-performant storage. In addition, your private cloud environment has multiple storage backend devices available to it — VMware VSAN and EMC VMAX 40K arrays. If you assign the VSAN cluster to “Storage Profile 1” and the VMAX arrays to “Storage Profile 2”, they could each exist within the same Mesos cluster. The figure below shows how two different frameworks with differing storage I/O needs would be placed on Mesos slaves meeting their requirements.



In order to provide this feature within a large-scale environment, being able to standardize and automate the deployment and configuration of the enhanced Apache Mesos cluster is key. I am already using VMware Big Data Extensions inside my private cloud environments, so naturally I looked to it first. The BDE framework allows for multiple datastores to be specified and each datastore can specify what type of storage it is — shared or local. A new Mesos cluster definition file can then be created to offer up two different node groups — one for shared storage and one for local storage. Corresponding entries through the BDE CLI or the vCenter Web Client interface can then be made to support two different types of storage.

The final piece then is creating an attribute that is assigned to each Mesos slave based on the type of storage it is deployed on. This can be accomplished by modifying the Chef cookbook for Mesos with with an entry in the /etc/mesos-slave/attributes directory. At which time, a new Mesos cluster can be deployed that has different storage I/O capabilities on the slaves.

Being able to allow certain frameworks to be placed on different physical or virtual machine nodes that present different storage characteristics is necessary in a multi-tenant, large-scale private cloud environment. Having the ability to deploy and scale the nodes in a standard, automated fashion is even more important. Fortunately the framework VMware Big Data Extensions offers is more than capable of providing the necessary tools for a cloud architect to support Apache Mesos with differing framework workloads.

Read More


There are a couple new versions for both Apache Mesos and Mesosphere Marathon. Since one of the great things about using VMware Big Data Extensions to handle the deployment and management of clusters is the ability to ensure control of the packages and versions within the SDLC. There were a few changes to be made within Chef to support Mesos v0.22.0 and Marathon 0.8.0* within the Big Data Extensions management server.

First thing, you need to grab the new RPM files and place them into your management server under /opt/serengeti/www/yum/repos/mesos/0.21.0 directory. For my own installation, I created a new directory called mesos/0.22.0 and copied the packages that had not been updated into it — but that decision is up to you. Once the files are there, you can choose whether to increment the version number in /opt/serengeti/www/distros/manifest file:

44   {
45     "name": "mesos",
46     "vendor": "MESOS",
47     "version": "0.22.0",
48     "packages": [
49       {
50         "package_repos": [
51           "https://bde.localdomain/yum/mesos.repo"
52         ],
53         "roles": [
54           "zookeeper",
55           "mesos_master",
56           "mesos_slave",
57           "mesos_docker",
58           "mesos_chronos",
59           "mesos_marathon"
60         ]
61       }
62     ]
63   },

If you choose to update the manifest file, make sure you restart tomcat on the BDE management server. You normally would be done at this point, but there were changes in where Mesos and Marathon install two files that we’ll need to account for. The old files were in subdirectories under /usr/local and are now placed in subdirectories of /usr.


  9 exec /usr/bin/marathon


 78 template '/usr/etc/mesos/mesos-slave-env.sh' do
 79   source 'mesos-slave-env.sh.erb'
 80   variables(
 81     zookeeper_server_list: zk_server_list,
 82     zookeeper_port: zk_port,
 83     zookeeper_path: zk_path,
 84     logs_dir: node['mesos']['common']['logs'],
 85     work_dir: node['mesos']['slave']['work_dir'],
 86     isolation: node['mesos']['isolation'],
 87   )
 88   notifies :run, 'bash[restart-mesos-slave]', :delayed
 89 end

Update the files in the Chef server by running ‘knife cookbook upload -a‘ and you are all set.

Go forth and deploy your updated Apache Mesos 0.22.0 and Marathon 0.8.0 clusters!

*Marathon 0.8.1 has been released but there has not been a RPM file created by Mesosphere yet.

Read More