Multiple Storage I/O Profiles in Apache Mesos & Mesosphere Marathon

Apache Mesos was initially created in order to support multiple frameworks within a single cluster. As I have used Mesos over the past six months it has worked amazing for segmenting multiple frameworks from one another, while consolidating the CPU and memory resources provided to the cluster into a single resource pool. Recently I began to ponder how to accommodate for multiple frameworks requiring differing storage I/O profiles. When you consider how different workloads are within an enterprise environment, being able to support storage I/O profiles within Apache Mesos is an important feature. Unfortunately a cursory Google search did not result in anything definitive around this subject.

After a bit of research and going through the white paper again, I discovered the functionality already exists within Apache Mesos and Mesosphere Marathon — it is just a matter of implementing it. The Placement Constraints feature within Mesosphere Marathon can be used to launch frameworks to Mesos slaves with specific storage profiles. An attribute on each Mesos slave node can then be tailored to specify what type of backend storage profile is available. The attribute can be specified in a file on each Mesos slave in the /etc/mesos-slave/attributes/ directory.

For example, let’s say you have a Hadoop framework that requires a high I/O storage profile and a NGiNX framework that can get by on slightly less-performant storage. In addition, your private cloud environment has multiple storage backend devices available to it — VMware VSAN and EMC VMAX 40K arrays. If you assign the VSAN cluster to “Storage Profile 1” and the VMAX arrays to “Storage Profile 2”, they could each exist within the same Mesos cluster. The figure below shows how two different frameworks with differing storage I/O needs would be placed on Mesos slaves meeting their requirements.



In order to provide this feature within a large-scale environment, being able to standardize and automate the deployment and configuration of the enhanced Apache Mesos cluster is key. I am already using VMware Big Data Extensions inside my private cloud environments, so naturally I looked to it first. The BDE framework allows for multiple datastores to be specified and each datastore can specify what type of storage it is — shared or local. A new Mesos cluster definition file can then be created to offer up two different node groups — one for shared storage and one for local storage. Corresponding entries through the BDE CLI or the vCenter Web Client interface can then be made to support two different types of storage.

The final piece then is creating an attribute that is assigned to each Mesos slave based on the type of storage it is deployed on. This can be accomplished by modifying the Chef cookbook for Mesos with with an entry in the /etc/mesos-slave/attributes directory. At which time, a new Mesos cluster can be deployed that has different storage I/O capabilities on the slaves.

Being able to allow certain frameworks to be placed on different physical or virtual machine nodes that present different storage characteristics is necessary in a multi-tenant, large-scale private cloud environment. Having the ability to deploy and scale the nodes in a standard, automated fashion is even more important. Fortunately the framework VMware Big Data Extensions offers is more than capable of providing the necessary tools for a cloud architect to support Apache Mesos with differing framework workloads.

Upgrading Apache Mesos & Marathon on VMware Big Data Extensions


There are a couple new versions for both Apache Mesos and Mesosphere Marathon. Since one of the great things about using VMware Big Data Extensions to handle the deployment and management of clusters is the ability to ensure control of the packages and versions within the SDLC. There were a few changes to be made within Chef to support Mesos v0.22.0 and Marathon 0.8.0* within the Big Data Extensions management server.

First thing, you need to grab the new RPM files and place them into your management server under /opt/serengeti/www/yum/repos/mesos/0.21.0 directory. For my own installation, I created a new directory called mesos/0.22.0 and copied the packages that had not been updated into it — but that decision is up to you. Once the files are there, you can choose whether to increment the version number in /opt/serengeti/www/distros/manifest file:

44   {
45     "name": "mesos",
46     "vendor": "MESOS",
47     "version": "0.22.0",
48     "packages": [
49       {
50         "package_repos": [
51           "https://bde.localdomain/yum/mesos.repo"
52         ],
53         "roles": [
54           "zookeeper",
55           "mesos_master",
56           "mesos_slave",
57           "mesos_docker",
58           "mesos_chronos",
59           "mesos_marathon"
60         ]
61       }
62     ]
63   },

If you choose to update the manifest file, make sure you restart tomcat on the BDE management server. You normally would be done at this point, but there were changes in where Mesos and Marathon install two files that we’ll need to account for. The old files were in subdirectories under /usr/local and are now placed in subdirectories of /usr.


  9 exec /usr/bin/marathon


 78 template '/usr/etc/mesos/' do
 79   source ''
 80   variables(
 81     zookeeper_server_list: zk_server_list,
 82     zookeeper_port: zk_port,
 83     zookeeper_path: zk_path,
 84     logs_dir: node['mesos']['common']['logs'],
 85     work_dir: node['mesos']['slave']['work_dir'],
 86     isolation: node['mesos']['isolation'],
 87   )
 88   notifies :run, 'bash[restart-mesos-slave]', :delayed
 89 end

Update the files in the Chef server by running ‘knife cookbook upload -a‘ and you are all set.

Go forth and deploy your updated Apache Mesos 0.22.0 and Marathon 0.8.0 clusters!

*Marathon 0.8.1 has been released but there has not been a RPM file created by Mesosphere yet.

HAProxy support for Mesos in vSphere Big Data Extensions

I realized late last night the current vSphere Big Data Extensions fling does not have HAProxy built into it for the Mesos cluster deployments. After a bit of reading and testing new pieces inside the Chef recipes, I have added support so that HAProxy is running on all of the Mesos nodes. The first thing is to add the HAProxy package to the /opt/serengeti/chef/cookbooks/mesos/recipes/install.rb file:

 72   %w( unzip libcurl haproxy ).each do |pkg|
 73     yum_package pkg do
 74       action :install
 75     end
 76   end

There is also a script that Mesosphere provides to modify the HAProxy configuration file and reload the rules when changes occur. You can find instructions on the file and how to incorporate it on the Mesosphere page.

Note: I had to edit ‘sudo’ out of the lines inside the script in order for Chef to execute it properly.

After copying the file haproxy-marathon-bridge into my Chef server, I added the following code to the same install.rb file to get things all setup and configured properly:

 82   directory "/etc/haproxy-marathon-bridge" do
 83     owner 'root'
 84     group 'root'
 85     mode '0755'
 86     action :create
 87   end
 89   template '/usr/local/bin/haproxy-marathon-bridge' do
 90     source 'haproxy-marathon-bridge.erb'
 91     action :create
 92   end
 94   master_ips = mesos_masters_ip
 95   slave_ips = mesos_slaves_ip
 97   all_ips = master_ips
 98   all_ips += slave_ips
100   template '/etc/haproxy-marathon-bridge/marathons' do
101     source 'marathons.erb'
102     variables(
103       haproxy_server_list: all_ips
104     )
105     action :create
106   end
108   execute 'configure haproxy' do
109     command 'chkconfig haproxy on; service haproxy start'
110   end
112   execute 'setup haproxy-marathon-bridge' do
113     command 'chmod 755 /usr/local/bin/haproxy-marathon-bridge; /usr/local/bin/haproxy-marathon-bridge install_cronjob'
114   end

There is also a bit of supporting code needed for lines 94-98 above that were added to /opt/serengeti/chef/cookbooks/mesos/libraries/default.rb:

  1 module Mesosphere
  3   def mesos_masters_ip
  4     servers = all_providers_fqdn_for_role("mesos_master")
  5"Mesos master nodes in cluster #{node[:cluster_name]} are: #{servers.inspect}")
  6     servers
  7   end
  9   def mesos_slaves_ip
 10     servers = all_providers_fqdn_for_role("mesos_slave")
 11"Mesos slave nodes in cluster #{node[:cluster_name]} are: #{servers.inspect}")
 12     servers
 13   end
 15 end
 17 class Chef::Recipe; include Mesosphere; end

The last thing needed is a new template file for the /etc/haproxy-marathon-bridge/marathons file that is needed by the script provided by Mesosphere. I created the file /opt/serengeti/chef/cookbooks/mesos/templates/default/marathons.erb:

  1 # Configuration file for haproxy-marathon-bridge script
  2 <%
  3   ha_url_list = []
  4   @haproxy_server_list.each do |ha_server|
  5     ha_url_list << "#{ha_server}"
  6   end
  7 %>
  8 <%= ha_url_list.join(":8080\n") + ":8080" %>

At this point, all of the modifications can be uploaded to the Chef server with the command knife cookbook upload -a and a new cluster can be deployed with HAProxy support.

After deploying a nginx workload, you scale it out and check the /etc/haproxy/haproxy.cfg file on a master node and see entries like:

[root@hadoopvm388 haproxy]# cat haproxy.cfg global
  log local0
  log local1 notice
  maxconn 4096
  log            global
  retries             3
  maxconn          2000
  timeout connect  5000
  timeout client  50000
  timeout server  50000
listen stats
  mode http
  stats enable
  stats auth admin:admin
listen nginx-80
  mode tcp
  option tcplog
  balance leastconn
  server nginx-10 hadoopvm382.localdomain:31000 check
  server nginx-9 hadoopvm390.localdomain:31000 check
  server nginx-8 hadoopvm387.localdomain:31000 check
  server nginx-7 hadoopvm389.localdomain:31000 check
  server nginx-6 hadoopvm386.localdomain:31000 check
  server nginx-5 hadoopvm383.localdomain:31000 check
  server nginx-4 hadoopvm378.localdomain:31001 check
  server nginx-3 hadoopvm381.localdomain:31000 check
  server nginx-2 hadoopvm385.localdomain:31000 check
  server nginx-1 hadoopvm378.localdomain:31000 check


VMware releases new Big Data Extensions fling!

vmware-sliderHot on the heels of my recent posts, and that from Andrew Nelson, the VMware Big Data Extensions team has released an official fling that extends the functionality to include Mesos, Marathon, Chronos, Docker and Kubernetes!

From the site:

“Big Data Extensions can be easily extended to deploy and manage all kinds of distributed or non-distributed applications. This release of the BDE-SE Fling adds support for deploying Mesos (with Chronos and Marathon) as well as Kubernetes clusters in addition to the Hadoop and HBase clusters.

Big Data Extensions simplifies the cluster deployment and provisioning process, gives you a real time view of the running services and the status of their virtual hosts, provides a central place from which to manage and monitor your clusters, and incorporates a broad range of tools to help you optimize cluster performance and utilization.

Big Data Extensions provides the following features:

  • Fast deployment, management, and scaling of Hadoop, Mesos and Kubernetes clusters. Big Data Extensions enable rapid deployment of Hadoop, Mesos and Kubernetes clusters on VMware vSphere. You can also quickly manage, scale out clusters, and scale up/down nodes subsequently.

  • Support for Docker. The Big Data Extensions for vSphere Standard Edition Fling includes support for Docker with Mesos, Marathon, Chronos, and Kubernetes.

  • Graphical User Interface Simplifies Management Tasks. The Big Data Extensions plug-in for vSphere, a graphical user interface integrated with vSphere Web Client, lets you easily perform common infrastructure and cluster management administrative tasks.

  • All-in-one solution. Big Data Extensions ships with installation package and configuration scripts for Apache Bigtop 0.8.0, Kubernetes 0.5.4, Mesos 0.21.0, Chronos 2.3.0 and Marathon 0.7.5. You can create and manage Hadoop, Mesos, and Kubernetes clusters out of box. You can also add new versions of these softwares into Big Data Extensions Server and create the cluster.”

Head over to the Flings page at VMware and download the latest to see how it all works! Great job by the BDE engineering team!

BDE + Mesosphere cluster code on GitHub

I have uploaded the necessary files to begin including the option for deploying a Mesosphere cluster with VMware Big Data Extensions v2.1. You can download the tarball or clone the repo via the following link:

As I begin work and provide further extensions for other clustering technologies, I will make them available via GitHub as well. To include this in your deployment, extract it directly into the /opt/serengeti folder — although be aware it will replace the default map and default manifest files as well. After the files are extracted (as user serengeti), simply run two commands on the BDE management server:

# knife cookbook upload -a
# service tomcat restart

If you have any questions, feel free to reach out to me over Twitter.