HAProxy support for Mesos in vSphere Big Data Extensions

I realized late last night the current vSphere Big Data Extensions fling does not have HAProxy built into it for the Mesos cluster deployments. After a bit of reading and testing new pieces inside the Chef recipes, I have added support so that HAProxy is running on all of the Mesos nodes. The first thing is to add the HAProxy package to the /opt/serengeti/chef/cookbooks/mesos/recipes/install.rb file:

 72   %w( unzip libcurl haproxy ).each do |pkg|
 73     yum_package pkg do
 74       action :install
 75     end
 76   end

There is also a script that Mesosphere provides to modify the HAProxy configuration file and reload the rules when changes occur. You can find instructions on the file and how to incorporate it on the Mesosphere page.

Note: I had to edit ‘sudo’ out of the lines inside the script in order for Chef to execute it properly.

After copying the file haproxy-marathon-bridge into my Chef server, I added the following code to the same install.rb file to get things all setup and configured properly:

 82   directory "/etc/haproxy-marathon-bridge" do
 83     owner 'root'
 84     group 'root'
 85     mode '0755'
 86     action :create
 87   end
 88 
 89   template '/usr/local/bin/haproxy-marathon-bridge' do
 90     source 'haproxy-marathon-bridge.erb'
 91     action :create
 92   end
 93 
 94   master_ips = mesos_masters_ip
 95   slave_ips = mesos_slaves_ip
 96 
 97   all_ips = master_ips
 98   all_ips += slave_ips
 99   
100   template '/etc/haproxy-marathon-bridge/marathons' do
101     source 'marathons.erb'
102     variables(
103       haproxy_server_list: all_ips
104     )
105     action :create
106   end
107   
108   execute 'configure haproxy' do
109     command 'chkconfig haproxy on; service haproxy start'
110   end
111   
112   execute 'setup haproxy-marathon-bridge' do
113     command 'chmod 755 /usr/local/bin/haproxy-marathon-bridge; /usr/local/bin/haproxy-marathon-bridge install_cronjob'
114   end

There is also a bit of supporting code needed for lines 94-98 above that were added to /opt/serengeti/chef/cookbooks/mesos/libraries/default.rb:

  1 module Mesosphere
  2 
  3   def mesos_masters_ip
  4     servers = all_providers_fqdn_for_role("mesos_master")
  5     Chef::Log.info("Mesos master nodes in cluster #{node[:cluster_name]} are: #{servers.inspect}")
  6     servers
  7   end
  8 
  9   def mesos_slaves_ip
 10     servers = all_providers_fqdn_for_role("mesos_slave")
 11     Chef::Log.info("Mesos slave nodes in cluster #{node[:cluster_name]} are: #{servers.inspect}")
 12     servers
 13   end
 14 
 15 end
 16 
 17 class Chef::Recipe; include Mesosphere; end

The last thing needed is a new template file for the /etc/haproxy-marathon-bridge/marathons file that is needed by the script provided by Mesosphere. I created the file /opt/serengeti/chef/cookbooks/mesos/templates/default/marathons.erb:

  1 # Configuration file for haproxy-marathon-bridge script
  2 <%
  3   ha_url_list = []
  4   @haproxy_server_list.each do |ha_server|
  5     ha_url_list << "#{ha_server}"
  6   end
  7 %>
  8 <%= ha_url_list.join(":8080\n") + ":8080" %>

At this point, all of the modifications can be uploaded to the Chef server with the command knife cookbook upload -a and a new cluster can be deployed with HAProxy support.

After deploying a nginx workload, you scale it out and check the /etc/haproxy/haproxy.cfg file on a master node and see entries like:

[root@hadoopvm388 haproxy]# cat haproxy.cfg global
  daemon
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096
defaults
  log            global
  retries             3
  maxconn          2000
  timeout connect  5000
  timeout client  50000
  timeout server  50000
listen stats
  bind 127.0.0.1:9090
  balance
  mode http
  stats enable
  stats auth admin:admin
listen nginx-80
  bind 0.0.0.0:80
  mode tcp
  option tcplog
  balance leastconn
  server nginx-10 hadoopvm382.localdomain:31000 check
  server nginx-9 hadoopvm390.localdomain:31000 check
  server nginx-8 hadoopvm387.localdomain:31000 check
  server nginx-7 hadoopvm389.localdomain:31000 check
  server nginx-6 hadoopvm386.localdomain:31000 check
  server nginx-5 hadoopvm383.localdomain:31000 check
  server nginx-4 hadoopvm378.localdomain:31001 check
  server nginx-3 hadoopvm381.localdomain:31000 check
  server nginx-2 hadoopvm385.localdomain:31000 check
  server nginx-1 hadoopvm378.localdomain:31000 check

Enjoy!

BDE + Mesosphere cluster code on GitHub

I have uploaded the necessary files to begin including the option for deploying a Mesosphere cluster with VMware Big Data Extensions v2.1. You can download the tarball or clone the repo via the following link:

https://github.com/virtualelephant/vmware-bde-mesos

As I begin work and provide further extensions for other clustering technologies, I will make them available via GitHub as well. To include this in your deployment, extract it directly into the /opt/serengeti folder — although be aware it will replace the default map and default manifest files as well. After the files are extracted (as user serengeti), simply run two commands on the BDE management server:

# knife cookbook upload -a
# service tomcat restart

If you have any questions, feel free to reach out to me over Twitter.

Apache Mesos Clusters – Part 3

The post includes the final pieces necessary to get a Mesosphere stack deployed through Big Data Extensions within a VMware environment. I’ve included the Chef cookbooks and commands required for tying all of the pieces together for a cluster deployment. The wonderful thing about the framework is the extensibility — once I had Mesos deploying, it became very clear how simple it is to extend the framework even further — look for future posts.

The idea that you can now turn a large cluster of VMs into a single Mesos cluster for use by a product, engineering team or operations team opens up an entirely new world within our environments. This is a very exciting place to be investing time.

Chef Roles

Big Data Extensions uses role definitions within the framework, so the first step was to create a new role for Mesos. If you remember from Part 2, we defined the role in the JSON file and called it ‘mesos’.

The role files can be found in /opt/serengeti/chef/roles. I created the roles for both mesos_master and mesos_worker through the command line interface:

Continue reading “Apache Mesos Clusters – Part 3”

Apache Mesos Clusters – Part 2

Building Mesosphere & Apache Mesos into BDE:

After playing with Mesosphere in AWS for the week, getting familiar with the packages and the deployment process, the real work has begun — getting the Mesosphere stack (Apache Mesos, Apache Zookeeper,  Mesosphere Marathon, Chronos and HAProxy) deployed through VMware Big Data Extensions. Fortunately, BDE v2.1 has some example JSON cluster definition files that can be used for deploying different types of clusters and these are perfect for modification in this use-case.

The example files are located in the directory /opt/serengeti/samples. I used the basic_cluster.json file in the directory as the template. From there, I modified the file based on what the Mesosphere stack deployed in AWS, with some slight modifications. I chose to have a base Mesos cluster include 3 master nodes and 6 worker nodes. The master nodes are allocated with 2vCPU, 8GB RAM and 50GB of disk space. The worker nodes are allocated with 2vCPU, 8GB RAM and 100GB of disk space.

The remainder of the post will go through all the various pieces that are necessary to utilize the Big Data Extensions framework to offer the Mesosphere stack within a VMware virtual environment.

Continue reading “Apache Mesos Clusters – Part 2”

Apache Mesos Clusters – Part 1

I watched a webinar today from Ken Sipe (@kensipe) from Mesosphere on Mesos, Marathon and Chronos. The topics covered included how Mesos works, configuring and standup of a Mesos cluster in various public cloud offerings. If you are unfamiliar with Mesos, I would direct you to Mesosphere and the Apache Mesos Project.

The basic explanation of from the Apache Mesos Project page states:

Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.

Think turning an entire datacenter of compute resources into a single pool to be consumed. Instead of carving out individual pieces of compute, Mesos handles the scheduling and helps you scale an application across all of the resources available to it.

So how quickly can you deploy a cluster and begin using Mesos?

Continue reading “Apache Mesos Clusters – Part 1”