Category: Mesosphere Marathon

 

Apache Mesos Chef Recipes

The recent update to v2.3.1 of VMware Big Data Extensions required a few updates on the management server to enable the proper deployment of Apache Mesos, Mesosphere Marathon and Docker. In order to have BDE deploy the latest code, the Apache Mesos Chef recipes needed a few updates. I prefer to use the Mesosphere CentOS repo for installing the packages, as it allows for keeping the clusters up-to-date with the proper packages. In order to do so, the following file should be created in /opt/serengeti/www/yum.

mesos.repo

[mesosphere]
name=Mesosphere Packages for EL 6 - $basearch
baseurl=http://repos.mesosphere.io/el/6/$basearch/
enabled=1
gpgcheck=1
gpgkey=https://bde.local.domain/yum/RPM-GPG-KEY-mesosphere

[mesosphere-noarch]
name=Mesosphere Packages for EL 6 - noarch
baseurl=http://repos.mesosphere.io/el/6/noarch/
enabled=1
gpgcheck=1
gpgkey=https://bde.local.domain/yum/RPM-GPG-KEY-mesosphere

[mesosphere-source]
name=Mesosphere Packages for EL 6 - $basearch - Source
baseurl=http://repos.mesosphere.io/el/6/SRPMS/
enabled=0
gpgcheck=1
gpgkey=https://bde.local.domain/yum/RPM-GPG-KEY-mesosphere

[cloudera-cdh4]
name=Cloudera's Distribution for Hadoop, Version 4
baseurl=http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/4/
gpgkey =  http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera
gpgcheck = 1

The file adds an extra repo for Zookeeper that Cloudier provides. These repos were necessary in order for the updated Chef recipes to properly install those packages. The remaining changes are strictly within the Chef recipes on the BDE management server.

/opt/serengeti/chef/cookbooks/mesos/recipes/install.rb

 69 when 'rhel', 'centos', 'amazon', 'scientific'
 70   %w( unzip libcurl ).each do |pkg|
 71     yum_package pkg do
 72       action :install
 73     end
 74   end
 75 
 76   template "/etc/pki/rpm-gpg/RPM-GPG-KEY-mesosphere" do
 77     source "RPM-GPG-KEY-mesosphere.erb"
 78   end
 79 
 80   package 'mesos'
 81   package 'chronos' if node.role?('mesos_chronos')
 82   package 'marathon' if node.role?('mesos_marathon')
 83 end

100 if distro == 'debian'
101   bash 'reload-configuration-debian' do
102     user 'root'
103     code <<-EOH
104     update-rc.d -f mesos-master remove
105     EOH
106     not_if { ::File.exist? '/usr/sbin/mesos-master' }
107   end
108 else
109   bash 'reload-configuration' do
110     user 'root'
111     code <<-EOH
112     initctl reload-configuration
113     EOH
114     not_if { ::File.exist? '/usr/sbin/mesos-master' }
115   end
116 end

Update:

A colleague pointed out the addition for the RPM-GPG-KEY-mesosphere.erb file was not as clear as it should have been. Make sure you add lines 76-78 AND copy the GPG key into /opt/serengeti/chef/cookbooks/mesos/templates/default with a .erb file extension.

/opt/serengeti/chef/cookbooks/mesos/templates/default/chronos.conf.erb

Several of the template files have an old path for executables that need to be updated. The Chronos configuration file needs the following change.

  9 exec /usr/bin/chronos

/opt/serengeti/chef/cookbooks/mesos/templates/default/mesos-slave.conf.erb

  6 exec /usr/bin/mesos-init-wrapper slave

/opt/serengeti/chef/cookbooks/mesos/templates/default/marathon.conf.erb

  9 exec /usr/bin/marathon

The final step was to update the Chef server so that the changes will take effect by executing the knife cookbook upload -a command. When all of the outlined steps have been completed, the Apache Mesos Chef recipes on the Big Data Extensions management server will be all set to deploy the latest code for Apache Mesos, Mesosphere Marathon, Chronos and Docker on CentOS.

There is a series of posts coming to show how to tie VMware NSX together with Apache Mesos, Apache Hadoop and VMware Big Data Extensions.

Enjoy!

Read More

docker

I had a need this week for a quick-and-easy IO benchmarking tool and decided to create a Docker container to achieve my goals. The Docker container itself is rather simple and is available for use at your leisure. The Docker container is based off of Ubuntu latest, installs the FIO package and grabs a simple test file from my website for generating load and benchmarking results.

Dockerfile

  1 # FIO benchmark on Ubuntu:latest
  2 
  3 FROM ubuntu:latest
  4 MAINTAINER Chris Mutchler <chris@virtualelephant.com>
  5 
  6 RUN apt-get update
  7 RUN apt-get -y install fio wget
  8 RUN wget http://virtualelephant.com/wp-content/uploads/2015/10/threads.txt
  9 
 10 CMD [ "/bin/bash" ]

Once I had the Docker image uploaded to my repo, I created a simple JSON file for deploying the container into my Mesos cluster.

iobench.json

{
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "chrismutchler/iobench"
    }    
  },
  "id": "iobench",
  "instances": 1,
  "cpus": 0.25,
  "mem": 256,
  "uris": [],
  "cmd": "while true; do date; /usr/bin/fio /threads.txt; sleep 10; done"
}

As noted, the Docker container will continue to run the FIO test infinitely until the app is destroyed in Marathon. The app can also be scaled to run across multiple Apache Mesos nodes, each running the FIO test independently. Be careful when running any sort of load or benchmarking test in your environments, it may have adverse effects.

Launching the application in Marathon was simple enough from the command line.

# curl -A POST http://mesos.local.domain:8080/v2/apps -d @iobench.json -H "Content-Type: application/json"

I used the test to create a noisy neighbor issue in my environment to test out VMware Storage IO Control (SIOC) settings and it worked adequately in this role. Once the application had been launched, the results are available for downloading or viewing in the Mesos UI, by selecting ‘Sandbox’ and then the STDOUT log file. To understand what FIO is performing for the IO benchmark, please read the blog post by Ben Martin where I copied the FIO test file from.

stdout

--container="mesos-20151009-234723-2583865536-5050-2901-S1.c0052620-2322-4d12-aa49-1e3b24402f50" --docker="docker" --help="false" --initialize_driver_logging="true" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/20151009-234723-2583865536-5050-2901-S1/frameworks/20151009-234723-2583865536-5050-2901-0001/executors/iobench.cb93c618-6eea-11e5-9209-0050569a4da8/runs/c0052620-2322-4d12-aa49-1e3b24402f50" --stop_timeout="0ns"
--container="mesos-20151009-234723-2583865536-5050-2901-S1.c0052620-2322-4d12-aa49-1e3b24402f50" --docker="docker" --help="false" --initialize_driver_logging="true" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/20151009-234723-2583865536-5050-2901-S1/frameworks/20151009-234723-2583865536-5050-2901-0001/executors/iobench.cb93c618-6eea-11e5-9209-0050569a4da8/runs/c0052620-2322-4d12-aa49-1e3b24402f50" --stop_timeout="0ns"
Registered docker executor on dhcp2-157.local.domain
Starting task iobench.cb93c618-6eea-11e5-9209-0050569a4da8
Sat Oct 10 01:04:40 UTC 2015
bgwriter: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
queryA: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=mmap, iodepth=1
queryB: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=mmap, iodepth=1
bgupdater: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
fio-2.1.3
Starting 4 processes
bgwriter: Laying out IO file(s) (1 file(s) / 256MB)
queryA: Laying out IO file(s) (1 file(s) / 256MB)
queryB: Laying out IO file(s) (1 file(s) / 256MB)
bgupdater: Laying out IO file(s) (1 file(s) / 32MB)

bgwriter: (groupid=0, jobs=1): err= 0: pid=8: Sat Oct 10 01:05:32 2015
  write: io=262144KB, bw=11578KB/s, iops=2894, runt= 22641msec
    slat (usec): min=16, max=4713.1K, avg=334.93, stdev=20011.69
    clat (usec): min=9, max=4750.2K, avg=10711.15, stdev=112034.10
     lat (usec): min=75, max=4750.6K, avg=11048.59, stdev=113823.09
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    6],
     | 30.00th=[    6], 40.00th=[    7], 50.00th=[    7], 60.00th=[    7],
     | 70.00th=[    8], 80.00th=[    8], 90.00th=[    9], 95.00th=[    9],
     | 99.00th=[   16], 99.50th=[   31], 99.90th=[  947], 99.95th=[ 1582],
     | 99.99th=[ 4752]
    bw (KB  /s): min=    5, max=39208, per=100.00%, avg=15073.13, stdev=7536.85
    lat (usec) : 10=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
    lat (usec) : 1000=0.01%
    lat (msec) : 2=0.05%, 4=12.47%, 10=85.23%, 20=1.46%, 50=0.34%
    lat (msec) : 250=0.05%, 500=0.19%, 750=0.05%, 1000=0.05%, 2000=0.05%
    lat (msec) : >=2000=0.05%
  cpu          : usr=2.60%, sys=8.83%, ctx=126969, majf=0, minf=26
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=65536/d=0, short=r=0/w=0/d=0
queryA: (groupid=0, jobs=1): err= 0: pid=9: Sat Oct 10 01:05:32 2015
  read : io=262144KB, bw=12106KB/s, iops=3026, runt= 21654msec
    clat (usec): min=26, max=4713.2K, avg=322.55, stdev=20018.65
     lat (usec): min=26, max=4713.2K, avg=322.77, stdev=20018.66
    clat percentiles (usec):
     |  1.00th=[   31],  5.00th=[   71], 10.00th=[   77], 20.00th=[  115],
     | 30.00th=[  126], 40.00th=[  155], 50.00th=[  159], 60.00th=[  165],
     | 70.00th=[  183], 80.00th=[  207], 90.00th=[  253], 95.00th=[  314],
     | 99.00th=[  724], 99.50th=[ 1012], 99.90th=[ 2576], 99.95th=[ 4384],
     | 99.99th=[272384]
    bw (KB  /s): min=    7, max=24840, per=67.92%, avg=16686.43, stdev=7338.70
    lat (usec) : 50=4.04%, 100=11.27%, 250=74.04%, 500=8.29%, 750=1.42%
    lat (usec) : 1000=0.45%
    lat (msec) : 2=0.36%, 4=0.09%, 10=0.03%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2000=0.01%, >=2000=0.01%
  cpu          : usr=2.93%, sys=7.36%, ctx=131487, majf=65536, minf=31
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=65536/w=0/d=0, short=r=0/w=0/d=0
queryB: (groupid=0, jobs=1): err= 0: pid=10: Sat Oct 10 01:05:32 2015
  read : io=262144KB, bw=11910KB/s, iops=2977, runt= 22010msec
    clat (usec): min=27, max=4714.3K, avg=325.85, stdev=20020.01
     lat (usec): min=27, max=4714.3K, avg=326.11, stdev=20020.01
    clat percentiles (usec):
     |  1.00th=[   32],  5.00th=[   72], 10.00th=[   82], 20.00th=[  117],
     | 30.00th=[  139], 40.00th=[  155], 50.00th=[  159], 60.00th=[  169],
     | 70.00th=[  187], 80.00th=[  213], 90.00th=[  262], 95.00th=[  318],
     | 99.00th=[  700], 99.50th=[  980], 99.90th=[ 2608], 99.95th=[ 4768],
     | 99.99th=[272384]
    bw (KB  /s): min=    2, max=23384, per=65.75%, avg=16152.90, stdev=6882.55
    lat (usec) : 50=2.90%, 100=9.19%, 250=76.15%, 500=9.50%, 750=1.34%
    lat (usec) : 1000=0.43%
    lat (msec) : 2=0.32%, 4=0.09%, 10=0.03%, 20=0.01%, 50=0.01%
    lat (msec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
    lat (msec) : >=2000=0.01%
  cpu          : usr=3.81%, sys=6.74%, ctx=131493, majf=65536, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=65536/w=0/d=0, short=r=0/w=0/d=0
bgupdater: (groupid=0, jobs=1): err= 0: pid=11: Sat Oct 10 01:05:32 2015
  read : io=16416KB, bw=3611.1KB/s, iops=902, runt=  4545msec
    slat (usec): min=81, max=285023, avg=520.41, stdev=5536.65
    clat (usec): min=1, max=2841, avg= 5.40, stdev=80.88
     lat (usec): min=115, max=285031, avg=528.39, stdev=5537.20
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    2], 50.00th=[    2], 60.00th=[    2],
     | 70.00th=[    2], 80.00th=[    3], 90.00th=[    3], 95.00th=[    3],
     | 99.00th=[    3], 99.50th=[    6], 99.90th=[ 1448], 99.95th=[ 2384],
     | 99.99th=[ 2832]
    bw (KB  /s): min= 2096, max= 4472, per=14.60%, avg=3587.00, stdev=1065.00
  write: io=16352KB, bw=3597.9KB/s, iops=899, runt=  4545msec
    slat (usec): min=74, max=254089, avg=489.47, stdev=3971.19
    clat (usec): min=1, max=2233, avg= 5.13, stdev=66.52
     lat (usec): min=77, max=254096, avg=497.09, stdev=3971.57
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    2], 50.00th=[    2], 60.00th=[    2],
     | 70.00th=[    2], 80.00th=[    3], 90.00th=[    3], 95.00th=[    3],
     | 99.00th=[    3], 99.50th=[    5], 99.90th=[ 1256], 99.95th=[ 1816],
     | 99.99th=[ 2224]
    bw (KB  /s): min= 2024, max= 4504, per=28.58%, avg=3514.88, stdev=1085.81
    lat (usec) : 2=0.04%, 4=99.26%, 10=0.24%, 50=0.18%, 100=0.04%
    lat (usec) : 250=0.06%, 500=0.02%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.07%, 4=0.06%
  cpu          : usr=8.30%, sys=4.71%, ctx=16460, majf=0, minf=25
  IO depths    : 1=99.8%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=4104/w=4088/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=540704KB, aggrb=24566KB/s, minb=3611KB/s, maxb=12106KB/s, mint=4545msec, maxt=22010msec
  WRITE: io=278496KB, aggrb=12300KB/s, minb=3597KB/s, maxb=11578KB/s, mint=4545msec, maxt=22641msec

Disk stats (read/write):
    dm-2: ios=135188/67188, merge=0/0, ticks=39356/28285, in_queue=67640, util=97.96%, aggrios=135217/70245, aggrmerge=0/0, aggrticks=1492/4673, aggrin_queue=6169, aggrutil=15.06%
    dm-0: ios=135217/70245, merge=0/0, ticks=1492/4673, in_queue=6169, util=15.06%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
  loop1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
  loop0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

If you would like to grab the Docker image and use it for your own needs, you can pull it from Docker Hub:

# docker pull chrismutchler/iobench

One of the next little Docker containers I will be building is a web page traffic generator — I’ve used all sorts of traffic testing applications in the past, but being able to spin up a test inside a Docker container and then scale it across a large Mesos cluster will simplify things even more!

Enjoy!

Read More

photon-dfad9617

VMware Photon TP2 was released on August 27. The new version contains native support for running Mesos and therefore should have allowed the Photon OS to run as a Mesos slave immediately after installation. I would like to think my earlier blog post detailing how to deploy Mesos on-top of Photon influenced this functionality.

Download the ISO here.

After conversations with people involved in the project, the idea is for Photon to act only as a Mesos slave, with external Mesos masters and Zookeeper running on an Ubuntu/CentOS/Red Hat nodes. Logically the architecture of a Mesos cluster with Photon OS would look like the following.

 

mesos-photon-cluster

 

In order to deploy the cluster in this fashion, I wanted to find a method for automating as much of it as possible. Currently, one limitation with VMware Big Data Extensions is the single template VM limit. How awesome would it be if you could have multiple template VMs within the vApp and choose which template to deploy based on a pre-defined role? Definitely something to look into.

Regardless, working within the current limitations of BDE, I will describe in detail how I am now deploying Photon OS nodes into a Mesos cluster as automated as possible.

Configuring Big Data Extensions

I decided to create a new cluster map for a Mesos cluster that only deployed the Zookeeper and Mesos master nodes. The idea is similar to a Compute-only Hadoop or HDFS-only Hadoop cluster deployment through BDE. All that is required to accomplish this is a JSON file with the new cluster definition and an entry in the /opt/serengeti/www/specs/map file.

/opt/serengeti/www/specs/Ironfan/mesos/master/spec.json

  1 {
  2   "nodeGroups":[
  3     {
  4       "name": "Zookeeper",
  5       "roles": [
  6         "zookeeper"
  7       ],
  8       "groupType": "zookeeper",
  9       "instanceNum": "[3,3,3]",
 10       "instanceType": "[SMALL]",
 11       "cpuNum": "[1,1,64]",
 12       "memCapacityMB": "[7500,3748,min]",
 13       "storage": {
 14         "type": "[SHARED,LOCAL]",
 15         "sizeGB": "[2,2,min]"
 16       },
 17       "haFlag": "on"
 18     },
 19     {
 20       "name": "Master",
 21       "description": "The Mesos master node",
 22       "roles": [
 23         "mesos_master",
 24         "mesos_chronos",
 25         "mesos_marathon"
 26       ],
 27       "groupType": "master",
 28       "instanceNum": "[2,1,2]",
 29       "instanceType": "[MEDIUM,SMALL,LARGE,EXTRA_LARGE]",
 30       "cpuNum": "[1,1,64]",
 31       "memCapacityMB": "[7500,3748,max]",
 32       "storage": {
 33         "type": "[SHARED,LOCAL]",
 34         "sizeGB": "[1,1,min]"
 35       },
 36       "haFlag": "on"
 37     }
 38   ]
 39 }

/opt/serengeti/www/specs/map

 17     "vendor" : "Mesos",
 18     "version" : "^(\\d)+(\\.\\w+)*",
 19     "type" : "Mesos Master-Only Cluster",
 20     "appManager" : "Default",
 21     "path" : "Ironfan/mesos/master/spec.json"
 22   },

Normally, editing the two files would have been all that was required, however I have modified the Chef cookbooks to include the HAProxy package. I had included it in the install.rb cookbook for Mesos and this causes a problem if there are no slave nodes. I moved the code to the master.rb cookbook and updated the Chef server.

/opt/serengeti/chef/cookbooks/mesos/recipes/master.rb

166 directory "/etc/haproxy-marathon-bridge" do
167   owner 'root'
168   group 'root'
169   mode '0755'
170   action :create
171 end
172 
173 template '/usr/local/bin/haproxy-marathon-bridge' do
174   source 'haproxy-marathon-bridge.erb'
175   action :create
176 end
177 
178 all_ips = mesos_masters_ip
179 
180 template '/etc/haproxy-marathon-bridge/marathons' do
181   source 'marathons.erb'
182   variables(
183     haproxy_server_list: all_ips
184   )
185   action :create
186 end
187 
188 execute 'configure haproxy' do
189   command 'chkconfig haproxy on; service haproxy start'
190 end
191 
192 execute 'setup haproxy-marathon-bridge' do
193   command 'chmod 755 /usr/local/bin/haproxy-marathon-bridge; /usr/local/bin/haproxy-marathon-bridge install_cronjob'
194 end
195 
196 template '/usr/local/bin/haproxy-marathon-bridge' do
197   source 'haproxy-marathon-bridge.erb'
198   action :create
199 end

Restart Tomcat on the management server and then the new cluster definition is available for use.

My new cluster, minus the slave nodes looks like this now.

mesos-no-slaves

Using the new deployment option to deploy the Apache Mesos cluster. Once the cluster is configured and available, note the IP addresses of the two Mesos master nodes. We are going to use those IP addresses within the Photon nodes to pre-populate configuration files so the Photon nodes automatically join the cluster.

Photon Node Configuration

The next step is to configure a Photon node template that will automatically join the Mesos cluster deployed previously. After installing a node with the new TP2 release of Photon, I enabled root login over SSH so that I could quickly configure the node — be sure to turn it back off after you perform the following tasks.

Unfortunately, the version of Mesos that shipped in the ISO file released is 0.22.0 and there is a known conflict with the newer versions of Docker. The Photon TP2 ISO included Docker version 1.8.1 and it threw the following error when I tried to start the node as a Mesos slave:

root [ /etc/systemd/system ]# /usr/sbin/mesos-slave --master=zk://192.168.1.126:2181,192.168.1.127:2181,192.168.1.128:2181/mesos_cell --hostname=$(/usr/bin/hostname) --log_dir=/var/log/mesos_slave --containerizers=docker,mesos --docker=/usr/bin/docker --executor_registration_timeout=5mins --ip=$(/usr/sbin/ip -o -4 addr list | grep eno | grep global | awk 'NR==1{print $4}' | cut -d/ -f1)
I0905 18:42:16.588754  4269 logging.cpp:172] INFO level logging started!
I0905 18:42:16.591898  4269 main.cpp:156] Build: 2015-08-20 20:33:22 by 
I0905 18:42:16.592162  4269 main.cpp:158] Version: 0.22.1
Failed to create a containerizer: Could not create DockerContainerizer: Insufficient version of Docker! Please upgrade to >= 1.0.0

The bug was already noted in the updated code on the Photon GitHub repo, however there is not an update ISO available. That meant I needed to build my own ISO file from the latest code on the repo.

Note: Make sure the Ubuntu node has plenty of CPU and memory for compiling the ISO image. I was using a 1vCPU and 1GB memory VM in my lab and it took a long time to build the ISO.

photon-iso

After successfully building an updated ISO image, I used it to build a new VM. I really enjoy how quickly the Photon OS builds, even in my limited home lab environment.

photon-build-time

I wanted to configure the mesos-slave service to start each time the VM is booted and automatically join the master-only Mesos cluster I deployed above using BDE. That meant I needed to configure the mesos-slave.service file on the Photon node.

/etc/systemd/system/mesos-slave.service

  1 [Unit]
  2 Description=Photon Mesos Slave node
  3 After=network.target,docker.service
  4 
  5 [Service]
  6 Restart=on-failure
  7 RestartSec=10
  8 TimeoutStartSec=0
  9 ExecStartPre=/usr/bin/rm -f /tmp/mesos/meta/slaves/latest
 10 ExecStart=/bin/bash -c "/usr/sbin/mesos-slave \
 11 --master=zk://192.168.1.126:2181,192.168.1.127:2181,192.168.1.128:2181/mesos_cell \
 12 --hostname=$(/usr/bin/hostname) \
 13 --log_dir=/var/log/mesos_slave \
 14 --containerizers=docker,mesos \
 15 --docker=/usr/bin/docker \
 16 --executor_registration_timeout=5mins \
 17 --ip=$(/usr/sbin/ip -o -4 addr list | grep eno | grep global | awk 'NR==1{print $4}' | cut -d/ -f1)"
 18 
 19 [Install]
 20 WantedBy=multi-user.target

After creating the service file for systemd, it was then possible to start service and see it join the Mesos cluster in the UI.

meson-running

mesos-cluster-1

I shutdown the VM and cloned it to a template for use with the next step.

Final step is now to run a workload on the cluster, with Photon providing the Docker containers.

Workload Deployment

Launching a container workload on the new cluster was rather straightforward. I used a simple NGiNX container and exposed it over port 80.

meson-running-workload

marathon-running-workload

 

A few things, like automatic hostname configuration within Photon based on the DHCP address, are still left to do. But this is a working solution and let’s me do some next-level deployment testing using Photon as the mechanism for deploying the Docker containers.

If you have any questions on what I did here, feel free to reach out to me over Twitter.

Read More

VMworld 2015

Andrew Nelson and Tom Twyman  spoke on Wednesday morning about Apache Mesos and Marathon at VMworld. During their session they showed a demo of a cluster deployment — although they experienced a couple technical difficulties. The session covered the overall basics of how to operationalize Cloud Native Applications using Apache Mesos, Mesosphere Marathon and Docker on a VMware private cloud.

Here is an alternate cut of the demo they showed yesterday.

 

The video walks a user through a deployment of a Apache Mesos cluster using VMware Big Data Extensions, shows the running UI for Apache Mesos, Mesosphere Marathon and Chronos. Behind the scenes, HAProxy has been installed to automatically add any workloads launched and Docker support on each node. After the deployment is complete, a NGiNX Docker workload is launched into Marathon using the API. The workload is scaled from 1 to 6 instances and shows the HAProxy ruleset being updated to include each instance that is running. Finally, the video shows the Apache Mesos cluster itself being scaled while the same NGiNX workload is still running.

A quick 3-minute video showing how versatile Cloud Native Apps on top of VMware infrastructure can be to enable developers to take advantages of the newest technologies for running containers.

Read More

MesosphereLogo

There are a couple new versions for both Apache Mesos and Mesosphere Marathon. Since one of the great things about using VMware Big Data Extensions to handle the deployment and management of clusters is the ability to ensure control of the packages and versions within the SDLC. There were a few changes to be made within Chef to support Mesos v0.22.0 and Marathon 0.8.0* within the Big Data Extensions management server.

First thing, you need to grab the new RPM files and place them into your management server under /opt/serengeti/www/yum/repos/mesos/0.21.0 directory. For my own installation, I created a new directory called mesos/0.22.0 and copied the packages that had not been updated into it — but that decision is up to you. Once the files are there, you can choose whether to increment the version number in /opt/serengeti/www/distros/manifest file:

44   {
45     "name": "mesos",
46     "vendor": "MESOS",
47     "version": "0.22.0",
48     "packages": [
49       {
50         "package_repos": [
51           "https://bde.localdomain/yum/mesos.repo"
52         ],
53         "roles": [
54           "zookeeper",
55           "mesos_master",
56           "mesos_slave",
57           "mesos_docker",
58           "mesos_chronos",
59           "mesos_marathon"
60         ]
61       }
62     ]
63   },

If you choose to update the manifest file, make sure you restart tomcat on the BDE management server. You normally would be done at this point, but there were changes in where Mesos and Marathon install two files that we’ll need to account for. The old files were in subdirectories under /usr/local and are now placed in subdirectories of /usr.

/opt/serengeti/chef/cookbooks/mesos/templates/default/marathon.conf.erb:

  9 exec /usr/bin/marathon

/opt/serengeti/chef/cookbooks/mesos/default/slave.rb:

 78 template '/usr/etc/mesos/mesos-slave-env.sh' do
 79   source 'mesos-slave-env.sh.erb'
 80   variables(
 81     zookeeper_server_list: zk_server_list,
 82     zookeeper_port: zk_port,
 83     zookeeper_path: zk_path,
 84     logs_dir: node['mesos']['common']['logs'],
 85     work_dir: node['mesos']['slave']['work_dir'],
 86     isolation: node['mesos']['isolation'],
 87   )
 88   notifies :run, 'bash[restart-mesos-slave]', :delayed
 89 end

Update the files in the Chef server by running ‘knife cookbook upload -a‘ and you are all set.

Go forth and deploy your updated Apache Mesos 0.22.0 and Marathon 0.8.0 clusters!

*Marathon 0.8.1 has been released but there has not been a RPM file created by Mesosphere yet.

Read More