Installing Apache Mesos on Photon Linux OS

Ever since the VMware Photon Technical Preview was made available, I have wanted to install Apache Mesos on a node. The Photon OS is a very minimal Linux installation and so my initial attempts to work through the process led me down the old-fashioned rabbit hole of manually compiling packages. It was very reminiscent of the later 1990’s and using Debian installed via floppy disk. I finally found the time to work my way down and back out of the rabbit hole and have been able to successfully get Mesos to install and run on Photon! This is a good first step towards building a Photon template to be used inside Big Data Extensions for deploying Cloud Native Apps with Mesos|Marathon|Chronos or Kubernetes.

The lab environment I used is running vSphere 5.5 and consists of a small set of nested ESXi hypervisors. I am not going to cover install Photon on a VM, but just be sure you have one that was installed with the full OS — not the minimal installation. After the Photon VM is configured to communicate with the Internet, you can follow these instructions to get Apache Mesos installed. The end result of the guide will be a working Mesos node running on Photon that can launch Docker containers.

Apache Mesos on Photon

Be sure to follow it in the order listed as the ordering of the packages is important. Also have a Photon VM with at least 3GB of Memory allocated to it for the compile processes.


Missing prerequisites
APR Library
# wget
# tar zxvf apr-1.5.2.tar.gz
# cd apr-1.5.2
# ./configure —prefix=/usr/local/lib/apr
# make
# make test
# make install

APR-UTIL Library
# wget
# tar zxvf apr-util-1.5.4.tar.gz
# cd apr-util-1.5.4
# ./configure —prefix=/usr/local/lib/apr —with-apr=/usr/local/lib/apr
# make
# make install

# wget
# tar zxvf subversion-1.8.13.tar.gz
# cd subversion-1.8.13
# ./configure —prefix=/usr/local/lib/subversion —with-apr=/usr/local/lib/apr —with-apr-util=/usr/local/lib/apr
# make
# make install

OpenJDK Java
Download the Java JDK source tarball from the Oracle website (
# tar zxvf jdk-7u79-linux-x64.tar.gz
# mv jdk1.7.0_79 /usr/local/java
# echo JAVA_HOME=/usr/local/java >> /etc/environment
# source /etc/environment

Apache Maven Library
# wget
# tar zxvf apache-ant-1.9.5-bin.tar.gz
# mv apache-ant-1.9.5 /usr/local
# ln -s /usr/local/apache-ant-1.9.5 /usr/local/apache-ant
# wget
# tar zxvf apache-maven-3.3.3-src.tar.gz
# cd apache-maven-3.3.3
# /usr/local/apache-ant/bin/ant -Dmaven.home=“/usr/local/maven-3.3.3"
# echo MAVEN_HOME=/usr/local/maven-3.3.3 >> /etc/environment
# export /etc/environment

Install Apache Mesos
# wget
# tar zxvf mesos-0.22.1.tar.gz
# cd mesos-0.22.1
# mkdir build
# cd build
# ../configure --prefix=/usr/local/mesos —with-apr=/usr/local/lib/apr —with-svn=/usr/local/lib/subversion
# make
# make check
# make install
After Apache Mesos is installed, you can start both the master and slave processes on the node to run a quick test.
# /usr/local/mesos/bin/ —ip= —work_dir=/var/lib/mesos
# /usr/local/mesos/bin/ —master=
Afterwards, open a web browser and point to to and you will see the Apache Mesos interface. The next step will be to deploy multiple Photon nodes and configure them to be a part of a single cluster. I will likely orchestrate all of this through Chef next and incorporate it into the Big Data Extensions framework.
There is always more to do, but I am glad to have gotten this working and sharing it with everyone. Reach out to me on Twitter if you have questions.

Building on Project Photon & Project Lightwave

The opportunities for VMware with Project Photon and Project Lightwave are significant. The press release stated:

Designed to help enterprise developers securely build, deploy and manage cloud-native applications, these new open source projects will integrate into VMware’s unified platform for the hybrid cloud — creating a consistent environment across the private and public cloud to support cloud-native and traditional applications. By open sourcing these projects, VMware will work with a broad ecosystem of partners and the developer community to drive common standards, security and interoperability within the cloud-native application market — leading to improved technology and greater customer choice.

What I always find interesting is the lack of discussion around the orchestration and automation of the supporting applications. The orchestration layer does not miraculously appear within a private cloud environment for the developers to consume. The pieces have to be in place in order for developers to consume the services a Mesos cluster offers them. For me, the choice is pretty obvious — expand what the Big Data Extensions framework is capable of providing. I alluded to this thought on Monday when the announcement was made.

Building on that thought and after seeing a diagram of VMware’s vision for how all the pieces tie together, I worked on a logical diagram of how the entire architecture could look like. I believe it looks something like this:



In this environment, Project Photon and Project Lightwave are able to be leveraged beyond just ESXi. By enhancing the deployment options for BDE to include ESXi on vCloud Air (not shown above), KVM and physical (through Ironic), the story is slightly changed. The story now sounds something like this:

For a developer, you choose what Cloud Native application orchestration layer (Mesos, Marathon, Chronos, CloudFoundry, etc.) you would like and communicate with it over the provided API. For operations, the deployment of the tenants within the private cloud environment can be deployed using the OpenStack API (with Heat templates). For both sides, SDLC consistency is maintained through the development process to production.

Simplicity is achieved by only interacting with two APIs — one for operations and one for development. There is large amount of work to do here. First, I need to continue to improve the OpenStack resource plugin to be production-ready. Second, testing of Project Photon inside BDE needs to take place — I imagine there will be some work to have it integrated correctly with the Chef server. Third, the deployment mechanism inside BDE needs to be enhanced to support other options. If the first two were a heavy lift, the last one is going to take a small army — but it is a challenge I am ready to take on!

Ultimately, I feel the gaps in OpenStack around Platform-as-a-Service orchestration can be solved though integrating Big Data Extensions. The framework is more robust and mature when compared to the Sahara offering. The potential is there, it just needs to be executed on.

Thoughts on the VMware Project Photon Announcement

project photon

VMware announced a new open source project call Project Photon today. The full announcement call be seen here. Essentially Project Photon is a lightweight Linux operating system built to support Docket and rkt (formerly Rocket) containers. The footprint is less than 400MB and can run containers immediately upon instantiation. I had heard rumors the announcement today was going to include some sort of OS, but I was not very excited about it until I started reading the material being released prior to the launch event in a few hours.

Having seen the demo and read the material, my mind went into overdrive for the possibilities both open source projects offer organizations who are venturing down the Cloud Native Apps (or Platform 3) road. I believe VMware has a huge opportunity here to cement themselves as the foundation for running robust, secure and enterprise-ready Cloud Native Applications. If you think about the performance gains vSphere 6.0 has provided, and then look at how they are playing in the OpenStack space with both VIO and NSX, the choice becomes obvious.

The area of focus now needs to be on tying all of the pieces together to offer organizations an enterprise-class end-to-end Platform-as-a-Service solution. This is where, I believe, the VMware Big Data Extensions framework should play an enormous part. The framework already allows deployment of Hadoop, Mesos and Kubernetes clusters. Partner the framework with Project Photon and you now have a minimal installation VM that can be launched within seconds with VMFork. From there, the resource plugin Virtual Elephant launched today could be mainstreamed (and improved) to allow for the entire deployment of a Mesos stack, backed by Project Photon, through the open source API OpenStack offers with Heat.

Epic win!

There is still work VMware could do with the Big Data Extensions framework to improve its capabilities, especially with newcomers SequenceIQ and their Cloudbreak offering stiff competition. Expanding BDE to be able to deploy clusters beyond an internal vSphere environment, but also to the major public cloud environments — including their own vCloud Air — will be key going forward. The code for BDE is already an open source project — by launching these two new open source projects they are showing the open source community they are serious.

This is a really exciting time in virtualization and I just got even more excited today!

SequenceIQ Cloudbreak Hadoop Cluster Deployments

I saw activity on Twitter talking about Cloudbreak from SequenceIQ as a method for deploying Hadoop clusters into several public cloud providers. My interest was peaked and I started reading through the Cloudbreak documentation on Saturday night. About 20 minutes later, I had an account setup, tied it to my AWS account and started deploying a cluster. Approximately 15 minutes later, the cluster was deployed and Ambari was setup running and I could log into the UI.

The interface is truly brilliant! I am pretty impressed with the functionality it provides, the ease of setup and the blueprinting functionality it provides. The project is only in a public beta state right now, but I plan to keep my eyes on the project and see how it develops. If you are interested in reading more about my experience with the very first cluster and some screenshots of the interface, keep reading!

Cloudbreak UI & Cluster Deployment

Upon deployment of the cluster, the Dashboard UI immediately began showing me status updates on the ribbon and a small square widget. When the widget was clicked, it expanded to show further details of the cluster and the status.

sequenceiq cloudbreak



I logged into Ambari after the cluster reported that it was deployed and Ambari was accessible. The Cloudbreak dashboard had a link sending me to the Ambari UI. Once logged in, I was pleased to see some basic status items displayed, along with a list of services available in the cluster.

Note: The Ambari login credentials for the Ambari UI are the defaults (admin/admin). The Cloudbreak UI did not tell me this anywhere that I saw, so I had to Google what the defaults were.

cloudbreak-screen-03Poking around the Ambari interface, it is pretty easy to see what nodes the Cloudbreak service had deployed inside AWS and their relevant information.


Noticing the metric widgets were reporting they were unable to report/collect data because the Ganglia service was not deployed, I started through the process of adding services to the running cluster. The Ambari interface went through  a multi-step process, allowing me to select the services I wanted and checking prerequisites for those services. In my case, the cluster also needed the Tez service in order to deploy the services I select.




Here is where things went a little sideways — the deployment of the new services stalled at 2/10.


After about 30 minutes and several attempts to refresh the page, I closed the page and went back into Ambari through the Cloudbreak dashboard. When Ambari loaded, I was presented with the following screen and no way to interact with Ambari — it appeared wedged.


This allowed me to test the termination functionality of Cloudbreak and seeing how it removes all of the objects from my AWS account.



All-in-all a great first experience with Cloudbreak. I am going to continue to play with it — especially since I am interested in how they are using Docker containers in the service.


Upgrading Apache Mesos & Marathon on VMware Big Data Extensions


There are a couple new versions for both Apache Mesos and Mesosphere Marathon. Since one of the great things about using VMware Big Data Extensions to handle the deployment and management of clusters is the ability to ensure control of the packages and versions within the SDLC. There were a few changes to be made within Chef to support Mesos v0.22.0 and Marathon 0.8.0* within the Big Data Extensions management server.

First thing, you need to grab the new RPM files and place them into your management server under /opt/serengeti/www/yum/repos/mesos/0.21.0 directory. For my own installation, I created a new directory called mesos/0.22.0 and copied the packages that had not been updated into it — but that decision is up to you. Once the files are there, you can choose whether to increment the version number in /opt/serengeti/www/distros/manifest file:

44   {
45     "name": "mesos",
46     "vendor": "MESOS",
47     "version": "0.22.0",
48     "packages": [
49       {
50         "package_repos": [
51           "https://bde.localdomain/yum/mesos.repo"
52         ],
53         "roles": [
54           "zookeeper",
55           "mesos_master",
56           "mesos_slave",
57           "mesos_docker",
58           "mesos_chronos",
59           "mesos_marathon"
60         ]
61       }
62     ]
63   },

If you choose to update the manifest file, make sure you restart tomcat on the BDE management server. You normally would be done at this point, but there were changes in where Mesos and Marathon install two files that we’ll need to account for. The old files were in subdirectories under /usr/local and are now placed in subdirectories of /usr.


  9 exec /usr/bin/marathon


 78 template '/usr/etc/mesos/' do
 79   source ''
 80   variables(
 81     zookeeper_server_list: zk_server_list,
 82     zookeeper_port: zk_port,
 83     zookeeper_path: zk_path,
 84     logs_dir: node['mesos']['common']['logs'],
 85     work_dir: node['mesos']['slave']['work_dir'],
 86     isolation: node['mesos']['isolation'],
 87   )
 88   notifies :run, 'bash[restart-mesos-slave]', :delayed
 89 end

Update the files in the Chef server by running ‘knife cookbook upload -a‘ and you are all set.

Go forth and deploy your updated Apache Mesos 0.22.0 and Marathon 0.8.0 clusters!

*Marathon 0.8.1 has been released but there has not been a RPM file created by Mesosphere yet.