Tag: Photon

docker

My 11-year old son has been after me to get a Minecraft server setup locally that he could connect to and play off of any of the computers in the house. Fortunately for him, my home lab environment is running a few different Docker container hosts, and I figured loading up a couple servers via Docker would be the simplest solution. A quick search on Docker Hub yielded several results for Minecraft and after looking through the most popular options, I settled on the itzg/minecraft-server image.

I deployed a new Photon VM to act as the container host and after a slow 180 second install, I was pulling the image down to the host. I wanted to run a couple different options for him, so that he could choose what type of server to log into. I created a couple simple shell scripts to load the different server types.

cre-minecraft.sh

  1 docker run -d -it \
  2 -e EULA=TRUE -e DIFFICULTY=normal -e VERSION=LATEST \
  3 -e MODE=creative -e PVP=false -e LEVEL_TYPE=LARGEBIOMES \
  4 -e 'JVM_OPTS=-Xmx4096M -Xms4096M' \
  5 -p 25566:25565 --name cre-minecraft \
  6 -v /opt/minecraft2:/data itzg/minecraft-server

adv-minecraft.sh

  1 docker run -d -it \
  2 -e EULA=TRUE -e DIFFICULTY=normal -e VERSION=LATEST \
  3 -e MODE=adventure -e PVP=false -e LEVEL_TYPE=AMPLIFIED \
  4 -e 'JVM_OPTS=-Xmx8192M -Xms8192M' \
  5 -p 25565:25565 --name adv-minecraft \
  6 -v /opt/minecraft:/data itzg/minecraft-server

After that it was just a matter of starting both containers and checking that they were listening on the correct host ports.

docker-minecraft-1

docker-minecraft-2

The Docker Minecraft containers worked wonderfully this evening. He was excited to see that I was able to spin up as many servers as he wanted to run and that he could easily hop between them in his Minecraft client. It was a fun little experiment with Docker volumes for me as well — I had not used them yet — resulting in a very happy kid!

Read More

ve-banner-logo-2

As noted in the previous post, I had to custom build an ISO file that included Mesos 0.23.0 in order for the nodes to join the Mesos cluster I had deployed using Big Data Extensions. There was no forking of the Photon OS code off of GitHub required, it was just a matter of building the ISO.

In order to save others time, I have made my build of Photon available on GitHub for downloading. You can download the ISO file here.

The updated SPEC file for Mesos on the official Photon GitHub repo:

mesos-0.23.0-photon-bug

Enjoy.

Read More

photon-dfad9617

VMware Photon TP2 was released on August 27. The new version contains native support for running Mesos and therefore should have allowed the Photon OS to run as a Mesos slave immediately after installation. I would like to think my earlier blog post detailing how to deploy Mesos on-top of Photon influenced this functionality.

Download the ISO here.

After conversations with people involved in the project, the idea is for Photon to act only as a Mesos slave, with external Mesos masters and Zookeeper running on an Ubuntu/CentOS/Red Hat nodes. Logically the architecture of a Mesos cluster with Photon OS would look like the following.

 

mesos-photon-cluster

 

In order to deploy the cluster in this fashion, I wanted to find a method for automating as much of it as possible. Currently, one limitation with VMware Big Data Extensions is the single template VM limit. How awesome would it be if you could have multiple template VMs within the vApp and choose which template to deploy based on a pre-defined role? Definitely something to look into.

Regardless, working within the current limitations of BDE, I will describe in detail how I am now deploying Photon OS nodes into a Mesos cluster as automated as possible.

Configuring Big Data Extensions

I decided to create a new cluster map for a Mesos cluster that only deployed the Zookeeper and Mesos master nodes. The idea is similar to a Compute-only Hadoop or HDFS-only Hadoop cluster deployment through BDE. All that is required to accomplish this is a JSON file with the new cluster definition and an entry in the /opt/serengeti/www/specs/map file.

/opt/serengeti/www/specs/Ironfan/mesos/master/spec.json

  1 {
  2   "nodeGroups":[
  3     {
  4       "name": "Zookeeper",
  5       "roles": [
  6         "zookeeper"
  7       ],
  8       "groupType": "zookeeper",
  9       "instanceNum": "[3,3,3]",
 10       "instanceType": "[SMALL]",
 11       "cpuNum": "[1,1,64]",
 12       "memCapacityMB": "[7500,3748,min]",
 13       "storage": {
 14         "type": "[SHARED,LOCAL]",
 15         "sizeGB": "[2,2,min]"
 16       },
 17       "haFlag": "on"
 18     },
 19     {
 20       "name": "Master",
 21       "description": "The Mesos master node",
 22       "roles": [
 23         "mesos_master",
 24         "mesos_chronos",
 25         "mesos_marathon"
 26       ],
 27       "groupType": "master",
 28       "instanceNum": "[2,1,2]",
 29       "instanceType": "[MEDIUM,SMALL,LARGE,EXTRA_LARGE]",
 30       "cpuNum": "[1,1,64]",
 31       "memCapacityMB": "[7500,3748,max]",
 32       "storage": {
 33         "type": "[SHARED,LOCAL]",
 34         "sizeGB": "[1,1,min]"
 35       },
 36       "haFlag": "on"
 37     }
 38   ]
 39 }

/opt/serengeti/www/specs/map

 17     "vendor" : "Mesos",
 18     "version" : "^(\\d)+(\\.\\w+)*",
 19     "type" : "Mesos Master-Only Cluster",
 20     "appManager" : "Default",
 21     "path" : "Ironfan/mesos/master/spec.json"
 22   },

Normally, editing the two files would have been all that was required, however I have modified the Chef cookbooks to include the HAProxy package. I had included it in the install.rb cookbook for Mesos and this causes a problem if there are no slave nodes. I moved the code to the master.rb cookbook and updated the Chef server.

/opt/serengeti/chef/cookbooks/mesos/recipes/master.rb

166 directory "/etc/haproxy-marathon-bridge" do
167   owner 'root'
168   group 'root'
169   mode '0755'
170   action :create
171 end
172 
173 template '/usr/local/bin/haproxy-marathon-bridge' do
174   source 'haproxy-marathon-bridge.erb'
175   action :create
176 end
177 
178 all_ips = mesos_masters_ip
179 
180 template '/etc/haproxy-marathon-bridge/marathons' do
181   source 'marathons.erb'
182   variables(
183     haproxy_server_list: all_ips
184   )
185   action :create
186 end
187 
188 execute 'configure haproxy' do
189   command 'chkconfig haproxy on; service haproxy start'
190 end
191 
192 execute 'setup haproxy-marathon-bridge' do
193   command 'chmod 755 /usr/local/bin/haproxy-marathon-bridge; /usr/local/bin/haproxy-marathon-bridge install_cronjob'
194 end
195 
196 template '/usr/local/bin/haproxy-marathon-bridge' do
197   source 'haproxy-marathon-bridge.erb'
198   action :create
199 end

Restart Tomcat on the management server and then the new cluster definition is available for use.

My new cluster, minus the slave nodes looks like this now.

mesos-no-slaves

Using the new deployment option to deploy the Apache Mesos cluster. Once the cluster is configured and available, note the IP addresses of the two Mesos master nodes. We are going to use those IP addresses within the Photon nodes to pre-populate configuration files so the Photon nodes automatically join the cluster.

Photon Node Configuration

The next step is to configure a Photon node template that will automatically join the Mesos cluster deployed previously. After installing a node with the new TP2 release of Photon, I enabled root login over SSH so that I could quickly configure the node — be sure to turn it back off after you perform the following tasks.

Unfortunately, the version of Mesos that shipped in the ISO file released is 0.22.0 and there is a known conflict with the newer versions of Docker. The Photon TP2 ISO included Docker version 1.8.1 and it threw the following error when I tried to start the node as a Mesos slave:

root [ /etc/systemd/system ]# /usr/sbin/mesos-slave --master=zk://192.168.1.126:2181,192.168.1.127:2181,192.168.1.128:2181/mesos_cell --hostname=$(/usr/bin/hostname) --log_dir=/var/log/mesos_slave --containerizers=docker,mesos --docker=/usr/bin/docker --executor_registration_timeout=5mins --ip=$(/usr/sbin/ip -o -4 addr list | grep eno | grep global | awk 'NR==1{print $4}' | cut -d/ -f1)
I0905 18:42:16.588754  4269 logging.cpp:172] INFO level logging started!
I0905 18:42:16.591898  4269 main.cpp:156] Build: 2015-08-20 20:33:22 by 
I0905 18:42:16.592162  4269 main.cpp:158] Version: 0.22.1
Failed to create a containerizer: Could not create DockerContainerizer: Insufficient version of Docker! Please upgrade to >= 1.0.0

The bug was already noted in the updated code on the Photon GitHub repo, however there is not an update ISO available. That meant I needed to build my own ISO file from the latest code on the repo.

Note: Make sure the Ubuntu node has plenty of CPU and memory for compiling the ISO image. I was using a 1vCPU and 1GB memory VM in my lab and it took a long time to build the ISO.

photon-iso

After successfully building an updated ISO image, I used it to build a new VM. I really enjoy how quickly the Photon OS builds, even in my limited home lab environment.

photon-build-time

I wanted to configure the mesos-slave service to start each time the VM is booted and automatically join the master-only Mesos cluster I deployed above using BDE. That meant I needed to configure the mesos-slave.service file on the Photon node.

/etc/systemd/system/mesos-slave.service

  1 [Unit]
  2 Description=Photon Mesos Slave node
  3 After=network.target,docker.service
  4 
  5 [Service]
  6 Restart=on-failure
  7 RestartSec=10
  8 TimeoutStartSec=0
  9 ExecStartPre=/usr/bin/rm -f /tmp/mesos/meta/slaves/latest
 10 ExecStart=/bin/bash -c "/usr/sbin/mesos-slave \
 11 --master=zk://192.168.1.126:2181,192.168.1.127:2181,192.168.1.128:2181/mesos_cell \
 12 --hostname=$(/usr/bin/hostname) \
 13 --log_dir=/var/log/mesos_slave \
 14 --containerizers=docker,mesos \
 15 --docker=/usr/bin/docker \
 16 --executor_registration_timeout=5mins \
 17 --ip=$(/usr/sbin/ip -o -4 addr list | grep eno | grep global | awk 'NR==1{print $4}' | cut -d/ -f1)"
 18 
 19 [Install]
 20 WantedBy=multi-user.target

After creating the service file for systemd, it was then possible to start service and see it join the Mesos cluster in the UI.

meson-running

mesos-cluster-1

I shutdown the VM and cloned it to a template for use with the next step.

Final step is now to run a workload on the cluster, with Photon providing the Docker containers.

Workload Deployment

Launching a container workload on the new cluster was rather straightforward. I used a simple NGiNX container and exposed it over port 80.

meson-running-workload

marathon-running-workload

 

A few things, like automatic hostname configuration within Photon based on the DHCP address, are still left to do. But this is a working solution and let’s me do some next-level deployment testing using Photon as the mechanism for deploying the Docker containers.

If you have any questions on what I did here, feel free to reach out to me over Twitter.

Read More

Ever since the VMware Photon Technical Preview was made available, I have wanted to install Apache Mesos on a node. The Photon OS is a very minimal Linux installation and so my initial attempts to work through the process led me down the old-fashioned rabbit hole of manually compiling packages. It was very reminiscent of the later 1990’s and using Debian installed via floppy disk. I finally found the time to work my way down and back out of the rabbit hole and have been able to successfully get Mesos to install and run on Photon! This is a good first step towards building a Photon template to be used inside Big Data Extensions for deploying Cloud Native Apps with Mesos|Marathon|Chronos or Kubernetes.

The lab environment I used is running vSphere 5.5 and consists of a small set of nested ESXi hypervisors. I am not going to cover install Photon on a VM, but just be sure you have one that was installed with the full OS — not the minimal installation. After the Photon VM is configured to communicate with the Internet, you can follow these instructions to get Apache Mesos installed. The end result of the guide will be a working Mesos node running on Photon that can launch Docker containers.

Apache Mesos on Photon

Be sure to follow it in the order listed as the ordering of the packages is important. Also have a Photon VM with at least 3GB of Memory allocated to it for the compile processes.

HOWTO Guide

Missing prerequisites
APR Library
# wget http://apache.claz.org//apr/apr-1.5.2.tar.gz
# tar zxvf apr-1.5.2.tar.gz
# cd apr-1.5.2
# ./configure —prefix=/usr/local/lib/apr
# make
# make test
# make install

APR-UTIL Library
# wget http://apache.claz.org//apr/apr-util-1.5.4.tar.gz
# tar zxvf apr-util-1.5.4.tar.gz
# cd apr-util-1.5.4
# ./configure —prefix=/usr/local/lib/apr —with-apr=/usr/local/lib/apr
# make
# make install

Subversion
# wget http://apache.osuosl.org/subversion/subversion-1.8.13.tar.gz
# tar zxvf subversion-1.8.13.tar.gz
# cd subversion-1.8.13
# ./configure —prefix=/usr/local/lib/subversion —with-apr=/usr/local/lib/apr —with-apr-util=/usr/local/lib/apr
# make
# make install

OpenJDK Java
Download the Java JDK source tarball from the Oracle website (http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.tar.gz)
# tar zxvf jdk-7u79-linux-x64.tar.gz
# mv jdk1.7.0_79 /usr/local/java
# echo JAVA_HOME=/usr/local/java >> /etc/environment
# source /etc/environment

Apache Maven Library
# wget http://apache.mirrors.ionfish.org//ant/binaries/apache-ant-1.9.5-bin.tar.gz
# tar zxvf apache-ant-1.9.5-bin.tar.gz
# mv apache-ant-1.9.5 /usr/local
# ln -s /usr/local/apache-ant-1.9.5 /usr/local/apache-ant
# wget http://apache.cs.utah.edu/maven/maven-3/3.3.3/source/apache-maven-3.3.3-src.tar.gz
# tar zxvf apache-maven-3.3.3-src.tar.gz
# cd apache-maven-3.3.3
# /usr/local/apache-ant/bin/ant -Dmaven.home=“/usr/local/maven-3.3.3"
# echo MAVEN_HOME=/usr/local/maven-3.3.3 >> /etc/environment
# export /etc/environment

Install Apache Mesos
# wget http://www.apache.org/dist/mesos/0.22.1/mesos-0.22.1.tar.gz
# tar zxvf mesos-0.22.1.tar.gz
# cd mesos-0.22.1
# mkdir build
# cd build
# ../configure --prefix=/usr/local/mesos —with-apr=/usr/local/lib/apr —with-svn=/usr/local/lib/subversion
# make
# make check
# make install
After Apache Mesos is installed, you can start both the master and slave processes on the node to run a quick test.
# /usr/local/mesos/bin/mesos-master.sh —ip=127.0.0.1 —work_dir=/var/lib/mesos
# /usr/local/mesos/bin/mesos-slave.sh —master=127.0.0.1:5050
Afterwards, open a web browser and point to to http://127.0.0.1:5050 and you will see the Apache Mesos interface. The next step will be to deploy multiple Photon nodes and configure them to be a part of a single cluster. I will likely orchestrate all of this through Chef next and incorporate it into the Big Data Extensions framework.
There is always more to do, but I am glad to have gotten this working and sharing it with everyone. Reach out to me on Twitter if you have questions.

Read More

The opportunities for VMware with Project Photon and Project Lightwave are significant. The press release stated:

Designed to help enterprise developers securely build, deploy and manage cloud-native applications, these new open source projects will integrate into VMware’s unified platform for the hybrid cloud — creating a consistent environment across the private and public cloud to support cloud-native and traditional applications. By open sourcing these projects, VMware will work with a broad ecosystem of partners and the developer community to drive common standards, security and interoperability within the cloud-native application market — leading to improved technology and greater customer choice.

What I always find interesting is the lack of discussion around the orchestration and automation of the supporting applications. The orchestration layer does not miraculously appear within a private cloud environment for the developers to consume. The pieces have to be in place in order for developers to consume the services a Mesos cluster offers them. For me, the choice is pretty obvious — expand what the Big Data Extensions framework is capable of providing. I alluded to this thought on Monday when the announcement was made.

Building on that thought and after seeing a diagram of VMware’s vision for how all the pieces tie together, I worked on a logical diagram of how the entire architecture could look like. I believe it looks something like this:

CNA

 

In this environment, Project Photon and Project Lightwave are able to be leveraged beyond just ESXi. By enhancing the deployment options for BDE to include ESXi on vCloud Air (not shown above), KVM and physical (through Ironic), the story is slightly changed. The story now sounds something like this:

For a developer, you choose what Cloud Native application orchestration layer (Mesos, Marathon, Chronos, CloudFoundry, etc.) you would like and communicate with it over the provided API. For operations, the deployment of the tenants within the private cloud environment can be deployed using the OpenStack API (with Heat templates). For both sides, SDLC consistency is maintained through the development process to production.

Simplicity is achieved by only interacting with two APIs — one for operations and one for development. There is large amount of work to do here. First, I need to continue to improve the OpenStack resource plugin to be production-ready. Second, testing of Project Photon inside BDE needs to take place — I imagine there will be some work to have it integrated correctly with the Chef server. Third, the deployment mechanism inside BDE needs to be enhanced to support other options. If the first two were a heavy lift, the last one is going to take a small army — but it is a challenge I am ready to take on!

Ultimately, I feel the gaps in OpenStack around Platform-as-a-Service orchestration can be solved though integrating Big Data Extensions. The framework is more robust and mature when compared to the Sahara offering. The potential is there, it just needs to be executed on.

Read More