The post includes the final pieces necessary to get a Mesosphere stack deployed through Big Data Extensions within a VMware environment. I’ve included the Chef cookbooks and commands required for tying all of the pieces together for a cluster deployment. The wonderful thing about the framework is the extensibility — once I had Mesos deploying, it became very clear how simple it is to extend the framework even further — look for future posts.
The idea that you can now turn a large cluster of VMs into a single Mesos cluster for use by a product, engineering team or operations team opens up an entirely new world within our environments. This is a very exciting place to be investing time.
Big Data Extensions uses role definitions within the framework, so the first step was to create a new role for Mesos. If you remember from Part 2, we defined the role in the JSON file and called it ‘mesos’.
The role files can be found in /opt/serengeti/chef/roles. I created the roles for both mesos_master and mesos_worker through the command line interface:
Continue reading “Apache Mesos Clusters – Part 3”
Building Mesosphere & Apache Mesos into BDE:
After playing with Mesosphere in AWS for the week, getting familiar with the packages and the deployment process, the real work has begun — getting the Mesosphere stack (Apache Mesos, Apache Zookeeper, Mesosphere Marathon, Chronos and HAProxy) deployed through VMware Big Data Extensions. Fortunately, BDE v2.1 has some example JSON cluster definition files that can be used for deploying different types of clusters and these are perfect for modification in this use-case.
The example files are located in the directory /opt/serengeti/samples. I used the basic_cluster.json file in the directory as the template. From there, I modified the file based on what the Mesosphere stack deployed in AWS, with some slight modifications. I chose to have a base Mesos cluster include 3 master nodes and 6 worker nodes. The master nodes are allocated with 2vCPU, 8GB RAM and 50GB of disk space. The worker nodes are allocated with 2vCPU, 8GB RAM and 100GB of disk space.
The remainder of the post will go through all the various pieces that are necessary to utilize the Big Data Extensions framework to offer the Mesosphere stack within a VMware virtual environment.
Continue reading “Apache Mesos Clusters – Part 2”
I watched a webinar today from Ken Sipe (@kensipe) from Mesosphere on Mesos, Marathon and Chronos. The topics covered included how Mesos works, configuring and standup of a Mesos cluster in various public cloud offerings. If you are unfamiliar with Mesos, I would direct you to Mesosphere and the Apache Mesos Project.
The basic explanation of from the Apache Mesos Project page states:
Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.
Think turning an entire datacenter of compute resources into a single pool to be consumed. Instead of carving out individual pieces of compute, Mesos handles the scheduling and helps you scale an application across all of the resources available to it.
So how quickly can you deploy a cluster and begin using Mesos?
Continue reading “Apache Mesos Clusters – Part 1”
It’s been very quiet around here since VMworld US ended a little over one month ago. I have had my head down studying for my VCP5-DCV exam — which I am taking on Wednesday. The rest of my spare time has been consumed getting ready for VMworld EMEA in Barcelona, Spain. I took the feedback received after VMworld US and will be showing a demo of Hadoop being deployed virtually through the vCAC orchestration workflows that interact with Big Data Extensions.
I am looking forward to my trip to Spain. I am planning on having several days to wander about and see some of what Europe has to offer — especially a FC Barcelona game on Saturday the 18th.
I do have some good things planned for the site, including posts on Isilon performance metrics with Hadoop, expanding BDE functionality to include Flume nodes, blueprints for deploying HDFS-only virtual clusters to be used for a unified data warehouse layer.
It is going to be a very busy winter here in Utah this year with all of the Hadoop work, next-generation Openstack (VMware Integrated OpenStack) and preparing for the VCAP5-DCA test in January.
The conference was completed just over two weeks ago, and since then I’ve had the opportunity to go through my notes, think about the sessions I attended and summarize what insight I gained while there.
The biggest takeaway I had for VMworld 2014 compared to last year revolved around lessons learned in 2013 were applied in 2014. The key insight in 2013 was that many other partners and customers of VMware were facing the same challenges around standardization, automation and self-service. It was helpful to learn that the things we were trying to accomplish within our department at Adobe were not unique to us.
This year, 2014, I learned that we have solved many of the challenges from the last year and now have great insight to offer out to the community. As we work towards building on the standardization, automation and self-service phases of offering both comprehensive IaaS and PaaS offerings, we are doing what we can to share that information with the broader community.
All of that is wonderful, but what are the next steps for our team, the market and others in the virtualization space? We heard a lot at the conference about OpenStack, Docker, VSAN and other emerging technologies. The focus I personally have for the next year is going to revolve around further implementation of the Hadoop ecosystem, using VMware technologies, and building out larger, comprehensive PaaS offerings.
There are many questions to be answered around how OpenStack and Docker plays in the space. I am looking forward to the challenges coming to us as we work with our engineering teams.
Should be an exciting year!