Ravello Nested ESXi Home Lab Environment


The past week I found myself in need of additional capacity for a Cloud Native App deployment that my current home lab lacked and it was a perfect opportunity to activate my vExpert Ravello Systems account. I had read multiple posts on how to run a nested VMware vSphere environment within Ravello and so I jumped straight into the deep end of the pool. After logging into the system, building an initial ESXi VM image, I started building out the lab environment. The application blueprinting functionality in the Ravello interface is pretty intuitive and left me wishing some other tools I’ve used in the past were as simplistic.

The blueprint includes a small external management cluster, an internal management cluster and the workgroup cluster. Here is a screenshot of the lab environment.

Ravello CNA Lab

External Management Cluster

The external management cluster runs the virtual machines services that are necessary to support the internal lab.

  • VMware vCenter Server virtual appliance (VCSA).
  • Linux VM running iSCSI for the ESXi hosts.
  • Linux VM running DHCPD for the 2nd-level nested VMs.
  • Windows 7 IE11 VM to act as a bastion host for accessing the environment.

Management Cluster

The internal management cluster run the components such as VMware Big Data Extensions and vRealize Operations.

  • 4 ESXi hosts running 5.5U2

Workgroup Cluster

The workgroup cluster runs the nested VMs deployed by Big Data Extensions, such as Hadoop, Mesos and Kubernetes clusters.

Configuration Items

There were a couple of issues I encountered when I first started using the lab environment. I was able to find a resolution for each one in various places on the internet, but I want to share them all here for future reference.

vCenter Server

The vCenter server will throw an error that will require user-intervention to boot due to the fact that when it is deployed as a 1st-level VM, it is running inside a KVM container. To solve the issue, follow these instructions from Ravello found on their site.


2nd-level Nested VMs

When I went to deploy 2nd-level VMs within the Ravello lab, I found they would not power on due to an incompatible hypervisor. To solve the issue, I had to add the Advanced Setting on each of the virtual machines deployed as a 2nd-level VM.

vmx.allowNested = “TRUE”


The next problem I encountered with 2nd-level VMs was the lack of DHCP functionality. The Ravello provided DHCP server for the networks within the application blueprint does not work beyond the 1st-level VMs. To solve the issue, I created a CentOS Linux VM (shown in the External Management Cluster above) and added a reserved IP address for it in the configuration.


Deploying OVF/OVA to the vCenter VM failed

After getting the vCenter configured with the ESXi hosts and HA/DRS clusters, the next step was to deploy the VMware Big Data Extensions vApp. During the configuration of the vApp (storage, network, etc), it failed to allow me to complete the deployment. After looking into the issue, it became apparent it was an issue of the public IP address being a NAT into the private network running inside Ravello.

To solve the issue, I deployed the Windows 7 IE11 VM and then opened an RDC session to the environment. From there I was able to access the vCenter server on the private IP network and perform the OVA deployment of BDE.

Note: If you need a free Windows VM that you can deploy any vSphere environment, log onto the Microsoft Edge site.

Overall Experience

My overall experience with Ravello this past week and consuming around 1500 CPU hours was pretty good. I liked the interface for creating the environment, being able to set a shutdown/startup schedule and overall deployment of the 1st-level VMs. The negatives I encountered were solely around the performance of the 2nd-level nested VMs. Deploying a Mesos cluster through BDE took a long time — I started a small 9 node cluster, went out to dinner with my wife, and it was finishing when we got home 1 1/2 hours later.

With it being VMworld this week, I went down to the Solutions Exchange area and talked to a few people at Ravello about the performance issues I encountered and they offered me some suggestions. I will be checking the environment to see if the settings they recommended make a noticeable difference. For now I am giving them the benefit of the doubt.

Overall, I think the system is a really great opportunity for VMware evangelists to spin up a low-cost lab environment to test new functionality as 1st-level VMs. Which makes it a great place to work on certification objectives for VCP, VCAP/VCIX exams. Trying to test out Cloud Native App frameworks (Kubernetes, Mesos, Docker Swarm) is probably not the best environment.

I will continue to use the environment — especially because of the vExpert program they offer — and I am hopeful the performance will increase over time for the 2nd-level workloads.

IGMP, Multicast and learning a lesson

In a recent conversation, it became clear to me that my knowledge of the inner workings of VXLAN and VSAN were not a deep as they could be. Since I am also studying for my VCAP exams, I knew additional time educating myself around these two technologies was a necessity. As a result, I’ve spent the last day diving into the IGMP protocol, multicast traffic and how they are utilized both within VXLAN and VMware VSAN. I wanted to capture what I’ve learned on a blog post as much for myself as for anyone else who might be interested in the subject. Writing what I’ve learned is one way I can absorb and retain information long-term.


IGMP is a layer 3 network protocol. It is a communications protocol use to establish multicast group memberships. It is encapsulated within an IP packet and does not use a transport layer — similar to ICMP. It is also used to register a router for receiving multicast traffic. There are two important pieces within the IGMP protocol that VXLAN and VSAN take advantage of — IGMP Querier and IGMP Snooping. Without these two pieces, IGMP would act astonishing more than a broadcast transmission and lack the efficiency required.

IGMP Querier

The IGMP Querier is the router or switch that acts as the master for the IGMP filter lists. It will check and track membership by sending queries on a timed interval.

IGMP Snooping

On a layer 2 switch, IGMP Snooping allows for the passive monitoring for IGMP packets sent between router(s) and host(s). It also does not send any additional network traffic across the wire, making it more efficient for multicast traffic passing through the network.


That’s IGMP in a nutshell — so how is it used in VXLAN?

In order for VXLAN to act as an overlay network, multicast traffic is used to enable the L2 over L3 connectivity — effectively spanning the entire logical network VXLAN has defined. When a virtual machine connects to a VXLAN logical network, it behaves as though everything is within a single broadcast domain. The ESXi hosts, configured for VXLAN, register themselves as VTEPs. Only those VTEPs that register with the VXLAN logical network participate in the multicast broadcasts. This is accomplished through IGMP Snooping and IGMP Querier. If you have 1000 ESXi hosts configured for VXLAN, but only a subset (say 100) of the hosts are concerned for a specific VXLAN logical network, you wouldn’t want to send multicast broadcasts out to all 1000 ESXi hosts — that would be inefficient by increasing the multicast traffic on the network unnecessarily.

There is a really good VMware Blog 4-part series on VXLAN and how it operates here.


The implementation for VSAN is very similar to that of VXLAN. The VSAN clusters require a methodology for learning what ESXi hosts are adjacent to each other and participating as a VSAN cluster. VMware uses layer 2 multicast traffic for the host discovery within VSAN.

Once again, IGMP Querier and IGMP Snooping are play a beneficial role. VMware states that implementing multicast flooding is not a best practice. By leveraging both IGMP Snooping and IGMP Querier, VSAN is able to understand who wants to participate within the multicast group. This is particularly beneficial when multiple network devices exist on the same VLAN that VSAN is operating on.

If you have multiple VSAN clusters operating on the same VLAN, it is recommended you change the broadcast address for the multicast traffic so they are not identical. This will prevent one VSAN cluster from receiving another clusters broadcasts. It can also help prevent the Misconfiguration detected error under the Network status sections of a VSAN cluster.

For a better understanding of how VSAN operates, please check out the VMware blog entry here.

For a season network professional, I highly doubt any of this was new or mind-blowing. For someone who does not generally dive into the various network protocols — but should probably start doing so — this information was both a good refresher on IGMP and helped me understand both VXLAN and VSAN a bit better.

Did I get something wrong? Let me know on Twitter.

Preparing for the VCAP-DCA exam

I have begun my preparations for taking (and passing) the VCAP-DCA exam during the first quarter of 2015. I consider myself fortunate to work in an environment where there is hardware that I can utilize as a lab environment. However, looking at the blueprint for the VCAP-DCA test, I realized there are bits there that would be more difficult to test in that environment. As such, I started looking for a low-cost solution for creating a home lab to play with iSCSI, VMware Replication and other topics covered in the exam. After doing research on the internet, I decided to focus on the Intel NUC platform as the basis for the lab systems.

I posted to Twitter earlier in the week and I was able to get several good pieces of advice for where to look for understanding how to run ESXi on the Intel NUC platform. The best articles I read were written by @virten at virten.net. He has tested several of the Intel NUC generations and individual platforms.

Continue reading “Preparing for the VCAP-DCA exam”

VCP5: Creating an iSCSI lab environment for vSphere

As I worked through the VCP5-DVC blueprint, the necessity to revisit iSCSI storage configuration and management became a key point of my study efforts. I had not used iSCSI storage before within a VMware vSphere environment, so learning how to tie it all into the infrastructure was totally new to me. In fact, the last time I had used iSCSI storage was with my previous employer 5+ years ago within a customized CentOS OpenVZ environment.

Fortunately, Google did not fail me and there were many resources readily available for teaching me how to implement iSCSI storage within a CentOS Linux virtual machine. From there it was a matter of creating a storage adapter within vCenter and exporting the iSCSI datastores to the environment.

This post will go through the steps to configure the iSCSI storage within a Linux VM, export it to vCenter and add it into the IaaS offering as a VMFS datastore. I found this extremely helpful in my preparation for the exam and in learning how to troubleshoot misconfiguration settings within the iSCSI VM — making mistakes are often the best way to learn!

Continue reading “VCP5: Creating an iSCSI lab environment for vSphere”