Install VMware Integrated OpenStack on VCF

I am currently pursuing my VCDX certification and the design I have submitted is based on VMware Cloud Foundation and VMware Integrated OpenStack. As part of the required documentation, I included a deployment guide — unfortunately, it is not as simple as laying down the SDDC components and the VIO vApp for the deployment.

This blog post will cover a couple items that are needed to get the two pieces playing together.

Shared Edge & Workload Cluster

The VCF architecture currently has a limitation that a vCenter Server can only have a single vSphere cluster — it’s a 1:1 relationship. VMware Integrated OpenStack requires either 3 clusters in a single vCenter Server or a management cluster in one vCenter Server instance and two clusters in a second vCenter Server. Neither of these options are compatible with VMware Integrated OpenStack.

In order to make it work, we are going to use a two vCenter Server deployment of VMware Integrated OpenStack and modify the OMS server to combine the NSX Edge and Workload Clusters into one. We do this by editing a single configuration file and restarting the oms service running on the VIO vApp Management (OMS) VM.

$ cd /opt/vmware/vio/etc
$ sudo vim

Add the following line to the end of the file:
oms.allow_shared_edge_cluster = true

$ sudo restart oms

VMware Integrated OpenStack can now be deployed on top of VMware Cloud Foundation.

VXLAN-backed External Network

This one is a bit trickier and is an obstacle whether or not you are using VMware Cloud Foundation as the infrastructure layer.

Logically, the end result for the OpenStack external network is to attach to a VXLAN port group created by NSX. The NSX logical switch network is attached to the internal interface on a NSX Distributed Logical Router.

The following is the logical diagram for the architecture.

external openstack

The issue is that during the deployment of an OpenStack instance using VMware Integrated OpenStack, you have to specify an external network. However, VMware Integrated OpenStack will not allow a vSphere Administrator to select a VXLAN port group during the deployment. I got around this by creating a non-VXLAN port group on the DVS used only for the deployment.

Once the OpenStack deployment is complete, I needed to attach the actual VXLAN-backed port group as the external network.

SSH to the OMS server
$ ssh -l viouser oms.domain.local

SSH to an OpenStack controller VM
$ ssh controller01
$ sudo cp /root/cloudadmin_v3.rc .
$ source cloudadmin_v3.rc
$ neutron

(neutron) net-list
(neutron) net-create --provider:network_type=portgroup --provider:physical_network=virtualwire-XX vio-external-network
(neutron) net-list

The network will now appear in the OpenStack network list. Go ahead and create your subnet for the external IP addresses, based on the network assignment in your environment.

If you have questions or issues with implementing these changes in your environment, please reach out.

Troubleshoot OpenStack Neutron + NSX


OpenStack Neutron likes to use some pretty awesome reference IDs for the tenant network objects. You know, helpful strings like ec43c520-bfc6-43d5-ba2b-d13b4ef5a760. The first time I saw that, I said to myself that is going to be a nightmare when trying to troubleshoot an issue.


Fortunately, VMware NSX also uses a similar character string when it creates logical switches. If NSX is being used in conjunction with OpenStack Neutron, magic happens. The logical switch is created with a string like vxw-dvs-9-virtualwire-27-sid-10009-ec43c520-bfc6-43d5-ba2b-d13b4ef5a760.


A keen eye will have noticed the OpenStack Neutron reference ID is included in the NSX logical switch name!

From there you can reference the NSX Edge virtual machines and see which interface the NSX logical switch is attached to. This tidbit of information proved useful today when I was troubleshooting an issue for a developer and is a piece of information going into my VCDX SOP document.



Using the VMware Validated Design Reference Material

Caution, this post is highly opinionated.

I am deep into the process of completing my VCDX design documentation and application for (hopefully) a Q2 2017 defense. As it so happens, a short conversation was had on Twitter today regarding a post on the VMware Communities site for the VMware Validation Design for SDDC 3.x, including a new design decision checklist.


The latest version of the VMware Validated Design (VVD) is a pretty awesome product for customers to reference when starting out on their private cloud journey. That being said, it is by no means a VCDX design or a set of materials that could simply be re-purposed for a VCDX design.

Why? Because there are no customer requirements.

For the same reason a hypothetical (or fake) design is often discouraged by people in the VCDX community, the VVD suffers from the same issue. In a vacuum you can make any decision you want, because there are no ramifications from your design decision. In the real-world this is simply not the case.

Taking a look at the Design Decisions Checklist, it goes through the over 200 design decisions the VVD made in the course of developing the reference architecture. The checklist does a good job of laying out the fields the design decision covers, like:

  • Design Decision
  • Design Justification
  • Design Implication

Good material. But if you’ve read my other post on design decisions, which you may or may not agree with, it highlights that a decision justification is made based on a requirement.

Let’s take a look at just one of the design decisions made by the VVD product and highlighted in the checklist.


The decision is to limit a single compute pod to a single physical rack, as in no cross-rack clusters. Sounds like a reasonable decision, especially if the environment had a restriction on L2 boundaries or some other requirement. But what if I have a customer requirement that said a compute node must be able to join any compute pod (cluster) regardless of physical rack location within a data center?

Should I ignore that requirement because the VVD says to do otherwise?

Of course not.

My issue with the Twitter conversation is two-fold:

  1. The VVD design decisions are not in fact design decisions, but design recommendations. They can be used to help a company, group or architect to determine, based on their requirements, which of these “decisions” should be leveraged within their environment. They are not die-hard decisions that must be adhered to.
  2. From a VCDX perspective, blindly assuming you could copy/paste any of these design decisions and use them in a VCDX defense is naive. You must have a justification for every design decision made and it has to map back to a customer requirement, risk or constraint.

I also do not think that is what  was saying when he initially responded to the Tweet about the checklist. I do think though that some people may actually think they can just take the VVD, wrap it in a bow and call it good.

My suggestion is to take the VVD design documentation and consider it reference material, just like the many other great books and online resources available to the community. It won’t work for everyone, because every design has different requirements, constraints and risks. Take the bits that work for you and expand upon them. Most importantly, understand why you are using or making that design decision.

Let me know what you think on Twitter.

Again, this post is highly opinionated from my own limited perspective. Do not mistake it for the opinion of VMware or any VCDX certified individuals.

NSX DLR Designated Instance

nsx designated instance

While a great show, we are going to talk about something slightly different — the NSX Distributed Logical Router (DLR) Designated Instance. NSX has many great features and also many caveats when implementing some of those great features — like having a Designated Instance when using a DLR.

So what is a Designated Instance? Honestly, I did not know what it was until a conversation earlier today with a few co-workers who are a bit more knowledgable with NSX than me. Essentially a Designated Instance is an elected ESXi host that will answer all new requests initially — also known as a single-point of failure.

Let’s look at the logical network diagram I posted yesterday.


Pretty sweet right?

The issue is when the DLR is connected directly to a VLAN. While technically not a problem — it does exactly what you’d expect it does — it results in having to have one of the ESXi hosts in the transport zone act as the Designated Instance. The result is that if the Designated Instance ESXi host encounters a failure, any new traffic will fail until the election process is complete and a new Designated Instance is chosen.

So is it possible to not need a Designated Instance when using a DLR? Yes.

It involves introducing another logical NSX layer into the virtual network design. If you saw my tweet earlier, this is what I meant.

I like , but sometimes I think it adds a little too much complexity for operational simplicity.

Adding a set of ECMP edges above the DLR and connecting the two together will eliminate the requirement for NSX to use a the Designated Instance. Here is what an alternative to the previous design would look like.

external openstack

Essentially what I’ve done is create another VXLAN, with a corresponding NSX Logical Switch and connect the uplink from the DLR to it. Then the ECMP Edges use the same Logical Switch as their internal interface. It is on the uplink side of the ECMP Edge where the P2V layer takes place and the VLAN is connected.

Using this design allows the environment to use a dynamic routing protocol between both the DLR and ECMP Edges and ECMP Edges and the upstream physical network — although mileage may vary depending on your physical network. The ECMP Edges introduce additional scalability — although limited to 8 — based on the amount of North-South network traffic and the bandwidth required to meet the tenant needs. Features like vSphere Anti-Affinity rules can mitigate a failure of a single ESXi host, which you cannot do when there is a Designated Instance. The design can also take into consideration a N+x scenario for when to scale the ECMP Edges.

So many options open up when NSX is introduced into an architecture, along with a lot of extra complexity. Ultimately the decision should be based on the requirements and the stakeholders risk acceptance. Relying on a Designated Instance may be acceptable to a stakeholder, while adding more complexity to the design may not be.

Until next time, enjoy!

Multi-tenant OpenStack with NSX – Part 3


This next post in the series about multi-tenant OpenStack with NSX will discuss the use of a Distributed Logical Router as the bridge between OpenStack and the physical network. If you have not read the previous posts in the series, you can catch up by reading this one.

Originally the plan had been to segment each OpenStack tenant off with their own HA-pair of NSX Edges. However, after discovering that OpenStack does not honor the tenant-id parameter, nor the disabling the Shared parameter within the external network object and adjustment had to be made. Working through the problem it became clear that a NSX Distributed Logical Router (DLR) could be leveraged and would also scale as the environment grows beyond a few dozen tenants. The new multi-tenant network design for OpenStack now looks like this:


The logical diagram shows how the uplink of the DLR is the upstream (north-south) boundary for the environment. The internal interface on the DLR is the external OpenStack network, and is leveraging VXLAN to provide the floating IP addresses the OpenStack tenants will consume. If you are unfamiliar with how a DLR operates, I recommend reading this post from Roie Ben Haim on his blog.

Basically, the DLR relies upon two components — the control VMs and a kernel module inside vSphere to inject routing on each ESXi host within the NSX transport zone. It is the inclusion of this kernel module on every ESXi host that will allow this virtual network design to scale as the environment does. In the previous design, the individual NSX Edges would merely be deployed in an HA-pair and only the active VM would be handling all of the traffic for that particular tenant. With the DLR, although all tenant traffic will be going through the single layer, that layer is distributed across the entire workload environment.

After the DLR is created, the routes can be see within each ESXi host, as shown in the following image.


The downside that still remains is the shared pool of IP addresses for all tenants to consume from. Operationally it will mean having to manage the tenant quotas for floating IP addresses and making sure there is no over-allocation. I would still like to see the OpenStack community take on the extra work of honoring the tenant-id parameter when creating an external network within OpenStack so that the option would exist to have individual tenant floating IP address pools.

Tuesday’s post will include detailed instructions on how to deploy and configure both the NSX portion of this deployment and the OpenStack pieces to tie it all together. Enjoy!