VMware Integrated OpenStack vs vCloud Director

boxers_header

If you have not already noticed, a lot of my work these days is in the OpenStack space, specifically using VMware Integrated OpenStack. As often happens when working on a new technology or service offering, I get the question

Is OpenStack going to replace VMware vCloud Director as my primary cloud management platform (CMP)?

I remember during TAM-day at VMworld 2014 the announcement for VMware Integrated OpenStack (VIO) and one of the members of the audience asked Carl Eschenbach why VMware was pivoting away from vRealize Automation (vRA) only a year after telling customers that was the go-forward strategy. It begged the question, just which CMP was VMware backing long-term? I don’t have an answer for you. Personally, I think the three CMPs serve different purposes and fit within different customer use-cases — some of which overlap, others don’t.

In order to answer the question, I put together some information comparing the two products to highlight both their similarities and differences. The following is not a complete list, but will highlight the key points for a team or business trying to determine which direction to use for their private cloud.

Comparison Chart

CategoryVMware Integrated OpenStackVMware vCloud Director
ComputeCombined CPU & Memory resourcesCombined CPU & Memory resources
ComputevSphere cluster endpointsvSphere cluster endpoints
vSphere Resource Pool endpoints
Compute2 vCenter Server maximum7+ vCenter Server support
ComputevSphere HA & DRS supportvSphere HA & DRS support
Compute25,000/40,000 tenant-VM maximum
StorageGlobal catalog service (Glance)Global catalog service
StoragePluggable block volumes (Cinder)
VMFS-backed storage
VMFS backed-storage
StoragevSphere Storage Policy enforcementTiered storage through storage policies
StoragevSphere Storage DRS supportvSphere Storage DRS support
StorageObject-based storage (Swift)
NetworkingVLAN-backed portgroup integration
- Single vCenter
NetworkingVMware NSX-v integration
- Single or multi-vCenter
VMware NSX-v integration
NetworkingFull NSX-v functionality
- NSX Edge tenant support.
- NSX DFW for Security Groups
- Load Balancer (as-a-Service)
Limited NSX-v functionality
- NSX Edge tenant support
- No NSX DFW support.
- No NSX Load Balancer support.
NetworkingShared external tenant networks
- No isolation between tenants
Per-tenant, isolated external tenant networks
NetworkingSingle cluster for Edge services for all tenants.Edge/Compute services present on all compute workload cluster.
Cloud ServicesGlobal & Tenant defined VM sizes (flavors)
Cloud ServicesGlobal & Tenant defined VM images.Global catalog for sharing images and vApps.
Cloud ServicesMetering & telemetry functionality for auto-scaling (Ceilometer)
Cloud ServicesApplication stack orchestration (Heat)
- Standard JSON & YAML support.
- Cross-cloud functional.
vApp deployment orchestration
Cloud ServicesStandard API framework
- Industry standard APIs
- Compatible with AWS & S3
Proprietary API framework
ManagementvApp plugin inside vCenter for administrative tasks.
ManagementDistributed Management stack
- Applications clustered within management stack (MongoDB, MariaDB, etc).
- Integrated load balancer for services.
Multiple vCD cell support
- External load balancer required.
- Database is not natively clustered.

Ultimately the answer to the question is going to depend on your use-case and your requirements. I believe either option could work in a majority of the private cloud architectures out there — especially considering how VMware Integrated OpenStack has simplified the deployment and lifecycle management of OpenStack.

Let me know what you think on Twitter! Enjoy.

Collecting VMware Integrated OpenStack Logs

banner-header-digital

From time to time it is necessary to escalate an issue to the VMware GSS team. Oftentimes they will ask for a support bundle to gather all of the relevant logs for a product in order to perform troubleshooting steps. VMware Integrated OpenStack provides a method for gathering these logs in an easy manner and VMware has published a KB 2125261 article to assist vSphere Administrators.

The long and short of it, there is a single command to be executed on the VIO OMS management virtual machine. The following screenshot shows the required command, the expected output and displays the notice of where the support bundle is saved.

vio_getlogs

Hopefully you are already using a vRealize Log Insight service for redirecting all of the VMware Integrated OpenStack logs from both the OpenStack management VMs and the OMS management VM. If not, I encourage you to do so — a future article with go over the vRealize Log Insight Management Pack for OpenStack.

Multi-tenant OpenStack with NSX – Part 2

multi-tenancy-network

The post yesterday discussed a method for having segmented multi-tenant networks inside of OpenStack. As a series of test cases were worked through with a setup of this nature, a large gaping hole in OpenStack came into view.

What does the previously described multiple external networks look like inside OpenStack?

admin-external-networks
Admin view with multiple external networks defined in OpenStack.
tenant1-external-networks
Networks from Tenant 1 view.
tenant2-external-networks
Networks from Tenant 2 view.

In the second and third screenshots, you can see the two tenants see both external networks, but they only see a subnet listed for the external network that was created with their respective tenant-id. At first glance, this would seem to be doing what was intended — each tenant receiving their own external network to consume floating IP addresses from. Unfortunately, it begins to breakdown when a tenant goes to Compute –> Access & Security –> Floating IPs in Horizon.

floating-ips
Multiple tenant floating IP addresses.

The above screenshot shows a tenant being assigned an floating IP address from what should have been an external network they did not have access to.
facepalm

I felt pretty much like Captain Picard after working through the test cases. Surely, OpenStack would allow a design where tenants have segmented external networks — right?

Unfortunately, OpenStack does not honor this type of segmented external networking design — it will allow any tenant consume/claim a floating IP address from any of the other external networks. To read how OpenStack fully implements external networks, you can read the documentation here. At issue here is highlighted here,

 

 

Nevertheless, the concept of ‘external’ implies some forms of sharing, and this has some bearing on the topologies that can be achieved. For instance it is not possible at the moment to have an external network which is reserved to a specific tenant.

Essentially, OpenStack Neutron thinks of external networks differently, than I believe most architects. It also does not clearly honor the tenant-id attribute that is specified when the network is created, nor when the shared attribute is not enabled on the external network. The methodology OpenStack Neutron uses is more in-line with the AWS consumption model — everyone drinks from the same pool and there is no segmentation between the tenants. I personally do not believe that model works in a private cloud where there are multiple tenants.

The next post in the series will discuss a potential design for working around the issue inside OpenStack Neutron.

Multi-tenant OpenStack with NSX – Part 1

openstack-nsx

I have been working on an OpenStack architecture design using VMware Integrated OpenStack (VIO) for the past several months. The design itself is being developed for an internal cloud service offer and is the design for my VCDX certification pursuit in 2017. As the design has gone through the Proof-of-Concept and later the Pilot phases, determining how to offer a multi-tenant/personal cloud offering has presented itself to be challenging. The design relies heavily on the NSX software defined networking (SDN) platform, both because of requirements from VIO and service offering requirements. As such, all north-south traffic goes through a set of NSX Edge devices. Prior to hitting the north-south boundary however, the east-west tenant traffic needed to be addressed.

I had seen other designs where a single, large (/22 or greater) external network was relied upon by all OpenStack tenants talked about. However, for a personal cloud or multi-tenant cloud offering based on OpenStack, I felt this was not a good design choice for this environment. I wanted to have a higher-level of separation between tenants and one option involved a secondary pair of NSX Edge devices for each tenant. The following diagram describes the logical approach.

OpenStack NSX

The above logical representation expresses how the deployed tenant NSX Edges are connected upstream to the security or L3 network boundary and downstream to the OpenStack Project that is created for each tenant. At a small to medium scale, I believe this model works operationally — the tenant NSX Edges create a logical separation between each OpenStack tenant and (if assigned to on a per-team basis) should remain relatively manageable as the environment scales to dozens of tenants.

The NSX Edge devices are deployed inside the very same OpenStack Edge cluster specified during the VIO deployment, however they are being managed outside of OpenStack and the tenant has no control over them. Each secondary pair of NSX Edge devices for the tenant are configured with two interfaces — one uplink to the north-south NSX Edge security gateway and a single interface that becomes the external network inside OpenStack. It is this internal interface that the OpenStack tenant deploys all of their own micro-segmented networks and consumes floating IP addresses from.

An upcoming post will describe the deployment and configuration of these tenant NSX Edge devices and their corresponding configuration inside OpenStack.

I am concerned with this approach once the scale of the environment grows to 50+ OpenStack tenants. It is at that point where operational management difficulties will begin to surface and cause challenges. Specifically, the lack of available automation for creating, linking and configuring both the NSX component deployment and tenant creation within OpenStack itself (projects, users, role-based access, external networks, subnets, etc).


Update

After writing the initial article in the post, further testing was performed that has influenced how multi-tenant OpenStack external networks can be implemented. As such, Part 2 of the series includes the new information. For completeness, I’ve decided to include that information here as well for archival purposes.

Part 2

The post yesterday discussed a method for having segmented multi-tenant networks inside of OpenStack. As a series of test cases were worked through with a setup of this nature, a large gaping hole in OpenStack came into view.

What does the previously described multiple external networks look like inside OpenStack?

admin-external-networks
Admin view with multiple external networks defined in OpenStack.
tenant1-external-networks
Networks from Tenant 1 view.
tenant2-external-networks
Networks from Tenant 2 view.

In the second and third screenshots, you can see the two tenants see both external networks, but they only see a subnet listed for the external network that was created with their respective tenant-id. At first glance, this would seem to be doing what was intended — each tenant receiving their own external network to consume floating IP addresses from. Unfortunately, it begins to breakdown when a tenant goes to Compute –> Access & Security –> Floating IPs in Horizon.

floating-ips
Multiple tenant floating IP addresses.

The above screenshot shows a tenant being assigned an floating IP address from what should have been an external network they did not have access to.
facepalm

I felt pretty much like Captain Picard after working through the test cases. Surely, OpenStack would allow a design where tenants have segmented external networks — right?

Unfortunately, OpenStack does not honor this type of segmented external networking design — it will allow any tenant consume/claim a floating IP address from any of the other external networks. To read how OpenStack fully implements external networks, you can read the documentation here. At issue here is highlighted here,

 

 

Nevertheless, the concept of ‘external’ implies some forms of sharing, and this has some bearing on the topologies that can be achieved. For instance it is not possible at the moment to have an external network which is reserved to a specific tenant.

Essentially, OpenStack Neutron thinks of external networks differently, than I believe most architects. It also does not clearly honor the tenant-id attribute that is specified when the network is created, nor when the shared attribute is not enabled on the external network. The methodology OpenStack Neutron uses is more in-line with the AWS consumption model — everyone drinks from the same pool and there is no segmentation between the tenants. I personally do not believe that model works in a private cloud where there are multiple tenants.

The next post in the series will discuss a potential design for working around the issue inside OpenStack Neutron.

VMware Integrated OpenStack – Collapse Compute & Edge Clusters

the_openstack_logo-svg

VMware Integrated OpenStack (VIO) introduced the ability to deploy to multiple vCenter Servers with version 2.5. The feature allowed the OpenStack management VMs to be deployed inside a control plane vCenter, while allowing the data plane to use a separate vCenter server. The architecture model still required three clusters:

  • Management Cluster (Management vCenter Server)
  • Compute Cluster(s) (Workload vCenter Server)
  • Edge Cluster (Workload vCenter Server)

The three cluster architecture follows the published best practices from both VIO and NSX. Having a dedicated Edge cluster should free up tenant resources and prevent potential issues with network noisy-neighbors. However, having a dedicated cluster just for NSX Edge VMs could be overkill in some environments from both a cost and compute perspective. If you are also using Virtual SAN to leverage hyper-converged infrastructure (HCI), the cost increases considerably with licensing costs for vSphere, NSX and Virtual SAN for hosts that will be extremely under-utilized.

So how can you collapse the compute and edge clusters in a VMware Integrated OpenStack environment?

In version 3.0 there is a configuration change that makes it possible to collapse these two cluster. Performing the following steps will allow you to deploy a smaller footprint OpenStack deployment using VIO.

$ sudo vim /opt/vmware/vio/etc/omjs.properties

Add the following lines to the end of the configuration file:

## Collapse the Edge/Compute clusters
oms.allow_shared_edge_cluster = true

Restart the OMS services
$ sudo restart oms

Once the OMS services have been restarted, the VIO Deployment UI will now allow you to deploy the Edge VMs inside the same Compute cluster on the control plane vCenter Server instance.

A couple caveats to this approach to be aware of:

  • All tenant deployed Edge VMs will live in the collapsed Edge/Compute cluster. As the environment scales to include multiple compute clusters, only this initial Edge/Compute cluster will have the Edge VMs.
  • The OpenStack Horizon UI is unaware of these tenant deployed Edge VMs, so reporting on utilization within the compute cluster is shown on the screen, the rates will have discrepancies — depending on how large the environment is.

Your mileage may vary, but this option allows for some additional flexibility when deploying VMware Integrated OpenStack.