Note: The following blog post is only relevant to VMware Integrated OpenStack deployments.
In a pilot environment running VMware Integrated OpenStack (VIO) v3.0, one of the ESXi management nodes experienced a network isolation event. As a result of the event, vSphere HA responded accordingly and began restarting the VMs on the isolated ESXi node onto other ESXi nodes. The isolated ESXi node happened to have the secondary VIO controller VM on it. When the VIO controller VM was restarted through vSphere HA on the new ESXi node, the operating system came online quickly. However, the vRealize Operations dashboard for OpenStack still reported the services in critical status.
When I logged into the secondary OpenStack controller VM, I noticed there were no OpenStack services running. That’s not good.
Digging in a bit deeper, I logged into the VIO management VM and ran a viocli command to check the status of the environment to see what other issues may be in existence.
It just so happens that this is expected behavior in the current VIO release when a vSphere HA event occurs. I wouldn’t classify that as ideal at all, but it’s software and sometimes we have to workaround limitations.
In order to restart the services of the secondary controller VM, there are a two options.
- Restart the entire OpenStack management stack.
- Restart just the affected controller VM.
Both require use of the vSphere Web Client with the corresponding VIO plugin.
Select the broken management VM and select the ‘All Actions’ drop-down menu at the top. Follow it up with selecting ‘Restart services’. A small pop-up window will appear verifying this is the action you wish to take.
Once the services are restarted the entire OpenStack management stack should once again begin functional.
One caveat worth noting, if the services have been stopped on the database nodes the restart of a single management VM through the UI may not re-establish the entire stack and a complete restart of the entire stack may still be required.
The vRealize Operations dashboards can play an integral part of a VMware Integrated OpenStack environment, allowing the services to be monitored remotely.
