Tag: vROPs

 

Note: The following blog post is only relevant to VMware Integrated OpenStack deployments.

In a pilot environment running VMware Integrated OpenStack (VIO) v3.0, one of the ESXi management nodes experienced a network isolation event. As a result of the event, vSphere HA responded accordingly and began restarting the VMs on the isolated ESXi node onto other ESXi nodes. The isolated ESXi node happened to have the secondary VIO controller VM on it. When the VIO controller VM was restarted through vSphere HA on the new ESXi node, the operating system came online quickly. However, the vRealize Operations dashboard for OpenStack still reported the services in critical status.

vRealize Operations OpenStack Services Dashboard

When I logged into the secondary OpenStack controller VM, I noticed there were no OpenStack services running. That’s not good.

Digging in a bit deeper, I logged into the VIO management VM and ran a viocli command to check the status of the environment to see what other issues may be in existence.

It just so happens that this is expected behavior in the current VIO release when a vSphere HA event occurs. I wouldn’t classify that as ideal at all, but it’s software and sometimes we have to workaround limitations.

In order to restart the services of the secondary controller VM, there are a two options.

  1. Restart the entire OpenStack management stack.
  2. Restart just the affected controller VM.

Both require use of the vSphere Web Client with the corresponding VIO plugin.


Select the broken management VM and select the ‘All Actions’ drop-down menu at the top. Follow it up with selecting ‘Restart services’. A small pop-up window will appear verifying this is the action you wish to take.

Once the services are restarted the entire OpenStack management stack should once again begin functional.

One caveat worth noting, if the services have been stopped on the database nodes the restart of a single management VM through the UI may not re-establish the entire stack and a complete restart of the entire stack may still be required.

The vRealize Operations dashboards can play an integral part of a VMware Integrated OpenStack environment, allowing the services to be monitored remotely.

Read More

monitoring-header

The previous post discussed the use of the vRealize Operations Management Pack for OpenStack and Endpoint Agent in order to provide detailed service-level monitoring within an environment. The management pack comes with nearly 200 pre-defined alerts for OpenStack that can be leveraged to understand what is occurring within the environment. As I’ve gone through the alerts, these are the key alerts that can be leveraged to understand when any of the OpenStack services are experiencing a partial or complete outage.

OpenStack Compute Alerts

ServiceAlert NameTriggers
NovaAll nova-network services are unavailableAll nova-network services are unavailable
NovaAll nova-xvpnc-proxy services are unavailableAll nova-xvpnc-proxy services are unavailable
NovaAll nova-scheduler services are unavailableAll nova-scheduler services are unavailable
NovaAll nova-api services are unavailableAll nova-api services are unavailable
NovaAll nova-consoleauth services are unavailableAll nova-consoleauth services are unavailable
NovaAll nova-cert services are unavailableAll nova-cert services are unavailable
NovaAll nova-compute services are unavailableAll nova-compute services are unavailable
NovaAll nova-conductor services are unavailableAll nova-conductor services are unavailable
NovaAll nova-console services are unavailableAll nova-console services are unavailable
NovaAll nova-novncproxy services are unavailableAll nova-novncproxy services are unavailable
NovaAll nova-objectstore services are unavailableAll nova-objectstore services are unavailable
NovaThe nova-compute service is unavailableNova-compute status is unknown
NovaThe nova-objectstore service is unavailableNova-objectstore status is unknown
NovaThe nova-conductor service is unavailableNova-conductor status is unknown
NovaThe nova-api service is unavailableNova-api status is unknown
NovaThe nova-cert service is unavailableNova-cert status is unknown
NovaThe nova-console service is unavailableNova-console status is unknown
NovaThe nova-consoleauth service is unavailableNova-consoleauth status is unknown
NovaThe nova-network service is unavailableNova-network status is unknown
NovaThe nova-novnc-proxy service is unavailableNova-novncproxy status is unknown
NovaThe nova-schedulerNova-scheduler status is unknown
NovaThe nova-xvpvnc-proxy service is unavailableNova-xvpvnc-proxy status is unknown

OpenStack Storage Alerts

ServiceAlert NameTriggers
GlanceAll glance-api services are unavailableAll glance-api services are unavailable
GlanceAll glance-registry services are unavailableAll glance-registry services are unavailable
GlanceThe glance-api service is unavailableGlance-api status is unknown
GlanceThe glance-registry service is unavailableGlance-registry status is unknown
CinderAll cinder-api services are unavailableAll cinder-api services are unavailable
CinderAll cinder-scheduler services are unavailableAll cinder-scheduler services are unavailable
CinderAll cinder-volume services are unavailableAll cinder-volume services are unavailable
CinderThe cinder-volume service is unavailableCinder-volume status is unknown
CinderThe cinder-api service is unavailableCinder-api status is unknown
CinderThe cinder-scheduler service is unavailableCinder-scheduler status is unknown

OpenStack Network Alerts

ServiceAlert NameTriggers
NeutronThe neutron-lbaas-agent service is unavailableNeutron-lbaas-agent status is unknown
NeutronThe neutron-server service is unavailableNeutron-server status is unknown
NeutronAll neutron-dhcp-agent services are unavailableAll neutron-dhcp-agent services are unavailable
NeutronAll neutron-l3-agent services are unavailableAll neutron-l3-agent services are unavailable
NeutronAll neutron-lbaas-agent services are unavailableAll neutron-lbaas-agent services are unavailable
NeutronAll neutron-metadata-agent services are unavailableAll neutron-metadata-agent services are unavailable
NeutronAll neutron-server services are unavailableAll neutron-server services are unavailable
NeutronThe neutron-dhcp-agent service is unavailableNeutron-dhcp-agent status is unknown
NeutronThe neutron-l3-agent service is unavailableNeutron-l3-agent status is unknown
NeutronThe neutron-lbaas-agent service is unavailableNeutron-lbaas-agent status is unknown
NeutronThe neutron-metadata-agent service is unavailableNeutron-metadata-agent status is unknown
NeutronThe neutron-server service is unavailableNeutron-server status is unknown

OpenStack Auxiliary Alerts

ServiceAlert NameTriggers
HeatAll heat-api services are unavailableAll heat-api services are unavailable
HeatAll heat-api-cfn services are unavailableAll heat-api-cfn services are unavailable
HeatAll heat-api-cloudwatch services are unavailableAll heat-api-cloudwatch services are unavailable
HeatAll heat-engine services are unavailableAll heat-engine services are unavailable
HeatThe heat-api service is unavailableHeat-api status is unknown
HeatThe heat-api-cfn service is unavailableHeat-api-cfn status is unknown
HeatThe heat-api-cloudwatch status is unavailableHeat-api-cloudwatch status is unknown
HeatThe heat-engine service is unavailableHeat-engine service is unknown
KeystoneAll keystone-all services are unavailableAll keystone-all services are unavailable
KeystoneThe keystone-all service is unavailableKeystone-al service is unknown
MySQLAll MySQL services are unavailableAll MySQL services are unavailable
MySQLThe MySQL Database service is unavailableMySQL status is unknown
ApacheAll Apache services are unavailableAll Apache services are unavailable
ApacheThe Apache service is unavailableApache status is unknown
JarvisAll Jarvis services are unavailableAll Jarvis services are unavailable
MemcachedAll Memcached services are unavailableAll Memcached services are unavailable
MemcachedThe memcached service is unavailableMemcached status is unknown
RabbitMQAll RabbitMQ services are unavailableAll RabbitMQ services are unavailable
RabbitMQThe Rabbit Messaging service is unavailableRabbit Message Queue status is unknown
OMSAll tc-oms services are unavailableAll tc-oms services are unavailable
OMSAll tc-osvmw services are unavailableAll tc-osvmw services are unavailable
vPostGresAll vPostGres services are unavailableAll vPostGres services are unavailable
vPostGresThe vpostgres service is unavailableVpostgres status is unknown
CeilometerThe ceilometer-agent-central service is unavailableCeilometer-agent-central status is unknown
CeilometerThe ceilometer-agent-compute service is unavailableCeilometer-agent-compute status is unknown
CeilometerThe ceilometer-agent-notification service is unavailableCeilometer-agent-notification status is unknown
CeilometerThe ceilometer-alarm-evaluator service is unavailableCeilometer-alarm-evaluator status is unknown
CeilometerThe ceilometer-alarm-notifier service is unavailableCeilometer-alarm-notifier status is unknown
CeilometerThe ceilometer-api service is unavailableCeilometer-api status is unknown
CeilometerThe ceilometer-collector service is unavailableCeilometer-collector status is unknown

Use of these alerts will help the environment be ready for a production deployment where an SLA can be attached. Enjoy!

Read More

banner-header-digital

Being able to monitor OpenStack is key when running a production private cloud. Fortunately, VMware has provided several tools for monitoring OpenStack — specifically VMware Integrated OpenStack — when deployed in a production environment. If your environment is already leveraging vRealize Operations, there is a management pack for OpenStack and NSX that when used together will provide dashboards and pre-defined alerts for OpenStack.

vRealize Operations Management Pack for OpenStack

Available on the VMware Solution Exchange website, the vRealize Operations Management Pack for OpenStack provides integration between vRealize Operations and VMware Integrated OpenStack. The management pack includes several pre-installed dashboards, collecting data through the native OpenStack APIs.

The management pack requires the vRealize Operations Management Pack for NSX also be installed, to correctly gather data related to the OpenStack Neutron service.

The management pack includes the following dashboards which can be leveraged to gain a deeper understanding of the OneCloud OpenStack environment from an operations standpoint.

  • OpenStack Services
  • OpenStack Compute Infrastructure
  • OpenStack Network Infrastructure
  • OpenStack vCenter Storage Infrastructure
  • OpenStack Tenants

The OpenStack Services dashboard displays the status the of the OpenStack services running on the VMware Integrated OpenStack management virtual machines.

vrops_mp_final

vRealize Operations Endpoint Agent for OpenStack

The vRealize Operations monitoring capabilities can be enhanced when the Endpoint Agent for OpenStack is installed on the OpenStack management virtual machines. As stated in the vRealize Operations Management Pack for OpenStack documentation, the Endpoint Agent can monitor the following services and displaying their status in the vRealize Operations OpenStack Services dashboard.

The VMware Integrated OpenStack OMS virtual machine provides an automated installation workflow for the Endpoint Agent on each of the management nodes. The workflow can be leveraged post-deployment to facilitate the installation of the required Endpoint Agent package on the local operating system.

epops_01 epops_02

 

 

Further details on the exact process of installing the Endpoint Agent (epops) can be viewed here.

Having these pieces of software included in your environment will help ensure the monitoring of the services and the capacity within an OpenStack cloud is being taken care of correctly. Of course there are other tools that can be leveraged as well, however I have found these to be extremely useful within my vSphere environments. The blog post tomorrow will be an overview of the alert definitions the vRealize Operations Management Pack for OpenStack includes.

Enjoy!

Read More