Category: OpenStack
Posted on

monitoring-header

OpenStack Neutron likes to use some pretty awesome reference IDs for the tenant network objects. You know, helpful strings like ec43c520-bfc6-43d5-ba2b-d13b4ef5a760. The first time I saw that, I said to myself that is going to be a nightmare when trying to troubleshoot an issue.

ve_nsx_neutron_troubleshoot01

Fortunately, VMware NSX also uses a similar character string when it creates logical switches. If NSX is being used in conjunction with OpenStack Neutron, magic happens. The logical switch is created with a string like vxw-dvs-9-virtualwire-27-sid-10009-ec43c520-bfc6-43d5-ba2b-d13b4ef5a760.

ve_nsx_neutron_troubleshoot02

A keen eye will have noticed the OpenStack Neutron reference ID is included in the NSX logical switch name!

From there you can reference the NSX Edge virtual machines and see which interface the NSX logical switch is attached to. This tidbit of information proved useful today when I was troubleshooting an issue for a developer and is a piece of information going into my VCDX SOP document.

ve_nsx_neutron_troubleshoot03

Enjoy!

Read More
Posted on

monitoring-header

Continuing the OpenStack + NSX series (Part 1, Part 2 and Part 3) on deploying a multi-tenant OpenStack environment that relies upon NSX, this post will cover the details of the deployment and configuration.

There have been a couple options discussed through the series, including the logical graphic with that relies on a NSX DLR w/o ECMP Edges:

nsx-dlr-openstack

Or a logical virtual network design with a DLR and ECMP Edges:

external openstack

Regardless of which virtual network design you choose, the configuration of the NSX Distributed Logical Router and the tie into OpenStack will need to be configured. In the course of building out a few VMware Integrated OpenStack labs, proof-of-concepts and pilot environments, I’ve learned a few things.

Rather than go through all 30+ steps to implement the entire stack, I want to simply highlight a few keys points. When you configure the DLR, you should end up with two interfaces — an uplink to either the ECMP layer or the physical VLAN and an internal interface to the OpenStack external VXLAN network.

ext_openstack_dlr14

Once the DLR is deployed, you can log into any of the ESXi hosts within the NSX transport zone and verify the routes are properly in place with a few simple CLI commands.

ext_openstack_dlr19

The implementation of tying the NSX components into OpenStack is now ready to be completed. I prefer to use the API method, using the neutron CLI — log into the VIO management server and then either of the Controller VMs.

ext_openstack_dlr20

ext_openstack_dlr22

Key points to remember here:

  • The physical_network parameter is the just the virtualwire-XX string from the NSX-created portgroup.
  • The name for the network must exactly match the NSX Logical Switch that was created for the OpenStack external network.

The commands I used here to create the network inside OpenStack:

$ source <cloudadmin_v3>
$ neutron net-list
$ neutron net-create --provider:network_type=portgroup --provide:physical_network=virtualwire-XX nsx_logical_switch_name
$ neutron net-list

All that remains is adding a subnet to the external network inside OpenStack, which can be performed through the Neutron CLI or the Horizon UI. All-in-all it is a pretty easy implementation, just make sure you remember to reference the proper the object names in NSX when creating the OpenStack network objects.

Enjoy!

Read More

design-sla-banner

This next post in the series about multi-tenant OpenStack with NSX will discuss the use of a Distributed Logical Router as the bridge between OpenStack and the physical network. If you have not read the previous posts in the series, you can catch up by reading this one.

Originally the plan had been to segment each OpenStack tenant off with their own HA-pair of NSX Edges. However, after discovering that OpenStack does not honor the tenant-id parameter, nor the disabling the Shared parameter within the external network object and adjustment had to be made. Working through the problem it became clear that a NSX Distributed Logical Router (DLR) could be leveraged and would also scale as the environment grows beyond a few dozen tenants. The new multi-tenant network design for OpenStack now looks like this:

nsx-dlr-openstack

The logical diagram shows how the uplink of the DLR is the upstream (north-south) boundary for the environment. The internal interface on the DLR is the external OpenStack network, and is leveraging VXLAN to provide the floating IP addresses the OpenStack tenants will consume. If you are unfamiliar with how a DLR operates, I recommend reading this post from Roie Ben Haim on his blog.

Basically, the DLR relies upon two components — the control VMs and a kernel module inside vSphere to inject routing on each ESXi host within the NSX transport zone. It is the inclusion of this kernel module on every ESXi host that will allow this virtual network design to scale as the environment does. In the previous design, the individual NSX Edges would merely be deployed in an HA-pair and only the active VM would be handling all of the traffic for that particular tenant. With the DLR, although all tenant traffic will be going through the single layer, that layer is distributed across the entire workload environment.

After the DLR is created, the routes can be see within each ESXi host, as shown in the following image.

ext_openstack_dlr19

The downside that still remains is the shared pool of IP addresses for all tenants to consume from. Operationally it will mean having to manage the tenant quotas for floating IP addresses and making sure there is no over-allocation. I would still like to see the OpenStack community take on the extra work of honoring the tenant-id parameter when creating an external network within OpenStack so that the option would exist to have individual tenant floating IP address pools.

Tuesday’s post will include detailed instructions on how to deploy and configure both the NSX portion of this deployment and the OpenStack pieces to tie it all together. Enjoy!

Read More

monitoring-header

The previous post discussed the use of the vRealize Operations Management Pack for OpenStack and Endpoint Agent in order to provide detailed service-level monitoring within an environment. The management pack comes with nearly 200 pre-defined alerts for OpenStack that can be leveraged to understand what is occurring within the environment. As I’ve gone through the alerts, these are the key alerts that can be leveraged to understand when any of the OpenStack services are experiencing a partial or complete outage.

OpenStack Compute Alerts

ServiceAlert NameTriggers
NovaAll nova-network services are unavailableAll nova-network services are unavailable
NovaAll nova-xvpnc-proxy services are unavailableAll nova-xvpnc-proxy services are unavailable
NovaAll nova-scheduler services are unavailableAll nova-scheduler services are unavailable
NovaAll nova-api services are unavailableAll nova-api services are unavailable
NovaAll nova-consoleauth services are unavailableAll nova-consoleauth services are unavailable
NovaAll nova-cert services are unavailableAll nova-cert services are unavailable
NovaAll nova-compute services are unavailableAll nova-compute services are unavailable
NovaAll nova-conductor services are unavailableAll nova-conductor services are unavailable
NovaAll nova-console services are unavailableAll nova-console services are unavailable
NovaAll nova-novncproxy services are unavailableAll nova-novncproxy services are unavailable
NovaAll nova-objectstore services are unavailableAll nova-objectstore services are unavailable
NovaThe nova-compute service is unavailableNova-compute status is unknown
NovaThe nova-objectstore service is unavailableNova-objectstore status is unknown
NovaThe nova-conductor service is unavailableNova-conductor status is unknown
NovaThe nova-api service is unavailableNova-api status is unknown
NovaThe nova-cert service is unavailableNova-cert status is unknown
NovaThe nova-console service is unavailableNova-console status is unknown
NovaThe nova-consoleauth service is unavailableNova-consoleauth status is unknown
NovaThe nova-network service is unavailableNova-network status is unknown
NovaThe nova-novnc-proxy service is unavailableNova-novncproxy status is unknown
NovaThe nova-schedulerNova-scheduler status is unknown
NovaThe nova-xvpvnc-proxy service is unavailableNova-xvpvnc-proxy status is unknown

OpenStack Storage Alerts

ServiceAlert NameTriggers
GlanceAll glance-api services are unavailableAll glance-api services are unavailable
GlanceAll glance-registry services are unavailableAll glance-registry services are unavailable
GlanceThe glance-api service is unavailableGlance-api status is unknown
GlanceThe glance-registry service is unavailableGlance-registry status is unknown
CinderAll cinder-api services are unavailableAll cinder-api services are unavailable
CinderAll cinder-scheduler services are unavailableAll cinder-scheduler services are unavailable
CinderAll cinder-volume services are unavailableAll cinder-volume services are unavailable
CinderThe cinder-volume service is unavailableCinder-volume status is unknown
CinderThe cinder-api service is unavailableCinder-api status is unknown
CinderThe cinder-scheduler service is unavailableCinder-scheduler status is unknown

OpenStack Network Alerts

ServiceAlert NameTriggers
NeutronThe neutron-lbaas-agent service is unavailableNeutron-lbaas-agent status is unknown
NeutronThe neutron-server service is unavailableNeutron-server status is unknown
NeutronAll neutron-dhcp-agent services are unavailableAll neutron-dhcp-agent services are unavailable
NeutronAll neutron-l3-agent services are unavailableAll neutron-l3-agent services are unavailable
NeutronAll neutron-lbaas-agent services are unavailableAll neutron-lbaas-agent services are unavailable
NeutronAll neutron-metadata-agent services are unavailableAll neutron-metadata-agent services are unavailable
NeutronAll neutron-server services are unavailableAll neutron-server services are unavailable
NeutronThe neutron-dhcp-agent service is unavailableNeutron-dhcp-agent status is unknown
NeutronThe neutron-l3-agent service is unavailableNeutron-l3-agent status is unknown
NeutronThe neutron-lbaas-agent service is unavailableNeutron-lbaas-agent status is unknown
NeutronThe neutron-metadata-agent service is unavailableNeutron-metadata-agent status is unknown
NeutronThe neutron-server service is unavailableNeutron-server status is unknown

OpenStack Auxiliary Alerts

ServiceAlert NameTriggers
HeatAll heat-api services are unavailableAll heat-api services are unavailable
HeatAll heat-api-cfn services are unavailableAll heat-api-cfn services are unavailable
HeatAll heat-api-cloudwatch services are unavailableAll heat-api-cloudwatch services are unavailable
HeatAll heat-engine services are unavailableAll heat-engine services are unavailable
HeatThe heat-api service is unavailableHeat-api status is unknown
HeatThe heat-api-cfn service is unavailableHeat-api-cfn status is unknown
HeatThe heat-api-cloudwatch status is unavailableHeat-api-cloudwatch status is unknown
HeatThe heat-engine service is unavailableHeat-engine service is unknown
KeystoneAll keystone-all services are unavailableAll keystone-all services are unavailable
KeystoneThe keystone-all service is unavailableKeystone-al service is unknown
MySQLAll MySQL services are unavailableAll MySQL services are unavailable
MySQLThe MySQL Database service is unavailableMySQL status is unknown
ApacheAll Apache services are unavailableAll Apache services are unavailable
ApacheThe Apache service is unavailableApache status is unknown
JarvisAll Jarvis services are unavailableAll Jarvis services are unavailable
MemcachedAll Memcached services are unavailableAll Memcached services are unavailable
MemcachedThe memcached service is unavailableMemcached status is unknown
RabbitMQAll RabbitMQ services are unavailableAll RabbitMQ services are unavailable
RabbitMQThe Rabbit Messaging service is unavailableRabbit Message Queue status is unknown
OMSAll tc-oms services are unavailableAll tc-oms services are unavailable
OMSAll tc-osvmw services are unavailableAll tc-osvmw services are unavailable
vPostGresAll vPostGres services are unavailableAll vPostGres services are unavailable
vPostGresThe vpostgres service is unavailableVpostgres status is unknown
CeilometerThe ceilometer-agent-central service is unavailableCeilometer-agent-central status is unknown
CeilometerThe ceilometer-agent-compute service is unavailableCeilometer-agent-compute status is unknown
CeilometerThe ceilometer-agent-notification service is unavailableCeilometer-agent-notification status is unknown
CeilometerThe ceilometer-alarm-evaluator service is unavailableCeilometer-alarm-evaluator status is unknown
CeilometerThe ceilometer-alarm-notifier service is unavailableCeilometer-alarm-notifier status is unknown
CeilometerThe ceilometer-api service is unavailableCeilometer-api status is unknown
CeilometerThe ceilometer-collector service is unavailableCeilometer-collector status is unknown

Use of these alerts will help the environment be ready for a production deployment where an SLA can be attached. Enjoy!

Read More

banner-header-digital

Being able to monitor OpenStack is key when running a production private cloud. Fortunately, VMware has provided several tools for monitoring OpenStack — specifically VMware Integrated OpenStack — when deployed in a production environment. If your environment is already leveraging vRealize Operations, there is a management pack for OpenStack and NSX that when used together will provide dashboards and pre-defined alerts for OpenStack.

vRealize Operations Management Pack for OpenStack

Available on the VMware Solution Exchange website, the vRealize Operations Management Pack for OpenStack provides integration between vRealize Operations and VMware Integrated OpenStack. The management pack includes several pre-installed dashboards, collecting data through the native OpenStack APIs.

The management pack requires the vRealize Operations Management Pack for NSX also be installed, to correctly gather data related to the OpenStack Neutron service.

The management pack includes the following dashboards which can be leveraged to gain a deeper understanding of the OneCloud OpenStack environment from an operations standpoint.

  • OpenStack Services
  • OpenStack Compute Infrastructure
  • OpenStack Network Infrastructure
  • OpenStack vCenter Storage Infrastructure
  • OpenStack Tenants

The OpenStack Services dashboard displays the status the of the OpenStack services running on the VMware Integrated OpenStack management virtual machines.

vrops_mp_final

vRealize Operations Endpoint Agent for OpenStack

The vRealize Operations monitoring capabilities can be enhanced when the Endpoint Agent for OpenStack is installed on the OpenStack management virtual machines. As stated in the vRealize Operations Management Pack for OpenStack documentation, the Endpoint Agent can monitor the following services and displaying their status in the vRealize Operations OpenStack Services dashboard.

The VMware Integrated OpenStack OMS virtual machine provides an automated installation workflow for the Endpoint Agent on each of the management nodes. The workflow can be leveraged post-deployment to facilitate the installation of the required Endpoint Agent package on the local operating system.

epops_01 epops_02

 

 

Further details on the exact process of installing the Endpoint Agent (epops) can be viewed here.

Having these pieces of software included in your environment will help ensure the monitoring of the services and the capacity within an OpenStack cloud is being taken care of correctly. Of course there are other tools that can be leveraged as well, however I have found these to be extremely useful within my vSphere environments. The blog post tomorrow will be an overview of the alert definitions the vRealize Operations Management Pack for OpenStack includes.

Enjoy!

Read More