openstack on Andrés Álvarez

Understanding Gnocchi Metrics

Thu, 18 May 2017 00:00:00 +0000

Metrics are one of the main object types in Gnocchi. They are identified by a UUID and they can also be attached to a resource by using a resource name. Metrics store measures, and the way they do this is defined by archive policies. These are concepts that I will cover in future articles.

Basically, a metric designates any thing that can be measured: the CPU usage of a server, the temperature of a room or the number of bytes sent by a network interface.

In the Gnocchi architecture, the storage back-end is responsible for storing measures of created metrics. It receives timestamps and values, and pre-computes aggregations according to the defined archive policies.

Diving Into OpenStack Gnocchi

Tue, 25 Apr 2017 00:00:00 +0000

Gnocchi is a multi-tenant timeseries, metrics and resources database. It provides an HTTP REST interface to create and manipulate the data. It is designed to store metrics at a very large scale while providing access to metrics and resources information and history.

It is the preferred storage method for metrics in Ceilometer, as of OpenStack Ocata.

In this post I want to dive into Gnocchi specifics such as its configuration, supported backends, APIs, daemons, and source code.

Configuration

Gnocchi's configuration is stored in a file called gnocchi.conf. Ideally, this file would be in ~/gnocchi.conf or /etc/gnocchi/gnocchi.conf. Let's take a look at a basic Gnocchi configuration:

[DEFAULT]
debug = true
verbose = true

[api]
workers = 1

[database]
backend = sqlalchemy

[indexer]
url = postgresql://gnocchi:gnocchi@127.0.0.1/gnocchi

[storage]
coordination_url = file:///home/ubuntu/gn/locks
driver = file
file_basepath = /home/ubuntu/gn

[cors]
allowed_origin = *
allow_credentials = false

The configuration above sets up Gnocchi to use Postgresql as the indexer, and use the file system for storage. Additionally it sets up CORS so that requests from any origin are allowed. You will want to configure CORS in a more secure manner when deploying to a production environment.

Database Setup

For this example, we are going to use Cloud 9 as our environment, and Postgresql as the database. This means that we need to first setup the database before we start using Gnocchi.

Make sure the Postgresql service is running:

sudo service postgresql start

We can enter the Postgresql command line using:

sudo sudo -u postgres psql

Now let's create a new Postgresql user:

CREATE USER gnocchi SUPERUSER PASSWORD 'gnocchi';

Then create the database:

CREATE DATABASE gnocchi WITH TEMPLATE = template0 ENCODING = 'UNICODE';

When the database is finally set up correctly and the configuration file is in place, we can initialize the indexer and storage:

gnocchi-upgrade

You should see the following output logs:

2017-05-17 05:28:50.917 3895 INFO gnocchi.cli [-] Upgrading indexer <gnocchi.indexer.sqlalchemy.SQLAlchemyIndexer object at 0x7ff76cff6190>
2017-05-17 05:28:50.982 3895 INFO alembic.runtime.migration [-] Context impl PostgresqlImpl.
2017-05-17 05:28:50.982 3895 INFO alembic.runtime.migration [-] Will assume transactional DDL.
2017-05-17 05:28:51.011 3895 INFO alembic.runtime.migration [-] Context impl PostgresqlImpl.
2017-05-17 05:28:51.011 3895 INFO alembic.runtime.migration [-] Will assume transactional DDL.
2017-05-17 05:28:51.154 3895 INFO gnocchi.cli [-] Upgrading storage <gnocchi.storage.file.FileStorage object at 0x7ff7688f9710>

Gnocchi REST API

Gnocchi's REST API is based on Pecan, a very lightweight Python web framework that provides object-dispatch style routing. We can confirm this in Gnocchi's rest/__init__.py file:

import pecan
from pecan import rest

Metrics

Gnocchi provides an object type that is called metric. A metric designates any thing that can be measured: the CPU usage of a server, the temperature of a room or the number of bytes sent by a network interface.

A metric only has a few properties: a UUID to identify it, a name, the archive policy that will be used to store and aggregate the measures.

Farther down the code in rest/__init__.py, we can find a metric controller which inherits from a Pecan REST controller:

class MetricController(rest.RestController):
    _custom_actions = {
        'measures': ['POST', 'GET']
    }

    def __init__(self, metric):
        self.metric = metric
        mgr = extension.ExtensionManager(namespace='gnocchi.aggregates',
                                         invoke_on_load=True)
        self.custom_agg = dict((x.name, x.obj) for x in mgr)

    def enforce_metric(self, rule):
        enforce(rule, json.to_primitive(self.metric))

    @pecan.expose('json')
    def get_all(self):
        self.enforce_metric("get metric")
        return self.metric

From the Pecan documentation, we can learn that Pecan uses a routing strategy known as object-dispatch to map an HTTP request to a controller, and then the method to call. Object-dispatch begins by splitting the path into a list of components and then walking an object path, starting at the root controller.

We can tell Pecan which methods in a class are publically-visible via expose(). If a method is not decorated with expose(), Pecan will never route a request to it. In the example above, the get_all() method is exposed to Pecan. Additionally, it makes use of Pecan's built-in support for a special JSON renderer, which translates template namespaces into rendered JSON text. Meaning that the returned content will be rendered as JSON.

Understanding oslo_config in OpenStack

Tue, 18 Apr 2017 00:00:00 +0000

The OpenStack Oslo library provides a set of Python libraries containing code shared by different OpenStack projects. This helps the many different OpenStack projects to follow a certain standard and convention when being developed.

In this post I want to go over oslo_config. This library helps with parsing of options found in configuration files or command line arguments.

Understanding Options

An option is defined using the Opt class or one of its sub-classes, from the cfg module:

from oslo_config import cfg

common_opts = [
    cfg.StrOpt('bind_host',
               default='0.0.0.0',
               help='IP address to listen on.'),
    cfg.Opt('bind_port',
            type=PortType,
            default=9292,
            help='Port number to listen on.')
]

In the example above we can see that we are using Opt and StrOpt, one of its sub-classes. You might notice that the second option's constructor has a type=PortType argument. This argument is a callable object that takes a string and either returns a value of that particular type, or raises ValueError if the value cannot be converted. The different types available can be found in the types module:

Understanding Ceilometer Publishers

Tue, 11 Apr 2017 00:00:00 +0000

Continuing with my study of the Ceilometer pipeline, this post now covers Ceilometer publishers. Publishers are components that make it possible to save the data into a persistent storage through the message bus, or to send it to one or more external consumers.

Publishers are specified in the publishers section for each pipeline that is defined in the pipeline.yaml and the event_pipeline.yaml files.

Many different publishers are available. The following are the most common and important publishers:

Gnocchi

Understanding Ceilometer Transformers

Fri, 07 Apr 2017 00:00:00 +0000

Ceilometer transformers are part of the Ceilometer pipeline, which is the mechanism by which data is processed. In Ceilometer there is a pipeline for samples and a pipeline for events.

Configurations for these pipelines can be found in /etc/ceilometer/pipeline.yaml and /etc/ceilometer/event_pipeline.yaml.

Transformers are responsible for mutating data points and passing them to publishers that will send the data to external systems.

Different types of transformers exist to mutate different types of data. The following is a table containing all the available transformer types:

Name of transformer	Reference name for configuration
Accumulator	accumulator
Aggregator	aggregator
Arithmetic	arithmetic
Rate of change	rate_of_change
Unit conversion	unit_conversion
Delta	delta

Let's take a more detailed look into the accumulator transformer.

Ceilometer and OpenStack Notifications

Thu, 06 Apr 2017 00:00:00 +0000

Ceilometer has different mechanisms to collect data from OpenStack. One of these mechanisms, are notifications, which will be explained in this post.

OpenStack Notifications

All OpenStack services (such as Nova Compute, Neutron, etc.) send notifications about executed operations or the state of the system. Many of these notifications carry information that can be metered such as CPU time of a virtual machine instance created by the compute service.

OpenStack services send these notifications through an AMQP message queue. However, not all these notifications are consumed by the Telemetry service, as the intention is only to capture the billable events and notifications that can be used for monitoring or profiling purposes. The notification agent filters by the event type. Each notification message contains the event type.

Here is a table with a few examples:

OpenStack Service	Event types
OpenStack Compute	`scheduler.run_instance.scheduled`, `scheduler.select_destinations`, `compute.instance.*`
OpenStack Image	`image.update`, `image.upload`, `image.delete`, `image.send`

In Ceilometer, the notification agent is reponsible for consuming these notifications from the AMQP message bus, and then transforming them into events and samples.

Setting Up an OpenStack Dev Environment

Wed, 22 Mar 2017 00:00:00 +0000

Before beginning to contribute to OpenStack, it is necessary that we setup an ideal development environment for a smoother workflow.

In this post I will cover how to setup a Devstack development environment in a Ubuntu 16.04 virtual machine using VMware Player.

Setting Up the Ubuntu Virtual Machine

Make sure you have VMware Player installed in your system. Then we need to choose an appropriate Ubuntu ISO image for our virtual machine. In my case I am using a Ubuntu Desktop 16.04.2 LTS (Xenial Xerus) for 32-bit PC i836.

In VMware Player, create a new virtual machine using the wizard. Assign your preferred amount of memory and storage space. Now start the machine, and select the Ubuntu .iso image file when prompted. Proceed with the installation and select the Erase Disk and Install Ubuntu option when prompted.

~> I tried using VirtualBox for virtualization but I could not successfully install Devstack with it. Probably due to issues on using the correct network adapter. I also tried using Ubuntu 14.04 without success as well. This is why I recommend Ubuntu 16.04.

Once the installation is finished, restart the virtual machine.

Getting Started on Contributing to OpenStack

Tue, 14 Mar 2017 00:00:00 +0000

I have started to become interested in contributing to the OpenStack project. Starting to contribute to such a massive project can seem like a daunting task, and it is. It is only by arming yourself with the proper tools and knowledge prior to contributing, that you will have a smoother process. I created this blog post to smooth that process.

Picking a Project

OpenStack is composed of many different projects for different purposes (compute, storage, networking, telemetry, etc.). Before you start contributing, you must have an idea of which project you actually want to contribute to. In my case I have decided on the Telemetry part, more specifically on Ceilometer.

After you have picked a project, it is time to start reading its documentation and developer documentation to understand how it works. The Ceilometer developer documentation offers good explanations and diagrams of its architecture and components.

Here is another nice guide on contributing to Ceilometer.

Processing Nova Live Migration Events in Zenoss

Mon, 23 Jan 2017 00:00:00 +0000

When monitoring OpenStack using the OpenStack Infrastructure ZenPack integrated with Ceilometer, it is possible to get fast virtual machine state changes (on/off powering, etc.) by receiving events sent from Ceilometer to Zenoss.

However I discovered that when trying to get the same effect for live migration (live migrating a virtual machine from one compute node to another) scenarios, this would not work. I proceeded to investigate why.

Ceilometer Dispatcher Live Migration Events

I decided that the first thing to check was if the Zenoss Ceilometer dispatcher was capturing and sending the live migration events to Zenoss. Indeed, the logs can be found in Ceilometer under /var/log/ceilometer/ceilometer-collector.log:

2017-01-23 14:22:02.593 25667 INFO ceilometer_zenoss.dispatcher.zenoss [-] record_events called (events=[<Event: 3538a01c-ab8d-4e64-b0ff-6e8fe270e06a, compute.instance.live_migration.post.dest.start, 2017-01-23 06:22:02.584049, <Trait: state_description 1 migrating> <Trait: memory_mb 2 512> <Trait: ephemeral_gb 2 0> <Trait: fixed_ips 1 [{u'version': 4, u'vif_mac': u'fa:16:3e:6e:02:e9', u'floating_ips': [], u'label': u'admin-net', u'meta': {}, u'address': u'192.168.0.15', u'type': u'fixed'}]> <Trait: user_id 1 6d581c230c86475abf70cce41440e8a1> <Trait: service 1 compute> <Trait: priority 1 info> <Trait: state 1 active> <Trait: launched_at 4 2017-01-23 05:06:41> <Trait: flavor_name 1 m1.tiny> <Trait: disk_gb 2 1> <Trait: display_name 1 pdcmtest> <Trait: root_gb 2 1> <Trait: tenant_id 1 10296907e44248d2a707689f77d59ef6> <Trait: instance_id 1 87be4b45-e214-4ca3-8f5c-1bd31159f9e4> <Trait: vcpus 2 1> <Trait: host_name 1 ndc27-3222> <Trait: request_id 1 req-d4c27d06-6329-4ce0-adb5-370b8ca83a22>>])

2017-01-23 14:22:02.771 25667 INFO ceilometer_zenoss.dispatcher.zenoss [-] record_events called (events=[<Event: 8db64329-606c-4184-9b5b-eb815166cb17, compute.instance.live_migration.post.dest.end, 2017-01-23 06:22:02.761138, <Trait: state_description 1 > <Trait: memory_mb 2 512> <Trait: ephemeral_gb 2 0> <Trait: fixed_ips 1 [{u'version': 4, u'vif_mac': u'fa:16:3e:6e:02:e9', u'floating_ips': [], u'label': u'admin-net', u'meta': {}, u'address': u'192.168.0.15', u'type': u'fixed'}]> <Trait: user_id 1 6d581c230c86475abf70cce41440e8a1> <Trait: service 1 compute> <Trait: priority 1 info> <Trait: state 1 active> <Trait: launched_at 4 2017-01-23 05:06:41> <Trait: flavor_name 1 m1.tiny> <Trait: disk_gb 2 1> <Trait: display_name 1 pdcmtest> <Trait: root_gb 2 1> <Trait: tenant_id 1 10296907e44248d2a707689f77d59ef6> <Trait: instance_id 1 87be4b45-e214-4ca3-8f5c-1bd31159f9e4> <Trait: vcpus 2 1> <Trait: host_name 1 ndc27-3222> <Trait: request_id 1 req-d4c27d06-6329-4ce0-adb5-370b8ca83a22>>])

2017-01-23 14:22:02.782 25667 INFO ceilometer_zenoss.dispatcher.zenoss [-] record_events called (events=[<Event: d0dc4f4a-454a-42c5-b767-386dc3e0d1f3, compute.instance.live_migration._post.end, 2017-01-23 06:22:02.776665, <Trait: state_description 1 migrating> <Trait: memory_mb 2 512> <Trait: ephemeral_gb 2 0> <Trait: fixed_ips 1 [{u'version': 4, u'vif_mac': u'fa:16:3e:6e:02:e9', u'floating_ips': [], u'label': u'admin-net', u'meta': {}, u'address': u'192.168.0.15', u'type': u'fixed'}]> <Trait: user_id 1 6d581c230c86475abf70cce41440e8a1> <Trait: service 1 compute> <Trait: priority 1 info> <Trait: state 1 active> <Trait: launched_at 4 2017-01-23 05:06:41> <Trait: flavor_name 1 m1.tiny> <Trait: disk_gb 2 1> <Trait: display_name 1 pdcmtest> <Trait: root_gb 2 1> <Trait: tenant_id 1 10296907e44248d2a707689f77d59ef6> <Trait: instance_id 1 87be4b45-e214-4ca3-8f5c-1bd31159f9e4> <Trait: vcpus 2 1> <Trait: host_name 1 ndc27-3205> <Trait: request_id 1 req-d4c27d06-6329-4ce0-adb5-370b8ca83a22>>])

Notice the live_migration.post.dest.start and live_migration.post.dest.end logs.

Keystone Interface in Juju Charms

Fri, 28 Oct 2016 00:00:00 +0000

Juju Charms interfaces make possible the interaction and exchange of data between services deployed by Juju. In this post I will explain how we can use the Keystone interface to make our custom charm service comunicate with the Keystone service within a Juju environment.

In the end, what we really want to achieve, is to link both services so that they can exchange data, using a Juju command:

juju add-relation keystone myservice

Juju Relations

Juju relations operate behind the concept that one service provides something, while another service requires it, and the interaction between the services is done through an interface. Since we want to interact with Keystone, this means that we need to use a suitable keystone interface that allows us to get what we need, the keystone Identity data. For this, we use the keystone-admin interface.

With this in mind, we configure our charm to require this relation and interface in metadata.yaml:

# ...
requires:
  identity-admin:
    interface: keystone-admin