sysadmin on Andrés Álvarez

Custom Triggers and Notifications in a ZenPack

Thu, 04 May 2017 00:00:00 +0000

In a previous post I talked about how to use triggers and command notifications in Zenoss to trigger custom actions when certain events occur. All of this was done using the Zenoss user interface.

In this post we will achieve something similar, but from a custom ZenPack. At the end of this post, our custom ZenPack will be able to create new custom triggers and notifications when installed.

Defining ZenPack Triggers & Notifications

Triggers and notifications within a ZenPack are actually defined using JSON. To do so, you must create a file named actions.json in a directory called zep (create it if it doesn't exist), within the ZenPack's top directory.

actions.json:

{
  "triggers": [
    {
      "name": "Critical_death_event",
      "uuid": "4c055067-98b7-483e-8f49-2820b4f2f721",
      "enabled": false,
      "rule": {
        "api_version": 1,
        "type": 1,
        "source": "(evt.event_class.startswith(\"/Status/Ping\")) and (evt.status == 0) and (evt.severity > 2)"
      }
    }
  ],

  "notifications": [
    {
      "id": "send_sms_message",
      "description": "Send SMS using Twilio",
      "action": "command",
      "guid": "2606439f-5ef7-40dc-90e4-3f3bee11cfe6",
      "enabled": false,
      "action_timeout": 60,
      "delay_seconds": 0,
      "repeat_seconds": 0,
      "send_initial_occurrence": false,
      "send_clear": false,
      "body_format": "echo \"Hello World!\"",
      "clear_body_format": "",
      "subscriptions": ["4c055067-98b7-483e-8f49-2820b4f2f721"]
    }
  ]
}

Working With RRDTool

Tue, 02 May 2017 00:00:00 +0000

Zenoss makes a lot of use of RRDTool for storing monitored performance data and generating graphs. In this post I want to go into more detail on the components of RRDTool and how works.

Round Robin Databases

RRDTool is a program that works with Round Robin databases (hence RRD). This type of database is more geared and limited towards storing time series data.

Moreover, rrdtool also doesn't store the data handed to it. Instead, it first re-samples the data and then stores this re-sampled data. Since rrdtool deals with time series data, it knows that certain data is meant to be stored every certain period of time (i.e 5 minutes). This newly re-sampled data will be the same as the original data, but the stored numbers will be different.

This means that if for example, we were to store some data every 5 minutes, a data sample that arrives at the 03 minute mark cannot go into the database, therefore it has to be re-adjusted into the correct slots. While rrdtool makes this adjustment, it will also change the value, so that this new value still represents the space in the curve.

In rrdtool, each database table is stored in a separate .rrd file. This means that one file represents one table.

We can create a database using the following command:

rrdtool create ex5.rrd --step=5 DS:input:GAUGE:10:0:1000000 RRA:LAST:0.5:1:603

Data Re-sampling: Step

One important parameter here is the step parameter. This means that rrdtool will re-sample all the data that is fed into the database to a 300 second interval. Still, data can be fed as quickly as desired (i.e a lot of data in one second), however the data will be re-sampled to 300 seconds on storage. This means that it only stores data every 300 seconds. Because of this re-sampling, rrdtool will not accept data from the past.

Understanding SNMP and Net-SNMP

Mon, 16 Jan 2017 00:00:00 +0000

Net-SNMP is a suite of tools used to implement SNMP (Simple Network Management Protocol). SNMP is a widely used protocol for monitoring the health and welfare of network equipment (eg. routers), computer equipment and even devices like UPSs. Net-SNMP includes tools with capabilities such as:

An extensible agent
An SNMP library
Tools to request or set information from SNMP agents
Tools to generate and handle SNMP traps
A version of the unix ‘netstat’ command using SNMP
A graphical Perl/Tk/SNMP based mib browser

How SNMP Works

SNMP allows a management station to treat its network as a distributed database of health and configuration information. SNMP contains a small set of operations:

GET: Retrieve data from a network node
GETNEXT: Retrieve the next element from a network node
SET: Send configuration or control commands to a network node
TRAP: A network node can send a notification to the management station
INFORM: An acknowledged trap (network nodes can try and send it again if no acknowledgement is received)

Fixing Zenoss Device Network Interface Graphs

Fri, 13 Jan 2017 00:00:00 +0000

When testing the monitoring of the OpenStack compute node devices in Zenoss (using OpenStack Infrastructure ZenPack), I noticed that I could not get any graphs for the network interfaces of the device, although we could still get and model all the available interfaces:

This was indeed strange because I could perfectly get the graphs for other devices. After a lot of head scratching and prying around the code and interface, I finally found the reason, which I explain below.

Working With Zenoss Events

Fri, 25 Nov 2016 00:00:00 +0000

Zenoss Events are among the most powerful and useful monitoring features that Zenoss Core provides. Many different types of events are automatically generated by Zenoss when important scenarios occur for the devices being monitored.

Zenoss Event Architecture

When an event arrives at Zenoss, it is parsed, associated with an event classification and then typically (but not always), it is inserted into the status table of the events database. Events can then be viewed by users using the Event Console of the Zenoss Graphical User Interface (GUI).

The events system has the concept of active status events and historical events (two different database tables in the MySQL events database).

Understanding Event Classes

Zenoss event classes are a simple organizational structure for the different types of events that Zenoss generates and receives. This organization is useful for driving alerting and reporting. You can, for example, create an alerting rule that sends you an email or pages you when the availability of a Web site or page is affected by filtering on the /Status/Web event class^[2].

Using RRDTool in Zenoss

Tue, 22 Nov 2016 00:00:00 +0000

rrdtool is an awesome high performance data logging and graphing tool for time series data. Zenoss Core uses RRDTool to collect, monitor, and graph peformance data for devices.

However Zenoss Core comes with built-in helper wrappers around RRDTool that makes using it within Zenoss much easier. These source code files can be found in $ZENHOME/Products/ZenRRD/.

RRDUtil

Located in $ZENHOME/Products/ZenRRD/RRDUtil.py, this Python module contains many wrapper methods around the rrdtool library. These methods can help us write to, create new, and read .rrd files using Python.

We can easily import this module into our code, using the following import statement:

from Products.ZenRRD.RRDUtil import RRDUtil

IPMI & Baseboard Management Controllers

Fri, 18 Nov 2016 00:00:00 +0000

An Intelligent Platform Management Interface (IPMI) is a set of interfaces that allow system administrators to manage and monitor systems, independently of the system's CPU, BIOS, and operating system.

IPMI can also help manage computers that are unresponsive or powered off, by using a network connection to the hardware, rather than to a operating system or login shell.

More specifically, IPMI can help us monitor, obtain data, and perform tasks such as:

System temperature
Voltages
Fans
Power supplies
Chassis intrusion
Perform recovery procedures
System rebooting
System powering

An IPMI sub-system consists of a main controller, called the baseboard management controller (BMC) and other management controllers distributed among different system modules that are referred to as satellite controllers. The satellite controllers within the same chassis connect to the BMC via another system interface (not covered in this post) called Intelligent Platform Management Bus/Bridge (IPMB)

Modifying the Zenoss Infrastructure Grid

Thu, 10 Nov 2016 00:00:00 +0000

Continuing from my previous post, where I explained how to modify the device detail bar from a ZenPack using ExtJS, in this post I will explain how we can modify the device list grid shown in the infrastructure page.

The Infrastructure Grid

This is the table in the infrastructure page that shows all devices being monitored by Zenoss. Default columns include device name, device class, IP address, production state, and events, as shown in the image below:

However it would be nice to also include the power status we added to the device detail bar in the previous post, maybe even add the ping status as well (why Zenoss doesn't do this by default is beyond me).

DevicePanels.js

The source code for the grid is found in $ZENHOME/Products/ZenUI/browser/resources/js/zenoss/DevicePanels.js. A quick glance at it and you will quickly find the definitions of the columns I mentioned earlier, defined in an array called deviceColumns.

Modifying the Zenoss Device Detail Bar

Tue, 08 Nov 2016 00:00:00 +0000

The Zenoss Core ExtJS graphic user interface is divided into many different components. In this post I will go over on how we can modify the device detail bar to display additional custom information for devices, by creating a custom ZenPack.

Zenoss Core 4 uses ExtJS 4 JavaScript framework to manage all the user interface components. These components can be found in $ZENHOME/Products/ZenUI3/browser/resources/js/zenoss, and as you would expect, the device detail bar component is also located there.

The Device Detail Bar

This is the detail bar located on the device view page which shows the device's icon, name, events, status, production state, and priority. The JavaScript source code for this component can be found in DeviceDetailBar.js

However since we are going to extend this component through a ZenPack, we will not modify that source. Instead, we will add new code to use ExtJS to add our custom data displays.

Extending The Component, Through ZenPacks

Let's assume we are starting with a freshly created ZenPack, which adds a new power_status integer field to certain devices, according to its zenpack.yaml file:

# ...

classes:
  CustomDevice:
    base: [zenpacklib.Device]
    label: Custom Device
    properties:
      power_status:
        type: boolean

What we want is to display this value in the device's detail bar, similar to the device's ping status display.

Monitoring CPU Utilization in Zenoss

Fri, 04 Nov 2016 00:00:00 +0000

While taking a look at the CPU Utilization graphs offered Zenoss Core's Linux Monitor ZenPack (v1.2.1), I noticed that the percentage values for Idle were ridiculously high:

This made sense since this particular device contains 16 cores. However, this then means that the monitoring template isn't really taking this into consideration, and instead just spits out the total value from all cores.