Andrés Álvarez

Implementing Rate Limiting in Chalice

Mon, 23 Jan 2023 00:00:00 +0000

Chalice (AWS's Python serverless micro-framework) does not offer built-in support for throttling/rate limiting. In this post I will show you how to implement your own custom rate limiting capabilities for your serverless API.

Setting Up a SQLAlchemy and Pytest Based Test Suite

Wed, 25 May 2022 00:00:00 +0000

✨ This is an improved article of the previous Python Transactional Tests Using SQL Alchemy, Pytest, and Factory Boy article.

In this post I will show you how you can setup a Pytest test suite that automatically sets up and teardowns your database using SQAlchemy on each test run. It will also provide transactional tests capabilities as described in Python Transactional Tests Using SQL Alchemy, Pytest, and Factory Boy.

How to Test Amazon SES Emails Using Pytest and Moto

Fri, 30 Jul 2021 09:58:59 -0600

In this post I will show you a very neat way to unit test your Amazon SES emails using Pytest and the awesome Moto library for mocking AWS services.

How I Define SES Emails in Application Code

I like to use a handy base class for preparing and sending SES emails within my application code. Here is a simple example:

How to create a 3 Stage Deployment Pipeline With Github Actions

Tue, 08 Jun 2021 14:34:36 -0600

In this post I will show you how to setup a nice 3 stage deployment pipeline that will lint and test your code before deploying to a specific stage, using Github Actions

I will be simulating a Python application for this post, so you should modify the Python related stuff according to your project.

🔧 Build And Test Workflow

This workflow will lint, type check, and test the project before deploying to any stage. Create a .github/workflows/build-and-test.yml file:

Python Transactional Tests Using SQL Alchemy, Pytest, and Factory Boy

Fri, 19 Feb 2021 13:59:36 -0600

🆕 Read the newer and improved post Setting Up a SQLAlchemy and Pytest Based Test Suite

📅 Last Updated on November 26 2021

Knowing how to integrate SQLAlchemy for your automated tests can be a bit tricky. Even after reading several blog posts on how to implement transactional tests with SQLAlchemy 1.3 and Pytest, it took me a bit of time to make it all play nicely with Factory Boy.

In this post I will teach you how you that is achieved.

How to Debug a Chalice App in Vscode

Thu, 24 Dec 2020 12:53:55 -0600

In this post I will show you how you can create a debug configuration in Visual Studio Code that will allow you to debug a Chalice application locally.

Django: Adding Context Data to Class Based Views Using Properties

Wed, 30 Sep 2020 09:58:57 -0600

When using Class-Based Views in Django, we normally add additional context data to the view by implementing the get_context_data() method:

class MyDetailView(DetailView):

    def get_context_data(self, *args, **kwargs):
        context = super().get_context_data(*args, **kwargs)
        context['foo'] = 'bar'

        return context

This works well. But I am very fond of using another approach, so as to not pollute the get_context_data method. We can achieve this using properties:

Django: Custom Middleware for Alerting High Database Querying

Wed, 06 May 2020 15:27:47 -0600

In this post I will show you how you can implement a custom middleware that you can use in your development environment to monitor queries to the database in your views when developing Django applications.

Let's begin by creating a middleware.py file (a nice convention) where we will place our new middleware. The middleware is relatively simple and small, this is how it looks like:

Custom Admin Action Buttons in Django

Wed, 29 Apr 2020 12:06:31 -0600

In this post I will explain how you can add custom action buttons in the admin detail view of objects in the Django admin.

This is the final result we want to achieve:

These buttons can trigger custom actions that we want to perform on these objects.

Gracefully Exiting Python Context Managers on Ctrl+C

Fri, 10 Apr 2020 16:27:41 -0600

In this post I will show you how you can exit gracefully from within a context manager in your Python application.

Suppose that we provide a context manager that can be used as a session to perform certain tasks. When the context manager is closed, there is some cleanup work to be done. We want this to happen even when the user interrupts the program with Ctrl+C key.

This is how we can achieve that:

Python Log Messages With Dynamic Formatter Variables

Thu, 02 Apr 2020 16:07:28 -0600

In this post I will explain how it is possible to replace logger formatter variables with dynamic values. Take the following log message for example:

[2020-04-02 15:37:01] myapp INFO: Logged in successfully.

What if we wanted to add some additional data after the INFO part, but this data is dynamic, or is only available at runtime. Like a username or an app name for example.

Let's take a look at how to achieve this.

Django: ORing A List of Q Objects

Sun, 04 Aug 2019 12:35:00 -0600

Django allows using the OR bitwise operator | with Q objects for more complex lookups:

Q(question__startswith='Who') | Q(question__startswith='What')

However this requires explicitely typing each Q object. How can we apply this OR opperation to all the Q objects in a list?

In Python 3 we can do so like this:

from functools import reduce
from operator import __or__


qs = MyModel.objects.filter(reduce(__or__, filters))

Where filters is a list containing multiple Q objects like the ones shown in the first example.

Coding Challenge: Rectangular Intersection

Tue, 09 Apr 2019 18:07:34 +0000

Difficulty: Medium

Problem

Given two rectangles, write a function that can find the rectangular intersection between the two rectangles. The rectangles are always straight and never diagonal (each side is parallel with either the x axis or the y axis). Consider only positive axis.

The rectangles are defined as a Rectangle class as shown below:

Rectangle(left_x, bottom_y, width, height)

# Example
r = Rectangle(2, 7, 30 8)

Your output rectangle should be returned using this format.

Coding Challenge: Longest Substring Without Repeating Characters

Mon, 08 Apr 2019 15:09:13 +0000

Difficulty: Easy

Problem

Given a string, find the length of the longest substring without repeating characters.

Examples:

Given abcabcbb, the answer is abc, which the length is 3.
Given bbbbb, the answer is b, with the length of 1.
Given pwwkew, the answer is wke, with the length of 3. Note that the answer must be a substring, pwke is a subsequence and not a substring.

Custom Log Files With Django and Elastic Beanstalk

Fri, 22 Feb 2019 17:49:21 +0000

With Amazon ElasticBeanstalk it is possible to view log files of your deployed application. However, ElasticBeanstalk by default only returns certain logs like /var/log/httpd/error_log or /var/log/httpd/access_log if you are using Apache httpd.

If you are generating custom logs in your Django application and using file handlers to save them to log files, you will probably want to be able to access and read them easily from the ElasticBeanstalk console.

In this post I will show you how to achieve this, using Django as our backend framework.

Making Functions Timeout in Python

Wed, 06 Feb 2019 16:02:35 +0000

Sometimes you need to execute a function that can take a lot of time to finish. You are not sure exactly when the function will finish, but you do not want to wait too long, or let your program “hang” waiting for a response.

We want our function to run for a certain period of time, and if this time limit is exceeded, we want to regain control of the program's execution.

We can achieve this by using a custom context manager and the signal module from the Python standard library.

Here is a complete example:

Dropdown Toolbar Buttons in Netbeans Platform

Tue, 05 Feb 2019 20:16:38 +0000

It is possible to add buttons that toggle a dropdown menu to the main toolbar in your Netbeans Platform application.

For example, the Netbeans IDE Run Project, Debug Project, and Profile Project buttons have a small arrow next to them. When clicked, a dropdown menu with more related actions is shown:

In this article, I will teach you how to implement this behavior.

Setting Up Elastic Beanstalk Health Checks With a Django Application

Tue, 15 Jan 2019 20:33:50 +0000

I was having an issue the past few days with Django and Elastic Beanstalk in a production environment that was driving me nuts.

Basically the Elastic Beanstalk environment was in a permament Severe/Degraded state. The health monitoring was reporting that 100% of the requests to the load balancer were 4xx requests:

This issue was probably caused by several reasons. In this post I will go over the possible causes, and how I managed to resolve this issue.

Media-Type Based Responses With Django REST Framework

Wed, 09 Jan 2019 17:28:17 +0000

Nowadays a typical REST API responds with data in JSON format. But what if you wanted to serve data in a different format (and not just a serialization format) using the same endpoint, but specified by the client?

Django REST Framework renderers can help us achieve this nice feature.

Use Case: ONIX For Books Compatible API

ONIX for books is an XML format for sharing bibliographic data pertaining to both traditional books and eBooks.

Let's say our application serves metadata about books (title, author, etc.) in the usual JSON format, but we also want to provide data in ONIX format, which is an XML standard that defines how the metadata should be described and put together for consumption.

Python Functional Testing for AWS Lambda

Wed, 26 Dec 2018 17:17:28 +0000

In this post I will explain and go over how I personally write functional tests for my AWS Lambda functions.

For our example use case, we will assume that our Lambda function is quite complex, as in that it involves downloading a file, processing it, and uploading a result. Additionally it will be called by an SQS event, and it make use of other non-Python related elements, as we will see later.

Let's get started!

Pytest Tricks for Better Python Tests

Fri, 21 Dec 2018 21:03:28 +0000

Pytest is my Python testing framework of choice. Very easy to use, and makes tests look much better.

In this article I'll show you some cool tricks I have incorporated into my test suites using Pytest.

Environment Variables

When your application must work with defined environment variables, the testing environment must have these variables defined as well, even if the values are not real.

With Pytest we can easily configure any necessary environment variables in our test environment. We simply create a fixture in conftest.py that will be loaded automatically:

# conftest.py

import pytest


@pytest.fixture(autouse=True)
def env_setup(monkeypatch):
    monkeypatch.setenv('MY_SETTING', 'some-value')
    monkeypatch.setenv('ANOTHER_SETTING', 'some-value')

monkeypatch is a built-in pytest fixture that allows us to set environment variables in the test runs. By enabling the autouse option, our custom environment setup fixture will be automatically called in every test without having to include it explicitly using the usual dependency injection mechanism.

CloudFront CORS Font Issues

Wed, 12 Dec 2018 16:32:22 +0000

I was experiencing an issue with CloudFront and some static assets of a Django application deployed to ElasticBeanstalk. The issue was happening when trying to retrieve some fonts that are stored in S3 and served by CloudFront, the browser dev tools would show some errors similar to the following:

Access to Font at 'http://CLOUDFRONT_HOSTNAME.cloudfront.net/FONT_PATH' from origin 'http://www.example.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

A CORS issue. And apparently, very common when trying to load fonts.

To fix this, there are a few possible things that you might have to try. I will list some of the fixes I found.

Cancellable Tasks in Netbeans Platform

Wed, 28 Nov 2018 00:00:00 +0000

I was trying to implement cancellable tasks in Netbeans Platform using this blog post as reference.

However I never managed to make it work. The tasks's thread would never get interrupted even after the confirmation dialog, the task would just keep running.

After some digging on the Netbeans Platform and IDE source code, I think I managed to find a way to properly and easily implement cancellable tasks.

Recursive Python AWS Lambda Functions

Tue, 18 Sep 2018 00:00:00 +0000

There are times where some processing task cannot be completed under the AWS Lambda timeout limit (a maximum of 5 minutes as of this writing). A possible solution for these kind of situations is to implement a recursive approach to perform the processing task.

Basically, if you are able to separate the task into multiple chunks or batches, then you could make each batch to be processed by a different Lambda function. The amount of Lambda functions necessary to finish the task will scale accordingly.

Displaying Images in Netbeans Platform Node Tooltips

Tue, 14 Aug 2018 00:00:00 +0000

Netbeans Platform allows you to display HTML inside a node's tooltip text. Naturally this means you can display images. For example, project nodes in the Netbeans IDE display icons related to errors (if there are any) and version control:

Achieving this is not very straight forward, since the documentation doesn't really say much on how to load this image. Fortunately it is very simple.

Troubleshooting Strange AWS Issues

Tue, 07 Aug 2018 00:00:00 +0000

We've all been there. The same things working fine one time but failing other times without any idea why.

This post is a list of troubleshooting strange issues I've encountered when using AWS.

ElasticBeanstalk Failing to Create RDS Security Group

I was having an issue where I couldn't create and attach an RDS database to an ElasticBeanstalk environment because an RDS security group was failing at being created. All this was being done using the AWS console.

Loosely Coupled Node Actions in Netbeans Platform

Wed, 25 Jul 2018 00:00:00 +0000

Nodes in Netbeans Platform can possess actions that can be executed with the node object as context. For example, the Project, Package, and Class nodes in the Netbeans IDE have a set of actions which can be seen by right clicking on the nodes:

In the Netbeans Platform tutorials I found a tutorial that explains how to add these actions to nodes. The problem is that in that tutorial the action class was created as an inner class of the node class. Making the action tightly coupled with the node.

I managed to implement a more modular and loosely coupled approach, which is what I will explain in this post.

Our loosely coupled approach will basically allow us to:

Keep the domain logic (Java Beans and POJOs) in its own domain module.
Keep the view logic (nodes that represent beans) in another module.
Keep the node actions logic in yet another module. This allows us to plug and unplug different action's that belong to separate modules. Making our application's node actions very modular.

Let's begin!

Netbeans Platform Status Bars

Thu, 19 Jul 2018 00:00:00 +0000

In desktop applications, status bars are a great way to convey information to the user about the current status of certain items in the application, such as connection status, total amount of items, current mode, current row and column number in an editor, etc.

Implementing custom status bars to a Netbeans Platform application is extremely easy. In this article I will go over how we can achieve really cool things with status bars and lookups.

Creating and Registering a New Status Bar

To create a new status bar, create a new Java class that implements the StatusLineElementProvider interface. To register it as a status bar, we use the @ServiceProvider annotation:

@ServiceProvider(service = StatusLineElementProvider.class, position = 1)
public class MyStatusBar implements StatusLineElementProvider {

}

Implementing Auto Updates for a Netbeans Platform Maven Application Using Github Pages

Tue, 17 Jul 2018 00:00:00 +0000

Being able to update an application when a new version is released is something of critical importance for desktop applications. Otherwise, the process of updating the application is extremely tedious since you have to download more installers all the time.

Luckily, Netbeans Platform comes with integrated support for enabling your custom applications to be able to be updated when a new version is released.

In this post I will go over how to implement this on a Maven based Netbeans Platform application.

How Netbeans Update Center Works

The Update Center is how we can achieve this functionality. The way it works is that you publish your module's NBM files to a web server, along with a XML catalog file that contains information about the modules such as latest release timestamp, latest version, etc.

When you configure an update center in your application, you will configure this web server's URL so that the update center client knows where to poll and where to download the new module files to install.

BytesIO File Uploads to Django Using Requests

Tue, 26 Jun 2018 00:00:00 +0000

It's very easy to post file data to Django using requests:

import requests


requests.post(url, files={'cover': open('imgpath.jpg', 'rb')})

However I was having a hard time getting that to work using BytesIO. The reason I wanted to use BytesIO was because I was reading the file binary data located in S3, from AWS Lambda. I didn't want to write the file to disk first and then do something like the code shown above.

Here's how to achieve that:

Using Listeners in Custom Netbeans Platform Wizards

Tue, 22 May 2018 00:00:00 +0000

This tutorial explains how you can add validation to your custom wizards in the Netbeans Platform. This works by setting a boolean variable to false and then throwing a WizardValidationException which will show an error message in the wizard, disabling the ability to click the Next or Finish buttons.

The problem with this is that after the above happens, there is no way to enable these buttons again. And the way to achieve this is not covered in that tutorial.

Therefore, in this blog post I will explain just how to do that.

Netbeans Platform Node Icon Badges

Thu, 03 May 2018 00:00:00 +0000

The Netbeans IDE has some very cool way of letting users know when a certain file contains errors. For example, if you are editing a Java file and the file contains syntax errors, Netbeans will change the file icons with a new icon indicating errors:

You can see that the error icons propagate all the way up to the top-most parent, the project itself.

Netbeans Platform Icon Badges

As a user of the Netbeans IDE and before even knowing what the Netbeans Platform was, I always assumed that the normal state of the icon and the error state of the icon were two completely different icons handled by Netbeans. A few days ago I learned that this is not the case.

Netbeans uses the concept of icon badges to display these kind of error icons. Icon badges are basically multiple images merged together at runtime into a single Image object.

In this article I want to quickly go over how you can apply this concept in your Netbeans Platform application when using the Nodes API.

Creating Toolbars for MultiView Elements in Netbeans Platform

Mon, 23 Apr 2018 00:00:00 +0000

On a previous post I talked about creating a new file type that Netbeans Platform can recognize and display the file's data in different ways by creating different visual editors that implement the MultiViewElement interface.

However, that tutorial did not cover how to add a custom toolbar to each multi view. Like the one Netbeans already provides when editing source code:

This is a very handy toolbar that can be used to add buttons that perform different kinds of actions with the current active file.

A Deeper Look Into the MultiViewElement Object

If you recall from the previous post, a JPanel that implements MultiViewElement was created after finishing the wizard. This MultiViewElement object has some very important characteristics that are key to this tutorial:

Django Formsets with Generic FormViews

Thu, 19 Apr 2018 00:00:00 +0000

Django Formsets is an extremely handy feature for handling multiple clones of forms in a single view. In this post I will go over how to implement them using one of Django's built-in generic class based views, the FormView.

The Scenario

To add some context to this tutorial, let's say we want to implement one of those kind of nested searches you see on some sites. Where you can select a field from a dropdown, and type in a search value for each field. Fields can be added and removed dynamically using JavaScript. Something like this:

For this tutorial, we will use the following model as an example:

class Book(models.Model):
    title = models.CharField()
    author = models.ForeignKey(Author)
    isbn = models.CharField()
    editorial = models.ForeignKey(Editorial)

Creating the Form

We will begin by creating the form that will be cloned. That is, the form that will represent one search query out of the many that will/could be nested together.

For our case, the form will only contain 3 fields:

The query field: Indicates which field of the model we will query.
The lookup type: Indicates which lookup is going to be used to make the query.
The query value: Indicates what we want to actually search for (i.e Harry Potter for the book title.)

Custom File Types in Netbeans Platform

Wed, 28 Mar 2018 00:00:00 +0000

If your desktop Java application needs to deal with very particular or special kinds of files, we can make use of Netbeans Platform's custom File Type feature to define a file that our application can recognize, and display visual and editing components to view and edit the file's data.

Moreover, we can also make use of Netbeans Platform's Favorites pane, which displays a tree view of a file system, where we can open the special files we need to work with.

Creating a New Module

For this article, we are going to create a new file type for CSV (Comma Separated Value) files. When we open these files, our application will display a visual representation in the form of an editable JTable.

All this functionaltiy will live in its own Netbeans Platform Module. Meaning that it can be plugged in into different applications in a modular fashion.

We are going to create the module and build it using Maven, since we are also going to be adding some functionalities that depend on external libraries that we can easily include in our project with Maven. But you can also just create the module without Maven as well. To create the module go to New Project > Maven > Netbeans Module

Testing Django and DRF With Pytest

Fri, 09 Feb 2018 00:00:00 +0000

Pytest has become my favorite Python testing framework. And in this article I want to go over on how I learned to write nice tests for Django and Django REST Framework.

We will be using the following tools:

Pytest: Python testing framework
pytest-django: Pytest extensions for Django
factoryboy: Factories for easy test data generation.

Building a Django Continuous Integration and Delivery Pipeline to AWS

Tue, 30 Jan 2018 00:00:00 +0000

Building a CI/CD (Continuous Integration / Continuous Delivery) flow for your web application is a must nowadays. With so many things that can go wrong, and the amount of time and effort the deployment process of an application takes, a flow like this will give you peace of mind that your application is running without (or with less) errors and that your application will be production ready on every new commit to its source code.

In this post I want to talk about building such a pipeline, with emphasis on deploying a Django 1.11 application to AWS ElasticBeanstalk, which is an Amazon service very similar to Heroku. Additionally I will talk about how to include other services to the delivery flow, such as Amazon S3 for storage of static files and Amazon RDS (Relational Database Service) for a relational database that will be connected to our application.

The pipeline will work as follows:

Code will be pushed to source control repository. For this article, I will use Github.
Project will be built and tested in a CI job at TravisCI.
TravisCI will perform a deployment to ElasticBeanstalk once the build passes successfully.

We will be using both the ElasticBeanstalk CLI (ebcli) and the AWS ElasticBeanstalk Console to achieve this.

Django Dynamic Queryset Filtering With Pagination

Wed, 17 Jan 2018 00:00:00 +0000

This blog article does a very good job at explaining how to setup dynamic queryset filtering in Django. However, it doesn't go into paginating the filtered queryset results.

I had a very hard time trying to implement this pagination feature using class based views. More specifically, using the FilterView that comes with the django-filter package.

In the end it turned out to be very simple. The documentation was a bit misleading on this so it took a while to understand and find the solution.

Django AJAX Search With Elasticsearch

Wed, 10 Jan 2018 00:00:00 +0000

In this post I want to go over how I implemented a very neat AJAX search using Elasticsearch in Django. Let's get started!

The Search Template

The starting point of our search feature will be the search template. That is, the template that contains the input field where the user will type in the search keywords.

This template is extremely simple and does not need a <form> tag. I will create it as a partial under templates/search.html:

{% csrf_token %}
<input id="search" name="q" placeholder="Search...">

Notice how we have given the input a name with the value of q. This will be important when our view receives the query from this input field.

I can then render this partial in the main template like this:

{% include 'search.html' %}

Java JTable Tricks

Wed, 20 Dec 2017 00:00:00 +0000

Java's JTable is a GUI component that has been around for many many years. While a bit difficult to work with, it's almost a mandatory component in an application that displays a lot of data.

In this post I share some nice tricks for working with JTables in Java applications.

Scrolling

A JTable by itself does not provide any scrolling when handling a lot of data. To fix this, we need to place the table inside a JScrollPane component. Additionally, we can place this JScrollPane into a JPanel in our window:

myPanel.add(new JScrollPane(myTable,
    JScrollPane.VERTICAL_SCROLLBAR_AS_NEEDED,
    JScrollPane.HORIZONTAL_SCROLLBAR_AS_NEEDED)
    );

Bulk Create With Django REST Framework

Mon, 27 Nov 2017 00:00:00 +0000

In this quick post I will go over how to create an endpoint that allows bulk creation of a resource using Django REST Framework.

The Serializer

Let's assume that we will be working with an existing Book model in our Django application. We will create a serializer for this model using Django REST's ModelSerializer helper class:

# serializers.py
from myapp.models import Book
from rest_framework import serializers


class BookSerializer(serializers.ModelSerializer):
    class Meta:
        model = Book
        fields = '__all__'

The "__all__" string indicates that we want to serialize all of the model's fields.

Building an OCR Service With TesseractJS in AWS Lambda

Tue, 21 Nov 2017 00:00:00 +0000

The past few days I was trying to make TesseractJS work in AWS Lambda so that I could do some OCR (Optical Character Recognition) on some images I had stored in an S3 bucket. However I am a bit new to NodeJS and I was running into some difficulties getting it to work in the Lambda environment. In this post I am going to go through some of these issues and how I solved them.

TesseractJS is a OCR library written in pure JavaScript. It can recognize the text in images, as well as provide information about the location of the paragraphs, lines, and words in the document.

We will be using a NodeJS 6.10 runtime in AWS Lambda. And I will be deploying the service with ClaudiaJS.

Downloading the TesseractJS Files

When running TesseractJS to recognize an image, TesseractJS will automatically begin downloading some files, which include tesseract language files, a core library file, and a worker file. These are all files that TesseractJS requires in order to correctly run.

The problem occurs when trying to download these inside AWS Lambda, since Lambda only allows writing to the /tmp/ directory, you will get an error like this in your logs:

Error: EROFS: read-only file system, open 'eng.traineddata'

Creating an Upload Progress Dialog in Java

Thu, 16 Nov 2017 00:00:00 +0000

Many Java file upload tutorials teach you how to show the current progress of a file upload using a JProgressBar in the user interface, usually placed in the main JFrame.

However I was trying make this much more visually appealing, like the way WinSCP does it:

This will require the use of another Swing class: JDialog. Basically the flow of the program will be like this:

Main class with main method calls SwingWorker class.
SwingWorker is in charge of doing the actual upload. It will also create an instance of a JDialog to show the current upload progress.
JDialog class will contain the progress bar and other useful upload information.

Querying Nested Documents in ElasticSearch

Mon, 23 Oct 2017 00:00:00 +0000

The other day I was using ElasticSearch to build an index that would contain book documents. These book documents would then contain many pages, each page indicating its page number in the book, and its content as text, extrated from its own PDF file.

I wanted to perform some queries on a specific book so that ElasticSearch would return all the pages in the book which contained a certain word in its text field.

Nested Documents

ElasticSearch 5 supports nested documents. These are internally separate documents that will belong to a certain parent, in this case, a book. Before creating any books, we must first define a nested object mapping for the index:

PUT /my_index

{
  "mappings": {
    "book": {
      "properties": {
        "pages": {
          "type": "nested",
          "properties": {
            "id":    { "type": "integer"  },
            "text": { "type": "text"  },
          }
        }
      }
    }
  }
}

Notice how the mapping is created using a PUT request at the index level.

Umask Permissions in a Puma Production Environment

Wed, 18 Oct 2017 00:00:00 +0000

Recently I was having an issue with a Sinatra application deployed in a staging server. The application was deployed with Puma and Nginx in the following location:

/var/www/

This web service would then try to access some files in another directory on the server, mounted as an SFTP directory. The Sinatra app would open these files and generate some new files from them, depending on the HTTP request received.

The problem was that the operation would fail due to a permissions issue. I was baffled since I had set read and write permissions to the directory and the files in it.

Umask: The Problem and Solution

It took around two days to find the culprit: umask. In Linux, umask acts as another set of permissions for processes and cannot be set for directories, basically speaking.

I realized that this probably meant that the process running the Puma application server had a umask configuration that was not allowing the generation of new files.

I decided to test this if this was the case. In the Puma documentation, I found an option to change the permissions of the UNIX socket using umask:

Easy Notification System in Rails Part 3

Tue, 12 Sep 2017 00:00:00 +0000

Read part 1 and part 2 of this series

In this post, we will be sending automatic e-emails every time notifications are created.

Creating the Mailer

We will work with one mailer that will send e-mails for every notification that is created. We can generate our mailer with this command:

rails g mailer NotificationsMailer

Our mailer will contain an action for each notifiable type that works with notifications in our application. In this series, we've been using comments and posts as examples.

Easy Notification System in Rails Part 2

Tue, 05 Sep 2017 00:00:00 +0000

Read Part 1 of this series here

In Part 1 we learned how to setup our models and controllers to create notifications using callbacks in our application. Then we displayed these notifications in a Bootstrap 3 navbar using JQuery written in CoffeeScript.

In this post we will be adding more functionality to our notification system.s

Mark as Read Feature

We will be adding a feature that allows the current user to mark a notification as read as well as all notifications, in the following manner:

For a single notification, it should be marked as read when it is clicked from the list of notifications.
For all notifications, they should be marked as read when a specific button is pressed.

Setting up the Routes

Let's begin adding the necessary routes. In part 1 we defined a notifications resource. We will add a collection POST route and a member POST route of the same name:

# config/routes.rb
Rails.application.routes.draw do
  # ...

  resources :notifications, only: [:index] do
    post :mark_as_read, on: :collection
    post :mark_as_read, on: :member
  end

end

Easy Notification System in Rails

Mon, 04 Sep 2017 00:00:00 +0000

Having a notification system is very common for a web application. In this post I will go over how to implement your own simple and easy notification system for your Rails application.

This post is heavily inspired by Chris Oliver's In-App Navbar Notifications tutorial, with a few personal changes and additions.

Read part 2 of this series.

Models and Routes

Before starting with the models, lets quickly define the application resources in our routes:

# config/routes.rb
Rails.application.routes.draw do

  devise_for :users

  resources :posts do
    resources :comments
  end

  resources :notifications, only: [:index]

end

The Notification Model

We will manually generate the notification model which will handle the storing of notifications in the database. We can generate a very basic model like this:

rails g model Notification recipient_id:integer actor_id:integer read_at:datetime action:string notifiable_id:integer notifiable_type:string

Here is an explanation for some of the fields for the above model:

recipient_id: Represents the user in your application which will receive the notification.
actor_id: Represents the user in your application which triggered the notification.
read_at: The time when the notification was read. A value of nil is used for unread notifications.
notifiable_id: The object that represents this notification (a post, comment, etc). This will be a polymorphic association.
notifiable_type: Type of the notifiable object. This is usually represented by a humanized (and optionally, internationalized) string form of the object's class.

Integrating Devise With Rolify in Rails

Thu, 24 Aug 2017 00:00:00 +0000

I have searched through tons of StackOverflow questions and blog posts on how to properly integrate Rolify with Devise when registering as a new user through the registration view provided by Devise. All proposed solutions and half-solutions seemed very messy to me, but I found a very simple way to achieve this.

Preparing the Roles

Assuming you already have initialized a Devise User model along with a Rolify Role model, a very common question I see is how should the roles be prepared so that the new users can have a list of already created roles to choose from when registering? In the Rolify documentation, most (if not all) examples involve implicitely creating roles by adding roles to user in the Rails console:

user = User.find(1)
user.add_role :admin

A new admin role will be created if it does not exist already. But in a fresh application, with no users, a list of existing roles should be available in order to register.

Answers to this dilemma involve different approaches. I personally prefer creating them when seeding the database:

# db/seeds.rb

Role.create!(name: 'admin')
Role.create!(name: 'doctor')
Role.create!(name: 'nurse')

Select2 With Simple Form in Rails

Mon, 14 Aug 2017 00:00:00 +0000

Select2 is a great JQuery plugin that customizes your select boxes to give a better user interface and experience.

Recently while working on a Rails project, I was experiencing some problems when using Select2 and Simple Form in my view.

In the view, I want to use Select2 to search for people in a Person table in the application's database. Select2 should use AJAX to hit a controller's action that performs the search. The user can then select one of the results found and it will be added to the field as a token (this is a multiple select field).

The controller action looks something like this:

def search
  @people = Person.all.where('name LIKE ? OR last_name LIKE ?',
    "%#{params[:q]}%", "%#{params[:q]}%")

  respond_to do |format|
    format.json { render json: @people.map { |p| { id: p.id, full_name: p.full_name } } }
  end
end

An Introduction to Game States in Love2D

Thu, 27 Jul 2017 00:00:00 +0000

After going through countless blog and video tutorials on getting started with Love2D, I found it sad that most people would not take the approach of implementing game states right from the start, or would try to mimick this functionality in a “dirty” way.

After lots of time of reading many forum posts and library documentations. I found a way to implement game states in my games that I am very satisfied with.

In this post I will go over how to implement game states and some of the most common game states seen in games, in such a way that your code is remains organized and neat.

Game States With Stateful.lua

We will be using a simple but great Lua library called Stateful, made by kikito. This library will allow us to easily implement and work with game states inside our game.

This library also depends on the Middleclass library (also by kikito). Middleclass will allow us to construct and develop our game in a OOP (Object Oriented Programing) fashion. If you are familiar with OOP then you will love this approach.

Getting Started With Love2D Basics

Wed, 19 Jul 2017 00:00:00 +0000

I recently started playing around with Love2D for game development. Although I am completely new to Lua, so far it's been pretty fun and I really like it.

I created this post to write down the most important basic concepts that I have encountered so far, so that I can use as quick reference later.

Setting Up a Basic IDE

So far I've been using Love2D in Windows. Initially I started using Sublime Text 2 as my editor, but then figured out how to make my Love2D development experience much better using Atom combined with Github for Windows.

In Atom there is a great package called love-ide which has many great tools and features for developming with Love.

Some of these features include Lua linting, autocomplete, and being able to run the Love application right from Atom.

Additionally, we will want to use Window's console for debugging. To do so, enable these options:

Working With Keyboard Keys

We can interact with user input through the keyboard by determining if a certain key has been pressed. This is done like this:

if love.keyboard.isDown("right") then
  -- ...
end

The love.keyboard.isDown function receives a KeyConstant string argument to indicate which key we want to test for. The complete list can be found here. Here are some examples:

Key string	Description
`space`	Space key
`backspace`	Backspace key
`down`	Down arrow key

Extending Device Python Classes From Other ZenPacks

Tue, 27 Jun 2017 00:00:00 +0000

Recently I was struggling with trying to make two custom ZenPacks work together with device objects in the Zenoss database. My issue was that a main ZenPack (which would be installed first) would create a custom class that inherits from Device. This ZenPack would add a lot of properties and relationships with custom components. Then, a second optional ZenPack could be installed which would extend this same class with an additional property that would work along with a Python data source.

The zenpack.yaml file for the first and main ZenPack would look something like this:

# ZenPack1

classes:
  SpecialServer:
    base: [zenpacklib.Device]
    label: Special Server

  SpecialComponent:
    base: [zenpacklib.Component]
    label: Special Component
    properties:
      # ...

  AmazingComponent:
    base: [zenpacklib.Component]
    label: Amazing Component
    properties:
      # ...

class_relationships:
  - SpecialServer 1:MC SpecialComponent
  - SpecialServer 1:MC AmazingComponent

Using Lambdas in Case Statements in Ruby

Mon, 12 Jun 2017 00:00:00 +0000

In a previous post I talked about how we could use Ruby ranges inside case statements. This was a very neat way of using case statements that would make our code more readable and less repetitive.

The other day I was trying to solve the Bob Ruby exercise in Exercism, and I came up with another very cool way to use case statements: with Lambdas.

The Problem

The Bob problem is very simple. The program receives some input, and it returns some output based on the contents of the input. A strong indication for using a case statement. From the problem's README file:

Bob is a lackadaisical teenager. In conversation, his responses are very limited.

Bob answers 'Sure.' if you ask him a question.

He answers 'Whoa, chill out!' if you yell at him.

He says 'Fine. Be that way!' if you address him without actually saying anything.

He answers 'Whatever.' to anything else.

Solutions

The test provided by Exercism indicates that we should create a Bob class with a hey class method that receives a remark. We could then use a case statement with this remark and evaluate if the remark is a question, yelling, or whatever else needs to be determined according to the instructions. We could create some class methods to determine each of these possibilities:

Zenoss Custom Notification Actions Using Subscribers

Wed, 07 Jun 2017 00:00:00 +0000

In a previous post I talked about how we could create custom notifications actions in Zenoss. Like a SMS notification, for example. In that post we required a valid cellphone number using a text field in the notification's content pane, as shown below:

In this post I want to change and improve this by using subscribers functionality instead. Similar to the e-mail (or pager) notification, which sends an e-mail to all the subscribed users using their e-mail address configured in their user settings. Since there is no SMS mobile number that we can configure in the user settings, we will have to use the Pager field instead.

Using Zenoss Core 4's pager action's source code as reference is a good starting point, since we are going to use the pager field for SMS.

We can see that this action class implements a executeOnTarget method instead of a execute method we used in the previous post.

Python Functional HTTP Testing With Gabbi

Tue, 06 Jun 2017 00:00:00 +0000

Continuing with my journey in contributing to Gnocchi, I have learned of an excellent tool which Gnocchi uses to test its API from a HTTP request approach. The tool is called Gabbi.

Coming from a Ruby, Rails, and Rspec background, I was very pleased on learning how Gabbi works and how to use it. Gabbi uses YAML to construct the tests that will hit the API endpoints, in a similar way to using Rspec's DSL when making request specs.

For my latest contribution to Gnocchi, I was working on an issue where newly created metrics were not being returned in the response from /v1/resource/generic/:id/metric. After applying the fixes and submitting the patch, it was also necessary to update some of the functional Gabbi tests that belong to this specific use case.

One of these original tests is shown below:

Testing JSON API Strong Parameters in Rails

Thu, 25 May 2017 00:00:00 +0000

Recently I was experiencing a problem with a Rails API where I would update a model with new attributes, and then proceeded to add the new attributes to the model's factory, and then add the corresponding model and request specs.

The problem was that I would forget to whitelist the new attributes in the controller. Then running the request specs would not detect the error since I only test against one updated attribute that was whitelisted long ago. This became really annoying since I had no way to know if the new attributes were being whitelisted or not, and my tests weren't saying anything about it. This problem becomes even more apparent when the list of whitelisted attributes starts becoming very long.

In an API following the JSON API specification, here is an example of how the strong parameters method would look like:

def thing_params
  params.require(:data).require(:attributes).permit(:color, :size, :age, :name,
      :owner, :origin, :location, :purpose, :price, :alive)
end

As you can see, the list of attributes is getting pretty long. And we do not want to explicitly keep adding more attributes to the test.

Testing Strong Parameters in Controller Specs

After the release of Rails 5 I started ditching controller specs in favor of request specs. I like this approach and I think it's great, but I think it would be too much unecessary work to test that all the necessary attributes are whitelisted, in the request spec. Therefore I proceeded to create very simple controller specs that would test that all attributes of importance are whitelisted in the controller.

Understanding Gnocchi Measures

Wed, 24 May 2017 00:00:00 +0000

Measures are simple objects that represent the timeseries data. They simply contain a timestamp and a value, and they belong to metrics. You could say that a measure has many metrics.

Adding Measures to a Metric

Using the REST API, we can easily add measures to a metric:

POST /v1/metric/511b61a1-8d67-42d5-8add-66d4209a0469/measures HTTP/1.1
Content-Type: application/json
Content-Length: 198

Understanding Gnocchi Metrics

Thu, 18 May 2017 00:00:00 +0000

Metrics are one of the main object types in Gnocchi. They are identified by a UUID and they can also be attached to a resource by using a resource name. Metrics store measures, and the way they do this is defined by archive policies. These are concepts that I will cover in future articles.

Basically, a metric designates any thing that can be measured: the CPU usage of a server, the temperature of a room or the number of bytes sent by a network interface.

In the Gnocchi architecture, the storage back-end is responsible for storing measures of created metrics. It receives timestamps and values, and pre-computes aggregations according to the defined archive policies.

Testing a Rails API With RSpec

Mon, 15 May 2017 00:00:00 +0000

As I continue to improve me API testing skills in Rails, I have come to point where I really feel comfortable with all the tools I need to correctly add useful tests to my API. This post explains how I usually test my Rails APIs using RSpec.

Setting Up the Necessary Tools

In addition to RSpec, there are a few tools that make my testing experience much easier. These are:

After Adding these gems to the Gemfile. I proceed to create a /support directory under the /spec directory. This directory will contain the additional configuration of the above tools, as well as some helper definitions that our specs will use.

/spec/support/shoulda.rb:

Shoulda::Matchers.configure do |config|
  config.integrate do |with|
    # Choose a test framework:
    with.test_framework :rspec

    # Or, choose the following (which implies all of the above):
    with.library :rails
  end
end

Custom Notification Action Types in Zenoss

Tue, 09 May 2017 00:00:00 +0000

In a previous post I talked about how we could create command type notifications to execute shell scripts when a trigger is fired by an event in Zenoss. Also, in a more recent post I explained how a ZenPack could automatically create these triggers and notifications when installed by defining them in JSON format.

In this post I will explain how to create additional custom action types that can be used by notifications for different purposes.

Zenoss Default Notification Actions

By default, Zenoss comes with a built-in notification action types. You can see them when creating a new notification from the user interface:

These action types are:

Command: Runs an executable script. Discussed in one of my previous posts.
Email: Sends an e-mail to a user.
Page: Sends a message to a user's pager, also known as beeper. Probably safe to say that this is not used by anyone anymore.
Syslog: Logs a message to syslog
SNMP Trap: Sends an SNMP trap.

All these notifications are sent when an event matches a trigger rule.

Creating the New Action

Next, we are going to learn how we can create and add a new fully functional custom action to this list, using a ZenPack. For this example, let's assume that we want to create a new SMS action to send SMS messages. Surely something more modern and useful than a Pager!

Rails API With Nested Resources

Mon, 08 May 2017 00:00:00 +0000

Cocoon is a very popular gem to add nested resources functionality in Rails. It allows you to add or remove “mini forms” of another model into a main model's form and create all objects at once when the form is saved.

I was recently finding myself trying to implement a similar functionality using only Rails 5 API and JSON API specification.

As an example, we will try to implement the commonly seen Work Experience and Educational Background features seen in sites such as LinkedIn, where the user can add many items for these categories. The user's profile and all items for those categories (which are separate models) should all be saved at once, once the form is submitted.

accepts_nested_attributes_for

accepts_nested_attributes_for allows us to save attributes on associated records (work experience, educational background) through a parent record (user). By default nested attribute updating is turned off and you can enable it using the #accepts_nested_attributes_for class method.

Custom Triggers and Notifications in a ZenPack

Thu, 04 May 2017 00:00:00 +0000

In a previous post I talked about how to use triggers and command notifications in Zenoss to trigger custom actions when certain events occur. All of this was done using the Zenoss user interface.

In this post we will achieve something similar, but from a custom ZenPack. At the end of this post, our custom ZenPack will be able to create new custom triggers and notifications when installed.

Defining ZenPack Triggers & Notifications

Triggers and notifications within a ZenPack are actually defined using JSON. To do so, you must create a file named actions.json in a directory called zep (create it if it doesn't exist), within the ZenPack's top directory.

actions.json:

{
  "triggers": [
    {
      "name": "Critical_death_event",
      "uuid": "4c055067-98b7-483e-8f49-2820b4f2f721",
      "enabled": false,
      "rule": {
        "api_version": 1,
        "type": 1,
        "source": "(evt.event_class.startswith(\"/Status/Ping\")) and (evt.status == 0) and (evt.severity > 2)"
      }
    }
  ],

  "notifications": [
    {
      "id": "send_sms_message",
      "description": "Send SMS using Twilio",
      "action": "command",
      "guid": "2606439f-5ef7-40dc-90e4-3f3bee11cfe6",
      "enabled": false,
      "action_timeout": 60,
      "delay_seconds": 0,
      "repeat_seconds": 0,
      "send_initial_occurrence": false,
      "send_clear": false,
      "body_format": "echo \"Hello World!\"",
      "clear_body_format": "",
      "subscriptions": ["4c055067-98b7-483e-8f49-2820b4f2f721"]
    }
  ]
}

Working With RRDTool

Tue, 02 May 2017 00:00:00 +0000

Zenoss makes a lot of use of RRDTool for storing monitored performance data and generating graphs. In this post I want to go into more detail on the components of RRDTool and how works.

Round Robin Databases

RRDTool is a program that works with Round Robin databases (hence RRD). This type of database is more geared and limited towards storing time series data.

Moreover, rrdtool also doesn't store the data handed to it. Instead, it first re-samples the data and then stores this re-sampled data. Since rrdtool deals with time series data, it knows that certain data is meant to be stored every certain period of time (i.e 5 minutes). This newly re-sampled data will be the same as the original data, but the stored numbers will be different.

This means that if for example, we were to store some data every 5 minutes, a data sample that arrives at the 03 minute mark cannot go into the database, therefore it has to be re-adjusted into the correct slots. While rrdtool makes this adjustment, it will also change the value, so that this new value still represents the space in the curve.

In rrdtool, each database table is stored in a separate .rrd file. This means that one file represents one table.

We can create a database using the following command:

rrdtool create ex5.rrd --step=5 DS:input:GAUGE:10:0:1000000 RRA:LAST:0.5:1:603

Data Re-sampling: Step

One important parameter here is the step parameter. This means that rrdtool will re-sample all the data that is fed into the database to a 300 second interval. Still, data can be fed as quickly as desired (i.e a lot of data in one second), however the data will be re-sampled to 300 seconds on storage. This means that it only stores data every 300 seconds. Because of this re-sampling, rrdtool will not accept data from the past.

Diving Into OpenStack Gnocchi

Tue, 25 Apr 2017 00:00:00 +0000

Gnocchi is a multi-tenant timeseries, metrics and resources database. It provides an HTTP REST interface to create and manipulate the data. It is designed to store metrics at a very large scale while providing access to metrics and resources information and history.

It is the preferred storage method for metrics in Ceilometer, as of OpenStack Ocata.

In this post I want to dive into Gnocchi specifics such as its configuration, supported backends, APIs, daemons, and source code.

Configuration

Gnocchi's configuration is stored in a file called gnocchi.conf. Ideally, this file would be in ~/gnocchi.conf or /etc/gnocchi/gnocchi.conf. Let's take a look at a basic Gnocchi configuration:

[DEFAULT]
debug = true
verbose = true

[api]
workers = 1

[database]
backend = sqlalchemy

[indexer]
url = postgresql://gnocchi:gnocchi@127.0.0.1/gnocchi

[storage]
coordination_url = file:///home/ubuntu/gn/locks
driver = file
file_basepath = /home/ubuntu/gn

[cors]
allowed_origin = *
allow_credentials = false

The configuration above sets up Gnocchi to use Postgresql as the indexer, and use the file system for storage. Additionally it sets up CORS so that requests from any origin are allowed. You will want to configure CORS in a more secure manner when deploying to a production environment.

Database Setup

For this example, we are going to use Cloud 9 as our environment, and Postgresql as the database. This means that we need to first setup the database before we start using Gnocchi.

Make sure the Postgresql service is running:

sudo service postgresql start

We can enter the Postgresql command line using:

sudo sudo -u postgres psql

Now let's create a new Postgresql user:

CREATE USER gnocchi SUPERUSER PASSWORD 'gnocchi';

Then create the database:

CREATE DATABASE gnocchi WITH TEMPLATE = template0 ENCODING = 'UNICODE';

When the database is finally set up correctly and the configuration file is in place, we can initialize the indexer and storage:

gnocchi-upgrade

You should see the following output logs:

2017-05-17 05:28:50.917 3895 INFO gnocchi.cli [-] Upgrading indexer <gnocchi.indexer.sqlalchemy.SQLAlchemyIndexer object at 0x7ff76cff6190>
2017-05-17 05:28:50.982 3895 INFO alembic.runtime.migration [-] Context impl PostgresqlImpl.
2017-05-17 05:28:50.982 3895 INFO alembic.runtime.migration [-] Will assume transactional DDL.
2017-05-17 05:28:51.011 3895 INFO alembic.runtime.migration [-] Context impl PostgresqlImpl.
2017-05-17 05:28:51.011 3895 INFO alembic.runtime.migration [-] Will assume transactional DDL.
2017-05-17 05:28:51.154 3895 INFO gnocchi.cli [-] Upgrading storage <gnocchi.storage.file.FileStorage object at 0x7ff7688f9710>

Gnocchi REST API

Gnocchi's REST API is based on Pecan, a very lightweight Python web framework that provides object-dispatch style routing. We can confirm this in Gnocchi's rest/__init__.py file:

import pecan
from pecan import rest

Metrics

Gnocchi provides an object type that is called metric. A metric designates any thing that can be measured: the CPU usage of a server, the temperature of a room or the number of bytes sent by a network interface.

A metric only has a few properties: a UUID to identify it, a name, the archive policy that will be used to store and aggregate the measures.

Farther down the code in rest/__init__.py, we can find a metric controller which inherits from a Pecan REST controller:

class MetricController(rest.RestController):
    _custom_actions = {
        'measures': ['POST', 'GET']
    }

    def __init__(self, metric):
        self.metric = metric
        mgr = extension.ExtensionManager(namespace='gnocchi.aggregates',
                                         invoke_on_load=True)
        self.custom_agg = dict((x.name, x.obj) for x in mgr)

    def enforce_metric(self, rule):
        enforce(rule, json.to_primitive(self.metric))

    @pecan.expose('json')
    def get_all(self):
        self.enforce_metric("get metric")
        return self.metric

From the Pecan documentation, we can learn that Pecan uses a routing strategy known as object-dispatch to map an HTTP request to a controller, and then the method to call. Object-dispatch begins by splitting the path into a list of components and then walking an object path, starting at the root controller.

We can tell Pecan which methods in a class are publically-visible via expose(). If a method is not decorated with expose(), Pecan will never route a request to it. In the example above, the get_all() method is exposed to Pecan. Additionally, it makes use of Pecan's built-in support for a special JSON renderer, which translates template namespaces into rendered JSON text. Meaning that the returned content will be rendered as JSON.

Understanding oslo_config in OpenStack

Tue, 18 Apr 2017 00:00:00 +0000

The OpenStack Oslo library provides a set of Python libraries containing code shared by different OpenStack projects. This helps the many different OpenStack projects to follow a certain standard and convention when being developed.

In this post I want to go over oslo_config. This library helps with parsing of options found in configuration files or command line arguments.

Understanding Options

An option is defined using the Opt class or one of its sub-classes, from the cfg module:

from oslo_config import cfg

common_opts = [
    cfg.StrOpt('bind_host',
               default='0.0.0.0',
               help='IP address to listen on.'),
    cfg.Opt('bind_port',
            type=PortType,
            default=9292,
            help='Port number to listen on.')
]

In the example above we can see that we are using Opt and StrOpt, one of its sub-classes. You might notice that the second option's constructor has a type=PortType argument. This argument is a callable object that takes a string and either returns a value of that particular type, or raises ValueError if the value cannot be converted. The different types available can be found in the types module:

Tech and Open Source Chinese Vocabulary

Mon, 17 Apr 2017 00:00:00 +0000

The following is my personal curated list of Chinese terms (in Traditional Chinese) and vocabulary used in web development, programming, technology and open source. Chinese Word Pinyin Definition 修復 xiū fù Repair, fix (bug) 發版 fā bǎn To release a version 報錯 baò cuò Error report/message 組件 zǔ jiàn Component 版本 bǎn běn Version 更新 gèng xīn To update/upgrade 顯示 xiǎn shì Show, display 表格 biaǒ gé Form, table 按鈕 àn niú Button, to press a button 重新 chóng xīn Retry (verb) 項目 xiàng mù Project 需求 xū qiú Request 代碼 dài mǎ Code 源碼 yuán mǎ Source code 固定 gù dìng Fixed, regular 內容 nèi róng Content 輸入 shū rù Input, to input 屬性 shǔ xìng attribute, property 路由 lù yóu routing 參考 cān kǎo Consult, refer to, reference 標題 biaō tí title, heading 一致 yī zhì consistent, identical 默認 mò rèn default 里程碑 lǐ chéng bēi milestone 圖標 tú biaō icon 接口 jiē koǔ interface, port 操作系統 Cāozuò xìtǒng operating system 支持 zhī chí support 佈局 bù jú layout

Debian Package Versioning

Thu, 13 Apr 2017 00:00:00 +0000

In a previous post I talked about managing Debian package dependencies. However I did not go into detail about the structure the versioning of packages must have.

To recap, let's see how we can specify a specific package version of a dependency that our package needs:

Depends: erlang (>= 1.2)

In the example above, the required version for erlang is simply 1.2 or greater. A package's version format however, can be a bit more complex than this.

Version Format and Components

The official format of a Debian package's version is:

[epoch:]upstream_version[-debian_revision]

Let's go into detail about each of the components in that format.

Understanding Ceilometer Publishers

Tue, 11 Apr 2017 00:00:00 +0000

Continuing with my study of the Ceilometer pipeline, this post now covers Ceilometer publishers. Publishers are components that make it possible to save the data into a persistent storage through the message bus, or to send it to one or more external consumers.

Publishers are specified in the publishers section for each pipeline that is defined in the pipeline.yaml and the event_pipeline.yaml files.

Many different publishers are available. The following are the most common and important publishers:

Gnocchi

Reworking Ace's HAML Syntax Highlighting

Mon, 10 Apr 2017 00:00:00 +0000

Ace is a great web text editor and the default editor for Cloud 9 IDE. I have been using it for many years without any complaints at all. However I was not very satisfied with the HAML syntax highlighting, which seemed to have some problems caused by indentation when highlighting some tokens. Additionally, it didn't support correct highlighting of some HAML stuff such as HAML comments (which begin with -#) or block comments.

This is how Ace's HAML syntax highlighting issues look like:

I proceeded to study Ace's logic for syntax highlighting. It consists basically on a lexer that reads the input through different regular expressions and proceeds to different stages depending on the regular expression caught. Basically, a state machine. The source where this happens is found in lib/ace/mode/haml_highlight_rules.js

Defining States

A few states have to be defined to represent where the lexer currently “stands” in regards to the code. For example, entering a multi-block comment could represent entering a new state, since everything parsed in this state would belong to this block comment until the block ends, this will also mean another change of state.

In Ace, all syntax highlighting lexers must begin with a start state. From this state we can switch to other defined states. The example below shows how we begin from the start state and can jump to a comment block state when the code matches a regular expression that represents this:

this.$rules = {
    "start": [
        {
            token: "comment.block", // multiline HTML comment
            regex: /^\/$/,
            next: "comment"
        },
        {
            token: "comment.block", // multiline HAML comment
            regex: /^\-#$/,
            next: "comment"
},

/* ... */

Notice that we define 2 different comment types, since HAML supports HAML (not rendered in HTML) and HTML (rendered in HTML) comments. Both of these different regular expressions will make the lexer parse them as comment.block tokens, and will also make the lexer jump to a comment state, denoted by the next keyword.

Understanding Ceilometer Transformers

Fri, 07 Apr 2017 00:00:00 +0000

Ceilometer transformers are part of the Ceilometer pipeline, which is the mechanism by which data is processed. In Ceilometer there is a pipeline for samples and a pipeline for events.

Configurations for these pipelines can be found in /etc/ceilometer/pipeline.yaml and /etc/ceilometer/event_pipeline.yaml.

Transformers are responsible for mutating data points and passing them to publishers that will send the data to external systems.

Different types of transformers exist to mutate different types of data. The following is a table containing all the available transformer types:

Name of transformer	Reference name for configuration
Accumulator	accumulator
Aggregator	aggregator
Arithmetic	arithmetic
Rate of change	rate_of_change
Unit conversion	unit_conversion
Delta	delta

Let's take a more detailed look into the accumulator transformer.

Ceilometer and OpenStack Notifications

Thu, 06 Apr 2017 00:00:00 +0000

Ceilometer has different mechanisms to collect data from OpenStack. One of these mechanisms, are notifications, which will be explained in this post.

OpenStack Notifications

All OpenStack services (such as Nova Compute, Neutron, etc.) send notifications about executed operations or the state of the system. Many of these notifications carry information that can be metered such as CPU time of a virtual machine instance created by the compute service.

OpenStack services send these notifications through an AMQP message queue. However, not all these notifications are consumed by the Telemetry service, as the intention is only to capture the billable events and notifications that can be used for monitoring or profiling purposes. The notification agent filters by the event type. Each notification message contains the event type.

Here is a table with a few examples:

OpenStack Service	Event types
OpenStack Compute	`scheduler.run_instance.scheduled`, `scheduler.select_destinations`, `compute.instance.*`
OpenStack Image	`image.update`, `image.upload`, `image.delete`, `image.send`

In Ceilometer, the notification agent is reponsible for consuming these notifications from the AMQP message bus, and then transforming them into events and samples.

Using Rails With Webpack in Cloud 9

Mon, 03 Apr 2017 00:00:00 +0000

The rails/webpacker project allows integration of Webpack with a Rails application. However, setting this up in a Cloud 9 development environment needs a few tweaks to be able to work correctly. This post explains how to achieve this.

Creating the Application

We will first create a regular Rails application and then use the webpacker gem to install Webpack.

rails new webpack-app

Add the gem to Gemfile:

# Gemfile

gem 'webpacker', github: 'rails/webpacker'

Install Webpacker

rails webpacker:install

Using Objects and Ranges With Cases in Ruby

Wed, 29 Mar 2017 00:00:00 +0000

Ruby's case statement provides some nice functionality to be used with objects and ranges. I was recently developing a small application in Ruby on Rails using blood test data where I had to check if a certain test item (i.e. HDL Cholesterol) value belonged to a certain range. I did not want to use a bunch of if statements and I actually found a pretty neat way to implement using the Ruby case.

In my application I have a Rails model called Entry which represents a blood test result with a lot of field data such as blood glucose level, blood pressure level, etc. etc. I want to display this data in the view using Bootstrap 3's nice progress bars and I want the progress bar's contextual class to be changed and determined by the range in which this data point sits in.

If we take total cholesterol as an example, a quick Google search indicates that an ideal value would be less than 200 mg/dL, a high value (not good) would be between 200-239 mg/dL, and a very high value (definitely not good) would be above 240 mg/dL.

To implement this, I created a Recommendations module under the /models directory. All classes belonging to this module will have different ranges to represent what I explained above. This is what the model directory tree looks like:

app/models/
├── application_record.rb
├── concerns
├── entry.rb
├── recommendations
│   ├── blood_glucose.rb
│   ├── diastolic_pressure.rb
│   ├── hdl.rb
│   ├── ldl.rb
│   ├── systolic_pressure.rb
│   └── total_cholesterol.rb
└── user.rb

Setting Up an OpenStack Dev Environment

Wed, 22 Mar 2017 00:00:00 +0000

Before beginning to contribute to OpenStack, it is necessary that we setup an ideal development environment for a smoother workflow.

In this post I will cover how to setup a Devstack development environment in a Ubuntu 16.04 virtual machine using VMware Player.

Setting Up the Ubuntu Virtual Machine

Make sure you have VMware Player installed in your system. Then we need to choose an appropriate Ubuntu ISO image for our virtual machine. In my case I am using a Ubuntu Desktop 16.04.2 LTS (Xenial Xerus) for 32-bit PC i836.

In VMware Player, create a new virtual machine using the wizard. Assign your preferred amount of memory and storage space. Now start the machine, and select the Ubuntu .iso image file when prompted. Proceed with the installation and select the Erase Disk and Install Ubuntu option when prompted.

~> I tried using VirtualBox for virtualization but I could not successfully install Devstack with it. Probably due to issues on using the correct network adapter. I also tried using Ubuntu 14.04 without success as well. This is why I recommend Ubuntu 16.04.

Once the installation is finished, restart the virtual machine.

Overriding Default Zenoss Pages

Fri, 17 Mar 2017 00:00:00 +0000

While testing the Layer 2 ZenPack, I noticed that the network map provided by the ZenPack was placed in the same page of the old default Zenoss's network map, replacing the latter which was made using Flash. After the ZenPack is installed, when you click on the Network Map secondary link in the Infrastructure page, you will get the new map instead.

This seemed pretty nice, and I was curious how they achieved this, so I started browsing around the source code. It turns out that this overriding is done in a file called overrides.zcml, which is placed in the ZenPack's top directory.

Getting Started on Contributing to OpenStack

Tue, 14 Mar 2017 00:00:00 +0000

I have started to become interested in contributing to the OpenStack project. Starting to contribute to such a massive project can seem like a daunting task, and it is. It is only by arming yourself with the proper tools and knowledge prior to contributing, that you will have a smoother process. I created this blog post to smooth that process.

Picking a Project

OpenStack is composed of many different projects for different purposes (compute, storage, networking, telemetry, etc.). Before you start contributing, you must have an idea of which project you actually want to contribute to. In my case I have decided on the Telemetry part, more specifically on Ceilometer.

After you have picked a project, it is time to start reading its documentation and developer documentation to understand how it works. The Ceilometer developer documentation offers good explanations and diagrams of its architecture and components.

Here is another nice guide on contributing to Ceilometer.

Python Tricks for Better Code

Wed, 08 Mar 2017 00:00:00 +0000

A compilation of very useful Python tricks for better code. All examples are Python 2.7 and Python 3 compatible unless stated otherwise.

Enumerate

When working with items in a list and we want to print or handle each item's index, instead of using a counter variable:

# Bad way
i = 0
for place in places:
    print(i, place)
    i += 1

We can use enumerate to improve our code:

for i, place in enumerate(places):
    print(i, place)

enumerate returns a list of indexes along with each item in the list.

Integrating TravisCI With Rails and Heroku

Tue, 07 Mar 2017 00:00:00 +0000

Adding TravisCI to your application's deployment flow will enable you to spot any build fails or test fails during deployment without affecting your production environment.

In this post I go over on how to integrate TravisCI with a Rails 4 application that uses Postgresql, Rspec, and is later deployed to Heroku if the Travis build is successful.

Setting up TravisCI

Assuming you already have an account in TravisCI, look for the Github repository of the Rails application and enable it. Then we will want to add a .travis.yml file to the application's root directory with the following contents:

language: Ruby
rvm:
  - 2.3.0

The configuration above tells TravisCI that our application will use Ruby 2.3.0. Once you commit and push the change above to the master branch, Travis will begin creating the build, which might take a few minutes.

Configure Postgresql

Seems our Rails application uses Postgresql, we need to configure TravisCI to run our build with a Postgresql database. To do this we will add the following configuration to .travis.yml:

services:
  - postgresql

Then we will add another configuration that will make Travis create a Postgresql database with username postgres and blank password for the build:

before_script:
  - psql -c 'create database travis_ci_test;' -U postgres

Chinese Onomatopoeia in Literature

Mon, 06 Mar 2017 00:00:00 +0000

It's been almost a year since I started reading the Harry Potter series completely in Traditional Chinese, currently I just recently started Harry Potter and the Order of the Phoenix (Book #5).

Through my Chinese reading journey I have come accross a lot of Chinese onomatopoeia that you don't really come across with in other types of writings such as Internet articles, conversations, and newspaper articles.

In this post I go into detail about each of these sounds I encounter in literature.

咕嚕 (gūlū)

Definition: rumble, roll.

Used a lot when describing the rumbling sound of the stomach when hungry.

Examples:

他的肚子咕嚕咕嚕直響 - His stomach kept rumbling).
粗大的圓木咕嚕咕嚕地從山坡上滾下來 - Large logs came rolling down the slope

啪啪 (pāpā)

Definition: The sound of clapping, slapping, a gunshot.

Examples:

啪的一聲, 瓶塞飛出來了 - The cork popped and flew out of the bottle.

Working With Zenoss Python Data Source Plugins

Thu, 23 Feb 2017 00:00:00 +0000

Using Python data source plugins in Zenoss is a great way to collect data, and probably a better way than using command datasources. Python data sources come with the introduction of the PythonCollector ZenPack, so this ZenPack is required in order to start using Python data sources in our own ZenPacks.

Python data source plugins work exceptionally great in replacing data collection logic in custom daemons written in Python. This means that the ZenPacks's code is greatly reduced because we do not have to create configuration service, and custom daemon code.

Moreover, while Python data source plugins are Python-based, we can still execute shell commands within the plugin.

Example Scenario: BMC Power Status

For this post, I will be using an example where we will be executing an ipmitool command to check the power chassis status of a BMC device.

If I run the following command from my shell:

ipmitool -H $BMC_IP -I lanplus -U admin -P admin power status

I get the following output:

Chassis Power is on

What we want is our data source plugin to periodically and continuously execute this command and obtain the value. Our ZenPack will then proceed to:

Map the value into a boolean property of a custon zenpacklib class.
Create a CRITICAL event if the power status is off or if the command fails. Create a CLEAR event if the power status is on.
Update the device model with the new property value.
Display the property's value in the device detail bar using JavaScript.

Useful Zendmd Tricks

Tue, 14 Feb 2017 00:00:00 +0000

The Zenoss Zendmd Tips Wiki page contains a few useful tricks using zendmd to perform tasks. In this post I am adding more tricks that I discover and learn along the way.

Removing Device Classes

We can easily remove device classes within zendmd with a simple command. Assuming we want to remove the default “KVM” device class:

dmd.Devices.manage_deleteOrganizer("/zport/dmd/Devices/KVM")
commit()

Change Zenoss User's Password

Let's say we want to change the default admin user's password (zenoss) in the Ubuntu auto deploy:

app.acl_users.userManager.updateUserPassword('admin', 'newpassword')
commit()

Getting Started With Juju Locally

Fri, 10 Feb 2017 00:00:00 +0000

Juju is a great tool that allows us to deploy many services in a local or cloud environment. Services are deployed into virtual containers and can be removed and re-deployed at any time. It is perfect for creating sandbox and test environments for specific kind of services that you develop for.

In this post I will explain how easy it is to get started with Juju in a local environment.

Installation

To start we will need to add the necessary repository from where we will retrieve the Juju packages:

sudo add-apt-repository ppa:juju/stable
sudo apt-get update

Next we will download and install the Juju core package and the package for local deployment:

sudo apt-get install juju-core
sudo apt-get install juju-local

Setting Up the Local Environment

Before we set up the local environment, let's first generate a configuration file. This file will contain the configuration for many kinds of environments (OpenStack, AWS, MaaS, etc.), including a local environment configuration. We will generate the file and then switch to the local environment:

juju generate-config
juju switch local

Once we have switched to the local environment, we can simply bootstrap the environment with the following command:

juju bootstrap

After the process is complete, we are now ready to start using Juju.

C Gotchas For Beginners

Tue, 07 Feb 2017 00:00:00 +0000

Similar to my post on some must-know concepts about JavaScript for beginners, this is a list of concepts that show how C differs from other programming languages. These are concepts that you must know if you are to become a good C programmer.

Boolean Types

C actually does not have a boolean type. They are usually defined like this:

#define BOOL char
#define FALSE 0
#define TRUE 1

However it is argued that it is a better practice to simply use integers instead (0 and 1).

Memory Address of Array Elements

In C, arrays have a contiguous memory allocation. We can verify this by printing the memory addresses of each element of an array. We can do so using & to access the memory location of the array element:

int main() {
  char vowels[] = {'a', 'e', 'i', 'o', 'u'};

  for (int i = 0; i < 5; i++) {
    printf("Address of vowel [%d]: %u\n", i, &vowels[i]);
  }

  return 0;
}

The above code will produces the following output for me:

Processing Nova Live Migration Events in Zenoss

Mon, 23 Jan 2017 00:00:00 +0000

When monitoring OpenStack using the OpenStack Infrastructure ZenPack integrated with Ceilometer, it is possible to get fast virtual machine state changes (on/off powering, etc.) by receiving events sent from Ceilometer to Zenoss.

However I discovered that when trying to get the same effect for live migration (live migrating a virtual machine from one compute node to another) scenarios, this would not work. I proceeded to investigate why.

Ceilometer Dispatcher Live Migration Events

I decided that the first thing to check was if the Zenoss Ceilometer dispatcher was capturing and sending the live migration events to Zenoss. Indeed, the logs can be found in Ceilometer under /var/log/ceilometer/ceilometer-collector.log:

2017-01-23 14:22:02.593 25667 INFO ceilometer_zenoss.dispatcher.zenoss [-] record_events called (events=[<Event: 3538a01c-ab8d-4e64-b0ff-6e8fe270e06a, compute.instance.live_migration.post.dest.start, 2017-01-23 06:22:02.584049, <Trait: state_description 1 migrating> <Trait: memory_mb 2 512> <Trait: ephemeral_gb 2 0> <Trait: fixed_ips 1 [{u'version': 4, u'vif_mac': u'fa:16:3e:6e:02:e9', u'floating_ips': [], u'label': u'admin-net', u'meta': {}, u'address': u'192.168.0.15', u'type': u'fixed'}]> <Trait: user_id 1 6d581c230c86475abf70cce41440e8a1> <Trait: service 1 compute> <Trait: priority 1 info> <Trait: state 1 active> <Trait: launched_at 4 2017-01-23 05:06:41> <Trait: flavor_name 1 m1.tiny> <Trait: disk_gb 2 1> <Trait: display_name 1 pdcmtest> <Trait: root_gb 2 1> <Trait: tenant_id 1 10296907e44248d2a707689f77d59ef6> <Trait: instance_id 1 87be4b45-e214-4ca3-8f5c-1bd31159f9e4> <Trait: vcpus 2 1> <Trait: host_name 1 ndc27-3222> <Trait: request_id 1 req-d4c27d06-6329-4ce0-adb5-370b8ca83a22>>])

2017-01-23 14:22:02.771 25667 INFO ceilometer_zenoss.dispatcher.zenoss [-] record_events called (events=[<Event: 8db64329-606c-4184-9b5b-eb815166cb17, compute.instance.live_migration.post.dest.end, 2017-01-23 06:22:02.761138, <Trait: state_description 1 > <Trait: memory_mb 2 512> <Trait: ephemeral_gb 2 0> <Trait: fixed_ips 1 [{u'version': 4, u'vif_mac': u'fa:16:3e:6e:02:e9', u'floating_ips': [], u'label': u'admin-net', u'meta': {}, u'address': u'192.168.0.15', u'type': u'fixed'}]> <Trait: user_id 1 6d581c230c86475abf70cce41440e8a1> <Trait: service 1 compute> <Trait: priority 1 info> <Trait: state 1 active> <Trait: launched_at 4 2017-01-23 05:06:41> <Trait: flavor_name 1 m1.tiny> <Trait: disk_gb 2 1> <Trait: display_name 1 pdcmtest> <Trait: root_gb 2 1> <Trait: tenant_id 1 10296907e44248d2a707689f77d59ef6> <Trait: instance_id 1 87be4b45-e214-4ca3-8f5c-1bd31159f9e4> <Trait: vcpus 2 1> <Trait: host_name 1 ndc27-3222> <Trait: request_id 1 req-d4c27d06-6329-4ce0-adb5-370b8ca83a22>>])

2017-01-23 14:22:02.782 25667 INFO ceilometer_zenoss.dispatcher.zenoss [-] record_events called (events=[<Event: d0dc4f4a-454a-42c5-b767-386dc3e0d1f3, compute.instance.live_migration._post.end, 2017-01-23 06:22:02.776665, <Trait: state_description 1 migrating> <Trait: memory_mb 2 512> <Trait: ephemeral_gb 2 0> <Trait: fixed_ips 1 [{u'version': 4, u'vif_mac': u'fa:16:3e:6e:02:e9', u'floating_ips': [], u'label': u'admin-net', u'meta': {}, u'address': u'192.168.0.15', u'type': u'fixed'}]> <Trait: user_id 1 6d581c230c86475abf70cce41440e8a1> <Trait: service 1 compute> <Trait: priority 1 info> <Trait: state 1 active> <Trait: launched_at 4 2017-01-23 05:06:41> <Trait: flavor_name 1 m1.tiny> <Trait: disk_gb 2 1> <Trait: display_name 1 pdcmtest> <Trait: root_gb 2 1> <Trait: tenant_id 1 10296907e44248d2a707689f77d59ef6> <Trait: instance_id 1 87be4b45-e214-4ca3-8f5c-1bd31159f9e4> <Trait: vcpus 2 1> <Trait: host_name 1 ndc27-3205> <Trait: request_id 1 req-d4c27d06-6329-4ce0-adb5-370b8ca83a22>>])

Notice the live_migration.post.dest.start and live_migration.post.dest.end logs.

Appending Modeler Plugins in ZenPacks

Fri, 20 Jan 2017 00:00:00 +0000

When creating ZenPacks using zenpacklib, assigning modeler plugins to device classes will cause all the device class's modeler plugins to be replaced by the new modeler plugins assigned by the ZenPack. Obviously most of the time this is not the behaviour we want. What we really want is that the new modeler plugins are simply added to the device class's list of modeler plugins.

In a previous post I explained how we can customize the ZenPacks installation process to perform additional tasks. The way to append new modeler plugins to a device class follows this concept.

Let's assume that we have the OpenStack Infrastructure ZenPack installed in our Zenoss Core. This ZenPack adds some custom modeler plugins to the /Server/SSH/Linux/NovaHost device class. We want our custom ZenPack to add new modeler plugins to this device class without replacing the ones added by the OpenStack ZenPack.

This will be done upon installation (like in the previous post) inside the __init__.py file in the ZenPack top directory. This is how the code looks like:

Customizing the ZenPack Installation Process

Tue, 17 Jan 2017 00:00:00 +0000

There are some circumstances where we need to perform certain tasks in our ZenPack the moment it is installed. We can achieve this using Python by placing this logic inside the ZenPack's top directory's __init__.py file. When creating a fresh ZenPack using zenpacklib, this __init__.py file will contain the following contents:

from . import zenpacklib

CFG = zenpacklib.load_yaml()

To add custom functionality that gets executed when the ZenPack is installed, we need to extend the install method of the ZenPack class. Below the original code, we can proceed to do so:

class ZenPack(schema.ZenPack):
    def install(self, app):
        # Our custom logic here

        super(ZenPack, self).install(app)

No other imports are necessary. Notice the last line of the install method, this is where the ZenPack gets installed.

For the purpuse of giving an example, Let's say that our ZenPack creates two new zProperties, one property to store an API key, and the other one to store a URL. Moreover let's say that we obtain these values from somewhere and we want to assign them automatically to the properties upon the installation of the ZenPack.

These two new properties are defined in zenpack.yaml:

name: ZenPacks.aalvarez.MyZenPack

zProperties:
  zMyApiUrl:
    category: MyApi
    type: string

  zMyApiKey:
    category: MyApi
    type: string

Understanding SNMP and Net-SNMP

Mon, 16 Jan 2017 00:00:00 +0000

Net-SNMP is a suite of tools used to implement SNMP (Simple Network Management Protocol). SNMP is a widely used protocol for monitoring the health and welfare of network equipment (eg. routers), computer equipment and even devices like UPSs. Net-SNMP includes tools with capabilities such as:

An extensible agent
An SNMP library
Tools to request or set information from SNMP agents
Tools to generate and handle SNMP traps
A version of the unix ‘netstat’ command using SNMP
A graphical Perl/Tk/SNMP based mib browser

How SNMP Works

SNMP allows a management station to treat its network as a distributed database of health and configuration information. SNMP contains a small set of operations:

GET: Retrieve data from a network node
GETNEXT: Retrieve the next element from a network node
SET: Send configuration or control commands to a network node
TRAP: A network node can send a notification to the management station
INFORM: An acknowledged trap (network nodes can try and send it again if no acknowledgement is received)

Fixing Zenoss Device Network Interface Graphs

Fri, 13 Jan 2017 00:00:00 +0000

When testing the monitoring of the OpenStack compute node devices in Zenoss (using OpenStack Infrastructure ZenPack), I noticed that I could not get any graphs for the network interfaces of the device, although we could still get and model all the available interfaces:

This was indeed strange because I could perfectly get the graphs for other devices. After a lot of head scratching and prying around the code and interface, I finally found the reason, which I explain below.

JavaScript Gotchas for Beginners

Fri, 06 Jan 2017 00:00:00 +0000

When starting to learn JavaScript, there are a few details and gotchas that are always important to remember. Memorizing and understanding these concepts will make you a better and faster JavaScript programmer overall and it is why I am listing a few of these concepts (which are best learned at the beginner level) in this post.

The Global Object

The global scope is the space in which global variables live and it can also be approached as an object. Each global variable is present as a property of this object. In browsers, the global scope object is stored in the window variable:

var v = 10;
console.log("v" in window);
// > true
console.log(window.v);
// > 10

Custom Pages in Zenoss

Mon, 26 Dec 2016 00:00:00 +0000

In a previous post I talked about how to create custom navigation links in Zenoss using a ZenPack. However we didn't get to creating the custom pages that these links would link to. This is what we will learn today.

Page Viewlets

Previously we created navigation item viewlets, this time we will be creating page viewlets. These type of viewlets allow us to create and insert new custom pages into Zenoss where we can display custom content. To create a new page, we add the following code to browser/configure.zcml:

<browser:page
  name="secondaryPage"
  for="*"
  permission="zenoss.View"
  template="templates/my_template.pt"
  />

It is important that the name attribute matches the URL of the navigation link we created previously. Another important attribute is the template attribute, this is a special file that will represent the base markup of the page using a combination of HTML, TAL, and METAL.

Also let's not forget to create a necessary python init file:

touch browser/__init__.py

Cracking Passwords With aircrack-ng

Sun, 25 Dec 2016 00:00:00 +0000

aircrack-ng is a powerful suite of tools that allows to assess wireless network security. The tools in the suite allows us to perform various actions such as packet monitoring in the environment, montoring of specific access points, de-authentication attacks on access points, and cracking of WEP and WPA PSK (WPA 1 and 2) passwords.

In this post I will explain on how to perform these actions.

Wireless Card Monitor Mode

Before we begin it is crucial that we change the mode of our wireless network card to monitor mode. BY default, our wireless network card's mode is managed mode. In managed mode, the network card generally only listens to packets addressed to the card, whereas in monitor mode it will hear every packet in the air. Additionally, monitor mode will also allows us to perform packet injection attacks (i.e. de-authentication attacks).

To set our network card into monitor mode, we can run the following commands in a terminal:

Extending Zenoss Navigation Bars

Wed, 21 Dec 2016 00:00:00 +0000

It is possible to extend the functionality of Zenoss's navigations from within our custom ZenPacks. This means that we can add or remove links to the navigation bars we frequently use to access the infrastructure page or event console.

configure.zcml

Zenoss ZenPacks can contain a file in the ZenPack top directory called configure.zcml. I've mentioned and talked about this particular file in previous posts. This file basically acts as a configuration glue between back-end functions and front-end components.

It is in this file where we will declare and create new navigational links from our ZenPack.

~> In this post we assume that the ZenPack is created using zenpacklib, which is a Python library that makes creating ZenPacks much easier. Zenpacklib also makes the integration between the back-end and front-end much easier as well.

Usually this file begins with some required boilerplate code:

configure.zcml:

<?xml version="1.0" encoding="utf-8"?>
<configure
  xmlns="http://namespaces.zope.org/zope"
  xmlns:browser="http://namespaces.zope.org/browser"
  xmlns:zcml="http://namespaces.zope.org/zcml">

  <!-- Our custom code here -->
  <include package=".browser" />
</configure>

Custom Zenoss API Endpoints

Tue, 20 Dec 2016 00:00:00 +0000

In a previous post I went over the Zenoss JSON API and how it works inside the Zenoss Core. In this post we will apply the concepts learned in that post in order to create custom API endpoints within Zenoss, which can be accessed by the JavaScript front-end, curl, API clients, etc. All this new functionality added from a basic ZenPack.

Creating an Endpoint

Assuming that we are starting with a new and fresh ZenPack created with zenpacklib, we will proceed to create a simple endpoint.

Let's go ahead and create a file named api.py (can be any name) under the ZenPack's top directory. In this file we will import necessary modules, implement interfaces, and define our routers and facades. If you still don't know what routers and facades are and what they do, I suggest you first take a look at The Zenoss JSON API post.

First, let's take a quick look at the imports:

import os.path
from urlparse import urlparse
import subprocess

from zope.event import notify
from zope.interface import implements
from ZODB.transact import transact

from Products.ZenUtils.Ext import DirectRouter, DirectResponse
from Products import Zuul
from Products.Zuul.catalog.events import IndexingEvent
from Products.Zuul.facades import ZuulFacade
from Products.Zuul.interfaces import IFacade
from Products.Zuul.utils import ZuulMessageFactory as _t
from Products.ZenUtils.Utils import zenPath

Some imports are probably not needed, but the ones that are of our interest are imports such as DirectRouter and DirectResponse, which you might remember from the previous post. Additionally we are also importing the necessary facade, events, and interfaces imports.

SSH Monitoring in ZenPacks

Wed, 14 Dec 2016 00:00:00 +0000

Zenoss usually monitors and collects information using SNMP and SSH methods. A good example of is the linux monitor ZenPack which can collect information such as hard disks, interfaces, and file systems using both methods.

SNMP works by installing and configuring an SNMP agent on the machine we want to monitor. This agent will poll the machine for data, and this data can be retrieved by Zenoss using net-snmp.

On the other hand, SSH works by configuring a username and password or a path to an SSH key zProperties. This will allow Zenoss to remotely access the host using SSH and execute the corresponding commands and return the corresponding information.

When developing ZenPacks, data collection using SNMP is a relatively common practice. There is even a very good tutorial on SNMP monitoring using zenpacklib. When using SNMP, the modeler plugins are SnmpPlugin plugins that typically work using a set of OIDs to determine exactly which data is to be modeled.

SSH in Modeler Plugins

In order to make a modeler plugin utilize SSH as it's polling method we need to use a different type of plugin, the CommandPlugin plugin. The structure of this type of plugin is very similar to the SnmpPlugin, however there is one critical variable that must be implemented.

IP Addresses - A Review

Fri, 02 Dec 2016 00:00:00 +0000

A review of IP addresses, as part of my Computer Networking Fundamentals series.

A lot of people know what IP addresses are, and a lot of people also probably know that each device that is connected to the Internet must have a unique IP Address, like 108.161.228.70

However, in this article I want to do a more in-depth review about IP addresses and how the work.

From a technical point of view and for reviewing purposes, it is worth remembering that IP Addresses (at least IPv4) are 32-bit addresses, because each number separated by each dot (called octet) is an 8-bit number. Since IP Addresses have 4 octets, these together make a sum of 32-bits.

Since the first two octets make a sum of 16 bits, 2^16 yields a total of 65,536 networks. At the same time, the remaining 16 bits also yield a total of 65,536 different hosts. But these are 65,536 different hosts per each network.

This raises a big issue because it makes it hard to decide how many bits in the IP address should be allocated to the network and how many bits should be allocated for the hosts, without wasting IP addresses. This is where subnet masks come in.

Working With Zenoss Events

Fri, 25 Nov 2016 00:00:00 +0000

Zenoss Events are among the most powerful and useful monitoring features that Zenoss Core provides. Many different types of events are automatically generated by Zenoss when important scenarios occur for the devices being monitored.

Zenoss Event Architecture

When an event arrives at Zenoss, it is parsed, associated with an event classification and then typically (but not always), it is inserted into the status table of the events database. Events can then be viewed by users using the Event Console of the Zenoss Graphical User Interface (GUI).

The events system has the concept of active status events and historical events (two different database tables in the MySQL events database).

Understanding Event Classes

Zenoss event classes are a simple organizational structure for the different types of events that Zenoss generates and receives. This organization is useful for driving alerting and reporting. You can, for example, create an alerting rule that sends you an email or pages you when the availability of a Web site or page is affected by filtering on the /Status/Web event class^[2].

The Zenoss JSON API

Thu, 24 Nov 2016 00:00:00 +0000

The Zenoss JSON API allows us to obtain very important information of what is going on in Zenoss, such as device information and events. This API can be queried using cURL or with some wrappers provided by Zenoss, available in languages such as Bashscript, Python, and Java.

The API documentation can be downloaded here. In the documentation you can see the available endpoints and methods that can be used to obtain the data we need.

In this post I will cover how the Zenoss back-end JSON API works, and how the Zenoss's front-end (made using ExtJS) interacts with it.

~> Even though Zenoss calls its API a JSON API (Simply because it returns data in JSON format), the API is not JSON API specification compliant.

Querying the API

To see the API in action, we can begin by making a simple query using cURL. Let's say we want to obtain all the available information of a specific device:

curl -u "admin:zenoss" -X POST -H "Content-Type: application/json" -d '{"action": "DeviceRouter", "method": "getInfo", "data": [{"uid": "/zport/dmd/Devices/Server/SSH/Linux/NovaHost/devices/$DEVICE_IP"}], "tid": 1}' http://$ZENOSS_HOST:8080/zport/dmd/device_router

The above request will return the information of the device in JSON format, assuming that we replace $ZENOSS_HOST with the IP address of our Zenoss server, and $DEVICE_IP with the IP address of the device we want to query.

This is all fine if all we need is querying the API and nothing else. But what if we really want to know how the API back-end works? How the endpoints are created? What are routers?

This is what we are gonna learn next.

Using RRDTool in Zenoss

Tue, 22 Nov 2016 00:00:00 +0000

rrdtool is an awesome high performance data logging and graphing tool for time series data. Zenoss Core uses RRDTool to collect, monitor, and graph peformance data for devices.

However Zenoss Core comes with built-in helper wrappers around RRDTool that makes using it within Zenoss much easier. These source code files can be found in $ZENHOME/Products/ZenRRD/.

RRDUtil

Located in $ZENHOME/Products/ZenRRD/RRDUtil.py, this Python module contains many wrapper methods around the rrdtool library. These methods can help us write to, create new, and read .rrd files using Python.

We can easily import this module into our code, using the following import statement:

from Products.ZenRRD.RRDUtil import RRDUtil

IPMI & Baseboard Management Controllers

Fri, 18 Nov 2016 00:00:00 +0000

An Intelligent Platform Management Interface (IPMI) is a set of interfaces that allow system administrators to manage and monitor systems, independently of the system's CPU, BIOS, and operating system.

IPMI can also help manage computers that are unresponsive or powered off, by using a network connection to the hardware, rather than to a operating system or login shell.

More specifically, IPMI can help us monitor, obtain data, and perform tasks such as:

System temperature
Voltages
Fans
Power supplies
Chassis intrusion
Perform recovery procedures
System rebooting
System powering

An IPMI sub-system consists of a main controller, called the baseboard management controller (BMC) and other management controllers distributed among different system modules that are referred to as satellite controllers. The satellite controllers within the same chassis connect to the BMC via another system interface (not covered in this post) called Intelligent Platform Management Bus/Bridge (IPMB)

Custom Markdown in Middleman

Thu, 17 Nov 2016 00:00:00 +0000

Middleman is an excellent static site generator written in Ruby. It is extremely practical for blogging as well, and it's the framework I am currently using for this blog.

Unfortunately Middleman's documentation is a bit unorganized and incomplete. I often find myself browsing through the source code to understand how to use or implement some custom features.

For example, using redcarpet I wanted to be able to create some re-usable “markdown components” that I could use in my blog posts. Something like bootstrap alerts, so that I could re-use them without having to add HTML or any code to my markdown.

And this post I am going to explain how to integrate this functionality into your blog using Middleman.

Creating a Custom Renderer

Middleman comes built in with a custom Redcarpet renderer which Middleman uses to parse the links and images (and possibly other things) in your blog's markdown. To be able to implement our custom markdown we need to create our own custom renderer, which should inherit from Redcarpet::Render::HTML. However, the key thing here is to actually inherit directly from Middleman's renderer (which of course, inherits from Redcarpet::Render::HTML).

To begin, we will create a lib (if it doesn't exist) under our blog's directory and create a file named markdown_helper.rb inside it. The initial contents of this helper will be:

module MarkdownHelper

  require 'middleman-core/renderers/redcarpet'

end

The required file above is what will allow us to inherit from Middleman's renderer.

Big O Notation - Omega & Theta

Wed, 16 Nov 2016 00:00:00 +0000

In a previous post I went over the basic and most common Big O notations along with respective common algorithms, and finally adding some Ruby benchmarks for each.

However I did not cover some other very important concepts in Big O notation: Omega and Theta.

Understanding Omega

The Omega symbol (Ω) represents a certain algorithm's best-case asymptotic complexity. For example, let's take again a look at the binary search algorithm:

Previously we determined that the Big O notation for this algorithm is O(log n). This represent the worst asymptotic complexity that the algorithm can possibly have.

Ruby Benchmarking & Big O Notation

Tue, 15 Nov 2016 00:00:00 +0000

The Ruby Benchmark library helps us to measure how long it takes for a block of code to run. Similar to when you subtract starting time and ending time in other languages such as Java.

I was doing some Big O Notation reviewing with Ruby the other day and did some common algorithm implementations for each of the most common Big O notations, just for fun.

Using Benchmark

To start benchmarking our code, we simply need to require benchmark and use the methods provided by the Benchmark module. For me, the most practical way is to use Benchmark#bm method, where we pass a block of code we want to measure:

require 'benchmark'

Benchmark.bm do |x|
  x.report do
    # our code here
  end

end

Inside the block we are using the report method of the block's variable, which will basically do a separate benchmark for each report.

However, we can improve this even more by adding a label to report and a label width to bm:

require 'benchmark'

Benchmark.bm(20) do |x|
  x.report("my_algorithm") do
    # our code here
  end
end

Let's see it in action with some common algorithms.

Infinite Sequences in Ruby Using Enumerator

Mon, 14 Nov 2016 00:00:00 +0000

The Enumerator class in Ruby allows us to generate some very smooth and useful infinite sequences. For example, in some Project Euler problems it is required to generate sequences without a specific limit. A good example of this is the Highly Divisible Triangular Number problem:

The sequence of triangle numbers is generated by adding the natural numbers. So the 7th triangle number would be 1 + 2 + 3 + 4 + 5 + 6 + 7 = 28. The first ten terms would be:

1, 3, 6, 10, 15, 21, 28, 36, 45, 55, ...

Let us list the factors of the first seven triangle numbers:

     1: 1
     3: 1,3
     6: 1,2,3,6
    10: 1,2,5,10
    15: 1,3,5,15
    21: 1,3,7,21
    28: 1,2,4,7,14,28

We can see that 28 is the first triangle number to have over five divisors.

What is the value of the first triangle number to have over five hundred divisors?

So first we would want to be able to generate a (infinite) sequence of triangular numbers that we can use to do many programming operations, such as iterating over the sequence.

Modifying the Zenoss Infrastructure Grid

Thu, 10 Nov 2016 00:00:00 +0000

Continuing from my previous post, where I explained how to modify the device detail bar from a ZenPack using ExtJS, in this post I will explain how we can modify the device list grid shown in the infrastructure page.

The Infrastructure Grid

This is the table in the infrastructure page that shows all devices being monitored by Zenoss. Default columns include device name, device class, IP address, production state, and events, as shown in the image below:

However it would be nice to also include the power status we added to the device detail bar in the previous post, maybe even add the ping status as well (why Zenoss doesn't do this by default is beyond me).

DevicePanels.js

The source code for the grid is found in $ZENHOME/Products/ZenUI/browser/resources/js/zenoss/DevicePanels.js. A quick glance at it and you will quickly find the definitions of the columns I mentioned earlier, defined in an array called deviceColumns.

Modifying the Zenoss Device Detail Bar

Tue, 08 Nov 2016 00:00:00 +0000

The Zenoss Core ExtJS graphic user interface is divided into many different components. In this post I will go over on how we can modify the device detail bar to display additional custom information for devices, by creating a custom ZenPack.

Zenoss Core 4 uses ExtJS 4 JavaScript framework to manage all the user interface components. These components can be found in $ZENHOME/Products/ZenUI3/browser/resources/js/zenoss, and as you would expect, the device detail bar component is also located there.

The Device Detail Bar

This is the detail bar located on the device view page which shows the device's icon, name, events, status, production state, and priority. The JavaScript source code for this component can be found in DeviceDetailBar.js

However since we are going to extend this component through a ZenPack, we will not modify that source. Instead, we will add new code to use ExtJS to add our custom data displays.

Extending The Component, Through ZenPacks

Let's assume we are starting with a freshly created ZenPack, which adds a new power_status integer field to certain devices, according to its zenpack.yaml file:

# ...

classes:
  CustomDevice:
    base: [zenpacklib.Device]
    label: Custom Device
    properties:
      power_status:
        type: boolean

What we want is to display this value in the device's detail bar, similar to the device's ping status display.

Creating Zenoss ZenPack Daemons

Mon, 07 Nov 2016 00:00:00 +0000

ZenPacks are powerful custom add-ons that can help us extend Zenoss's functionality. In this post I will go over on how to create a ZenPack that adds a custom daemon to the existing Zenoss daemons and runs on a configured cycle time to perform custom tasks. We will achieve this by creating the ZenPack using zenpacklib, but it is also possible to create it from the Zenoss user interface.

About zenpacklib

zenpacklib is a Python library developed by the Zenoss team to facilitate the process of creating ZenPacks, specially ZenPacks that deal with modeling and monitoring devices and components. Most of the newer ZenPacks that are being released are now being built with zenpacklib.

We can obtain zenpacklib by running the following commands:

wget http://zenpacklib.zenoss.com/zenpacklib.py
chmod 755 zenpacklib.py

This will download the zenpacklib Python library and give it executable permissions.

Creating the ZenPack

Using the zenpacklib file we just downloaded, we proceed to create a new fresh ZenPack:

./zenpacklib.py create ZenPacks.<your_namespace>.<zenpack_name>

This will create the ZenPack base directory with the necessary base files.

Now we go into this directory to begin.

cd ZenPacks.<your_namespace>.<zenpack_name>

Creating the ZenPack Daemon

Our custom daemon declaration will be located in a directory named daemons, inside our ZenPack directory:

mkdir ZenPacks.<your_namespace>.<zenpack_name>/<your_namespace>/<zenpack_name>/daemons

Here we will create a new Bashscript file with the name of our daemon. In this case we will name it mydaemon:

#! /usr/bin/env bash

DAEMON_NAME="mydaemon"

. $ZENHOME/bin/zenfunctions

MYPATH=`python -c "import os.path; print os.path.realpath('$0')"` THISDIR=`dirname $MYPATH` PRGHOME=`dirname $THISDIR` PRGNAME=$DAEMON_NAME.py CFGFILE=$CFGDIR/$DAEMON_NAME.conf

generic "$@"

Once the ZenPack is installed, files under this daemons directory will become executable (chmod 0755), a symlink to the file will be created in $ZENHOME/bin, and a configuration file will be generated in $ZENHOME/etc/<daemon_name>.conf

NOTE: If you created your ZenPack using the Zenoss user interface, the daemons directory will also be automatically created, and will contain an example daemon file named zenexample with code similar to the one above. In this case you should simply replace the necessary values.

Monitoring CPU Utilization in Zenoss

Fri, 04 Nov 2016 00:00:00 +0000

While taking a look at the CPU Utilization graphs offered Zenoss Core's Linux Monitor ZenPack (v1.2.1), I noticed that the percentage values for Idle were ridiculously high:

This made sense since this particular device contains 16 cores. However, this then means that the monitoring template isn't really taking this into consideration, and instead just spits out the total value from all cores.

Zenoss Monitoring Template Data Points

Thu, 03 Nov 2016 00:00:00 +0000

Content from this post is mostly obtained from Zenoss Core Administration guide.

In Zenoss Core monitoring templates, data sources can return data for one or more performance metrics. Each metric retrieved by a data source is represented by a data point.

When creating a data point, there are some important fields that we can define for our data point:

Name: Displays the name you entered in the Add a New DataPoint dialog.
RRD Type: Specify the RRD data source type to use for storing data for this data point. (Zenoss Core uses RRDTool to store performance data.) Available options are:
- COUNTER: Saves the rate of change of the value over a step period. This assumes that the value is always increasing (the difference between the current and the previous value is greater than 0). Traffic counters on a router are an ideal candidate for using COUNTER.
- GAUGE: Does not save the rate of change, but saves the actual value. There are no divisions or calculations. To see memory consumption in a server, for example, you might want to select this value.
- DERIVE: Same as COUNTER, but additionally allows negative values. If you want to see the rate of change in free disk space on your server, for example, then you might want to select this value.
- ABSOLUTE: Saves the rate of change, but assumes that the previous value is set to 0. The difference between the current and the previous value is always equal to the current value. Thus, ABSOLUTE stores the current value, divided by the step interval.
Create Command: Enter an RRD expression used to create the database for this data point. If you do not enter a value, then the system uses a default applicable to most situations. For details about the rrdcreate command, go here.
RRD Minimum: Enter a value. Any value received that is less than this number is ignored.
RRD Maximum: Enter a value. Any value received that is greater than this number is ignored.

Data Point Aliases

Performance reports pull information from various data points that represent a metric. The report itself knows which data points it requires, and which modifications are needed, if any, to put the data in its proper units and format.

The addition of a data point requires changing the report.

To allow for more flexibility in changes, some reports use data point aliases. Data point aliases group data points so they can be more easily used for reporting. In addition, if the data points return data in different units, then the plugin can normalize that data into a common unit.

Triggering Commands From Events in Zenoss

Mon, 31 Oct 2016 00:00:00 +0000

In Zenoss Core 4, we can configure some email notifications that produce and send emails based on conditions defined in a trigger, when an event is created.

Additionally, Zenoss Core 4 also provides command type notifications that allow us to execute commands in the Zenoss machine when the trigger criterias are met.

For example, let's say we want to execute a script everytime we ping a device and get no response. This scenario would involve the following:

The device is suddenly down.
Zenoss pings the device and gets no response.
The device status is then changed to DOWN.
An Event with an event class of /Status/Ping and a severity of Critical is created.

With this information, we can create a trigger that will represent this exact scenario. Moreover, we can configure a command notification to be executed when this trigger is fired.

Configuring a Trigger

Navigate to Events > Triggers
Create a new trigger.
Add the rules that represent the scenario mentioned before. In this case, all of these rules must apply:

=> Because we want to make sure the device is truly DOWN, and has been so for quite a while, we also want to add a condition that checks that the event count is greater than a certain number, only then the trigger will be fired, and consequently, the command.

Keystone Interface in Juju Charms

Fri, 28 Oct 2016 00:00:00 +0000

Juju Charms interfaces make possible the interaction and exchange of data between services deployed by Juju. In this post I will explain how we can use the Keystone interface to make our custom charm service comunicate with the Keystone service within a Juju environment.

In the end, what we really want to achieve, is to link both services so that they can exchange data, using a Juju command:

juju add-relation keystone myservice

Juju Relations

Juju relations operate behind the concept that one service provides something, while another service requires it, and the interaction between the services is done through an interface. Since we want to interact with Keystone, this means that we need to use a suitable keystone interface that allows us to get what we need, the keystone Identity data. For this, we use the keystone-admin interface.

With this in mind, we configure our charm to require this relation and interface in metadata.yaml:

# ...
requires:
  identity-admin:
    interface: keystone-admin

Zenoss Renderers

Thu, 27 Oct 2016 00:00:00 +0000

Zenoss makes use of some very interesting graphical components called renderers. These are used to manipulate the way data is shown in the Zenoss user interface.

For example, a value of total bytes used could be 6080626688 in bytes, which is a very high number and doesn't really convey much meaning. However, we can use a built-in Zenoss renderer called bytesString which will convert this value in bytes to the closest representation:

As we can see, this is a much better and meaningful way of displaying the data.

The built-in Zenoss renderers and source code can be found in $ZENHOME/Products/ZenUI3/browser/resources/js/zenoss/Renderers.js

They are a list of registered Javascript functions that can be assigned in our YAML definitions file by adding the renderer property. For example:

HardDisk:
  base: [zenpacklib.Component]
  label: Hard Disk
  properties:
     location:
        label: Location

     capacity:
        label: Capacity

     raid_name:
        label: Raid Name

     raid_level:
        label: Raid Level

     status:
        label: Status
        renderer: Zenoss.render.pingStatus

Zenoss.render.pingStatus is one of the default renderers that come built-in within Zenoss Core 4, similar to Zenoss.render.bytesString.

Changing Zenoss Dashboard Portlets Using CLI

Wed, 26 Oct 2016 00:00:00 +0000

The Zenoss ZenPacks.zenoss.Dashboard ZenPack adds a very nice and new dashboard to our Zenoss Core 4 deployment, replacing the old default dashboard that comes with the installation.

Configuring a local apt repository

Mon, 17 Oct 2016 00:00:00 +0000

Setting up a local apt repository can allow you to use apt-get install command to install your own packages. This is a better approach than using dpkg -i because apt will fetch and install the required dependencies (which should also be located in your local repository) in an offline fashion.

Add Local Repository to Sources

First we need to add our local repository to apt's list of source repositories, this can be found in /etc/apt/sources.list. To add our local repository, we add the following line at the top of the file:

/path/to/local-deb-repo/ ./

Where local-deb-repo is the name of our local repository directory.

Add Debian Packages

Now we create local repository's directory just like we specified it in the previous step:

mkdir /path/to/local-deb-repo/

Once we've done that, we can proceed to put the Debian package files (.deb) and their dependencies inside the directory.

Managing Debian Package Dependencies

Fri, 14 Oct 2016 00:00:00 +0000

In a previous post, I talked about building a simple Debian package in Ubuntu, however I did not go into details on how to manage the dependencies that your Debian package might require. In this post I will cover exactly that.

dpkg and Dependencies

Previously we installed our package using the dpkg -i command. However, the problem with dpkg is that by itself, it is not capable of managing repositories. Therefore, higher level tools (such as apt-get) are required to fetch dependencies from repositories.

dkpg is only the core tool that installs/removes/configures packages, taking care of dependencies and other factors. apt-get and aptitude are tools that manage repositories, download data from them, and use dkpg to install/remove packages from them. This means, that apt-get and aptitude can resolve dependencies and get required packages from repository, but dpkg cannot, because it knows nothing about repositories.

Creating Python Packages

Thu, 13 Oct 2016 00:00:00 +0000

For a project involving Zenoss Core 4 and a HaaS solution maintained by another team, we needed a module to interact with the Zenoss JSON API to get the list of events for specific devices. Browsing around I found this python-zenoss module to work with the Zenoss JSON API. However I was experiencing some issues when installing it, so I decided to create my own Python package to provide a different way to interact with the Zenoss JSON API according to our specific needs.

The package would be installed locally using pip, and all its functionality should be easily accessible using from and import commands in Python.

To start working in our package we will create a base directory. The directory's name will resemble our package's name. For naming, we should following these guidelines:

All lowercase
Underscore-separated or no word separators at all (don’t use hyphens)

The package contents are the following:

my_api/
├── __init__.py
├── api.py
└── setup.py

Debian Packaging in Ubuntu

Tue, 11 Oct 2016 00:00:00 +0000

Debian packaging is a nice way to organize our software so that it can be installed and uninstalled with more ease. In this post we will go through the basics of creating a simple Debian package in Ubuntu 14.04 LTS.

Package Source & Structure

We will first create our working space for our package along with the necessary basic structure needed. For this example, we will create a directory for our package source, using the package title and 3 digit versioning in the directory's title:

mkdir mypackage-1.0.0

Inside our package, we will need a directory where we will hold the control files that give the Debian package the desired behavior:

cd mypackage-1.0.0
mkdir DEBIAN
touch DEBIAN/control

This control file will describe our package to the Debian package manager (dpkg). Inside, we can specify information about the package such as version, architecture, dependencies, maintainer, etc, as shown below:

Package: mypackage
Version: 1.0-0
Section: base
Priority: optional
Architecture: all
Depends: python2.7
Maintainer: Andres Alvarez <myemail@gmail.com>
Description: A new and improved C++ Compiler

With this basic setup, we now have a basic working package that can be handled by the Debian package manager, albeit, with no contents.