I’ve used Grafana for several years. Ever since the first time I used it, I have wanted to sit down and write a server which would provide metrics to it through the Simple JSON datasource plugin. I’m happy to announce that I have finally gotten around to doing just that. The tool is called PyGraf and will be opened sourced in the very near future!
In the meantime, I’d like to share my findings as there’s not a lot of documentation on the topic.
The Simple JSON docs tells us that our backend must support the following endpoints:
/ should return 200 ok. Used for “Test connection” on the datasource config page.
/search used by the find metric options on the query tab in panels.
/query should return metrics based on input.
/annotations should return annotations.
That’s great to know, but how do we go about testing them? Let’s run through that now.
In the previous post, Choosing a configuration method, we looked at the three ways in which a Kubernetes cluster can be configured. In this post we’ll go through the components which make up the cluster.
Kubernetes clusters have two types of servers: masters and nodes. The masters provide the control plane for the cluster while the nodes take care of the workloads. As you probably guessed, having one or more of either provides high availability.
The components documentation gives us a great rundown on the components that make up each of these two nodes, though it might be a little confusing for those who are just starting out. In this post we’ll look through each component’s documentation while adding additional information as required. Furthermore, to help really get the point across, we’ll build a diagram to show where each component resides.
virtualenv, as the name suggests, creates virtual Python environments. If you’re familiar with server virtualisation, virtualenv acts in a similar fashion to virtual machines in that the environments share the same physical hardware, but they’re completely separated from one another.
virtualenv is very useful for when you’re working on multiple projects or using applications that require different versions of the same module. In this post I’ll demonstrate how to install and usage of virtualenv on Windows, though the process is similar for Linux too.
If you haven’t already, install virtualenv like so:
C:\>pip install virtualenv
Downloading virtualenv-15.1.0-py2.py3-none-any.whl (1.8MB)
100% |################################| 1.8MB 461kB/s
Installing collected packages: virtualenv
Successfully installed virtualenv-15.1.0
GitLab Runner is used as part of GitLab CI/CD pipelines. On a side note, it also supports GitHub and BitBucket too! But I digress…
In this post we’ll cover how to install, configure and register Runner.
So many choices!
Runner can be installed on various operating systems/tools (Linux, Windows, Mac, Kubernetes, Docker), to name a few. If you’re interested, a full list can be found in the documentation. For the purpose of this post, we’ll use the Dockerised version on a CoreOS instance which is running in AWS. (Boy, that was a mouthful!)
URL & Token
Before we can spin up our Runner, we’ll first need to retrieve our GitLab URL and registration token. We can obtain both of these by:
- Selecting a repository
- Clicking “Settings” –> “CI/CD”
- Navigating to the “Specific Runners” section
In the previous post, What is it?, we gained an understanding of Kubernetes and container orchestrations in general. In this post we’ll cover the three ways in which Kubernetes can be configured.
Kubernetes’ configuration is simply a bunch of Kubernetes objects. Let’s take a quick look at what these objects are, and what they’re used for. The following quotes are from the Kubernetes object documentation:
Kubernetes Objects are persistent entities in the Kubernetes system. Kubernetes uses these entities to represent the state of your cluster. Specifically, they can describe:
- What containerized applications are running (and on which nodes)
- The resources available to those applications
- The policies around how those applications behave, such as restart policies, upgrades, and fault-tolerance
If you’ve looked into containers before, you’ve likely heard the name Kubernetes. This post will tackle what it is at a high level, while subsequent posts will delve deeper into the details.
Let’s kick off this post with a couple of quotes from the Kubernetes website:
Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.
It groups containers that make up an application into logical units for easy management and discovery. Kubernetes builds upon 15 years of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community.
There’s a lot of great information in this quote, so let’s go ahead and take a closer look.
One server per application
In the not too distant past, it was normal to have a one to one relationship between applications and servers. For example, your mail and web applications would reside on two separate, physical servers.
This approach was greatly inefficient and costly too. Each server would be purchased with years of predicted growth in mind. This resulted in racks full of overly powerful and underutilised servers which would drive up cooling, power and data centre space costs.
Furthermore, applications would often fall short of their forecasted growth, leaving businesses in a stalemate. They would want to move their applications to smaller servers to save on costs, but they can’t guarantee they won’t break their applications in the process. If they do break, then that’ll end up costing the business too. It really was a lose-lose situation.
As you’ve probably guessed by Docker posts, I’m a huge fan of containerisation. Therefore instead of installing Prometheus on a host, let’s instead spin it up in a container. As described on the Prometheus website, we can accomplish this by issuing only a single command:
docker run -p 9090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
Let’s break down each component of this command to make sure we fully understand what it is doing:
docker run: Spin up a container
-p 9090:9090: Bind a port on our Docker host to a container port. This enables devices outside of the Docker host to reach the container on port 9090
-v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml: Binds the
/tmp/prometheus.yml file stored on the Docker host to
/etc/prometheus/prometheus.yml inside of the container
<user_account>/<container_name> that we want to use
If you’ve used Grafana, or even heard of it, chances are you’ve also heard of InfluxDB and Prometheus too. As I haven’t touched on the latter yet, I figured now is a good time to start. In case you haven’t heard of some, or all of these applications, let’s start off with a quick description on what they can do for us.
Note: You might also want to have a read of the My Monitoring Journey: Cacti, Graphite, Grafana & Chronograf post too.
Grafana is a frontend web app that is used to create beautiful dashboards. It does this by retrieving metrics which are stored on backend database servers such as InfluxDB, Prometheus MySQL, PostgreSQL and Graphite (to name just a few). It then uses metrics to create graphs which are displayed on the aforementioned dashboards.
In the previous post we may have started running before we could walk. In this post we’ll first take a few steps back to make sure we cover the basics before diving deeper.
Building an image
When we examined the
python:3.6.3-alpine3.6 image’s dockerfile, we saw the components which are used to create a Docker image:
- A single
OK, so we already know that the
FROM alpine3.6 command means that the Python image is going to use the Apline 3.6 as its Linux operating system.
ENV, as its name suggests, is used to specify environment variables for the image. That leaves us with