CI/CD: Setting up GitLab Runners on AWS using CoreOS & Terraform

GitLab Runner is used as part of GitLab CI/CD pipelines. On a side note, it also supports GitHub and BitBucket too! But I digress… 

In this post we’ll cover how to install, configure and register Runner.

So many choices!

Runner can be installed on various operating systems/tools (Linux, Windows, Mac, Kubernetes, Docker), to name a few. If you’re interested, a full list can be found in the documentation. For the purpose of this post, we’ll use the Dockerised version on a CoreOS instance which is running in AWS. (Boy, that was a mouthful!)

URL & Token

Before we can spin up our Runner, we’ll first need to retrieve our GitLab URL and registration token. We can obtain both of these by:

  1. Selecting a repository
  2. Clicking “Settings” –> “CI/CD”
  3. Navigating to the “Specific Runners” section

Manual method

First, we’ll need to spin up a CoreOS AMI. Next,  we’ll need to install the Runner. While the documentation uses a Ubuntu image with Runner installed, we’ll use the Alpine version instead:

Looking at our container’s logs though, it seems that things didn’t go too well:

This is because the Runner is expecting to find its configuration on our host at  /srv/gitlab-runner/config.toml and /etc/gitlab-runner/config.toml on our container. We could put a configuration together ourselves by looking at the advanced configuration documentation, but that may be a little too advanced for newcomers. For now, let’s use the wizard instead.

We do this by:

  1. Logging into our Runner container
  2. Issuing the gitlab-runner register command
  3. Entering the “Specific Runners” data we retrieved earlier

Performing the above steps looks like this:

If we take a look at our logs now, we should see that the Runner has successfully started, as per the last line in the output below:

Looking at the /etc/gitlab-runner/config.toml file on our Runner container, as well as the /srv/gitlab-runner/config/config.toml file of our CoreOS instance, we see that the config was generated automatically for us:

And there we have it! Our Runner is good to go… But let’s not stop here. We live in a DevOps world, surely there’s a way we can automate this process!

Automated (Terraform) method

Automating the deployment of our Runner not only makes our lives easier, but it also enables us to take an Infrastructure as Code (IaC) approach to the deployment. As a result, we’ll have visibility of every revision of our Runner’s configuration. We’ll also be able to spin it up and down as required and be guaranteed that it will be exactly the same every time.

Below is a quick configuration I put together to demonstrate how this can be done using Terraform. (Note: This version of the code is very static.  I intend to re-write it in the near future to make it more flexible.)

The above code does the following:

  1. Spins up a CoreOS instance on AWS
  2. SSH’s into it
  3. Spins up a container using the gitlab/gitlab-runner:alpine-v10.6.0  image on the CoreOS instance and passes it some environment variables
  4. Issues the gitlab-runner register command on the container

Let’s now focus our attention on the 3rd step as it is the most important in regards to automating this process. Recall that when we ran through the process manually, we had to answer a series of questions provided by the wizard. However, this time we didn’t. Why is this?

It’s because we passed the REGISTER_NON_INTERACTIVE=true environment variable to our container. However, if we passed only this variable we’d receive an error error because the Runner would have no configuration. That is why we passed the other environment variables too.

Using environment variables enables us to avoid writing our own config.toml in much the same way as using the wizard did. In fact, the Runner also converts our environment variables into a configuration file, as it did with the wizard’s configuration too.

On the topic of environment variables – Earlier in this post it was mentioned that the config.toml  configuration parameters can be found in the advanced configuration section of the documentation. However, the corresponding variables are not. Therefore, I have posted them below for convenience.