Notes on setting up gitlab CI

At Maker Labs we’ve been working on a Django project with a javascript frontend using React and Redux. Both projects have a significant number of tests and I wanted to have a CI setup running tests on each branch and updating the branch status. As it turns out it was so easy to add deployment that I ended up setting up continuous deployment too.

How Gitlab CI works

Gitlab monitors your repo and every time a change is made which requires a build it looks for a “runner” to run the tests. Runners run on a separate machine and register themselves with the Gitlab instance, letting it know when they are available to run jobs. There are two kinds of runner, specific runners and shared runners.

Shared runners can be used by any project in the gitlab instance whereas specific runners have to be registered (on the machine where the runner is running) for each project you wish to run them with. You can read more about the runner types here

We’re using gitlab.com, which has a number of shared runners. However, it also has this warning:

So, we need a specific runner.

Runner Requirements

This project has both frontend and backend test. The backend is django on Python 3.5 and the frontend is bundled by webpack but the tests run using Mocha on nodejs. So, we need an environment which has both Python 3.5 and Node available.

Gitlab allows you to specify a docker image to run the build inside (provided docker is installed on the runner machine). I like the idea of using a docker image because it makes the build environment easily reproducible. The actual environment we need is easy to achieve, the Dockerfile looks like this:

FROM python:3.5
RUN curl -sL https://deb.nodesource.com/setup_5.x | bash -
RUN apt-get install --yes nodejs
RUN apt-get install --yes build-essential

I’ve pushed this to alexjg/node-python3 on Docker Hub.

Spinning up a runner and connecting to Gitlab

We need to actually run our CI runner somewhere, I chose Digital Ocean because it’s cheap and easy. Once you’ve got a Droplet running Ubuntu 15.10 we need to do three things.

1. Install Docker
2. Install the gitlab runner
3. Register the gitlab runner with our project

1 is perfectly documented on the docker website. 2 is straightforward enough, instructions are here.

For the third step, run sudo gitlab-ci-multi-runner register. This will prompt you for your projects coordinator URL and registration token, which can be found in project -> edit project -> runners. It will also prompt you for the docker image to use, so in this case I entered alexjg/node-python3.

Once this step is complete you should be able to go to the gitlab runners page and the runner will be listed in available specific runners.

Writing a .gitlab-ci.yml

To actually tun tests we need to tell gitlab CI what to do. Jobs are defined in the .gitlab-ci.yml file, which looks like this:

image: alexjg/node-python3

services:
  - postgres

variables:
  DATABASE_URL: postgres://postgres:postgres@postgres/postgres

frontend-test:
  script:
    - npm install
    - npm test

backend-test:
  script:
    - pip install -r requirements.txt
    - pip install -r requirements-dev.txt
    - python manage.py test

There are a few things going on here.

Set the image

image: alexjg/node-python3

This tells Gitlab CI to run the test scripts in a container built from this image.

Define Services

services:
  - postgres

This tells Gitlab CI to run the postgres image in the same network as the test container.

Define Variables

variables:
  - DATABASE_URL: postgres://postgres:postgres@postgres/postgres

This sets the environment variable DATABASE_URL for the container the tests run in. This variable is needed because the Django application is set up to get it’s database config from the environment variable in line with the 12 factor app principles. Note that the postgres container we told Gitlab to run as a service is available under the hostname postgres, the password, user, and default database postgres are all set by the official postgres image.

Define Jobs

frontend-test:
  script:
    - npm install
    - npm test

This defines a job called frontend-test, it will install the npm dependencies and then run the tests.

backend-test:
  script:
    - pip install -r requirements.txt
    - pip install -r requirements-dev.txt
    - python manage.py test

Similarly this defines a job to run for the backend tests.

At this point pushing to the repo will run both builds on the runner and mark the branch as “passed” or “failed” depending on the results. This is what I set out to do and I’m pretty happy with it, however, it turns out to be very easy to add deployment.

Deployment

This particular app is running on Heroku, deployment to Heroku is ludicrously easy with Gitlab CI. The additional entry in .gitlab-ci.yml looks like this:

staging:
  type: deploy
  script:
    - apt-get update -qy
    - apt-get install -y rubygems
    - gem install dpl
    - dpl --provider=heroku --app=<app-name> --api-key=$HEROKU_STAGING_API_KEY
  only:
    - dev

By marking this job as deploy we ensure that it will only run after the test jobs have passed (jobs are type: test by default). This job installs rubygems and then install dpl - which is the library Travis CI uses to deploy - to deploy the app to heroku. The only key restricts this to the dev branch. The $HEROKU_STAGING_API_KEY is a variable set in the project settings on Gitlab, which means it doesn’t have to be kept in version management anywhere.

Conclusion

At the end of all this we have a continuous deployment system which runs tests on each branch and on a successful (tests passing) merge to dev deploys the code to our staging environment. We’re using Heroku’s pipelines feature for this application so once we’ve kicked the tyres for a while it’s very easy for someone to promote the application to production with the click of a button.