Notes on setting up gitlab CI31 Mar 2016
How Gitlab CI works
Gitlab monitors your repo and every time a change is made which requires a build it looks for a “runner” to run the tests. Runners run on a separate machine and register themselves with the Gitlab instance, letting it know when they are available to run jobs. There are two kinds of runner, specific runners and shared runners.
Shared runners can be used by any project in the gitlab instance whereas specific runners have to be registered (on the machine where the runner is running) for each project you wish to run them with. You can read more about the runner types here
We’re using gitlab.com, which has a number of shared runners. However, it also has this warning:
So, we need a specific runner.
This project has both frontend and backend test. The backend is django on Python 3.5 and the frontend is bundled by webpack but the tests run using Mocha on nodejs. So, we need an environment which has both Python 3.5 and Node available.
Gitlab allows you to specify a docker image to run the build inside (provided docker is installed on the runner machine). I like the idea of using a docker image because it makes the build environment easily reproducible. The actual environment we need is easy to achieve, the Dockerfile looks like this:
FROM python:3.5 RUN curl -sL https://deb.nodesource.com/setup_5.x | bash - RUN apt-get install --yes nodejs RUN apt-get install --yes build-essential
I’ve pushed this to
alexjg/node-python3 on Docker Hub.
Spinning up a runner and connecting to Gitlab
We need to actually run our CI runner somewhere, I chose Digital Ocean because it’s cheap and easy. Once you’ve got a Droplet running Ubuntu 15.10 we need to do three things.
- Install Docker
- Install the gitlab runner
- Register the gitlab runner with our project
For the third step, run
sudo gitlab-ci-multi-runner register. This will prompt you for your projects coordinator URL and registration token, which can be found in
project -> edit project -> runners. It will also prompt you for the docker image to use, so in this case I entered
Once this step is complete you should be able to go to the gitlab runners page and the runner will be listed in
available specific runners.
Writing a .gitlab-ci.yml
To actually tun tests we need to tell gitlab CI what to do. Jobs are defined in the
.gitlab-ci.yml file, which looks like this:
image: alexjg/node-python3 services: - postgres variables: DATABASE_URL: postgres://postgres:postgres@postgres/postgres frontend-test: script: - npm install - npm test backend-test: script: - pip install -r requirements.txt - pip install -r requirements-dev.txt - python manage.py test
There are a few things going on here.
Set the image
This tells Gitlab CI to run the test scripts in a container built from this image.
services: - postgres
This tells Gitlab CI to run the
postgres image in the same network as the test container.
variables: - DATABASE_URL: postgres://postgres:postgres@postgres/postgres
This sets the environment variable
DATABASE_URL for the container the tests run in. This variable is needed because the Django application is set up to get it’s database config from the environment variable in line with the 12 factor app principles. Note that the
postgres container we told Gitlab to run as a service is available under the hostname
postgres, the password, user, and default database
postgres are all set by the official postgres image.
frontend-test: script: - npm install - npm test
This defines a job called
frontend-test, it will install the npm dependencies and then run the tests.
backend-test: script: - pip install -r requirements.txt - pip install -r requirements-dev.txt - python manage.py test
Similarly this defines a job to run for the backend tests.
At this point pushing to the repo will run both builds on the runner and mark the branch as “passed” or “failed” depending on the results. This is what I set out to do and I’m pretty happy with it, however, it turns out to be very easy to add deployment.
This particular app is running on Heroku, deployment to Heroku is ludicrously easy with Gitlab CI. The additional entry in
.gitlab-ci.yml looks like this:
staging: type: deploy script: - apt-get update -qy - apt-get install -y rubygems - gem install dpl - dpl --provider=heroku --app=<app-name> --api-key=$HEROKU_STAGING_API_KEY only: - dev
By marking this job as deploy we ensure that it will only run after the test jobs have passed (jobs are
type: test by default). This job installs rubygems and then install
dpl - which is the library Travis CI uses to deploy - to deploy the app to heroku. The
only key restricts this to the
dev branch. The
$HEROKU_STAGING_API_KEY is a variable set in the project settings on Gitlab, which means it doesn’t have to be kept in version management anywhere.
At the end of all this we have a continuous deployment system which runs tests on each branch and on a successful (tests passing) merge to
dev deploys the code to our staging environment. We’re using Heroku’s pipelines feature for this application so once we’ve kicked the tyres for a while it’s very easy for someone to promote the application to production with the click of a button.