Exercise 6: Change the order of the pipeline steps¶

Objective¶

Learn how to change the order of pipeline stages, and how to build a parallel pipeline

Change the order of stages¶

Build stages have dependencies, which give them access to the artifacts from other stages, but the order is determined by the stages: declaration in the .gitlab-ci.yml file. After the last exercise, your stages: section will look something like this:

> cat .gitlab-ci.yml | grep --after-context=5 --before-context=1 stages

stages:
  - build
  - test
  - deploy
  - test-docker

If you simply swap the order of the test and deploy stages, commit, tag and push your code, you should see that the build pipeline runs in a different order, with the deploy (create and push the docker image) happening before the test (running the executable).

It’s up to you to decide if it makes sense to ‘deploy’ before testing, or after, this simply illustrates how to do it.

Sequential and parallel builds¶

In our example, testing the executable and testing the docker image could, in principle, happen in parallel. If the tests were long-running, and we have enough resources in our gitlab runners, this can save time. We can do this very simply by removing the test-docker stage from the stages: piece and changing the run-docker step to execute the test stage, instead of having it execute the test-docker stage.

In other words, there will be two steps, run and run-docker, both of which run the test stage. They will happen in the correct order, after the deploy stage, but they are eligible to run in parallel if there are sufficient resources. If there are not sufficient resources for the steps to run in parallel, they will simply happen sequentially, in a random order.

Make this change to your YAML file, commit it, tag it, and push it to the server. Take a look at the pipeline page and see how it shows the parallel steps. It should look like this:

../../../_images/gitlab-04-parallel-steps.png

Conclusion¶

You now know how to write pipelines which can run stages sequentially, or in parallel.
Each stage of a pipeline can have many steps that run that stage.
Each step in a stage must succeed, or the whole pipeline fails
Each step in a stage must complete before the next stage is free to start
Parallel stages can speed up testing, e.g. by running tests in many environments at the same time

Best Practices¶

Think carefully about what you are trying to achieve in your pipeline. Keep the steps small, and parallelise where it makes sense.

Cheat-sheet¶

In case you get stuck, this is what your .gitlab-ci.yml file should now look like. Don’t peek until you’ve had a go yourself though!

> cat .gitlab-i.yml
variables:
  DOCKER_TLS_CERTDIR: ""
  GIT_STRATEGY: clone
  REGISTRY_USER: tonywildish
  APPLICATION: tiny-test
  LATEST_IMAGE: $CI_REGISTRY/$REGISTRY_USER/$APPLICATION:latest
  RELEASE_IMAGE: $CI_REGISTRY/$REGISTRY_USER/$APPLICATION:$CI_BUILD_REF_NAME
  DOCKER_DRIVER: overlay

before_script:
  - echo "Starting..."
  - export DOCKER_IMAGE=$RELEASE_IMAGE
  - if [ "$CI_BUILD_REF_NAME" == "master" ]; then export DOCKER_IMAGE=$LATEST_IMAGE; fi
  - echo "Build docker image $DOCKER_IMAGE"

stages:
  - build
  - deploy
  - test

compile:
  stage: build
  image: gcc:6
  services:
    - docker:dind
  artifacts:
    name: "${CI_BUILD_NAME}_${CI_BUILD_REF_NAME}"
    untracked: true
    expire_in: 1 week
  script:
    - make

run:
  stage: test
  dependencies:
  - compile
  only:
    - tags
  script:
    - echo "Testing application. First, list the files here, to show we have the git repo + the artifacts from the build step"
    - ls -l
    - echo 'Now try running it'
    - ./hello
    - echo "If that failed you won't see this because you'll have died already"

install:
  stage: deploy
  image: docker:latest
  services:
  - docker:dind
  dependencies:
  - compile
  script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
    - echo Building $DOCKER_IMAGE
    - docker build -t $DOCKER_IMAGE .
    - echo Deploying $DOCKER_IMAGE
    - docker push $DOCKER_IMAGE

run-docker:
  stage: test
  image: docker:latest
  services:
  - docker:dind
  script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
    - docker run $DOCKER_IMAGE

after_script:
  - echo "Congratulations, this step succeeded"