DevOps toolchain from GitLab to Docker Hub for Container Build

The default builder on Docker Hub has short timeout when building an image. This can be a problem when an image is large or network is slow. Build can be moved to a GitLab runner to solve the problem. As the CI/CD pipeline is relative simple, there is no need for a dashboard or library. The toolchain for Docker build pipelines consists of the following tools:

1. IntelliJ IDEA CE
2. GitLab
3. GitHub
4. Docker Hub

The integration of tools are the same as for documentation and ECP above. The overall workflow is show as the diagram below:

The toolchain can pull Dockerfile from a GitHub repository. It builds the Docker image on a GitLab runner. It pushes the image to Docker Hub when a build is done. This is useful when an image is too big for Docker Hub to build it.

The easiest way to integrate with an external repository for CI/CD is to click CI/CD for external repo when creating a new project. Alternatively, mirroring repositories at Settings > Repository.

Building Docker image

There is only one file .gitlab-ci.yml needed to create a toolchain to build a docker image with GitLab at EBI, instead of DockerHub. It runs the build engine on the GitLab runners for EBI. It is a production-grade Kubernetes cluster managed by the cloud team. There is no extra configuration needed.

The .gitlab-ci.yml file can be edited with online editor in GitLab or any text editor. Developers typically use an IDE such as IntelliJ integrated with GitLab. The file contains the following major sections.

Add official Docker image to a runner with a Kubernetes cluster and require docker-in-docker service:

# Official docker image.
image: docker:latest

services:
  - docker:dind

Log into Docker Hub and install git. Leave CI_REGISTRY blank for Docker Hub:

before_script:
  - docker login -u "${CI_REGISTRY_USER}" -p "${CI_REGISTRY_PASSWORD}" ${CI_REGISTRY}
  - apk add git
  - mkdir -p ${ARTIFACT_DIR}

Clone the repository for Docker build onto the runner. Build, tag and push a new Docker image:

docker.image:
  stage: build
  variables:
    CI_GIT_REPO_NAME: Metagenomics-Assembly
    CI_GIT_URL: https://github.com/Medalibi/${CI_GIT_REPO_NAME}.git
    CI_REGISTRY_IMAGE: davidyuyuan/metagenomics
    CI_COMMIT_REF_SLUG: assembly
    CI_SOURCE_IMAGE: metagenomics
    CI_DOCKER_FILE: ${CI_PROJECT_DIR}/${CI_GIT_REPO_NAME}/Dockerfile
  script:
    - git clone ${CI_GIT_URL}
    - cd ${CI_GIT_REPO_NAME}
    - docker build -f ${CI_DOCKER_FILE} -t ${CI_SOURCE_IMAGE} . | tee ${ARTIFACT_DIR}/build.log
    - docker tag ${CI_SOURCE_IMAGE} ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG}
    - docker push ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG} | tee ${ARTIFACT_DIR}/push.log
  only:
    - master
  artifacts:
    name: "$CI_COMMIT_REF_NAME-$CI_JOB_STAGE-$CI_JOB_NAME"
    paths:
      - ${ARTIFACT_DIR}

It is a good practise to tee build log and push log onto the ARTIFACT_DIR. They can then be accessed and reviewed under the pipeline or job generating thme.

Appendix: Command line instructions

Git global setup:

git config --global user.name "User Name"
git config --global user.email "username@ebi.ac.uk"

Create a new repository:

git clone git@gitlab.ebi.ac.uk:davidyuan/adv-k8s.git
cd adv-k8s
touch README.md
git add README.md
git commit -m "add README"
git push -u origin master

Existing folder:

cd existing_folder
git init
git remote add origin git@gitlab.ebi.ac.uk:davidyuan/adv-k8s.git
git add .
git commit -m "Initial commit"
git push -u origin master

Existing Git repository:

cd existing_repo
git remote rename origin old-origin
git remote add origin git@gitlab.ebi.ac.uk:davidyuan/adv-k8s.git
git push -u origin --all
git push -u origin --tags