DevOps toolchain from GitLab to Docker Hub for Container Build
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The default builder on Docker Hub has short timeout when building an image. This can be a problem when an image is large or network is slow. Build can be moved to a GitLab runner to solve the problem. As the CI/CD pipeline is relative simple, there is no need for a dashboard or library. The toolchain for Docker build pipelines consists of the following tools:

#. `IntelliJ IDEA CE <https://tsi-ccdoc.readthedocs.io/en/master/Tech-tips/Cloud-Consulting-Team-toolbox.html#intellij-idea>`_
#. `GitLab <https://gitlab.ebi.ac.uk/tsi/>`_
#. `GitHub <https://github.com/>`_
#. `Docker Hub <https://hub.docker.com/>`_

The integration of tools are the same as for documentation and ECP above. The overall workflow is show as the diagram below:

.. image:: https://www.lucidchart.com/publicSegments/view/d93750dd-81c2-4647-b448-25ae8649faa8/image.png

The toolchain can pull Dockerfile from a GitHub repository. It builds the Docker image on a GitLab runner. It pushes the image to Docker Hub when a build is done. This is useful when an image is too big for Docker Hub to build it.

The easiest way to integrate with an external repository for CI/CD is to click `CI/CD for external repo` when creating a `new project <https://gitlab.ebi.ac.uk/projects/new>`_. Alternatively, mirroring repositories at `Settings` > `Repository`.

Building Docker image
---------------------

There is only one file `.gitlab-ci.yml` needed to create a toolchain to build a docker image with GitLab at EBI, instead of DockerHub. It runs the build engine on the GitLab runners for EBI. It is a production-grade Kubernetes cluster managed by the cloud team. There is no extra configuration needed.

The `.gitlab-ci.yml` file can be edited with online editor in GitLab or any text editor. Developers typically use an IDE such as IntelliJ integrated with GitLab. The file contains the following major sections.

Add official Docker image to a runner with a Kubernetes cluster and require docker-in-docker service::

  # Official docker image.
  image: docker:latest

  services:
    - docker:dind

Log into Docker Hub and install git. Leave CI_REGISTRY blank for Docker Hub::

  before_script:
    - docker login -u "${CI_REGISTRY_USER}" -p "${CI_REGISTRY_PASSWORD}" ${CI_REGISTRY}
    - apk add git
    - mkdir -p ${ARTIFACT_DIR}

Clone the repository for Docker build onto the runner. Build, tag and push a new Docker image::

  docker.image:
    stage: build
    variables:
      CI_GIT_REPO_NAME: Metagenomics-Assembly
      CI_GIT_URL: https://github.com/Medalibi/${CI_GIT_REPO_NAME}.git
      CI_REGISTRY_IMAGE: davidyuyuan/metagenomics
      CI_COMMIT_REF_SLUG: assembly
      CI_SOURCE_IMAGE: metagenomics
      CI_DOCKER_FILE: ${CI_PROJECT_DIR}/${CI_GIT_REPO_NAME}/Dockerfile
    script:
      - git clone ${CI_GIT_URL}
      - cd ${CI_GIT_REPO_NAME}
      - docker build -f ${CI_DOCKER_FILE} -t ${CI_SOURCE_IMAGE} . | tee ${ARTIFACT_DIR}/build.log
      - docker tag ${CI_SOURCE_IMAGE} ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG}
      - docker push ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG} | tee ${ARTIFACT_DIR}/push.log
    only:
      - master
    artifacts:
      name: "$CI_COMMIT_REF_NAME-$CI_JOB_STAGE-$CI_JOB_NAME"
      paths:
        - ${ARTIFACT_DIR}

It is a good practise to tee build log and push log onto the `ARTIFACT_DIR`. They can then be accessed and reviewed under the pipeline or job generating thme.

Appendix: Command line instructions
-----------------------------------
Git global setup::

  git config --global user.name "User Name"
  git config --global user.email "username@ebi.ac.uk"

Create a new repository::

  git clone git@gitlab.ebi.ac.uk:davidyuan/adv-k8s.git
  cd adv-k8s
  touch README.md
  git add README.md
  git commit -m "add README"
  git push -u origin master

Existing folder::

  cd existing_folder
  git init
  git remote add origin git@gitlab.ebi.ac.uk:davidyuan/adv-k8s.git
  git add .
  git commit -m "Initial commit"
  git push -u origin master

Existing Git repository::

  cd existing_repo
  git remote rename origin old-origin
  git remote add origin git@gitlab.ebi.ac.uk:davidyuan/adv-k8s.git
  git push -u origin --all
  git push -u origin --tags