Welcome to Cloud Consultancy Team

This site is maintained by the Cloud Consultancy Team at EBI

Table of contents

• ResOps Training - ResOps Training
• Recommended Tools - a list of recommended tools, and to get the most out of them
• EBI Cloud Portal - the EBI multi-cloud way of deploying applications
• Architectural Design Patterns - how to build cloud-native applications
• About this documentation
• Contact us
• Site map

ResOps Training

• Notice
  • Please check the course pre-requisite on the agenda page.
• Agenda for ResOps 2020
• Agenda for ResOps 2019

Recommended Tools

• Ansible:

• Cloud Consultancy Team toolbox:
  • How to install tools
• Common Workflow Language
  • CWL self-contained training course
• Docker:
  • Docker best practice
• GitLab:
  • Overview
    • How to create docker image
    • How to deploy ECP instance
    • How to publish documentation
    • GitLab best practice
  • Useful links
    • GitLab CI with Maven

• Google Cloud Platform:
  • Accessing GCP node from CLI
• HPC in the cloud:
  • HPC with Slurm on GCP
  • Azure CycleCloud
    • HPC with CycleCloud on Azure
    • Useful links:
      • CycleCloud Overview
      • Setting up CycleCloud as a container
• Kubernetes:
  • Deployment of Kubernetes cluster on various clouds
  • Kubeflow for Machine Learning
  • Useful links:

• OneData:
  • Overview
  • Oneclient
  • User Guide
  • Onedata API
  • EBI setup details
  • EBI onedata portal

• Terraform:
  • Best Practice
• Supervisor:
  • Description: Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.
  • Reference documentation

EBI Cloud Portal

• EMBL-EBI Cloud Portal documentation
  • Using the EMBL-EBI Cloud Portal
  • Packaging Applications for the EMBL-EBI Cloud Portal
  • Avoid security credentials on git public repository
  • API Endpoint documentation

Architectural Design Patterns

• How to convert monolithic design to microservices

About this documentation

This documentation is maintained at in a gitlab repository at EBI. Feel free to clone a local copy. You would need to run a local build to generate a local HTML site from the source files of *.md and *.rst. Install Sphinx with the instructions of Getting started with sphinx. Follow the instructions of Using markdown with sphinx and Local build with Sphinx to generate your local site.

The master branch will always be the latest, most up-to-date version. Other versions may be available through the selection drop-down on the bottom-left corner of this page.

If you find any mistakes, have suggestions for improvement, have questions, or wish to talk to us about your project, please contact us. Pull requests or issues filed against the git repository are also welcome.

Contact us

You can contact us by email or come find us in pod 14 of the South Building.

Site map

• EMBL-EBI Cloud Portal documentation
• ResOps Training & Workshop - 2020
  • Introduction
  • Course objectives
  • Prerequisite
  • Course videos
  • Course guidelines
• ResOps Training & Workshop - 2019
  • Introduction
  • Course objectives
  • Prerequisite
  • Course guidelines
• Creating VMs with Terraform on OSK for ResOps
  • Assumptions
  • Process
• Docker exercises
• Exercise 1: Running images
  • Objective
  • Terminology
  • Running a simple command in a container
  • Running an interactive command in an image
  • Making changes to a container
  • Starting a long-running service in a container
  • Conclusion
  • Best practices
• Exercise 2: Running a bioinfomatics application
  • Objective
  • Running velvet from the bioboxes repository
  • Conclusion
  • Best practices
• Exercise 3: Creating your own docker image
  • Objective
  • Access to the tutorial material
  • Creating an image, step by step
  • Inspecting an image to find out how it was built
  • Adding our own programs to the image
  • Passing arguments to an application in an image
  • Running an application by default
  • Optimizing builds
  • Conclusion
  • Best practices
• Exercise 4: Optimising a docker image
  • Objective
  • Introduction
  • Optimise a Dockerfile
  • Are you installing stuff you don’t need?
  • The centos image is rather large
  • There’s still more stuff you don’t need in there
  • Now to make better use of the cache
  • Conclusion
  • Best Practices
• Exercise 5: Using the ‘builder’ pattern to build small images
  • Objective
  • The Builder pattern
  • Conclusion
  • Best practices
• Exercise 6: Running as root or a non-root user
  • Objective
  • Containers run as root by default
  • Adding a non-root user to an image
  • Conclusion
  • Best practices
• Exercise 7: Using metadata
  • Objective
  • The four docker environments
  • Using ARG to control your image builds
  • Conclusion
  • Best practices
• Exercise 8: Removing images from your machine (optional)
  • Objective
  • Cleaning up
  • Conclusion
  • Best practices
• Gitlab exercises
• Exercise 1: Log into gitlab, set up your SSH key
  • Objective
  • Introduction
  • Create a Personal Access Token
  • Upload your SSH key (optional!)
  • Configure email notifications (optional!)
  • Conclusion
  • Best Practices
• Exercise 2: Create a new project
  • Objective
  • Create the project and upload the code, watch the build fail
  • Fix the build and try again
  • Conclusion
  • Best Practices
• Exercise 3: Download and run the docker image from your gitlab project
  • Objective
  • Introduction
  • Deploy tokens
  • Conclusion
  • Best Practices
• Exercise 4: Use git tags to create a named version of a docker image
  • Objective
  • Using the git tag to tag the docker image
  • Using the git tag to control the build
  • Check your Registry to see the named docker image
  • Bonus exercise: git branches
  • Conclusion
  • Best Practices
• Exercise 5: Extend the pipeline by adding further steps
  • Objective
  • Introduction
  • Conclusion
  • Best Practices
  • Cheat-sheet
• Exercise 6: Change the order of the pipeline steps
  • Objective
  • Change the order of stages
  • Sequential and parallel builds
  • Conclusion
  • Best Practices
  • Cheat-sheet
• Exercise 7: Pass secrets to the build pipeline
  • Objective
  • Gitlab Variables
  • Masking the variable values
  • Taking it to the next level
  • Conclusion
  • Best Practices
• Error-Tracking
  • Why Error-Tracking is required
  • Implementation
  • Gitlab Integration
• Gitlab to Github pipeline
  • Code Pipeline
  • Release Automation
  • Implementation
• Gitlab DevOps
  • Pre-requisites
  • DevOps - Sounds familiar?
  • Why Gitlab for DevOps?
  • Auto DevOps
• Tracing
  • Why Tracing is required
  • What Tracing addresses
  • Implementation
  • Gitlab Integration
• Auto DevOps tutorial
  • Objective
  • Forking Repo
  • Adding Kubernetes Cluster
  • Enabling Auto-DevOps
  • Checking Deployment
  • Checking Security Dashoard
  • Checking Monitoring Dashoard
  • Conclusion
  • Best Practices
• Additional considerations for research pipelines
  • Reading 1: Bringing compute close to data
  • Reading 2: Resource consumption
  • Reading 3: Docker builds
  • Reading 4: CI/CD toolchain
• Kubernetes on OpenStack
  • Credentials and Floating IP [FIP]
  • Kubectl
  • Kubernetes Dashboard
  • Prometheus and Grafana
  • Application for web (NGINX)
  • Data Loading via S3 (Minio)
  • End-to-end: integrating NGINX, NFS and Minio
• Kubernetes Practical
  • Reading 0: Adding Minikube to the new VMs
  • Exercise 0.0: Starting Minikube
  • Exercise 0.1: (Optional) Enabling GUI for VNC
  • Exercise 1: Creating NGINX
  • Exercise 2: Adding HTML to pods
  • Exercise 3: Persistence and disaster recovery
  • Exercise 4: Problem determination step-by-step
  • Homework: Additional exercises after the workshop
• Scaling up Kubernetes for research pipelines
  • Reading 0: Preparing Minikube VM to support NFS volumes
  • Exercise 1: ReadWriteMany for shared output
  • Exercise 2: ReadOnlyMany for data source
  • Exercise 3: ReadWriteOnce for private workspace
  • Exercise 4: Initialising persistent volumes
  • Exercise 5: Kubernetes secret & S3 interface
  • Exercise 6: Horizontal scaling
  • Reading 1: Vertical scaling
  • Exercise 7: Autoscaling
• Accessing GCP node from CLI
  • SSH or SCP to a node without public IP
• Arvados on Kubernetes
  • Arvados installation options
  • Kubenetes cluster
  • Kubernetes cluster on GKE
  • Helm installation
  • Clone Git repository
  • Start a cluster
  • Final note
• Cloud Consulting Team toolbox
  • Git client
  • SFTP client
  • SSH client
  • IntelliJ IDEA
  • Terraform client
  • Ansible
  • Linux screen sharing - TMUX
  • Yasha
  • Missing xcrun after upgrading to Mojave
  • Shell utility for json
  • Virtual environment
  • Find command for command line search
  • Slack and GoogleDocs
• DevOps toolchain from GitLab to Docker Hub for Container Build
  • Building Docker image
  • Appendix: Command line instructions
• DevOps toolchain from Gitlab to OpenStack for pipelines on ECP
  • Adding test case onto the Dashboard
  • Creating pipeline
  • Implementing ECP test cases
  • Downloading and viewing logs
• DevOps toolchain from IntelliJ to ReadtheDocs for publishing
  • Local build with Sphinx
  • Using markdown with sphinx
  • Reference
• HPC with Azure CycleCloud
  • Cost
  • HPC on CycleCloud
  • Configuration
  • CLI
  • Subscription
  • Service principle
  • Resource group
  • Vnet and subnet
  • Container instance
• HPC with Slurm on GCP
  • Accessing with CLI
  • Enable GCSFuse
  • Monitoring
  • References
• Installing OpenStack CLI on Mac OS X
  • Prerequisite
  • Installation
  • Configuration
  • Verification
  • Configuration
  • Reference
• Kubeflow for Machine Learning
  • Setting up Kubeflow on GKE
  • Setting up Jupyter notebook
  • Customising images for Notebook servers
  • Pipeline SDK
  • Authentication
  • Accessing Git repository
  • Accessing data via Tensorflow
  • Accessing data via OneData
  • GPU quotas
  • Separate GPU node pool
  • GPU availability at europe-west
  • GPU pricing
  • References
• Tips and tricks with Docker
  • Adding $USER to docker group
  • Mounting a volume from host
  • Size of a Docker image
  • Running processes in a container as nonroot
  • Containers should be immutable & ephemeral
  • Paying attention to build context
  • Best practices for Dockerfile
  • Recommendations for the packaging and containerizing of bioinformatics software
• Tips and tricks with Terraform
  • Getting started
  • Benefit to use Terraform
  • Drawback to use Terraform
  • What, not how
  • Plan ahead
  • Custom modules and submodules
  • Provisioners
  • Reference