Welcome to Cloud Consultancy Team¶
This site is maintained by the Cloud Consultancy Team at EBI
Table of contents¶
- ResOps Training - ResOps Training
- Recommended Tools - a list of recommended tools, and to get the most out of them
- EBI Cloud Portal - the EBI multi-cloud way of deploying applications
- Architectural Design Patterns - how to build cloud-native applications
- About this documentation
- Contact us
- Site map
ResOps Training¶
- Notice
- Please check the course pre-requisite on the agenda page.
- Agenda for ResOps 2020
- Agenda for ResOps 2019
Recommended Tools¶
- Ansible:
- Cloud Consultancy Team toolbox:
- Common Workflow Language
- Docker:
- GitLab:
- Google Cloud Platform:
- HPC in the cloud:
- HPC with Slurm on GCP
- Azure CycleCloud
- Kubernetes:
- OneData:
- Overview
- Oneclient
- User Guide
- Onedata API
- EBI setup details
- EBI onedata portal
- Terraform:
- Supervisor:
- Description: Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.
- Reference documentation
EBI Cloud Portal¶
Architectural Design Patterns¶
About this documentation¶
This documentation is maintained at in a gitlab repository at EBI. Feel free to clone a local copy. You would need to run a local build to generate a local HTML site from the source files of *.md and *.rst. Install Sphinx with the instructions of Getting started with sphinx. Follow the instructions of Using markdown with sphinx and Local build with Sphinx to generate your local site.
The master branch will always be the latest, most up-to-date version. Other versions may be available through the selection drop-down on the bottom-left corner of this page.
If you find any mistakes, have suggestions for improvement, have questions, or wish to talk to us about your project, please contact us. Pull requests or issues filed against the git repository are also welcome.
Contact us¶
You can contact us by email or come find us in pod 14 of the South Building.
Site map¶
- EMBL-EBI Cloud Portal documentation
- ResOps Training & Workshop - 2020
- ResOps Training & Workshop - 2019
- Creating VMs with Terraform on OSK for ResOps
- Docker exercises
- Exercise 1: Running images
- Exercise 2: Running a bioinfomatics application
- Exercise 3: Creating your own docker image
- Exercise 4: Optimising a docker image
- Exercise 5: Using the ‘builder’ pattern to build small images
- Exercise 6: Running as root or a non-root user
- Exercise 7: Using metadata
- Exercise 8: Removing images from your machine (optional)
- Gitlab exercises
- Exercise 1: Log into gitlab, set up your SSH key
- Exercise 2: Create a new project
- Exercise 3: Download and run the docker image from your gitlab project
- Exercise 4: Use git tags to create a named version of a docker image
- Exercise 5: Extend the pipeline by adding further steps
- Exercise 6: Change the order of the pipeline steps
- Exercise 7: Pass secrets to the build pipeline
- Error-Tracking
- Gitlab to Github pipeline
- Gitlab DevOps
- Tracing
- Auto DevOps tutorial
- Additional considerations for research pipelines
- Kubernetes on OpenStack
- Kubernetes Practical
- Reading 0: Adding Minikube to the new VMs
- Exercise 0.0: Starting Minikube
- Exercise 0.1: (Optional) Enabling GUI for VNC
- Exercise 1: Creating NGINX
- Exercise 2: Adding HTML to pods
- Exercise 3: Persistence and disaster recovery
- Exercise 4: Problem determination step-by-step
- Homework: Additional exercises after the workshop
- Scaling up Kubernetes for research pipelines
- Reading 0: Preparing Minikube VM to support NFS volumes
- Exercise 1: ReadWriteMany for shared output
- Exercise 2: ReadOnlyMany for data source
- Exercise 3: ReadWriteOnce for private workspace
- Exercise 4: Initialising persistent volumes
- Exercise 5: Kubernetes secret & S3 interface
- Exercise 6: Horizontal scaling
- Reading 1: Vertical scaling
- Exercise 7: Autoscaling
- Accessing GCP node from CLI
- Arvados on Kubernetes
- Cloud Consulting Team toolbox
- DevOps toolchain from GitLab to Docker Hub for Container Build
- DevOps toolchain from Gitlab to OpenStack for pipelines on ECP
- DevOps toolchain from IntelliJ to ReadtheDocs for publishing
- HPC with Azure CycleCloud
- HPC with Slurm on GCP
- Installing OpenStack CLI on Mac OS X
- Kubeflow for Machine Learning
- Tips and tricks with Docker
- Adding $USER to docker group
- Mounting a volume from host
- Size of a Docker image
- Running processes in a container as nonroot
- Containers should be immutable & ephemeral
- Paying attention to build context
- Best practices for Dockerfile
- Recommendations for the packaging and containerizing of bioinformatics software
- Tips and tricks with Terraform