Kubernetes Practical

This practical provides step-by-step guide to create a Web front-end in Kubernetes. It is designed to help a developer new to Kubernetes to create a work environment from scratch. A virtual machine running Ubuntu 18.04 LTS is assumed. The following exercises are built on top of each other:

  1. Reading 0: Adding Minikube to the new VMs
  2. Exercise 0: Starting Minikube under your own ID
  3. Exercise 1: Creating NGINX
  4. Exercise 2: Adding HTML to pods
  5. Exercise 3: Persistence and disaster recovery
  6. Exercise 4: Problem determination step-by-step
  7. Homework: Additional exercises after the workshop

This practical intends to use a simple example to demonstrate the features and characteristics of Kubernetes as an orchestrator for container-based workloads. You should be able to create your own work environment and to start implementing your own project after finishing this practical.

The code for the practical is under https://gitlab.ebi.ac.uk/TSI/tsi-ccdoc/tree/master/tsi-cc/ResOps/scripts. The deployment would look like the following once it is completed:

../../_images/K8SPractical.png

Reading 0: Adding Minikube to the new VMs

This original exercise was designed to show user how to build a sandbox for an individual developer. This is automated to give users more time to focus on cloud-specific subjects. Read through this section so that you can build your own sandbox after the workshop.

Kubernetes has comprehensive documentation on if and how to use Minikube https://kubernetes.io/docs/setup/minikube/.

Access the VM assigned to you from your SSH client. The command is ssh <user_id>@<IP_of_VM_assigned_to_you>. Your user ID, password and IP address will be given to you in the workshop.

Install Docker as the runtime for Minikube:

sudo apt-get update
sudo apt install -y docker.io

Install kubectl as the client to access the Kubernetes cluster in Minikube:

sudo snap install kubectl --classic

Download, install and configure Minikube:

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube
sudo cp minikube /usr/local/bin && rm minikube

Note that /home/ubuntu/.kube/config is owned by root. Run chown to avoid using sudo:

sudo chown -R $USER $HOME/.kube $HOME/.minikube

If a sudor other than ubuntu is used, move the following two directories:

sudo mv /home/ubuntu/.kube /home/ubuntu/.minikube $HOME

Exercise 0: Starting Minikube under your own ID

Start Minikube without a hypervisor (a.k.a Docker runtime):

sudo minikube start --vm-driver=none

The following message should be displayed when Minikube is started successfully:

😄  minikube v1.0.1 on linux (amd64)
🔥  Creating none VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
📶  "minikube" IP address is 10.0.0.5
🐳  Configuring Docker as the container runtime ...
🐳  Version of container runtime is 18.09.2
✨  Preparing Kubernetes environment ...
❌  Unable to load cached images: loading cached images: loading image /home/ubuntu/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner_v1.8.1: stat /home/ubuntu/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner_v1.8.1: no such file or directory
💾  Downloading kubeadm v1.14.1
💾  Downloading kubelet v1.14.1
🚜  Pulling images required by Kubernetes v1.14.1 ...
🚀  Launching Kubernetes v1.14.1 using kubeadm ...
⌛  Waiting for pods: apiserver proxy etcd scheduler controller dns
🔑  Configuring cluster permissions ...
🤔  Verifying component health .....
🤹  Configuring local host environment ...

⚠️  The 'none' driver provides limited isolation and may reduce system security and reliability.
⚠️  For more information, see:
👉  https://github.com/kubernetes/minikube/blob/master/docs/vmdriver-none.md

⚠️  kubectl and minikube configuration will be stored in /home/ubuntu
⚠️  To use kubectl or minikube commands as your own user, you may
⚠️  need to relocate them. For example, to overwrite your own settings:

    ▪ sudo mv /home/ubuntu/.kube /home/ubuntu/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

💡  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
💗  kubectl is now configured to use "minikube"
🏄  Done! Thank you for using minikube!

(Optional) Note that ~/.kube/config is owned by root. Run chown to avoid using sudo all the time:

sudo chown -R $USER $HOME/.kube $HOME/.minikube

Verify that Minicube is working and you can access it. You should see the following message:

kubectl get node

NAME       STATUS   ROLES    AGE     VERSION
minikube   Ready    master   3m59s   v1.14.3

Now, you have a Kubernetes environment to develop and to test your workload. Never use it for production though.

Exercise 1: Creating NGINX

Create a Kubernetes manifest file with the following command:

nano ~/nginx.yml

You can copy and paste the manifest from https://gitlab.ebi.ac.uk/TSI/tsi-ccdoc/blob/master/tsi-cc/ResOps/scripts/minikube/nginx.yml to the editor. Review the file carefully to see what resources are to be created and how parts are connected with each other.

../../_images/nginx.yml.png

This manifest creates a StatefulSet with a corresponding service, where NGINX will be listening on port 80 for HTTP. There are two pods to be created in the set for load balancing, minimising downtime and to certain extend disaster recovering. The container NGINX is pulled from Docker Hub. It will be started to serve HTML pages from persistent volumes private to each pod.

Apply the manifest to create a service and statefulset:

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl apply -f nginx.yml
service/nginx created
statefulset.apps/web created

This creates additional resources needed to for a cluster of NGINX servers(persistent volume claims, persistent volumes, pods, and services):

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Bound    pvc-4cd5c31b-7718-11e9-8b0b-fa163ede6c1a   1Gi        RWO            standard       39s
www-web-1   Bound    pvc-5ac9090b-7718-11e9-8b0b-fa163ede6c1a   1Gi        RWO            standard       16s

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-4cd5c31b-7718-11e9-8b0b-fa163ede6c1a   1Gi        RWO            Delete           Bound    default/www-web-0   standard                41s
pvc-5ac9090b-7718-11e9-8b0b-fa163ede6c1a   1Gi        RWO            Delete           Bound    default/www-web-1   standard                18s

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl get pod
NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          52s
web-1   1/1     Running   0          29s

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   61m
nginx        ClusterIP   10.104.206.229   <none>        80/TCP    76s

You now have a cluster of two NGINX pods running in Minicube. They are listening on port 80 with one cluster IP. You have just built a web service infrastructure with redundancy managed by Kubernetes.

Exercise 2: Adding HTML to pods

Each of the two NGINX server has its own storage, running in its own pod but share the same cluster IP, for example 10.104.206.229. The cluster IP is listed with kubectl get svc in the previous exercise. If you try to access the home page, you will get HTTP403:

ubuntu@resops-1-k8s-node-nf-1:~$ curl http://10.104.206.229
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.15.12</center>
</body>
</html>

Let’s fix this on one of the two NGINX servers. Recall that the volume www is mounted on /usr/share/nginx/html/, which is the default document root for NGINX server. Connect to pod web-0 and you can see that there is no file to be served by NGINX. That’s why the HTTP403 is sent back:

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl exec -it web-0 -- /bin/bash
root@web-0:/# cd /usr/share/nginx/html/
root@web-0:/usr/share/nginx/html# ls -la
total 8
drwxrwxrwx 2 root root 4096 May 15 13:49 .
drwxr-xr-x 3 root root 4096 May  8 03:01 ..

Create a simple /usr/share/nginx/html/index.html with a title and heading “Hello World from web-0”:

cat <<EOF > /usr/share/nginx/html/index.html
<html>
<head><title>Hello World from web-0</title></head>
<body>
<center><h1>Hello World from web-0</h1></center>
<hr><center>nginx/1.15.12</center>
</body>
</html>
EOF

Exit out of the pod web-0:

root@web-0:/usr/share/nginx/html# exit
exit

Repeat the process in pod web-1. It is important to use a title and heading “Hello World from web-1” so that the two pods have two different index.html files:

kubectl exec -it web-1 -- /bin/bash

cat <<EOF > /usr/share/nginx/html/index.html
<html>
<head><title>Hello World from web-1</title></head>
<body>
<center><h1>Hello World from web-1</h1></center>
<hr><center>nginx/1.15.12</center>
</body>
</html>
EOF

Exit out of the pod web-1. Send HTTP request to the cluster IP again. You will find that the two NGINX servers take turns to serve the home page. The HTTP 403 error is gone:

ubuntu@resops-1-k8s-node-nf-1:~$ curl http://10.104.206.229
<html>
<head><title>Hello World from web-1</title></head>
<body>
<center><h1>Hello World from web-1</h1></center>
<hr><center>nginx/1.15.12</center>
</body>
</html>

ubuntu@resops-1-k8s-node-nf-1:~$ curl http://10.104.206.229
<html>
<head><title>Hello World from web-0</title></head>
<body>
<center><h1>Hello World from web-0</h1></center>
<hr><center>nginx/1.15.12</center>
</body>
</html>

Note that Kubernetes tend to route the requests to the same pod for better performance. You may keep seeing your HTML page served from the same pod, for example web-0. If this is happening, rename the index.html in web-0. Then you will see the page gets served from the other pod web-1:

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl exec -it web-0 -- /bin/bash
root@web-0:/# mv /usr/share/nginx/html/index.html /usr/share/nginx/html/index.html.bak
root@web-0:/# exit
exit

ubuntu@resops-1-k8s-node-nf-1:~$ curl http://10.104.206.229
<html>
<head><title>Hello World from web-1</title></head>
<body>
<center><h1>Hello World from web-1</h1></center>
<hr><center>nginx/1.15.12</center>
</body>
</html>

Change the index.html page in web-0 back:

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl exec -it web-0 -- /bin/bash
root@web-0:/# mv /usr/share/nginx/html/index.html.bak /usr/share/nginx/html/index.html
root@web-0:/# exit
exit

From exercises 2 & 3, we understand that the NGINX serves independent copies from /usr/share/nginx/html persisted on two separate volumes via two pods web-0 and web-1. See output by kubectl get commands in exercise 2 and curl http://10.104.206.229 command in exercise 3 again.

Exercise 3: Persistence and disaster recovery

If a stateful set is removed, the persistent volumes are preserved by design. Run kubectl delete -f nginx.yml to simulate scheduled outage. The service and pods are deleted but persistent volumes are saved:

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl delete -f nginx.yml
service "nginx" deleted
statefulset.apps "web" deleted

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Bound    pvc-4cd5c31b-7718-11e9-8b0b-fa163ede6c1a   1Gi        RWO            standard       110m
www-web-1   Bound    pvc-5ac9090b-7718-11e9-8b0b-fa163ede6c1a   1Gi        RWO            standard       110m

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-4cd5c31b-7718-11e9-8b0b-fa163ede6c1a   1Gi        RWO            Delete           Bound    default/www-web-0   standard                110m
pvc-5ac9090b-7718-11e9-8b0b-fa163ede6c1a   1Gi        RWO            Delete           Bound    default/www-web-1   standard                110m

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   173m

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl get pod
No resources found.

Run kubectl apply -f nginx.yml to recreate the stateful set and service. The each new pod recreated will mount its original volume as if it was never deleted. The pod web-0 still has the index.html with the message of “Hello World from web-0”:

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl apply -f nginx.yml
service/nginx created
statefulset.apps/web created

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl get pod
NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          14s
web-1   1/1     Running   0          7s

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl exec -it web-0 -- /bin/bash
root@web-0:/# cat /usr/share/nginx/html/index.html
<html>
<head><title>Hello World from web-0</title></head>
<body>
<center><h1>Hello World from web-0</h1></center>
<hr><center>nginx/1.15.12</center>
</body>
</html>

root@web-0:/# exit
exit

Run kubectl delete pod web-1 to simulate unscheduled outage. The recovery happens really fast. We need to chain two kubectl command to see what is happening:

kubectl delete pod web-1 && kubectl get pod

Kubernete tries to restart pod web-1 immediately. After a little while web-1 will be running again as if noting happened. It will be mounted to its original volume. The index.html in used by pod web-1 still has the same message of “Hello World from web-1”:

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl delete pod web-1 && kubectl get pod
pod "web-1" deleted
NAME    READY   STATUS              RESTARTS   AGE
web-0   1/1     Running             0          6m50s
web-1   0/1     ContainerCreating   0          0s

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl get pod
NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          11m
web-1   1/1     Running   0          4m27s

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl exec -it web-1 -- /bin/bash
root@web-1:/# cat /usr/share/nginx/html/index.html
<html>
<head><title>Hello World from web-1</title></head>
<body>
<center><h1>Hello World from web-1</h1></center>
<hr><center>nginx/1.15.12</center>
</body>
</html>

As you can see, Kubernetes restarts web-1 immediately. The newly started pod still mount to the same persistent volume as if the pod was never killed.

Exercise 4: Problem determination step-by-step

Frequently, a manifest file that we have created does not behave as we expected or contains bugs. Here is a micky-mouse example how we can investigate what is going on.

First, repeat the first step in the previous exercise to remove the deployment:

ubuntu@resops-1-k8s-node-nf-1:~$ kubectl delete -f nginx.yml
service "nginx" deleted
statefulset.apps "web" deleted

Run kubernetes get commands to confirm if the resources are deleted. Check the instructions in the previous exercise if you do not remember the details.

Second, inspect https://gitlab.ebi.ac.uk/TSI/tsi-ccdoc/blob/master/tsi-cc/ResOps/scripts/minikube/mickymouse.yml to see if you can spot any problems in the manifest.

Third, apply mickymouse.yml:

resops49@resops-k8s-node-17:~$ kubectl apply -f https://gitlab.ebi.ac.uk/TSI/tsi-ccdoc/raw/master/tsi-cc/ResOps/scripts/minikube/mickymouse.yml
service/nginx created
statefulset.apps/web created

Cool! Or, is it? Let’s repeat the kubernetes get commands to see if there is anything unusual. Here is what you would likely see:

resops49@resops-k8s-node-17:~$ kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Bound    pvc-4290e18c-d2b1-4714-b960-2f892121fd5b   1Gi        RWO            standard       6m
www-web-1   Bound    pvc-dcbd9627-9b02-4d5a-8be1-c40045c0670c   1Gi        RWO            standard       5m48s

resops49@resops-k8s-node-17:~$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-4290e18c-d2b1-4714-b960-2f892121fd5b   1Gi        RWO            Delete           Bound    default/www-web-0   standard                6m6s
pvc-dcbd9627-9b02-4d5a-8be1-c40045c0670c   1Gi        RWO            Delete           Bound    default/www-web-1   standard                5m54s

resops49@resops-k8s-node-17:~$ kubectl get pod
No resources found.

resops49@resops-k8s-node-17:~$ kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        6d22h
nginx        NodePort    10.107.235.150   <none>        80:31541/TCP   3m54s

We are expecting two pods running but nothing is created. PVs and PVCs are bounded. However, you would notice the mismatch of the timestamp between Service and PersistentVolumes. This is because, again, StatefulSet always leaves PersistentVolumes untouched by design even if the reclaim policy is delete. We need to delete these volumes explicitly to fully understand how naughty mickymouse.yml is:

resops49@resops-k8s-node-17:~$ kubectl delete -f https://gitlab.ebi.ac.uk/TSI/tsi-ccdoc/raw/master/tsi-cc/ResOps/scripts/minikube/mickymouse.yml
service "nginx" deleted
statefulset.apps "web" deleted

resops49@resops-k8s-node-17:~$ kubectl delete pvc www-web-0
persistentvolumeclaim "www-web-0" deleted

resops49@resops-k8s-node-17:~$ kubectl delete pvc www-web-1
persistentvolumeclaim "www-web-1" deleted

resops49@resops-k8s-node-17:~$ kubectl get pv
No resources found.

resops49@resops-k8s-node-17:~$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   6d23h

Because of the reclaim policy, deleting the PVCs cleans out PVs as well. Now the environment is clean. Let’s apply mickymouse.yml, again:

resops49@resops-k8s-node-17:~$ kubectl apply -f https://gitlab.ebi.ac.uk/TSI/tsi-ccdoc/raw/master/tsi-cc/ResOps/scripts/minikube/mickymouse.yml
service/nginx created
statefulset.apps/web created

resops49@resops-k8s-node-17:~$ kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Bound    pvc-9d3be2c4-3605-4f86-95d2-0e1f7328ded4   1Gi        RWO            standard       45s

resops49@resops-k8s-node-17:~$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-9d3be2c4-3605-4f86-95d2-0e1f7328ded4   1Gi        RWO            Delete           Bound    default/www-web-0   standard                54s

resops49@resops-k8s-node-17:~$ kubectl get pod
No resources found.

resops49@resops-k8s-node-17:~$ kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        6d23h
nginx        NodePort    10.105.177.74   <none>        80:30482/TCP   74s

The Service and StatefulSet seem created successfully again. The timestamps of Service, PV and PVC are close enough now. We still do not have any pods. In addition, we only see one pair of PV and PVC instead of two pairs. Things are going from bad to worse.

Fourth, check the events recorded by Kubernetes to see if there have been any failures. The event log can be long and messy. Always remember to sort the events by last seen or to apply various filters. See kubectl get –help to see how to apply filters:

resops49@resops-k8s-node-17:~$ kubectl get event --sort-by='{.lastTimestamp}'
LAST SEEN   TYPE      REASON                  OBJECT                            MESSAGE
60m         Normal    Starting                node/minikube                     Starting kube-proxy.
60m         Normal    RegisteredNode          node/minikube                     Node minikube event: Registered Node minikube in Controller
46m         Warning   FailedScheduling        pod/web-0                         pod has unbound immediate PersistentVolumeClaims
46m         Normal    ProvisioningSucceeded   persistentvolumeclaim/www-web-0   Successfully provisioned volume pvc-4290e18c-d2b1-4714-b960-2f892121fd5b
46m         Normal    Provisioning            persistentvolumeclaim/www-web-0   External provisioner is provisioning volume for claim "default/www-web-0"
46m         Normal    ExternalProvisioning    persistentvolumeclaim/www-web-0   waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
46m         Normal    SuccessfulCreate        statefulset/web                   create Pod web-0 in StatefulSet web successful
46m         Normal    SuccessfulCreate        statefulset/web                   create Claim www-web-0 Pod web-0 in StatefulSet web success
46m         Normal    Scheduled               pod/web-0                         Successfully assigned default/web-0 to minikube
46m         Normal    Pulling                 pod/web-0                         Pulling image "nginx"
46m         Normal    Pulled                  pod/web-0                         Successfully pulled image "nginx"
46m         Normal    SuccessfulCreate        statefulset/web                   create Claim www-web-1 Pod web-1 in StatefulSet web success
46m         Normal    ExternalProvisioning    persistentvolumeclaim/www-web-1   waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
46m         Normal    Provisioning            persistentvolumeclaim/www-web-1   External provisioner is provisioning volume for claim "default/www-web-1"
46m         Normal    ProvisioningSucceeded   persistentvolumeclaim/www-web-1   Successfully provisioned volume pvc-dcbd9627-9b02-4d5a-8be1-c40045c0670c
46m         Normal    SuccessfulCreate        statefulset/web                   create Pod web-1 in StatefulSet web successful
46m         Warning   FailedScheduling        pod/web-1                         pod has unbound immediate PersistentVolumeClaims
46m         Normal    Created                 pod/web-0                         Created container nginx
46m         Normal    Started                 pod/web-0                         Started container nginx
46m         Normal    Scheduled               pod/web-1                         Successfully assigned default/web-1 to minikube
46m         Normal    Pulling                 pod/web-1                         Pulling image "nginx"
46m         Normal    Pulled                  pod/web-1                         Successfully pulled image "nginx"
46m         Normal    Started                 pod/web-1                         Started container nginx
46m         Normal    Created                 pod/web-1                         Created container nginx
45m         Normal    Killing                 pod/web-1                         Stopping container nginx
45m         Normal    Killing                 pod/web-0                         Stopping container nginx
32m         Warning   FailedCreate            statefulset/web                   create Pod web-0 in StatefulSet web failed error: Pod "web-0" is invalid: spec.containers[0].volumeMounts[0].name: Not found: "html"
18m         Normal    ExternalProvisioning    persistentvolumeclaim/www-web-0   waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
18m         Normal    Provisioning            persistentvolumeclaim/www-web-0   External provisioner is provisioning volume for claim "default/www-web-0"
18m         Normal    ProvisioningSucceeded   persistentvolumeclaim/www-web-0   Successfully provisioned volume pvc-9d3be2c4-3605-4f86-95d2-0e1f7328ded4
18m         Normal    SuccessfulCreate        statefulset/web                   create Claim www-web-0 Pod web-0 in StatefulSet web success
7m5s        Warning   FailedCreate            statefulset/web                   create Pod web-0 in StatefulSet web failed error: Pod "web-0" is invalid: spec.containers[0].volumeMounts[0].name: Not found: "html"

From the event log, we can see an error message create Pod web-0 in StatefulSet web failed error: Pod “web-0” is invalid: spec.containers[0].volumeMounts[0].name: Not found: “html”. Checking the mickymouse.yml by following the path spec.containers[0].volumeMounts[0].name, we see a mismatch of names between volumeMounts.name=html and volumeClaimTemplates.metadata.name=www. This needs to be fixed.

Delete everything created by mickymouse.yml. Remember to delete the PVC as well. By now, you should be able to find relevant commands in this practical to complete the task.

Fifth, clone the repository and start modifying mickymouse.yml locally:

resops49@resops-k8s-node-17:~$ git clone https://gitlab.ebi.ac.uk/TSI/tsi-ccdoc.git
Cloning into 'tsi-ccdoc'...
remote: Enumerating objects: 388, done.
remote: Counting objects: 100% (388/388), done.
remote: Compressing objects: 100% (196/196), done.
remote: Total 4488 (delta 200), reused 302 (delta 145)
Receiving objects: 100% (4488/4488), 102.69 MiB | 30.49 MiB/s, done.
Resolving deltas: 100% (1904/1904), done.

resops49@resops-k8s-node-17:~$ cd tsi-ccdoc/tsi-cc/ResOps/scripts/minikube/

resops49@resops-k8s-node-17:~/tsi-ccdoc/tsi-cc/ResOps/scripts/minikube$ nano ./mickymouse.yml

Change name from html to www. Apply mickymouse.yml to see if the problem is fixed. Things certainly look much better. Everything seems created successfully. Is everything working? If not, why? How are you going to find out? How are you going to fix the problems?

resops49@resops-k8s-node-17:~/tsi-ccdoc/tsi-cc/ResOps/scripts/minikube$ kubectl apply -f ./mickymouse.yml
service/nginx created
statefulset.apps/web created

resops49@resops-k8s-node-17:~/tsi-ccdoc/tsi-cc/ResOps/scripts/minikube$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-8b5f0d99-f4dd-461d-8f46-2514d23e784c   1Gi        RWO            Delete           Bound    default/www-web-0   standard                8s
pvc-ad55fa2d-d194-4b58-8b6c-815da976d038   1Gi        RWO            Delete           Bound    default/www-web-1   standard                1s

resops49@resops-k8s-node-17:~/tsi-ccdoc/tsi-cc/ResOps/scripts/minikube$ kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Bound    pvc-8b5f0d99-f4dd-461d-8f46-2514d23e784c   1Gi        RWO            standard       18s
www-web-1   Bound    pvc-ad55fa2d-d194-4b58-8b6c-815da976d038   1Gi        RWO            standard       11s

resops49@resops-k8s-node-17:~/tsi-ccdoc/tsi-cc/ResOps/scripts/minikube$ kubectl get pod
NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          27s
web-1   1/1     Running   0          20s

resops49@resops-k8s-node-17:~/tsi-ccdoc/tsi-cc/ResOps/scripts/minikube$ kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        7d
nginx        NodePort    10.106.111.118   <none>        80:31542/TCP   55s

Homework: Additional exercises after the workshop

There are two more problems and one enhancement for your exercise after the workshop. If you get stuck, compare mickymouse.yml against nginx.yml. It might be easier if you know what you are looking for.

Here are some general guidance:

  1. Check the event log again to confirm that deployment was successful. Use kubectl get event –help to learn how to use it.
  2. Check the runtime logs to see if there are errors. Use kubectl logs –help to learn how to use it.
  3. Explore on Kubernetes Dashboard to see if you can spot anything unusual. Note that Kubernetes Dashboard would not work in the sandbox as the VM does not have GUI enabled.
  4. Connect to pods and containers in the pods to see if everything is configured properly.
  5. Test runtime behaviour to see if the system acts as designed. In this case, run curl http:// to see what happens.

Here are some steps to look for issues. You may wan to continue working on this after the workshop:

resops49@resops-k8s-node-17:~/tsi-ccdoc/tsi-cc/ResOps/scripts/minikube$ kubectl get event --sort-by='{.lastTimestamp}'
LAST SEEN   TYPE      REASON                  OBJECT                            MESSAGE
34m         Warning   FailedCreate            statefulset/web                   create Pod web-0 in StatefulSet web failed error: Pod "web-0" is invalid: spec.containers[0].volumeMounts[0].name: Not found: "html"
10m         Normal    SuccessfulCreate        statefulset/web                   create Claim www-web-0 Pod web-0 in StatefulSet web success
10m         Normal    ExternalProvisioning    persistentvolumeclaim/www-web-0   waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
10m         Normal    ProvisioningSucceeded   persistentvolumeclaim/www-web-0   Successfully provisioned volume pvc-8b5f0d99-f4dd-461d-8f46-2514d23e784c
10m         Normal    Provisioning            persistentvolumeclaim/www-web-0   External provisioner is provisioning volume for claim "default/www-web-0"
10m         Warning   FailedScheduling        pod/web-0                         pod has unbound immediate PersistentVolumeClaims
10m         Normal    SuccessfulCreate        statefulset/web                   create Pod web-0 in StatefulSet web successful
10m         Normal    Scheduled               pod/web-0                         Successfully assigned default/web-0 to minikube
10m         Normal    Pulling                 pod/web-0                         Pulling image "nginx"
10m         Normal    Pulled                  pod/web-0                         Successfully pulled image "nginx"
10m         Normal    Created                 pod/web-0                         Created container nginx
10m         Normal    Started                 pod/web-0                         Started container nginx
10m         Normal    ExternalProvisioning    persistentvolumeclaim/www-web-1   waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
10m         Normal    Provisioning            persistentvolumeclaim/www-web-1   External provisioner is provisioning volume for claim "default/www-web-1"
10m         Warning   FailedScheduling        pod/web-1                         pod has unbound immediate PersistentVolumeClaims
10m         Normal    SuccessfulCreate        statefulset/web                   create Claim www-web-1 Pod web-1 in StatefulSet web success
10m         Normal    SuccessfulCreate        statefulset/web                   create Pod web-1 in StatefulSet web successful
10m         Normal    ProvisioningSucceeded   persistentvolumeclaim/www-web-1   Successfully provisioned volume pvc-ad55fa2d-d194-4b58-8b6c-815da976d038
10m         Normal    Scheduled               pod/web-1                         Successfully assigned default/web-1 to minikube
10m         Normal    Pulling                 pod/web-1                         Pulling image "nginx"
10m         Normal    Created                 pod/web-1                         Created container nginx
10m         Normal    Pulled                  pod/web-1                         Successfully pulled image "nginx"
10m         Normal    Started                 pod/web-1                         Started container nginx

resops49@resops-k8s-node-17:~/tsi-ccdoc/tsi-cc/ResOps/scripts/minikube$ kubectl logs web-0 --all-containers=true

resops49@resops-k8s-node-17:~/tsi-ccdoc/tsi-cc/ResOps/scripts/minikube$ kubectl exec -it web-0 -- bash
root@web-0:/# ls -l /usr/share/nginx/html
total 0

resops49@resops-k8s-node-17:~/tsi-ccdoc/tsi-cc/ResOps/scripts/minikube$ kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        7d
nginx        NodePort    10.106.111.118   <none>        80:31542/TCP   15m

resops49@resops-k8s-node-17:~/tsi-ccdoc/tsi-cc/ResOps/scripts/minikube$ curl http://10.106.111.118
curl: (7) Failed to connect to 10.106.111.118 port 80: Connection refused

resops49@resops-k8s-node-17:~/tsi-ccdoc/tsi-cc/ResOps/scripts/minikube$ curl http://10.106.111.118:31542
curl: (7) Failed to connect to 10.106.111.118 port 31542: Connection timed out