Accessing GCP node from CLI

The following focuses on accessing a GCP node without public IP. If a public IP is available, simply drop the optional flag –tunnel-through-iap from a command.

SSH or SCP to a node without public IP

With GCP Identity-Aware Proxy (IAP), you can SSH to any node without public IP with your G Suite ID at EBI. There are many ways to access nodes. They require little or no extra effort.

../_images/SSHbutton.png

SSH via a web browser window

Pros: Simplest option. No installation or configuration required.

Cons: Only a dumb terminal in a web browser.

Click an SSH button for the node in GCP Cloud Console. This pops up a browser window emulating an SSH terminal.

SSH via Cloud Shell

Pros: Fully functioning VM for development to keep your laptop clean.

Cons: One time configuration required. Files can be copied between your laptop / EBI cluster and Cloud Shell via GUI or SCP.

Click a drop-down button next to SSH in GCP Cloud Console. Click “View gcloud command”. Click “RUN IN CLOUD SHELL”. This starts your private Cloud Shell VM for you to SSH to a node. You may be asked to generate keys for the first time. This option keeps your laptop clean.

SSH from your laptop

Pros: Direct connection from your laptop. Convenient if it is your default development environment already.

Cons: Installation of GCP SDK required. One time configuration and authentication required.

Start a terminal window. Copy and paste the gcloud command (e.g. gcloud compute ssh –zone $ZONE $LOGIN_HOST –tunnel-through-iap –project $PROJECT) into it. You may have to install Cloud SDK first by following the instructions on Installing Google Cloud SDK. You may also need to authenticate with your G Suite ID by following instructions in the terminal window.

SSH from EBI cluster

Pros: Direct connection from EBI cluster. Convenient if you need to transfer files between GCP and EBI cluster.

Cons: Most complicated installation, configuration and authentication. Ancient Python version in EBI cluster making life harder.

Log into EBI cluster (e.g. ssh ebi-cli or ssh ebi-login) from a terminal window. Run the following commands to install Miniconda3 as instructed by https://docs.conda.io/en/latest/miniconda.html. Answer yes to all the questions:

[davidyuan@noah-login-03 ~]$ cd "${HOME}" && curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
[davidyuan@noah-login-03 ~]$ chmod +x ./Miniconda3-latest-Linux-x86_64.sh
[davidyuan@noah-login-03 ~]$ ./Miniconda3-latest-Linux-x86_64.sh
[davidyuan@noah-login-03 ~]$ . "${HOME}"/.bashrc

This should create a base conda environment with Python 3.8 installed. You can confirm that with the following commands:

(base) [davidyuan@noah-login-03 ~]$ python --version
Python 3.8.3
(base) [davidyuan@noah-login-03 ~]$ which python
~/miniconda3/bin/python

Now you can install Google Cloud SDK with Miniconda3 as instructed by https://anaconda.org/conda-forge/google-cloud-sdk. Again, answer yes when asked:

(base) [davidyuan@noah-login-03 ~]$ conda install -c conda-forge google-cloud-sdk

It is always a good idea to double-check what you have done:

(base) [davidyuan@noah-login-03 ~]$ which gcloud
~/miniconda3/bin/gcloud
(base) [davidyuan@noah-login-03 ~]$ gcloud --version
Google Cloud SDK 310.0.0

You can SSH from EBI cluster to any node on GCP (e.g. gcloud compute ssh –zone $ZONE $LOGIN_HOST –tunnel-through-iap –project $PROJECT).

Notes:

  1. You can use IAP with SCP in a similar fashion (e.g. gcloud compute scp –zone $ZONE –tunnel-through-iap –project $PROJECT <normal_scp_parameters>). It can be handy to push or pull files between EBI cluster and GCP nodes via SCP.
  2. You can also use gsutil to upload files to or to download objects from the storage buckets (e.g. gsutil ls gs://<bucket_name>).