Accessing GCP node from CLI¶
The following focuses on accessing a GCP node without public IP. If a public IP is available, simply drop the optional flag –tunnel-through-iap from a command.
SSH or SCP to a node without public IP¶
With GCP Identity-Aware Proxy (IAP), you can SSH to any node without public IP with your G Suite ID at EBI. There are many ways to access nodes. They require little or no extra effort.
SSH via a web browser window¶
Pros: Simplest option. No installation or configuration required.
Cons: Only a dumb terminal in a web browser.
Click an SSH button for the node in GCP Cloud Console. This pops up a browser window emulating an SSH terminal.
SSH via Cloud Shell¶
Pros: Fully functioning VM for development to keep your laptop clean.
Cons: One time configuration required. Files can be copied between your laptop / EBI cluster and Cloud Shell via GUI or SCP.
Click a drop-down button next to SSH in GCP Cloud Console. Click “View gcloud command”. Click “RUN IN CLOUD SHELL”. This starts your private Cloud Shell VM for you to SSH to a node. You may be asked to generate keys for the first time. This option keeps your laptop clean.
SSH from your laptop¶
Pros: Direct connection from your laptop. Convenient if it is your default development environment already.
Cons: Installation of GCP SDK required. One time configuration and authentication required.
Start a terminal window. Copy and paste the gcloud command (e.g. gcloud compute ssh –zone $ZONE $LOGIN_HOST –tunnel-through-iap –project $PROJECT) into it. You may have to install Cloud SDK first by following the instructions on Installing Google Cloud SDK. You may also need to authenticate with your G Suite ID by following instructions in the terminal window.
SSH from EBI cluster¶
Pros: Direct connection from EBI cluster. Convenient if you need to transfer files between GCP and EBI cluster.
Cons: Most complicated installation, configuration and authentication. Ancient Python version in EBI cluster making life harder.
Log into EBI cluster (e.g. ssh ebi-cli or ssh ebi-login) from a terminal window. Run the following commands to install Miniconda3 as instructed by https://docs.conda.io/en/latest/miniconda.html. Answer yes to all the questions:
[davidyuan@noah-login-03 ~]$ cd "${HOME}" && curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
[davidyuan@noah-login-03 ~]$ chmod +x ./Miniconda3-latest-Linux-x86_64.sh
[davidyuan@noah-login-03 ~]$ ./Miniconda3-latest-Linux-x86_64.sh
[davidyuan@noah-login-03 ~]$ . "${HOME}"/.bashrc
This should create a base conda environment with Python 3.8 installed. You can confirm that with the following commands:
(base) [davidyuan@noah-login-03 ~]$ python --version
Python 3.8.3
(base) [davidyuan@noah-login-03 ~]$ which python
~/miniconda3/bin/python
Now you can install Google Cloud SDK with Miniconda3 as instructed by https://anaconda.org/conda-forge/google-cloud-sdk. Again, answer yes when asked:
(base) [davidyuan@noah-login-03 ~]$ conda install -c conda-forge google-cloud-sdk
It is always a good idea to double-check what you have done:
(base) [davidyuan@noah-login-03 ~]$ which gcloud
~/miniconda3/bin/gcloud
(base) [davidyuan@noah-login-03 ~]$ gcloud --version
Google Cloud SDK 310.0.0
You can SSH from EBI cluster to any node on GCP (e.g. gcloud compute ssh –zone $ZONE $LOGIN_HOST –tunnel-through-iap –project $PROJECT).
Notes:
- You can use IAP with SCP in a similar fashion (e.g. gcloud compute scp –zone $ZONE –tunnel-through-iap –project $PROJECT <normal_scp_parameters>). It can be handy to push or pull files between EBI cluster and GCP nodes via SCP.
- You can also use gsutil to upload files to or to download objects from the storage buckets (e.g. gsutil ls gs://<bucket_name>).