Running Jupyter on CREATE

In order to run Jupyter from the HPC nodes it is necessary to request a batch job and then create a tunnel to the running process.

One way to do this is to use a virtual environment. This example uses virtualenv although it is also just as easy with conda.

Make Jupyter available in a virtualenv

k1234567@erc-hpc-login1:/scratch/users/k1234567$ module load python/3.11.6-gcc-13.2.0
k1234567@erc-hpc-login1:/scratch/users/k1234567$ virtualenv jvenv -p `which python`
k1234567@erc-hpc-login1:/scratch/users/k1234567$ source jvenv/bin/activate
k1234567@erc-hpc-login1:/scratch/users/k1234567$ pip install jupyterlab
Collecting jupyterlab
...
Successfully built json5
...
(jvenv) k1234567@erc-hpc-login1:/scratch/users/k1234567$ deactivate

Create a script to be submitted using sbatch

#!/bin/bash -l

#SBATCH --job-name=ops-jupyter
#SBATCH --partition=cpu
#SBATCH --ntasks=1
#SBATCH --mem=2G
#SBATCH --signal=USR2
#SBATCH --cpus-per-task=1

module load python/3.11.6-gcc-13.2.0

# get unused socket per https://unix.stackexchange.com/a/132524
readonly IPADDRESS=$(hostname -I | tr ' ' '\n' | grep '10.211.4.')
readonly PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')
cat 1>&2 <<END
1. SSH tunnel from your workstation using the following command:

   Linux and MacOS:
   ssh -NL 8888:${HOSTNAME}:${PORT} ${USER}@hpc.create.kcl.ac.uk

   Windows:
   ssh -m hmac-sha2-512 -NL 8888:${HOSTNAME}:${PORT} ${USER}@hpc.create.kcl.ac.uk

   and point your web browser to http://localhost:8888/lab?token=<add the token from the jupyter output below>

When done using the notebook, terminate the job by
issuing the following command on the login node:

      scancel -f ${SLURM_JOB_ID}

END

source jvenv/bin/activate
jupyter-lab --port=${PORT} --ip=${IPADDRESS} --no-browser

printf 'notebook exited' 1>&2

Submit the script

k1234567@erc-hpc-login1:/scratch/users/k1234567$ sbatch ops-jupyter.sh

Use the connection string (and token) from the output file

Once the job has started an output file will be generated that will have the details required for you to be able to connect.

R Kernel for Jupyter

Using Singularity containers

Info

For a introductory setup on how to get started with Singularity on CREATE HPC refer to our guides here.

Singularity can be used to launch interactive Jupyter notebooks for R on CREATE HPC. The installation requirements for both R and Jupyter are available in the following Binder image provided by the Rocker Project: https://hub.docker.com/r/rocker/binder.

Setting up the container

You must first pull your required container image from DockerHub:

singularity pull docker://rocker/binder:4.3.0

This will create a Singularity image file called: binder_4.3.0.sif. It should be noted in the above example, a 4.3.0 tag was used to specify the image, this is if you wish to match the same default latest version of R that is currently available through modules on CREATE HPC.

Creating the batch script

Tip

Refer to our guide documentation here on how to bind the non-default personal library locations, that you have created for R on the CREATE HPC host, to your container.

#!/bin/bash -l

#SBATCH --job-name=ops-r-jupyter
#SBATCH --partition=cpu
#SBATCH --ntasks=1
#SBATCH --mem=2G
#SBATCH --signal=USR2
#SBATCH --cpus-per-task=1

module load python/3.11.6-gcc-13.2.0

# get unused socket per https://unix.stackexchange.com/a/132524
readonly IPADDRESS=$(hostname -I | tr ' ' '\n' | grep '10.211.4.')
readonly PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')
cat 1>&2 <<END
1. SSH tunnel from your workstation using the following command:

   Linux and MacOS:
   ssh -NL 8888:${HOSTNAME}:${PORT} ${USER}@hpc.create.kcl.ac.uk

   Windows:
   ssh -m hmac-sha2-512 -NL 8888:${HOSTNAME}:${PORT} ${USER}@hpc.create.kcl.ac.uk

   and point your web browser to http://localhost:8888/lab?token=<add the token from the jupyter output below>

When done using the notebook, terminate the job by
issuing the following command on the login node:

      scancel -f ${SLURM_JOB_ID}

END

singularity exec \
    binder_4.3.0.sif \
    jupyter-lab --ip=${IPADDRESS} --port=${PORT} --no-browser

printf 'notebook exited' 1>&2

Once you have submitted the script through sbatch, the slurm output file will have all of the information needed to launch Jupyter for R through Singularity.

Using conda environments

The following conda environment setup can be used to run Jupyter with an R kernel:

conda create -n rkernel-env -c conda-forge ipykernel r=4.2 r-essentials r-irkernel

For further instrucitons on how to set this up using a non-default cache location, saving space in your home directory, then please refer to our documentation here for conda.

Once the required environment installations have completed, and you have activated your new conda environment, you can then install a new kernel for R with the following commands:

$ conda activate rkernel-env
(rkernel-env)$ R
> IRkernel::installspec(name="rkernel-env", displayname="r/4.2")

The name and displayname set above are only examples, and can be renamed to whatever you prefer, however, please note these do help with identifying and selecting your kernel from those listed in your Jupyter. With regards to changes required in the batch script for launching jupyter, you will just need to ensure that you also load the required anaconda3 module:

module load anaconda3/2022.10-gcc-13.2.0

As well as, replace the virtual environment sourced with your new R kernel evironment:

source activate rkernel-env

Once you have submitted your updated script via sbatch and opened your Jupyter in the browser, you should then be able to select and switch to your new kernel and use R in Jupyter.