Skip to content

Running Jupyter on CREATE

In order to run Jupyter from the HPC nodes it is necessary to request a batch job and then create a tunnel to the running process.

One way to do this is to use a virtual environment. This example uses virtualenv although it is also just as easy with conda.

Make Jupyter available in a virtualenv

1
2
3
4
5
6
7
8
9
k1234567@erc-hpc-login1:/scratch/users/k1234567$ module load python/3.11.6-gcc-13.2.0
k1234567@erc-hpc-login1:/scratch/users/k1234567$ virtualenv jvenv -p `which python`
k1234567@erc-hpc-login1:/scratch/users/k1234567$ source jvenv/bin/activate
k1234567@erc-hpc-login1:/scratch/users/k1234567$ pip install jupyterlab
Collecting jupyterlab
...
Successfully built json5
...
(jvenv) k1234567@erc-hpc-login1:/scratch/users/k1234567$ deactivate

Create a script to be submitted using sbatch

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#!/bin/bash -l

#SBATCH --job-name=ops-jupyter
#SBATCH --partition=cpu
#SBATCH --ntasks=1
#SBATCH --mem=2G
#SBATCH --signal=USR2
#SBATCH --cpus-per-task=1

module load python/3.11.6-gcc-13.2.0

# get unused socket per https://unix.stackexchange.com/a/132524
readonly IPADDRESS=$(hostname -I | tr ' ' '\n' | grep '10.211.4.')
readonly PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')
cat 1>&2 <<END
1. SSH tunnel from your workstation using the following command:

   Linux and MacOS:
   ssh -NL 8888:${HOSTNAME}:${PORT} ${USER}@hpc.create.kcl.ac.uk

   Windows:
   ssh -m hmac-sha2-512 -NL 8888:${HOSTNAME}:${PORT} ${USER}@hpc.create.kcl.ac.uk

   and point your web browser to http://localhost:8888/lab?token=<add the token from the jupyter output below>

When done using the notebook, terminate the job by
issuing the following command on the login node:

      scancel -f ${SLURM_JOB_ID}

END

source jvenv/bin/activate
jupyter-lab --port=${PORT} --ip=${IPADDRESS} --no-browser

printf 'notebook exited' 1>&2

Submit the script

1
k1234567@erc-hpc-login1:/scratch/users/k1234567$ sbatch ops-jupyter.sh

Use the connection string (and token) from the output file

Once the job has started an output file will be generated that will have the details required for you to be able to connect.

R Kernel for Jupyter

The following conda environment setup can be used to run Jupyter with an R kernel:

1
conda create -n rkernel-env -c conda-forge ipykernel r=4.2 r-essentials r-irkernel

For further instrucitons on how to set this up using a non-default cache location, saving space in your home directory, then please refer to our documentation here for conda.

Once the required environment installations have completed, and you have activated your new conda environment, you can then install a new kernel for R with the following commands:

1
2
3
$ conda activate rkernel-env
(rkernel-env)$ R
> IRkernel::installspec(name="rkernel-env", displayname="r/4.2")

The name and displayname set above are only examples, and can be renamed to whatever you prefer, however, please note these do help with identifying and selecting your kernel from those listed in your Jupyter. With regards to changes required in the batch script for launching jupyter, you will just need to ensure that you also load the required anaconda3 module:

1
module load anaconda3/2022.10-gcc-13.2.0

As well as, replace the virtual environment sourced with your new R kernel evironment:

1
source activate rkernel-env

Once you have submitted your updated script via sbatch and opened your Jupyter in the browser, you should then be able to select and switch to your new kernel and use R in Jupyter.