Running Jupyter on CREATE
In order to run Jupyter from the HPC nodes it is necessary to request a batch job and then create a tunnel to the running process.
One way to do this is to use a virtual environment. This example uses virtualenv although it is also just as easy with conda.
Make Jupyter available in a virtualenv
| k1234567@erc-hpc-login1:/scratch/users/k1234567$ module load python/3.8.12-gcc-9.4.0
k1234567@erc-hpc-login1:/scratch/users/k1234567$ virtualenv jvenv -p `which python`
k1234567@erc-hpc-login1:/scratch/users/k1234567$ source jvenv/bin/activate
k1234567@erc-hpc-login1:/scratch/users/k1234567$ pip install jupyterlab
Collecting jupyterlab
...
Successfully built json5
...
(jvenv) k1234567@erc-hpc-login1:/scratch/users/k1234567$ deactivate
|
Create a script to be submitted using sbatch
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32 | #!/bin/bash -l
#SBATCH --job-name=ops-jupyter
#SBATCH --partition=cpu
#SBATCH --ntasks=1
#SBATCH --mem=2G
#SBATCH --signal=USR2
#SBATCH --cpus-per-task=1
module load python/3.8.12-gcc-9.4.0
# get unused socket per https://unix.stackexchange.com/a/132524
readonly IPADDRESS=$(hostname -I | tr ' ' '\n' | grep '10.211.4.')
readonly PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')
cat 1>&2 <<END
1. SSH tunnel from your workstation using the following command:
ssh -NL 8888:${HOSTNAME}:${PORT} ${USER}@hpc.create.kcl.ac.uk
and point your web browser to http://localhost:8888/lab?token=<add the token from the jupyter output below>
When done using the notebook, terminate the job by
issuing the following command on the login node:
scancel -f ${SLURM_JOB_ID}
END
source jvenv/bin/activate
jupyter-lab --port=${PORT} --ip=${IPADDRESS} --no-browser
printf 'notebook exited' 1>&2
|
Submit the script
| k1234567@erc-hpc-login1:/scratch/users/k1234567$ sbatch ops-jupyter.sh
|
Use the connection string (and token) from the output file
Once the job has started an output file will be generated that will have the details required for you to be able to connect.