TRE HPC

TRE HPC combines the security of a Trusted Research Environment with the familiar job scheduling process of SLURM and the Spack package manager. This guide explains how to load software, submit jobs, use virtual environments, and troubleshoot common issues.

Note

CREATE TRE HPC is not part of a default TRE-configuration. If you require a TRE HPC implementation, please email support@er.kcl.ac.uk.

Running jobs on CREATE TRE HPC

1. Create a submit_job script

CREATE TRE HPC uses the same modules as CREATE HPC meaning the same packages are available.

Running a job on CREATE TRE HPC creates a temporary directory /job_scratch.

Output written to /job_scratch by jobs in CREATE TRE HPC can be found in the directory /hpc_jobs/<k-number>/job_<slurm_job_id> from the HPC virtual desktops in the TRE.

Note

The -l flag is not required for scripts submitted using submit_job to the CREATE TRE HPC.

Interactive sessions cannot be run on the CREATE TRE HPC, scripts must be created and the wrapper submit_job needs to be used in place of sbatch.

Example script

#!/bin/bash
#SBATCH --job-name=hello_tre_hpc
#SBATCH --output=/job_scratch/%x.out
#SBATCH --error=/job_scratch/error.out
#SBATCH --partition=cpu
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --time=HH:MM:SS

module load x
module load y
module load z

2. Submitting job scripts using submit_job

submit_job <sbatch_script.sh>

Resource requests and SBATCH flags are the same as on CREATE HPC. See the SLURM documentation for details.

Example

submit_job -p gpu --gres=gpu:1 jobscript.sh

Note

Slurm utilities such as squeue, sacct, scontrol and sinfo all work in the same way as on CREATE HPC.

Virtual Environments

Python venv works with a minor modification to pip install commands. pip can install packages from local and main module repositories, but not from sources other than PyPi.

Creating and activating a virtual environment using venv

python3 -m venv ~/myenv
source ~/myenv/bin/activate
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org --upgrade pip

Installing PyTorch with CUDA support

pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org torch torchvision torchaudio

Conda environments currently do not work.

Troubleshooting

• Add some basic additional output to the submission script, e.g.

module list
echo ""
echo "which python ..."
which python
exho ""
echo "python --version ..."
python --version

• Check paths and permissions.

echo "long list working directory:"
ls -lh /job_scratch/
echo ""
echo "Present working directory:"
pwd
echo ""

• Request additional support via support@er.kcl.ac.uk.