Skip to content

TRE HPC

TRE HPC combines the security of a Trusted Research Environment with the familiar job scheduling process of SLURM and the Spack package manager. This guide explains how to load software, submit jobs, use virtual environments, and troubleshoot common issues.

Note

This is a newly developed service created by e-Research and is not the default TRE-configuration. If you believe you would benefit from a TRE HPC implementation, please email support@er.kcl.ac.uk where a member of the team will be able to guide you.

One critical difference between CREATE HPC and the TRE HPC is the use of a 'true scratch' working directory /job_scratch and a /hpc_jobs directory for job output. This can be evoked as part of your 'SBATCH' script using #SBATCH --workdir=/job_scratch. The /job_scratch directory is faster than your /home storage.

1. Loading Software Modules

TRE HPC uses the same module environment system as CREATE HPC. This allows you to load modules and packages needed for your jobs without manual builds or lengthy approval processes.

Example SBATCH script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
#!/bin/bash
#SBATCH --job-name=<name>
#SBATCH --output=<outputname>.out
#SBATCH --error=<error>.out
#SBATCH --partition=<partition_name>
#SBATCH --cpus-per-task=<n>
#SBATCH --mem=<n>
#SBATCH --time=HH:MM:SS

module load x
module load y
module load z

Warning

Do not use the -l flag in your SBATCH script; it is not required for job submission.

You can check module versions interactively or within your job script:

1
2
module load anaconda
conda --version

Note

If using a GPU, ensure you load the appropriate CUDA modules.

2. Submitting Jobs

Unlike CREATE HPC, srun and sbatch are disabled on TRE HPC. Instead, use the submit_job wrapper:

1
submit_job -p <partition_name> <sbatch_script.sh>

Resource requests and SBATCH flags are the same as on CREATE HPC. See the SLURM documentation for details.

Example:

1
submit_job -p gpu --gres=gpu:1 jobscript.sh

Monitor your jobs with:

1
squeue -u <your_username>

3. Example: Matrix Multiplication Benchmark

Create a Python script to compare CPU and GPU performance:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
import tensorflow as tf
import time

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    print(f"GPUs available: {len(gpus)}")
    for gpu in gpus:
        print(f" - {gpu}")
else:
    print("No GPU found! Using CPU instead.")

matrix_size = 10000
A = tf.random.normal([matrix_size, matrix_size])
B = tf.random.normal([matrix_size, matrix_size])

# GPU multiplication
start_time = time.time()
with tf.device('/GPU:0'):
    C = tf.matmul(A, B)
    print("Matrix multiplication on GPU completed.")
gpu_time = time.time() - start_time
print(f"Time taken on GPU: {gpu_time:.4f} seconds")

# CPU multiplication
start_time = time.time()
with tf.device('/CPU:0'):
    C = tf.matmul(A, B)
    print("Matrix multiplication on CPU completed.")
cpu_time = time.time() - start_time
print(f"Time taken on CPU: {cpu_time:.4f} seconds")

print(f"GPU is {cpu_time / gpu_time:.2f}x faster than the CPU in this operation.")

Corresponding SBATCH script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
#!/bin/bash
#SBATCH --job-name=matrix_benchmark
#SBATCH --output=matrix_benchmark.out
#SBATCH --error=matrix_benchmark.err
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=00:30:00

module load python
module load cuda
module load cudnn
module load py-tensorflow

python matrix_benchmark.py

4. Using Virtual Environments

  • Python venv module works as expected.
  • Conda environments currently do not work due to package issues.
  • pip can install packages from local and main module repositories, but not from outside PyPi.

Creating and activating a virtual environment using venv:

1
2
3
python3 -m venv ~/myenv
source ~/myenv/bin/activate
pip install --upgrade pip

Installing PyTorch with CUDA support:

1
pip install torch torchvision torchaudio

Note

If you need additional software, email support@er.kcl.ac.uk.

Proxy errors:
If you see errors like ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')), contact the e-Research team with details.

5. Example: PyTorch Benchmark with Virtualenv

Python script (pytorch_test.py):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import torch
import time

def matmul_test(device, size=2048):
    print(f"\nTesting on {device}...")
    x = torch.randn(size, size, device=device)
    y = torch.randn(size, size, device=device)
    if device == "cuda":
        torch.cuda.synchronize()
    start = time.time()
    result = torch.mm(x, y)
    if device == "cuda":
        torch.cuda.synchronize()
    end = time.time()
    print(f"Time for matrix multiplication ({size}x{size}): {end - start:.4f} seconds")
    return result

print("PyTorch version:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())

matmul_test("cpu")
if torch.cuda.is_available():
    matmul_test("cuda")
else:
    print("\nCUDA not available, skipping GPU test.")

SBATCH script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
#!/bin/bash
#SBATCH --job-name=pytorch_compare
#SBATCH --output=pytorch_compare.out
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --gres=gpu:1
#SBATCH --time=00:10:00

source ~/myenv/bin/activate
python ~/pytorch_test.py

Tip:
Run submit_job from the directory containing your scripts, or provide the correct path.

6. Troubleshooting

  • Ensure modules are loaded in your SBATCH script.
  • Check paths and permissions.
  • For proxy or package errors, contact support.

7. Support

For help with software installation or access issues, email: support@er.kcl.ac.uk

Last updated: 06/2025