Transferring files to (and from) CREATE¶

The scp, rsync and sftp methods laid out below are for transferring limited datasets (up to ~500GB) to (and from) CREATE HPC.

Datasets over 500GB in size can be pulled from sources using an interactive session (srun) or a job submitted using squeue. See the final section on this page for further details and an alternative.

Using scp¶

One of the easiest and quickest ways to transfer files is using scp (secure copy).

It is strongly recommended that you use an ~/.ssh/config file as outlined in the relevant sections for MacOS and linux or Windows on the Accessing CREATE page.

Using the config file in the same format as the example given means that to copy a file called hello_world.sh from your local machine to your home directory on CREATE would be as simple as running the following command from your local machine in the same directory the file was in:

Linux and MacOS:

1	`scp hello_world.sh create:/users/k1234567/hello_world.sh`

Windows:

scp -o MACs=hmac-sha2-512 hello_world.sh create:/users/k1234567/hello_world.sh

To copy a file called hello_er.out from the scratch users area to the directory the shell was open in would be:

Linux and MacOS:

1	`scp create:/scratch/users/k1234567/hello_er.out ./hello_er.out`

Windows:

scp -o MACs=hmac-sha2-512 create:/scratch/users/k1234567/hello_er.out ./hello_er.out

Take note

scp has been reactivated on the login nodes, to use scp you no longer need to use erc-hpc-dm1

rsync¶

For more advanced copy and file transfer capabilities rsync is available and well documented here.

SFTP¶

It is strongly recommended that you use an ~/.ssh/config file as outlined in the relevant sections for MacOS and linux or Windows on the Accessing CREATE page.

SFTP ensures that the data you send is encrypted.

If you have created a config file as specified above then to open an SFTP connection you can simply use:

$ sftp create

else use:

$ sftp <your k-number>@hpc.create.kcl.ac.uk

To use the FileZilla GUI client see here

Larger file transfers¶

For larger file transfers an interactive session, script submitted using squeue or erc-hpc-dm1 can be used. When making large file transfers immediate sequential file operations should be avoided (like mv or rename). The file system is fast, however, the nodes can be faster and can cause issues e.g. if they try to rename something that has not been replicated to all stores ceph may continue to try to replicate the original file. When connecting to erc-hpc-dm1 use the usual password for your k-number.

k1234567@erc-hpc-login1:~$ ssh erc-hpc-dm1
k1234567@erc-hpc-dm1's password:
...
k1234567@erc-hpc-dm1:~$rsync -avHS /rosalind_ro/scratch/users/k1234567/example_project /scratch/prj/operations/

An example of using rsync to copy a directory called example_project from users k1234567 scratch space on Rosalind to a project space on CREATE for a fictional project called operations is included.