Transferring files to (and from) CREATE¶
The scp, rsync and sftp methods laid out below are for transferring limited datasets (up to ~500GB) to (and from) CREATE HPC.
Datasets over 500GB in size can be pulled from sources using an interactive session (srun) or a job submitted using squeue. See the final section on this page for further details and an alternative.
One of the easiest and quickest ways to transfer files is using scp (secure copy).
Using the config file in the same format as the example given means that to copy a file called hello_world.sh from your local machine to your home directory on CREATE would be as simple as running the following command from your local machine in the same directory the file was in:
Linux and MacOS:
To copy a file called hello_er.out from the scratch users area to the directory the shell was open in would be:
Linux and MacOS:
scp has been reactivated on the login nodes, to use
scp you no longer need to use
For more advanced copy and file transfer capabilities rsync is available and well documented here.
SFTP ensures that the data you send is encrypted.
If you have created a config file as specified above then to open an SFTP connection you can simply use:
$ sftp create
$ sftp <your k-number>@hpc.create.kcl.ac.uk
To use the FileZilla GUI client see here
Larger file transfers¶
For larger file transfers an interactive session, script submitted using squeue or erc-hpc-dm1 can be used. When making large file transfers immediate sequential file operations should be avoided (like mv or rename). The file system is fast, however, the nodes can be faster and can cause issues e.g. if they try to rename something that has not been replicated to all stores ceph may continue to try to replicate the original file. When connecting to erc-hpc-dm1 use the usual password for your k-number.
1 2 3 4
An example of using rsync to copy a directory called example_project from users k1234567 scratch space on Rosalind to a project space on CREATE for a fictional project called operations is included.