To set up password-free file transfers, see Log in without Password. Then there is also no need to specify your cluster username.

Transfer Files and Directories #

For storage mounted directly on the C4 cluster, files can be transferred to and from the cluster using regular file copying, e.g. cp and rsync. For non-mounted storage, files may be transferred to and from the cluster via secure copying, e.g. scp and sftp, which can be utilized via rsync. For file downloads from online resources, tools such as curl, wget, and rsync may be used.

Compute nodes have only HTTP/HTTPS internet access via a proxy server, i.e. they cannot be used for file transfers other than between mount points. Although file transfers can also be done via the login nodes, it is recommended to use one of the dedicated data transfer nodes, which provides ten times faster transfer rates.

TIPS: Try to use scp -c aes128-gcm@openssh.com ... to speed up the transfer rates. There can be a fair bit of CPU overhead when SCP encrypts the data before transferring it - this option makes it use a faster encryption cipher.

From your local machine to the C4 file system #

Copy a single file on your local file system to your C4 home directory #

To copy a single file to your home directory (~/) on the cluster, use

{local}$ scp one_file.tsv alice@c4-dt1.ucsf.edu:~/

Copy one or more files to a folder on the cluster #

To copy multiple files to C4 so they appear directly under ~/study/files/, use

{local}$ scp *.txt *.R ../some/path/another_file.tsv alice@c4-dt1.ucsf.edu:study/files/

Recursively copy a directory to a folder on the cluster #

To copy all content of directory dataset/ to C4 so that it appears as ~/study/dataset/, use

{local}$ scp -r dataset/ alice@c4-dt1.ucsf.edu:study/

From the C4 file system to your local machine #

Copy a single file from your C4 home directory to your local machine #

To copy a single file in your C4 home directory to the working directory of your local machine, use

{local}$ cd /path/project
{local}$ scp alice@c4-dt1.ucsf.edu:one_file.tsv .

Note: Don’t forget that period (.) at the end - it indicates copy [the file] “to the current directory”.

Copy one or more files from the cluster #

To copy multiple files from ~/study/files/ on the cluster to ~/study/ on your local machine, do:

{local}$ scp alice@c4-dt1.ucsf.edu:study/files/*.txt alice@c4-dt1.ucsf.edu:study/files/*.R ~/study/

Recursively copy a folder from the cluster #

To copy all content of directory dataset/ on the cluster so that it appears as dataset/ in your local working directory, use

{local}$ cd /path/project
{local}$ scp -r alice@c4-dt1.ucsf.edu:dataset/ .

Note: Don’t forget that period (.) at the end - it indicates copy [the folder] “to the current directory”.