Data Transfer¶

Data are transferred to and from the cluster through the login node login.hpc.imtm.cz using the SSH protocol, with the same username and private key you use to connect to the cluster.

Tool	Platform	Best for
scp	Linux, macOS, Windows	Copying individual files or directories.
rsync	Linux, macOS	Large datasets, synchronization, resuming transfers.
sftp	Linux, macOS, Windows	Interactive browsing and transfer of files.
SSHFS	Linux, macOS	Mounting cluster directories as a local disk.
WinSCP	Windows	Graphical file transfer.

Warning

Transfer data into your home or project directory, never into scratch. Scratch directories exist only while a job is running.

SCP¶

scp my_data.tar.gz your_user_name@login.hpc.imtm.cz:~/            # Upload a file to home
scp -r my_dataset your_user_name@login.hpc.imtm.cz:/project/my_project/   # Upload a directory
scp your_user_name@login.hpc.imtm.cz:~/results.tar.gz .           # Download a file

Rsync¶

Rsync transfers only files that have changed, so it is ideal for large datasets and for resuming interrupted transfers.

rsync -avP my_dataset/ your_user_name@login.hpc.imtm.cz:/project/my_project/my_dataset/   # Upload
rsync -avP your_user_name@login.hpc.imtm.cz:~/results/ ./results/                       # Download

Option	Meaning
`-a`	Archive mode, preserves permissions, times and symbolic links.
`-v`	Verbose output.
`-P`	Shows progress and keeps partially transferred files, so an interrupted transfer can be resumed.

Note

A trailing slash on the source directory (my_dataset/) copies the contents of the directory. Without it, the directory itself is created inside the destination.

Parallel Transfer¶

A single transfer stream may not use the full network bandwidth. When transferring many files, you can run several rsync processes in parallel.

cd my_dataset
ls | xargs -P 4 -I {} rsync -aP {} your_user_name@login.hpc.imtm.cz:/project/my_project/my_dataset/

This example transfers the contents of my_dataset using 4 parallel rsync processes.

Tip

Long transfers started on the login node stop when your SSH connection drops. Run them inside a tmux or screen session, which keeps running after you disconnect. Start a session with tmux, run the transfer, detach with Ctrl+B D and later reattach with tmux attach.

SFTP¶

sftp your_user_name@login.hpc.imtm.cz

sftp> put my_data.tar.gz         # Upload a file
sftp> get results.tar.gz         # Download a file
sftp> exit

SSHFS¶

SSHFS mounts a cluster directory on your local computer, so you can work with remote files as if they were local. It is convenient for browsing and editing small files, but slower than rsync for large transfers.

mkdir -p ~/hpc
sshfs your_user_name@login.hpc.imtm.cz:/home/your_user_name ~/hpc   # Mount home directory
fusermount -u ~/hpc                                                 # Unmount (Linux)
umount ~/hpc                                                        # Unmount (macOS)

Windows¶

On Windows, you can use the built-in scp and sftp commands in PowerShell, or a graphical client such as WinSCP.

In WinSCP, create a new site with the SFTP protocol, host name login.hpc.imtm.cz and your username. Under Advanced → SSH → Authentication, select your private key file. WinSCP can convert a key generated by OpenSSH to the PuTTY .ppk format automatically.

Tip

Compress many small files into a single archive before transferring them, e.g. tar -czf my_dataset.tar.gz my_dataset. Transferring one large file is much faster than thousands of small ones.

Downloading Data From the Internet¶

Both the login node and the compute nodes have access to the internet, so you can download public data directly to the cluster, without transferring it through your computer.

cd /project/my_project/reference
wget https://ftp.ensembl.org/pub/current_fasta/homo_sapiens/dna/Homo_sapiens.GRCh38.dna.primary_assembly.fa.gz
wget -c <url>                               # Continue an interrupted download

Tip

Run long downloads inside Tmux, so they continue after you disconnect. Downloads of many large files, e.g. whole sequencing projects, can also run as a job with one or two CPUs.

Sequencing Data From SRA and ENA¶

The SRA Toolkit downloads sequencing data from the NCBI Sequence Read Archive. Install it into your own Conda environment:

conda create -n sra -c conda-forge -c bioconda sra-tools
conda activate sra

prefetch SRR000001                          # Download the run
fasterq-dump --threads 4 SRR000001          # Convert it to FASTQ

Run fasterq-dump in a job or an interactive job, it needs several CPUs and creates large temporary files.

Checking Data Integrity¶

After transferring or downloading large data, check that the files were not damaged. Compare checksums of the files on both sides:

md5sum *.fastq.gz > checksums.md5           # Create checksums before the transfer
md5sum -c checksums.md5                     # Verify them after the transfer

Many public archives publish checksums of their files, e.g. MD5SUM or CHECKSUMS files, which you can check the same way.