Storage¶
Our cluster offers multiple types of data storage, each designed to meet different performance, capacity, and accessibility needs.
| Storage | Path | Purpose | Visible on | Data lifetime |
|---|---|---|---|---|
| Home | ~ |
Code, configuration, important files | Login and compute nodes | Persistent |
| Project | /project/<project> |
Shared project data (on request) | Login and compute nodes | Persistent |
| Personal project | /project/<username> |
Your own large data (on request) | Login and compute nodes | Persistent |
| Memory scratch | $SCRATCH |
Very fast temporary data, up to 128 GB per node | One compute node | Deleted when the job ends |
| Local scratch | $SCRATCH |
Fast temporary data, up to 5 TB per node | One compute node | Deleted when the job ends |
| Shared scratch | $SCRATCH |
Temporary data shared by all nodes of a job, 40 TB in total | All nodes of the job | Deleted when the job ends |
Home¶
The home directory is shared across all cluster nodes and is intended for storing code, configurations, and important files that need to persist between sessions.
Every user has their own home directory under /home.
Project¶
Project storage is designed to facilitate data sharing and collaboration among users within the cluster. Each user can be assigned to one or more projects, allowing them to access shared datasets, contribute their own files, and work collaboratively with other project members.
This ensures that teams can efficiently manage and access common resources throughout the course of their research or development activities.
The /project storage contains two kinds of directories:
| Directory | Owner | Accessible to |
|---|---|---|
/project/<project> |
Project group | All members of the project |
/project/<username> |
You | Only you |
Note
Space on /project is not assigned automatically. Both a project directory and a personal directory are created only on request, see Requesting Project Storage.
Project Directory¶
A project directory belongs to the project group, so all members of the project can access the data stored there. Use it for datasets and results the whole team works with. See Sharing Data for how to set permissions on files you create.
Personal Directory¶
A personal directory /project/<username> is accessible only to you. Use it for your own large datasets and results that do not belong to any project, instead of filling your home directory.
You can check which directories you already have with:
ls -ld /project/${USER} # Your personal directory, if assigned
groups # Projects (groups) you are a member of
Requesting Project Storage¶
To get space on /project, contact the cluster administrator at jan.lostak@upol.cz.
| Request | Include |
|---|---|
| New project | Project name, research purpose, responsible person, usernames of members and estimated storage capacity |
| Joining an existing project | Project name and your username, ideally with the approval of the responsible person |
| Personal directory | Your username, purpose and estimated storage capacity |
| More space | Project name or username, current usage (du -sh) and the required capacity |
Warning
This storage is not suitable for I/O-intensive computations, as it has high latency and overhead. It is intended for long-term data storage, not active processing.
Always copy your data to a scratch directory before running calculations, and move results back after your job completes.
Scratch¶
Scratch is used for temporary files during computations and is optimized for performance, but data stored there is not backed up and will be deleted after job completion.
Scratch directories can be found on compute nodes under /scratch; inside a job, use the $SCRATCH variable.
Info
Use scratch for temporary working data, project storage for shared long-term data and home storage for essential files only.
Typical Workflow¶
A typical job works with data on all three storage types:
- Upload input data to your project directory, see Data Transfer.
- At the start of the job, copy the input data from the project directory to
$SCRATCH. - Compute in
$SCRATCH, where reading and writing is fast. - At the end of the job, copy the results back to the project directory.
#!/bin/bash
#PBS -l select=1:ncpus=8:mem=16gb:scratch_local=100gb
#PBS -l walltime=04:00:00
cp -r /project/my_project/input ${SCRATCH}/ # Stage input data to scratch
cd ${SCRATCH} # Work in scratch
module load mysoftware # Load any necessary modules
./my_program input/ results/ # Run your application
cp -r ${SCRATCH}/results /project/my_project/results_${PBS_JOBID} # Save results
Checking Disk Usage¶
To find out how much space your data take and which directories are the largest, use the du command:
du -sh ~ # Total size of your home directory
du -sh /project/my_project # Total size of a project directory
du -sh ~/* ~/.[!.]* 2>/dev/null | sort -h | tail -20 # Largest items in your home directory
Besides size, the number of files also matters. Directories with hundreds of thousands of small files, such as Conda environments or unpacked datasets, are slow to work with and to transfer. Count files in a directory with:
Tip
Pack large collections of small files you do not work with every day into an archive, e.g. tar -czf my_dataset.tar.gz my_dataset. One large archive is much faster to copy, transfer and back up than thousands of small files.
Sharing Data¶
Project directories are bound to the project group. To allow other project members to read and modify the files you created, give the group the corresponding permissions:
chmod -R g+rwX /project/my_project/shared_data # Group can read and write, enter directories
chmod g+s /project/my_project/shared_data # New files inherit the project group
ls -l /project/my_project # Check owner, group and permissions
Warning
Do not make your files writable for everyone with chmod 777 or chmod o+w. Use group permissions to share data only with members of your project.
Sharing With Individual Users¶
To share a directory with a single user who is not a member of your project, use access control lists (ACL):
setfacl -R -m u:colleague:rX /project/my_project/shared_data # Read access for user "colleague"
setfacl -R -d -m u:colleague:rX /project/my_project/shared_data # Also for files created later
getfacl /project/my_project/shared_data # Show current ACL
setfacl -R -x u:colleague /project/my_project/shared_data # Remove access again
The user also needs permission to enter all parent directories, e.g. setfacl -m u:colleague:X /project/my_project.
Warning
Before sharing data with somebody outside your project, make sure you are allowed to do so, especially in the case of sensitive or patient data.
Tip
To share data from your personal directory with your team, move or copy it into the project directory, e.g. mv /project/${USER}/results /project/my_project/, instead of changing permissions of your personal directory.
Long-term Archiving¶
The cluster storage is intended for data you actively work with. Keep at least one additional copy of irreplaceable data, such as raw sequencing data, outside the cluster, e.g. in your institutional storage or at CESNET Data Storage, which is available to Czech academic users.