Compute Nodes¶

The cluster has 17 compute nodes in two classes. All nodes are dual-socket servers with AMD EPYC 9004 (Zen 4) processors, so a program compiled for one class runs on the other as well.

Node class	Nodes	Hostnames	Processors	Cores / threads	RAM
Density	12	`pbs-mom1` – `pbs-mom12`	2 × AMD EPYC 9654	192 / 384	512 GiB
Frequency	5	`pbs-mom13` – `pbs-mom17`	2 × AMD EPYC 9474F	96 / 192	768 GiB

Note

Part of the resources of each node is reserved for the operating system. The amount of CPUs, memory and scratch available to jobs is listed in Node Resources.

Node Classes¶

Density¶

Density nodes provide the highest number of cores per node. They are the best choice for well-parallelized workloads, MPI applications and capacity computing with many independent tasks.

2 × AMD EPYC 9654, 96 cores each, 192 cores per node
512 GiB RAM per node, about 2.7 GiB per core
Local NVMe scratch

Frequency¶

Frequency nodes have fewer cores with higher clock speed and more memory per core. They are the best choice for applications that use only a few threads, have long serial parts, or need a lot of memory.

2 × AMD EPYC 9474F, 48 cores each, 96 cores per node
768 GiB RAM per node, 8 GiB per core
Local NVMe scratch

Tip

By default, the scheduler places your job on any node with enough free resources. A chunk requesting more memory than a Density node provides can start only on Frequency nodes, and a chunk requesting more CPUs than a Frequency node provides can start only on Density nodes. See Node Resources for the exact limits.

Processor Architecture¶

Both processor models are based on the AMD Zen 4 (Genoa) architecture and support the same instruction set, including AVX-512.

Parameter	AMD EPYC 9654	AMD EPYC 9474F
Used in	Density nodes	Frequency nodes
Cores / threads	96 / 192	48 / 96
Base frequency	2.4 GHz	3.6 GHz
Max boost frequency	up to 3.7 GHz	up to 4.1 GHz
L3 cache	384 MB	256 MB
Memory channels	12 × DDR5-4800	12 × DDR5-4800
FP64 FLOPs per cycle	16 per core	16 per core
Instruction sets	x86-64, AVX2, AVX-512 (incl. VNNI, BF16)	x86-64, AVX2, AVX-512 (incl. VNNI, BF16)

Tip

To get the best performance, compile your programs with -march=znver4, which enables all instructions of these processors. See Compiling Software.

Hardware Threads¶

Each physical core runs two hardware threads (simultaneous multithreading). The scheduler counts threads, so ncpus=8 gives your job 4 physical cores with 2 threads each.

Compute-intensive applications, e.g. linear algebra or molecular dynamics, often do not run faster with two threads per core. For such applications, you can start only one thread per physical core:

#PBS -l select=1:ncpus=16:mem=32gb          # 8 physical cores, 16 threads

export OMP_NUM_THREADS=$((NCPUS / 2))       # One thread per physical core
export OMP_PLACES=cores                     # Bind each thread to its own core
export OMP_PROC_BIND=close

Note

Whether your application benefits from hardware threads depends on the application. Run a short test with both settings and compare the walltime.

Running On a Specific Node¶

You can see the state and free resources of all nodes with:

pbsnodes -aSj

In rare cases, e.g. for benchmarks, you may need to run your job on a specific node. Use the host resource in the chunk:

#PBS -l select=1:ncpus=96:mem=256gb:host=pbs-mom13

Warning

Requesting a specific node usually makes your job wait longer, because it cannot start anywhere else. Do not use it for regular jobs.