Nextflow¶

Nextflow is a workflow manager used by many bioinformatics pipelines, including the community pipelines of nf-core. Nextflow can submit every step of a pipeline as a separate PBS job, and runs the tools inside Singularity containers or Conda environments.

Everything on this page is installed and configured in your own home or project directory.

Installation¶

Nextflow is available as a module, which already includes Java:

module load Nextflow
nextflow -version

If you need a different version of Nextflow, or the nf-core tools for downloading and managing nf-core pipelines, install them into your own Conda environment:

source ${HOME}/miniforge3/etc/profile.d/conda.sh
conda create -n nextflow -c conda-forge -c bioconda nextflow nf-core
conda activate nextflow
nextflow -version

Configuration¶

Create a configuration file ~/.nextflow/imtm.config, which tells Nextflow to submit jobs to PBS and to use Singularity containers:

process {
    executor = 'pbspro'
    cpus     = 1
    memory   = '2 GB'
    time     = '1h'
}

executor {
    queueSize    = 50                       // Maximum number of jobs in the queue at the same time
    pollInterval = '30 sec'
    submitRateLimit = '10/1min'             // Do not flood the scheduler
}

singularity {
    enabled    = true
    autoMounts = true
    cacheDir   = "${System.getenv('HOME')}/singularity/nextflow"
}

Nextflow translates cpus, memory and time of each process into select=1:ncpus=...:mem=... and walltime of the PBS job.

Tip

Container images of nf-core pipelines can take tens of gigabytes. If you have a project directory, set cacheDir there, e.g. /project/my_project/singularity, so the whole team shares one copy of the images.

Running a Pipeline¶

The main Nextflow process only plans and submits jobs, so it can run on the login node inside Tmux, which keeps it running after you disconnect:

tmux new -s nf
module load Nextflow                        # Or: conda activate nextflow
export NXF_OPTS="-Xms512m -Xmx2g"          # Limit memory of the main Nextflow process

nextflow run nf-core/rnaseq \
    -r 3.14.0 \
    -profile singularity \
    -c ~/.nextflow/imtm.config \
    -work-dir /project/my_project/nf-work \
    --input samplesheet.csv \
    --outdir /project/my_project/rnaseq-results \
    --genome GRCh38 \
    -resume

Option	Meaning
`-r`	Version of the pipeline. Always set it, so your results are reproducible.
`-profile`	Predefined profile of the pipeline, here run tools in Singularity containers.
`-c`	Your cluster configuration from above.
`-work-dir`	Directory for intermediate files of all steps.
`-resume`	Reuse results of steps that already finished, e.g. after a failure.

Tip

qsub also works on compute nodes, so instead of tmux you can run the main Nextflow process in a small job with 1 to 2 CPUs, 4 GB of memory and a walltime long enough for the whole pipeline.

Detach from tmux with Ctrl+B D and check the pipeline later with tmux attach -t nf. Jobs submitted by Nextflow are named nf-<process> and you can see them with qstat -u $USER.

Limiting Resources of Pipelines¶

nf-core pipelines set default resources for each step, which can be higher than what a node provides. Limit them in your configuration file to fit the cluster, see Node Resources:

process {
    resourceLimits = [ cpus: 64, memory: 500.GB, time: 72.h ]
}

To change resources of a single step, use withName:

process {
    withName: 'STAR_ALIGN' {
        cpus   = 32
        memory = '96 GB'
    }
}

Cleaning Up¶

The work directory contains copies of all intermediate files and quickly grows to a large size. After the pipeline has finished successfully and you have checked the results, remove it:

nextflow clean -f                           # Remove work files of the last run
rm -rf /project/my_project/nf-work          # Or remove the whole work directory

Warning

Do not delete the work directory while you may still need to -resume the pipeline. Without it, Nextflow has to run all steps again.