Examples not working?

Using Local /scratch (TMPDIR) on Compute Nodes #

All nodes (compute and development) have their own locally storage mounted as /scratch. The /scratch storage is fast - faster than system-wide storage such as home folders but also /c4/scratch - which make it ideal for holding intermediate data files. This will also lower the load on the system-wide storage and the local network. Using local /scratch is a win-win for everyone.

Instructions #

Here is how to use /scratch:

Use environment variable TMPDIR - on compute nodes, it points to an already created job-specific folder under local /scratch. Specifically, for each job it is /scratch/$SLURM_JOB_USER/$SLURM_JOB_ID On other machines, you need to set it.
Write intermediate files to folder $TMPDIR. Bonus: most software already acknowledges TMPDIR for their internal temporary files.
When done, make sure to copy TMPDIR files to be kept to your persistent storage, e.g. to $HOME.
The job-specific TMPDIR folder (e.g. /scratch/alice/1012) will be deleted automatically when the job terminates.
Specify how much local scratch (TMPDIR) storage your job will need. Local storage is limited to 0.05-11 TiB/node. If your job will use up to 200 GiB of disk space, you can specify this resource as --gres=scratch:200G (in units of GiB) when submitting the job. A node with 800 GiB of scratch space can support up to four --gres=scratch:200G jobs running at the same time.

Example #

Here is a script called ex-scratch.sh that requests 300 GiB of local /scratch space and 4 cores. The script also shows how to copy input files over from the home (= ~/) folder to the local scratch folder (TMPDIR) of whatever node the job ends up running on. After processing of the input files is complete, the output files are moved from the local scratch (TMPDIR) to HOME.

#!/bin/env bash
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --gres=scratch:300G

## 0. In case TMPDIR is not set, e.g. on development nodes, set
##    it to local /scratch, if it exists, otherwise to /tmp
if [[ -z "$TMPDIR" ]]; then
  if [[ -d /scratch ]]; then TMPDIR=/scratch/$USER; else TMPDIR=/tmp/$USER; fi
  mkdir -p "$TMPDIR"
  export TMPDIR
fi

## 1. Use a temporary working directory
cd "$TMPDIR"

## 2. Copy input files from global disk to local scratch
cp ~/sample.fq .
cp ~/reference.fa .

## 3. Process input files
/path/to/my_pipeline --cores=${SLURM_NTASKS:-1} reference.fa sample.fq > output.bam

## 4. Move output files back to global disk
mv output.bam ~