Cluster Specifications #

Overview #

Compute nodes

43 (of which 9 are communal)

Logical cores

3024 (of which 680 are communal)

RAM

32-1024 GiB/node

Local scratch

0.05-11 TiB/node

Global scratch

240 TiB

User home storage

1 TiB/user (196 TiB in total)

Summary of Compute Environment #

All nodes on the cluster runs Rocky 8 which is updated on a regular basis. The job scheduler is Slurm 21.08 (Simple Linux Utility for Resource Management) which provides partitions for both communal and lab-priority tasks.

Details #

Login Nodes #

The cluster can be accessed via SSH to one of two login nodes:

1. c4-log1.ucsf.edu
2. c4-log2.ucsf.edu

Data Transfer Nodes #

For transferring large data files, it is recommended to use one of the dedicate data transfer nodes:

1. c4-dt1.ucsf.edu
2. c4-dt2.ucsf.edu

which have 10 Gbps connection - providing a file transfer speed of up to (theoretical) 1.25 GB/s = 4.5 TB/h. Please note, as is true for all ethernet, 80% of line speed is doing pretty good. As for the login nodes, the transfer nodes can be accessed via SSH.

Comment: You can also transfer data via the login nodes, but since those only have 1 Gbps connections, you will see much lower transfer rates.

Development Nodes #

The cluster has development nodes for the purpose of validating scripts, prototyping pipelines, compiling software, and more. Development nodes can be accessed from the login nodes.

Comment: Please use the GPU development node only if you need to build or prototype GPU software. The CPU x86-64 level is the x86-64 microarchitecture levels supported by the nodes CPU.

Compute Nodes #

The majority of the compute nodes have Intel processors, while a few have AMD processes. Each compute node has a local /scratch drive (see below for size), which is either a hard disk drive (HDD), a solid state drive (SSD), or a Non-Volatile Memory Express (NVMe) drive. Each node has a tiny /tmp drive (8-8 GiB).

The compute nodes can only be utilized by submitting jobs via the scheduler.

GPU Nodes #

GPU Usage Guide — A100 MIG Configuration on c4-n38 and c4-n39 #

Overview #

The servers c4-n38 and c4-n39 each contain a single NVIDIA A100 80GB PCIe GPU.
Each GPU is currently configured into seven MIG (Multi-Instance GPU) devices, each providing roughly 10 GB of GPU memory.

This configuration allows multiple GPU jobs to run concurrently on the same GPU, improving utilization and keeping workloads isolated from one another.

What Is MIG (Multi-Instance GPU)? #

MIG, or Multi-Instance GPU, is a feature of NVIDIA’s A100 architecture.
It allows a single GPU to be divided into multiple isolated GPU “instances.”
Each instance behaves like an independent GPU with its own:

• Dedicated HBM2 memory
• Compute cores (SMs)
• L2 cache and memory bandwidth
• Scheduler and power/thermal controls

As a result, multiple users or jobs can safely share one GPU without interfering with each other.

Why We Use 7 MIG Instances #

Each A100 80 GB GPU is split into 7 × 1g.10gb instances:

• Each instance provides about 10 GB of GPU RAM.
• Together they use the full 80 GB of VRAM.

Benefits #

1. Better Utilization
   Without MIG, only a single process can occupy the GPU—even if it uses a small portion of its capacity.
   With MIG, up to seven smaller jobs can run at once, keeping the GPU busy.

2. Isolation
   Each MIG device is hardware-isolated.
   A crash or memory overflow in one instance does not affect others.

3. Good Fit for Current Workloads
   Most research and inference tasks use less than 10 GB of GPU memory, so this layout allows multiple experiments to run simultaneously.

4. Flexibility
   The configuration can be changed if a project requires a larger GPU slice.
   You can request the full 80 GB GPU from the system administrator if needed.

Checking GPU and MIG Usage #

Run:

$ nvidia-smi 
Thu Oct 16 07:14:58 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100 80GB PCIe          On  |   00000000:CA:00.0 Off |                   On |
| N/A   67C    P0            183W /  300W |   45269MiB /  81920MiB |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| MIG devices:                                                                            |
+------------------+----------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |                     Memory-Usage |        Vol|        Shared         |
|      ID  ID  Dev |                       BAR1-Usage | SM     Unc| CE ENC  DEC  OFA  JPG |
|                  |                                  |        ECC|                       |
|==================+==================================+===========+=======================|
|  0    7   0   0  |              13MiB /  9728MiB    | 14      0 |  1   0    0    0    0 |
|                  |                 0MiB / 16383MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0    8   0   1  |            7548MiB /  9728MiB    | 14      0 |  1   0    0    0    0 |
|                  |                 2MiB / 16383MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0    9   0   2  |            7548MiB /  9728MiB    | 14      0 |  1   0    0    0    0 |
|                  |                 2MiB / 16383MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0   11   0   3  |            7540MiB /  9728MiB    | 14      0 |  1   0    0    0    0 |
|                  |                 2MiB / 16383MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0   12   0   4  |            7540MiB /  9728MiB    | 14      0 |  1   0    0    0    0 |
|                  |                 2MiB / 16383MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0   13   0   5  |            7540MiB /  9728MiB    | 14      0 |  1   0    0    0    0 |
|                  |                 2MiB / 16383MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0   14   0   6  |            7540MiB /  9728MiB    | 14      0 |  1   0    0    0    0 |
|                  |                 2MiB / 16383MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0    8    0    3479490      C   python                                       7528MiB |
|    0    9    0    3479493      C   python                                       7528MiB |
|    0   11    0    3479524      C   python                                       7520MiB |
|    0   12    0    3479535      C   python                                       7520MiB |
|    0   13    0    3479546      C   python                                       7520MiB |
|    0   14    0    3479557      C   python                                       7520MiB |
+-----------------------------------------------------------------------------------------+

File System #

Scratch Storage #

The C4 cluster provides two types of scratch storage:

• Local /scratch/ - 0.05-11 TiB/node storage unique to each compute node (can only be accessed from the specific compute node).
• Global /c4/scratch/ - 240 TiB storage accessible from everywhere.

There are no per-user quotas in these scratch spaces. Files not added or modified during the last two weeks will be automatically deleted on a nightly basis. Note, files with old timestamps that were “added” to the scratch place during this period will not be deleted, which covers the use case where files with old timestamps are extracted from tar.gz file. (Details: tmpwatch --ctime --dirmtime --all --force is used for the cleanup.)

User and Lab Storage #

• /c4/home: 196 TiB storage space

Each user may use up to 1 TiB disk space in the home directory. It is not possible to expand user’s home directory. Many labs have purchased their own storage servers which were then mounted on the cluster. If you’d like more information please contact Adam Olshen or Harry Putnam.

Network #

The compute nodes are connected to the local network with 1 Gbps and 10 Gbps network cards.