Slurm Cheatsheet for CCS Cluster

Useful commands for managing Slurm jobs and checking CCS cluster status.

Job Management

Check Queue

# Show your active/pending jobs
squeue -u $USER

# Show all jobs on the specific partition
squeue -p H8V141_SAP112M2000_L

# Estimate start time for pending jobs
squeue --start -u $USER

Control Jobs

# Cancel a specific job
scancel <JOB_ID>

# Cancel all your jobs
scancel -u $USER

# Hold a job (prevent it from starting)
scontrol hold <JOB_ID>

# Release a held job
scontrol release <JOB_ID>

# Requeue a job from the beginning
scontrol requeue <JOB_ID>

Update a pending job

Change an existing allocation request without cancelling or resubmitting. The job ID and dependencies remain unchanged.

# Check that the job is still pending
scontrol show job <JOB_ID>

# Set the total time limit to 24 hours (days-hours:minutes:seconds)
scontrol update JobId=<JOB_ID> TimeLimit=1-00:00:00

# Request four GPUs
scontrol update JobId=<JOB_ID> Gres=gpu:4

# Update both together
scontrol update JobId=<JOB_ID> Gres=gpu:4 TimeLimit=1-00:00:00

# Verify the updated request and estimated start time
scontrol show job <JOB_ID>
squeue -u "$USER" -o "%.18i %.10T %.12l %.20b %.35E"
squeue --start -u "$USER"

These commands set new limits, not extra hours. Cluster permissions and partition or QoS limits still apply. Increasing a running job's time limit may require an administrator. Start times and scheduled nodes are estimates, not guarantees.

Editing #SBATCH lines in your local script does not update an already submitted job; Slurm keeps the submitted script. Use scontrol for pending resource changes and edit the script separately for future submissions. CPU and memory requests do not automatically shrink when you reduce GPUs.

Update any external configuration before the job reads it. The launcher must use the allocated GPU count rather than a hard-coded process count. If coordinating several edits, hold the pending job first and release it after verifying the request and config.

Job arrays

An array submits many copies of the same job script. Slurm assigns each copy a different SLURM_ARRAY_TASK_ID, which the script can use to select a config, input file, or random seed. In 0-9%4, 0-9 creates ten tasks and %4 limits the array to four concurrent tasks. Common uses include parameter sweeps, dataset shards, cross-validation folds, and repeated runs with different seeds.

Submit an array

sbatch --array=0-9%4 job.sh

Use the task ID inside job.sh:

python train.py "configs/${SLURM_ARRAY_TASK_ID}.yaml"

Change parameters per task

CONFIGS=(cnn.yaml vit.yaml impala.yaml)
SEEDS=(10 20 30)
INDEX="$SLURM_ARRAY_TASK_ID"

python train.py "${CONFIGS[$INDEX]}" "training.seed=${SEEDS[$INDEX]}"

Pass a shared variable

sbatch --array=0-2 --export=ALL,DATASET=gradients_train.h5 job.sh

Inside job.sh:

python train.py "${CONFIGS[$SLURM_ARRAY_TASK_ID]}" "data.h5_path=$DATASET"

Choose task ranges

sbatch --array=0-9 job.sh      # Tasks 0 through 9
sbatch --array=1,4,8 job.sh    # Only tasks 1, 4, and 8
sbatch --array=0-20:5 job.sh   # Tasks 0, 5, 10, 15, and 20
sbatch --array=0-20%4 job.sh   # At most four concurrent tasks

Separate task logs

%A is the array job ID and %a is the task ID.

#SBATCH --output=logs/%A_%a.out

Cancel array tasks

scancel <ARRAY_JOB_ID>_<TASK_ID>
scancel <ARRAY_JOB_ID>

Array tasks share the same CPU, GPU, memory, and time requests. Use separate arrays when runs need different resources.

Dependent jobs

# Submit only after the first job succeeds
FIRST_JOB=$(sbatch --parsable first.sh)
SECOND_JOB=$(sbatch --parsable --dependency="afterok:${FIRST_JOB}" second.sh)

# Wait for two jobs
sbatch --dependency="afterok:${JOB_A}:${JOB_B}" next.sh

# Run after completion regardless of success or failure
sbatch --dependency="afterany:${JOB_ID}" cleanup.sh

# Run only if the dependency fails
sbatch --dependency="afternotok:${JOB_ID}" recover.sh

Job details and logs

Check before submission

Validate the request and estimate when it could start without submitting it:

sbatch --test-only job.sh

Inspect pending and running jobs

# Show requested resources, dependencies, and pending reason
scontrol show job <JOB_ID>

# Show pending reason and estimated priority
squeue -j <JOB_ID> -o "%.18i %.10T %.30R"
sprio -j <JOB_ID>

# Show live CPU and memory use
sstat -j <JOB_ID>.batch --format=JobID,AveCPU,MaxRSS,MaxVMSize

# Compact queue view
squeue -u "$USER" -o "%.18i %.24j %.10T %.12M %.30R"

Review job history

# One job
sacct -j <JOB_ID> --format=JobID,State,Elapsed,ExitCode,MaxRSS,AllocTRES

# All jobs submitted today
sacct -u "$USER" -S today \
  --format=JobID,JobName,Partition,State,Elapsed,ExitCode,MaxRSS

Inspect scripts and logs

# Print the submitted batch script
scontrol write batch_script <JOB_ID> -

# Follow output and error logs
tail -f logs/out_<JOB_ID>.log
tail -f logs/err_<JOB_ID>.log

Common environment variables

# Show every Slurm variable available to the job
env | sort | grep '^SLURM_'

# Common examples
echo "$SLURM_JOB_ID"
echo "$SLURM_JOB_NAME"
echo "$SLURM_SUBMIT_DIR"
echo "$SLURM_JOB_NODELIST"
echo "$SLURM_CPUS_PER_TASK"
echo "$SLURM_ARRAY_JOB_ID"
echo "$SLURM_ARRAY_TASK_ID"

Node & Resource Status

Check Node Details

# Show detailed status of a specific node (e.g., ghnode001)
scontrol show node <NODE_NAME>
# Look for 'AllocTRES' to see used resources vs 'CfgTRES' for total.

Check Partition Status

# Summary of all partitions
sinfo

# Partition, availability, time limit, nodes, state, CPUs, memory, and GPUs
sinfo -o "%P %a %l %D %t %C %m %G"

# Detailed node list for the partition
scontrol show partition H8V141_SAP112M2000_L

Interactive Jobs

Start an Interactive Session

# Request an interactive shell on a compute node (adjust time/resources as needed)
salloc --partition=H8V141_SAP112M2000_L --gres=gpu:1 --time=1:00:00
# Once allocated, run 'srun --pty bash' to get the shell

Enter an existing allocation

srun --jobid=<JOB_ID> --pty bash

Troubleshooting

Inspect Running Job

# SSH into the node running your job (find Nodelist from squeue)
ssh <NODE_NAME>

# Check GPU usage on the node
nvidia-smi

# Check process status
top -u $USER