Flatiron Cluster Intro

Lehman Garrison
Scientific Computing Core, Flatiron Institute

CCA Fluid Dynamics Summer School

July 31, 2023

https://slides.com/d/jGAavrM/live

https://slides.com/lehmangarrison/flatiron-cluster-intro-summer-2023

Intro to rusty

• This week you'll be logging into the "rusty" cluster
• A partition of nodes has been set aside for you!
  • 24 x Intel Skylake nodes, 40 cores each, 768 GB RAM
  • Each user can allocate 10 CPUs
  • Submit to the "temp" partition—we'll see how shortly
• Two filesystems:
  • Home ($HOME): keep code, scripts, git repos, etc, here
  • Ceph ($HOME/ceph): keep data here
  • Storage will be purged on Saturday
• First, let's set up your accounts

https://wiki.flatironinstitute.org/SCC/Hardware/Rusty

rusty Account Setup

rusty Account Setup

1. Find the FIDO email with your login link
   • Email is from fido@flatironinstitute.org
2. Use the PIN from the spreadsheet to login
   • Lehman will show the spreadsheet now
3. Set a new password
4. On FIDO, click "Retrieve google-authenticator backup code". Copy the code.
5. SSH into the cluster
   • ssh -p 61022 USERNAME@gateway.flatironinstitute.org
   • Use password and backup code
6. Run the following to generate a QR code

   • google-authenticator -td -w3 -r3 -R30 -f

   • Scan the QR code with Google Authenticator (or any authenticator app)

7. Done! Try closing your terminal and logging in again.

Instructions on Wiki

Intro to Slurm

• You'll submit jobs to the cluster using Slurm
• Let's try it now! First SSH to rusty from gateway:
  • ssh rusty
• Create a file called hello.sbatch
• Submit it with:
  • sbatch hello.sbatch
• Check the status with:
  • squeue --me
• Look for a file named
  • "slurm-<JODIB>.out"

#!/bin/bash
#SBATCH -p temp  # partition

echo Hello from $(hostname)

hello.sbatch:

Wiki info on Slurm

Intro to Slurm (2)

• Take a look at the more realistic example on the right
• How many times will you see "Hello from $(hostname)" in the output?
• Your instructors will give you Slurm scripts during the classes
• Remember, each summer school user uses the "temp" partition and is limited to 10 cores

#!/bin/bash
#SBATCH -p temp  # partition
#SBATCH -t 0-1   # 1 hour

#SBATCH -n 5     # 5 tasks

#SBATCH -c 2     # 2 cores/task

#SBATCH --mem-per-cpu 16G

echo Hello from $(hostname)

parallel.sbatch:

Wiki info on Slurm

JupyterHub

• Easiest way to run Jupyter notebooks on the cluster is through JupyterHub
  • jupyter.flatironinstitute.org
• Use the "temp" partition

Wiki info on JupyterHub

Create a Jupyter Kernel

To install Python packages and make them visible to Jupyter, need to create a venv and custom kernel

Wiki info on JupyterHub

module load python
python -m venv fluidenv --system-site-packages
source fluidenv/bin/activate

1. Create a venv:

module load jupyter-kernels
python -m make-custom-kernel fluid

2. Create a kernel:

3. Open a notebook on JupyterHub and select the fluid kernel

4. In the future, to install a package, run:

source fluidenv/bin/activate
pip install ...

rusty Cheat Sheet

• Connecting
  • Use "-p 61022" when SSH-ing to gateway to select the right port
  • Use "ssh rusty" from gateway
• Storage
  • Scripts and code go in your home directory ($HOME)
  • Data goes in ceph ($HOME/ceph)
• Slurm
  • Submit to the "temp" partition with "-p temp"
  • Ask for <= 10 cores
• Jupyter
  • Launch Jupyter sessions from jupyter.flatironinstitute.org
  • Activate your venv before installing pip packages
• Have fun!