Cron

@SudoNeuroSci

Learning Objectives

  • cron syntax
  • use cron on argon
  • tip/tricks for cron

Cron: Origins

  • CReate ON utility
  • Command Run On Notice
  • Command Run Over Night
  • Chronos (greek god)
  • Chron (greek prefix for time)

Cron: Origins

  • Download data from XNAT
  • Run mriqc
  • Run fmriprep
  • Update website

Cron: use cases

Download data from XNAT

Cron: use cases

0 * * * * /home/jdkent/cronScripts/xnat_download.sh

crontab (work computer)

#!/usr/bin/env bash

docker run --rm \
-v /home/jdkent:/config \
-v /home/jdkent/mnt/vosslabhpc/Projects/BETTER/3-Experiment/2-data/bids/sourcedata/code:/json \
-v /home/jdkent/mnt/vosslabhpc/Projects/BETTER/3-Experiment/2-data/bids:/out \
jdkent/xnat_downloader:v0.2.2 -c /config/xnat_config.json -i /json/better.json

xnat_download.sh

Download data from XNAT

Cron: use cases

Run mriqc

Cron: use cases

55 * * * * /Shared/vosslabhpc/Projects/BETTER/3-Experiment/2-data/bids/derivatives/code/mriqc/cron_scripts/cron_mriqc.sh

crontab (argon)

#!/usr/bin/env bash

source ~/.bash_profile
SourceData=/Shared/vosslabhpc/Projects/BETTER/3-Experiment/2-data/bids
SourceJobs=${SourceData}/derivatives/code/mriqc/job_scripts
Template=${SourceJobs}/TEMPLATE_mriqc.job

# gets sub-whatever_ses-whatever
subject_nii=$(echo ${SourceData}/sub-*/ses-* | egrep -o 'sub-[A-Za-z0-9]+/ses-[A-Za-z0-9]+' | sed -e "s|/|_|" | sort)

# should be listed as sub-whatever_ses-whatever.job
subject_job=$(echo ${SourceJobs}/sub-*_ses-* | xargs -d' ' -I {} basename {} | sed -e "s|.job||" | sort)

run_list=($(comm -23 <(printf '%s\n' ${subject_nii[@]}) <(printf '%s\n' ${subject_job[@]})))

for subject_session in ${run_list[@]}; do
    subject=$(echo ${subject_session} | awk -F "_" '{print $1}' | sed -e "s|sub-||")
    session=$(echo ${subject_session} | awk -F "_" '{print $2}' | sed -e "s|ses-||")
    sed -e "s/SUBJECT/${subject}/g" -e "s/SESSION/${session}/g" ${Template} > ${SourceJobs}/sub-${subject}_ses-${session}.job
    hold_jid=$(qsub -terse ${SourceJobs}/sub-${subject}_ses-${session}.job)
done

if [ ! -z ${hold_jid} ]; then
   qsub -hold_jid ${hold_jid} ${SourceJobs}/group_mriqc.job
fi

cron_mriqc.sh

Run mriqc

Cron: use cases

Run mriqc

Cron: use cases

#!/bin/sh

#$ -pe smp 16
#$ -q UI
#$ -m bea
#$ -M james-kent@uiowa.edu
#$ -o /Shared/vosslabhpc/Projects/BETTER/3-Experiment/2-data/bids/derivatives/code/mriqc/out
#$ -e /Shared/vosslabhpc/Projects/BETTER/3-Experiment/2-data/bids/derivatives/code/mriqc/err
OMP_NUM_THREADS=10
singularity run -H ${HOME}/singularity_home -B /Shared/vosslabhpc:/mnt \
/Shared/vosslabhpc/UniversalSoftware/SingularityContainers/poldracklab_mriqc_0.10.4-2018-03-23-26b58c18952f.img \
/mnt/Projects/BETTER/3-Experiment/2-data/bids/ /mnt/Projects/BETTER/3-Experiment/2-data/bids/derivatives/mriqc \
participant --participant_label SUBJECT --session-id SESSION \
-w /mnt/Projects/BETTER/3-Experiment/2-data/bids/derivatives/work/mriqc \
--n_procs 10 --mem_gb 35 --write-graph \
--fd_thres 0.5 --start-idx 4

TEMPLATE_mriqc.job

Run fmriprep

Cron: use cases

55 * * * * /Shared/vosslabhpc/Projects/BETTER/3-Experiment/2-data/bids/derivatives/code/fmriprep/cron_scripts/cron_fmriprep.sh

crontab (argon)

Run fmriprep

Cron: use cases

#!/usr/bin/env bash

source ~/.bash_profile
SourceData=/Shared/vosslabhpc/Projects/BETTER/3-Experiment/2-data/bids
SourceJobs=${SourceData}/derivatives/code/fmriprep/job_scripts
Template=${SourceJobs}/TEMPLATE_fmriprep.job

subject_nii=$(echo ${SourceData}/sub-*/ses-* | xargs -d' ' -I {} dirname {} | xargs -I {} basename {} | awk -F"-" '{print $2}' | grep -v test | grep -v dallas | sort | uniq -d)
# who's in list?
# echo contents of array in bash for i in ${subject_nii[@]}; do echo $i; done

subject_job=$(find ${SourceJobs} -mindepth 1 -maxdepth 1 -name "sub-*" | xargs -I {} basename {} | awk -F"-" '{print $2}' | sed -e "s|.job||" | sort)

# make a list of subject_nii who don't have job files yet
run_list=($(comm -23 <(printf '%s\n' ${subject_nii[@]}) <(printf '%s\n' ${subject_job[@]})))

for subject in ${run_list[@]}; do
    # use sed stream editor to search and replace template job with subjects in the run_list
    sed -e "s/SUBJECT/${subject}/g" ${Template} > ${SourceJobs}/sub-${subject}.job
    qsub ${SourceJobs}/sub-${subject}.job
done

cron_fmriprep.sh

Run fmriprep

Cron: use cases

#!/bin/bash

#$ -pe smp 16
#$ -q UI
#$ -m bea
#$ -M james-kent@uiowa.edu
#$ -o /Shared/vosslabhpc/Projects/BETTER/3-Experiment/2-data/bids/derivatives/code/fmriprep/job_scripts/out
#$ -e /Shared/vosslabhpc/Projects/BETTER/3-Experiment/2-data/bids/derivatives/code/fmriprep/job_scripts/err
OMP_NUM_THREADS=10
singularity run -H ${HOME}/singularity_home -B /Shared/vosslabhpc:/mnt \
/Shared/vosslabhpc/UniversalSoftware/SingularityContainers/poldracklab_fmriprep_1.1.1-2018-06-07-86609a14355e.img \
/mnt/Projects/BETTER/3-Experiment/2-data/bids/ /mnt/Projects/BETTER/3-Experiment/2-data/bids/derivatives \
participant --participant_label SUBJECT \
-w /mnt/Projects/BETTER/3-Experiment/2-data/bids/derivatives/work/fmriprep \
--write-graph --mem_mb 35000 --omp-nthreads 10 --nthreads 16 --use-aroma \
--output-space template \
--template MNI152NLin2009cAsym \
--fs-license-file /mnt/UniversalSoftware/freesurfer_license.txt

TEMPLATE_fmriprep.job

Update website

Cron: use cases

0 5 * * * bash /home/jdkent/Programming/Projects/ProjectBetterAging/code/cron_cp_files.sh

crontab (work computer)

Update website

Cron: use cases

#!/usr/bin/env bash

src_dir="/home/jdkent/mnt/vosslabhpc/Projects/BETTER/3-Experiment/2-data/bids/derivatives/mriqc/reports"
repo_dir="/home/jdkent/Programming/Projects/ProjectBetterAging"
dst_dir="${repo_dir}/_indivMRIQC"
group_dir="${repo_dir}/groupMRIQC"

# find the unique individual reports that haven't been updated
src_htmls=$(echo ${src_dir}/sub-* | xargs -d' ' -I {} basename {} | sort)
dst_htmls=$(echo ${dst_dir}/sub-* | xargs -d' ' -I {} basename {} | sort)

missing_htmls=($(comm -23 <(printf '%s\n' ${src_htmls[@]}) <(printf '%s\n' ${dst_htmls[@]})))

for html in ${missing_htmls[@]}; do
    sed -e '1s|^|---\n---\n|' ${src_dir}/${html} > ${dst_dir}/${html}
done

# copy the group reports
cp ${src_dir}/T1w_group.html ${group_dir}/T1w_group.html
cp ${src_dir}/bold_group.html ${group_dir}/bold_group.html

# add new files and push to master
cd ${repo_dir}
git add .
git commit -m "$(date)"

cron_cp_files.sh

Cron: syntax

crontab guru

Cron: tutorial

  1. login into argon
ssh hawkid@argon-login-1.hpc.uiowa.edu

Note:

type argon-login-1 (or argon-login-2) instead of argon

Cron: tutorial

2. clone the github repo

git clone https://github.com/jdkent/cronTutorial.git

Cron: tutorial

3. cd into cronTutorial and read README.md

cd ./cronTutorial
cat README.md

Cron: tutorial

4. (optional) change your crontab default text editor

nano ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi

# User specific environment and startup programs

# change the default text editor
export EDITOR=nano # THIS IS WHAT WE ADDED
.bash_profile

Cron: tutorial

5. make a new entry to your crontab

crontab -e
MAILTO=<name>@uiowa.edu 
# replace <name> with your email name
* * * * * /Users/<hawkid>/cronTutorial/cron_rsync.sh 
# replace <hawkid> with your hawkid



crontab

Cron: tutorial

6. Success?

 

 

Did trash.txt disappear and treasure.txt appear in dirB?

Cron: tutorial

7. Delete the entry

 

 

use crontab -e to remove the job (you probably don't want an email every minute for the rest of eternity)

cron: gotchas

  • cronjobs don't load your .bash_profile/.bashrc by default (afaik)
    • If you defined something in those files that is needed by your script, then you will need source your .bashrc and .bash-profile in your script.

cron: gotchas

  • Specify absolute paths to the scripts you are using

DO

DO NOT

* * * * * /full/path/to/script.sh
* * * * * script.sh

cron: gotchas

  • You can't make inline comments

DO

DO NOT

# this job backs up my data every minute
* * * * * /full/path/to/script.sh

* * * * * /full/path/to/script.sh # this job backs up my data every minute

cron: gotchas

  • You can't use linebreaks for commands

DO

DO NOT

# this job backs up my data every minute
* * * * * /full/path/to/script.sh -a arg1 -b arg2 -c arg3

# this job backs up my data every minute
* * * * * /full/path/to/script.sh \
-a arg1 \ 
-b arg2 \
-c arg3

Learning Objectives

 

Questions?

@SudoNeuroSci

Cron

By James Kent

Made with Slides.com

Cron

Basic Intro to cron, what it is, why it's helpful, how to use it.

  • 872

More from James Kent