Reproducible, transparent, and adaptable data analysis with Snakemake
Johannes Köster
2023
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/7796524/DKTK.png)
dataset
results
Data analysis
"Let me do that by hand..."
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/3236000/1.gif)
dataset
results
dataset
dataset
dataset
dataset
dataset
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/3236000/1.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/3235999/2.gif)
"Let me do that by hand..."
Data analysis
- check computational validity
- apply same analysis to new data
- check methodological validity
- understand analysis
Data analysis
Reproducibility
Transparency
- modify analysis
- extend analysis
Adaptability
>700k downloads since 2015
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2646163/nature_genetics.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2646162/cancer_cell.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2646179/nature_methods.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2646177/genome_biology.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2646170/bioinformatics.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2646184/cell.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2646193/molecular_cell.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2646200/plos_genetics.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/5293194/ng50_7.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/5293229/nar_46_12.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/5293228/isme_12_8.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/5293223/nm_3_2.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/5293204/nar_46_14.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/5293277/msb_13_5.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/5293330/mbe_34_5.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/5293331/pnas_114_5.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/5293364/nbt-v34-n5.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/5293426/molcel_66_3.gif)
>2000 citations
>10 citations per week in 2022
- automation
- scalability
- portability
- readability
- documentation
- traceability
Data analysis
Reproducibility
Transparency
- readability
- portability
- scalability
Adaptability
Data analysis
- automation
- scalability
- portability
- readability
- documentation
- traceability
Reproducibility
Transparency
- readability
- portability
- scalability
Adaptability
dataset
results
dataset
dataset
dataset
dataset
dataset
Define workflows
in terms of rules
Define workflows
in terms of rules
rule mytask:
input:
"path/to/{dataset}.txt"
output:
"result/{dataset}.txt"
script:
"scripts/myscript.R"
rule myfiltration:
input:
"result/{dataset}.txt"
output:
"result/{dataset}.filtered.txt"
shell:
"mycommand {input} > {output}"
rule aggregate:
input:
"results/dataset1.filtered.txt",
"results/dataset2.filtered.txt"
output:
"plots/myplot.pdf"
script:
"scripts/myplot.R"
Define workflows
in terms of rules
Define workflows
in terms of rules
rule mytask:
input:
"data/{sample}.txt"
output:
"result/{sample}.txt"
shell:
"some-tool {input} > {output}"
rule name
how to create output from input
define
- input
- output
- log files
- parameters
- resources
rule mytask:
input:
"path/to/{dataset}.txt"
output:
"result/{dataset}.txt"
script:
"scripts/myscript.R"
rule myfiltration:
input:
"result/{dataset}.txt"
output:
"result/{dataset}.filtered.txt"
shell:
"mycommand {input} > {output}"
rule aggregate:
input:
"results/dataset1.filtered.txt",
"results/dataset2.filtered.txt"
output:
"plots/myplot.pdf"
script:
"scripts/myplot.R"
Define workflows
in terms of rules
Boilerplate-free integration of scripts
rule mytask:
input:
"data/{sample}.txt"
output:
"result/{sample}.txt"
script:
"scripts/myscript.py"
reusable scripts:
- Python
- R
- Julia
- Rust
import pandas as pd
data = pd.read_table(snakemake.input[0])
data = data.sort_values("id")
data.to_csv(snakemake.output[0], sep="\t")
Python:
data <- read.table(snakemake@input[[1]])
data <- data[order(data$id),]
write.table(data, file = snakemake@output[[1]])
Boilerplate-free integration of scripts
R:
import polar as pl
pl.read_csv(&snakemake.input[0])
.sort()
.to_csv(&snakemake.output[0])
Rust:
Jupyter notebook integration
rule mytask:
input:
"data/{sample}.txt"
output:
"result/{sample}.txt"
notebook:
"notebooks/mynotebook.ipynb"
- Integrated interactive edit mode.
- Automatic generalization for reuse in other jobs.
Reusable wrappers
rule map_reads:
input:
"{sample}.bam"
output:
"{sample}.sorted.bam"
wrapper:
"0.22.0/bio/samtools/sort"
reuseable wrappers from central repository
Reusable wrappers
Output handling
rule mytask:
input:
"data/{sample}.txt"
output:
temp("result/{sample}.txt")
shell:
"some-tool {input} > {output}"
Output handling
rule mytask:
input:
"data/{sample}.txt"
output:
protected("result/{sample}.txt")
shell:
"some-tool {input} > {output}"
Output handling
rule mytask:
input:
"data/{sample}.txt"
output:
pipe("result/{sample}.txt")
shell:
"some-tool {input} > {output}"
Using and combining workflows
configfile: "config/config.yaml"
module dna_seq:
snakefile:
github("snakemake-workflows/dna-seq-gatk-variant-calling", path="workflow/Snakefile", tag="v1.17.0")
config:
config
use rule * from dna_seq
# easily extend the workflow
rule plot_vafs:
input:
"filtered/all.vcf.gz"
output:
"results/plots/vafs.svg"
notebook:
"notebooks/plot-vafs.py.ipynb"
Using and combining workflows
configfile: "config/config.yaml"
module dna_seq:
snakefile:
github("snakemake-workflows/dna-seq-gatk-variant-calling", path="workflow/Snakefile", tag="v1.17.0")
config:
config["dna-seq"]
use rule * from dna_seq as dna_seq_*
# easily extend the workflow
rule plot_vafs:
input:
"filtered/all.vcf.gz"
output:
"results/plots/vafs.svg"
notebook:
"notebooks/plot-vafs.py.ipynb"
module rna_seq:
snakefile:
github("snakemake-workflows/rna-seq-kallisto-sleuth", path="workflow/Snakefile", tag="v1.0.0")
config:
config["rna-seq"]
use rule * from rna_seq as rna_seq_*
Data analysis
- automation
- scalability
- portability
- readability
- documentation
- traceability
Reproducibility
Transparency
- readability
- portability
- scalability
Adaptability
job selection
job resource usage
free resources
job temp file consumption
temp file lifetime fraction
job priority
job thread usage
Scheduling
temp file size
temp file deletion
DAG partitioning
--groups a=g1 b=g1
--groups a=g1 b=g1 --group-components g1=2
--groups a=g1 b=g1 --group-components g1=5
Scalable to any platform
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2640416/workstation.jpeg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2640415/server.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2640414/cluster.jpeg)
workstation
compute server
cluster
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2640415/server.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2640415/server.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2640415/server.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2640415/server.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2640415/server.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/2640415/server.png)
grid computing
cloud computing
![](https://s3.amazonaws.com/media-p.slid.es/uploads/362168/images/4125298/gcp-logo.jpg)
MPI support
rule calc_pi:
output:
"pi.calc",
resources:
tasks=10,
mpi="mpiexec",
shell:
"{resources.mpi} -n {resources.tasks} calc-pi-mpi 10 > {output}"
Between workflow caching
dataset
results
dataset
dataset
dataset
dataset
dataset
shared data
Between workflow caching
Data analysis
- automation
- scalability
- portability
- readability
- documentation
- traceability
Reproducibility
Transparency
- readability
- portability
- scalability
Adaptability
rule mytask:
input:
"path/to/{dataset}.txt"
output:
"result/{dataset}.txt"
conda:
"envs/some-tool.yaml"
shell:
"some-tool {input} > {output}"
Conda integration
channels:
- conda-forge
dependencies:
- some-tool =2.3.1
- some-lib =1.1.2
Container integration
rule mytask:
input:
"path/to/{dataset}.txt"
output:
"result/{dataset}.txt"
container:
"docker://biocontainers/some-tool#2.3.1"
shell:
"some-tool {input} > {output}"
Containerization
containerized:
"docker://username/myworkflow:1.0.0"
rule mytask:
input:
"path/to/{dataset}.txt"
output:
"result/{dataset}.txt"
conda:
"envs/some-tool.yaml"
shell:
"some-tool {input} > {output}"
snakemake --containerize > Dockerfile
Data analysis
- automation
- scalability
- portability
- readability
- documentation
- traceability
Reproducibility
Transparency
- readability
- portability
- scalability
Adaptability
Self-contained HTML reports
Snakemake workflow catalog
Conclusion
With
- the human readable specification language
- reusable modularization capabilities
- seamless execution on all platforms without adaptation of the workflow definition
- integrated package management and containerization
- interactive reporting
Snakemake covers all aspects of fully reproducible, transparent, and adaptable data analysis.
Sustainable data analysis with Snakemake
By Johannes Köster
Sustainable data analysis with Snakemake
Non-bio-related talk about Snakemake
- 1,498