Singularity and workflow management systems

Singularity + (snakemake || nextflow)

Michael Hall - Iqbal Group (EBI)

https://slides.com/mbhall88/singularity-and-workflow-management-systems

Singularity + Snakemake

Run entire pipeline inside a given Singularity container

https://snakemake.readthedocs.io/en/stable/snakefiles/deployment.html#running-jobs-in-containers

singularity: "docker://quay.io/biocontainers/bwa:0.7.3a--h84994c4_4"

rule map:
    input: 
        query = "data/query.fastq",
        ref = "data/ref.fa"
    output:
        "data/query.sam"
    shell:
        "bwa mem {input.ref} {input.query} > {output}"

Singularity + Snakemake

Run single rule inside a given Singularity container

https://snakemake.readthedocs.io/en/stable/snakefiles/deployment.html#running-jobs-in-containers

rule map:
    input: 
        query = "data/query.fastq",
        ref = "data/ref.fa"
    output:
        "data/query.sam"
    singularity: "docker://quay.io/biocontainers/bwa:0.7.3a--h84994c4_4"
    shell:
        "bwa mem {input.ref} {input.query} > {output}"

Singularity + Snakemake

When executing the pipeline you just need to add the use-singularity option.

https://snakemake.readthedocs.io/en/stable/snakefiles/deployment.html#running-jobs-in-containers

snakemake --use-singularity

Singularity + Snakemake

Inception

singularity: "docker://continuumio/miniconda3:4.4.10"

rule plot:
    input:
        "table.txt"
    output:
        "plots/myplot.pdf"
    conda:
        "envs/ggplot.yaml"
    script:
        "scripts/plot-stuff.R"

https://snakemake.readthedocs.io/en/stable/snakefiles/deployment.html#combining-conda-package-management-with-containers

singularity: "docker://continuumio/miniconda3:4.4.10"

rule plot:
    input:
        "table.txt"
    output:
        "plots/myplot.pdf"
    conda:
        "envs/ggplot.yaml"
    script:
        "scripts/plot-stuff.R"

Will run the whole pipeline inside a container with miniconda installed. For our plot rule we then specify a conda environment.

Note: Still need conda installed locally and use --use-conda when running snakemake.

Singularity + Nextflow

nextflow run <your script> -with-singularity [singularity image file]

Run whole workflow in a container

Or specify the container in the nextflow.config file

process.container = '/path/to/singularity.img'
singularity.enabled = true

https://www.nextflow.io/docs/latest/singularity.html

Singularity + Nextflow

process {
    withName:foo {
        container = 'image_name_1'
    }
    withName:bar {
        container = 'image_name_2'
    }
}
singularity {
    enabled = true
}

https://www.nextflow.io/docs/latest/singularity.html

Or specify multiple containers in the nextflow.config file