for beginners

Agenda:

What is the challenge
What is Nextflow (NF)
Nextflow basics
Superpowers
Existing pipelines and good boilerplates
Q&A

What is the challenge?

Data

Output

Data

Function

Software

- Version

- Options

Data

Function

Software

- Version

- Options

Output

Data

Any change in:

- Data

- Software version

- Command options

Can result in different output data

Results should be reproducible!

Parallelisation of tasks

is efficient

Parallelisation of tasks

is efficient

There are different HPC scheduling systems/executors

We should isolate software dependencies to give exact version of software to user

It is hard and error-prone to install software manually

Goal

Data

Packaged software

Guidelines how to process data with given software

Output

DSL - Domain specific Language

What is

Data

Software package

Guidelines how to process data with given software

Guidelines how to process data with given software using specific executor

Submit tasks in parallel to the specified executor

Output

Nextflow basics

- Processes

- Channels

Process

process makeSTARindex {
    input:
    file fasta from ch_fasta_for_star_index

    output:
    file "star" into star_index

    script:
    """
    mkdir star
    STAR \\
        --runMode genomeGenerate \\
        --runThreadN ${task.cpus} \\
        --sjdbGTFfile $gtf \\
        --genomeDir star/ \\
        --genomeFastaFiles $fasta \\
        $avail_mem
    """
}

Takes input
Executes script
Passes output of script

Has its own needs
Has one specific job
Fires automatically when input is available

Channel

process makeSTARindex {
    input:
    file fasta from ch_fasta_for_star_index # I am a channel

    output:
    file "star" into star_index # I am also a channel

    script:
    """
    mkdir star
    STAR \\
        --runMode genomeGenerate \\
        --runThreadN ${task.cpus} \\
        --sjdbGTFfile $gtf \\
        --genomeDir star/ \\
        --genomeFastaFiles $fasta \\
        $avail_mem
    """
}

Passes data between processes asynchronously
There are multiple operation you can do with channels
- merge, mix, fork, combine, divide etc.
There are different types of channels

Channel

process plink_to_vcf{
    input:
    set file(bed), file(bim), file(fam) from harmonised_genotypes

    output:
    file "harmonised.vcf.gz" into harmonised_vcf_ch

    script:
    """
    plink2 --bfile ${bed.simpleName} --recode vcf-iid --out ${bed.simpleName}
    bgzip harmonised.vcf
    """
}

process vcf_fixref{
    input:
    file input_vcf from harmonised_vcf_ch
    
    output:
    file "fixref.vcf.gz" into filter_vcf_input

    script:
    """
    bcftools index ${input_vcf}
    bcftools +fixref ${input_vcf} -- -f ${fasta} -i ${vcf_file} | \
     bcftools norm --check-ref x -f ${fasta} -Oz -o fixref.vcf.gz
    """
}

Superpowers?

deck

By kerimoff