Scientific Workflows

https://www.slideshare.net/soilandreyes/

stain@biggie-mint ~/src/taverna-prov/example $ executeworkflow -embedded \
 -provbundle helloanyone.bundle.zip \
 -inputvalue name fred helloanyone.t2flow 

Provenance bundle zip will be saved to: /home/stain/src/taverna-prov/example/helloanyone.bundle.zip

stain@biggie-mint ~/src/taverna-prov/example $ mkdir helloanyone.bundle ; cd helloanyone.bundle
stain@biggie-mint ~/src/taverna-prov/example/helloanyone.bundle $ unzip ../helloanyone.bundle.zip 
Archive:  ../helloanyone.bundle.zip
 extracting: mimetype                
   creating: inputs/
  inflating: inputs/name.txt         
   creating: outputs/
  inflating: outputs/greeting.txt    
   creating: intermediates/
   creating: intermediates/3a/
  inflating: intermediates/3a/3a82e39d-a537-40cf-91a0-2c89d4a2e62b.txt  
  inflating: workflowrun.prov.ttl    
  inflating: workflow.wfbundle       
   creating: .ro/
   creating: .ro/annotations/
  inflating: .ro/annotations/workflow.wfdesc.ttl  
  inflating: .ro/annotations/a2f03983-8836-4c36-bfb2-d713d9a1928f.ttl  
  inflating: .ro/manifest.json

https://github.com/apache/incubator-taverna-engine
/tree/master/taverna-prov

https://doi.org/10.5281/zenodo.51314

http://www.commonwl.org/

cwlVersion: v1.0
class: Workflow
inputs:
  inp: File
  ex: string

outputs:
  classout:
    type: File
    outputSource: compile/classfile

steps:
  untar:
    run: tar-param.cwl
    in:
      tarfile: inp
      extractfile: ex
    out: [example_out]

  compile:
    run: arguments.cwl
    in:
      src: untar/example_out
    out: [classfile]

https://w3id.org/cwl/v1.0/

https://view.commonwl.org/

https://github.com/rabix/composer

Copyright © 2013 W3C® (MIT, ERCIM, Keio, Beihang), All Rights Reserved. W3C liability, trademark and document use rules apply.

PROV Model Primer

W3C Working Group Note 30 April 2013

http://www.w3.org/TR/prov-primer/

Which PROV format?

<prov:wasGeneratedBy>
  <prov:entity prov:ref="ex:ent1"/>
  <prov:activity prov:ref="ex:act1"/>
  <prov:time>2017-10-26T21:32:52Z</prov:time>
  <ex:port>p1</ex:port>
</prov:wasGeneratedBy>

wasGeneratedBy(ent1, act1, 
  2017-10-26T21:32:52Z, ex:port="p1")

:ent1
  a prov:Entity;
  prov:wasGeneratedBy :act1;
  prov:generatedAtTime "2017-10-26T21:32:52Z"^^xsd:dateTime ;
  ex:port "p1" .

    "wasGeneratedBy": {
        "ex:gen1": {
            "prov:entity": "ent1",
            "prov:activity": "act1",
            "prov:time": "2017-10-26T21:32:52Z",
            "ex:port": "p1"
        },
    },

{ "@context": { .. }, 
  "@id": "ent1",
  "@type": "prov:Entity",
  "ex:port": "p1",
  "prov:generatedAtTime":  "2017-10-26T21:32:52Z",
  "prov:wasGeneratedBy": {
    "@id": "act1",
    "@type": "prov:Activity"
  } 
}

PROV-N

PROV-XML

PROV-JSON

PROV-O Turtle

PROV-O JSON-LD

https://www.w3.org/TR/prov-overview/

Tooling to the rescue

https://provenance.ecs.soton.ac.uk/

http://rdf.greggkellogg.net/

How to identify the workflow?

Permalink URI scheme

https://w3id.org/cwl/view/{scheme}/{commit}/{path}#{fragment}

• https://w3id.org/cwl/view/ fixed prefix at permalink service https://w3id.org/
• {scheme} - source code management protocol, currently only git supported:
  • {commit} - full git commit sha1 id (no branches or short commits allowed)
  • {path} - relative path to .cwl file within a checkout of that git commit
  • #{fragment} - optional part within CWL file , e.g. #main

Any git permalinks are resolved using https://view.commonwl.org/git which - if it knows about that particular git commit - will content-negotiate to provide various representations.

Anyone can mint these permalinks for .cwl files for a given commit, in any public or private git repository, given no uncommitted files or git submodules.

https://w3id.org/cwl/view/git/933bf2a1a1cce32d88f88f136275535da9df0954
/workflows/hello/hello.cwl#main

https://w3id.org/cwl/view

wasAssociatedWith(run:2e1287e0-6dfb-11e7-8acf-0242ac110002, 
      engine:b2210211-8acb-4d58-bd28-2a36b18d3b4f, 
      https://w3id.org/cwl/view/git/933bf2a1a1cce32d88f88f136275535da9df0954/workflows/hello/hello.cwl#main)

agent(engine:b2210211-8acb-4d58-bd28-2a36b18d3b4f, 
      prov:type='prov:SoftwareAgent', prov:type='wfprov:WorkflowEngine', 
      prov:label="cwltool v1.2.5")

A workflow definition (prospective provenance)
can be executed multiple times (retrospective provenance)

(and on different machines)

workflow definition - a recipe (prov:Plan)

workflow instance - a recipe, fully configured to run (prov:Plan)

workflow run - an execution of a workflow instance (prov:Activity)

Run a Command line tool

cwlVersion: v1.0
class: CommandLineTool

baseCommand: [ esl-sfetch, --index ]


inputs:
  sequences:
    type: File
    inputBinding:
      position: 1
      valueFrom: $(self.basename)

Where does esl-sfetch come from?
Which version? How is it configured?

Containers to the rescue

http://singularity.lbl.gov/

http://docker.com/

https://bioconda.github.io/

https://hub.docker.com/r/openphacts/
identitymappingservice/

Step in workflow ~= tool execution?

Scatter/gather

To use scatter/gather, ScatterFeatureRequirement must be specified in the workflow or workflow step requirements.

A "scatter" operation specifies that the associated workflow step or subworkflow should execute separately over a list of input elements. Each job making up a scatter operation is independent and may be executed concurrently.

• dotproduct specifies that each of the input arrays are aligned and one element taken from each array to construct each job. It is an error if all input arrays are not the same length.

• nested_crossproduct specifies the Cartesian product of the inputs, producing a job for every combination of the scattered inputs. The output must be nested arrays for each level of scattering, in the order that the input arrays are listed in the scatter field.

• flat_crossproduct specifies the Cartesian product of the inputs, producing a job for every combination of the scattered inputs. The output arrays must be flattened to a single level, but otherwise listed in the order that the input arrays are listed in the scatter field.

http://www.commonwl.org/v1.0/Workflow.html#Scatter/gather

3.5 Expressions

An expression is a fragment of Javascript/ECMAScript 5.1 code evaluated by the workflow latform to affect the inputs, outputs, or behavior of a process.

Expressions are denoted by the syntax $(...) or ${...}.
A code fragment wrapped in the $(...) syntax must be evaluated as a ECMAScript expression.
A code fragment wrapped in the ${...} syntax must be evaluated as a ECMAScript function body for an anonymous, zero-argument function.
Expressions must return a valid JSON data type: one of null, string, number, boolean, array, object. 

http://www.commonwl.org/v1.0/Workflow.html#Expressions

Conditional branching

  step2:
    in: [threshold]
    out: [out]
    switch:
      "$(inputs.threshold > 2)": high.cwl
      "$(inputs.threshold == 1)": low.cwl
      default:
        result:
          out: 0

https://github.com/common-workflow-language/
common-workflow-language/issues/494

Nested workflows

A single activity unrolled to multiple steps

activity(run:2e1287e0-6dfb-11e7-8acf-0242ac110002, , , 
   [prov:type='wfprov:WorkflowRun', prov:label="Run of workflow/packed.cwl#main"])    
    // main workflow run started outside somehow (we're don't know how)
    wasStartedBy(run:4305467e-6dfb-11e7-885d-0242ac110002, -, -, 
                 -, 2017-10-27T15:00:00Z)
    // ...
    // step is a nested workflow, so also a WorkflowRun
    activity(run:4305467e-6dfb-11e7-885d-0242ac110002, -, -, 
      [prov:type='wfprov:WorkflowRun', prov:label="Run of workflow/packed.cwl#main/nested1"])
        // started by the mother activity
        wasStartedBy(run:4305467e-6dfb-11e7-885d-0242ac110002, -, -, 
                     run:2e1287e0-6dfb-11e7-8acf-0242ac110002, 2017-10-27T15:00:30Z)
    
        // inner step of nested workflow, ProcessRun as this is a command line execution
        activity(run:c42dc36e-6dfd-11e7-bc24-0242ac110002, -, - 
          [prov:type='wfprov:ProcessRun', prov:label="Run of workflow/packed.cwl#nested/innerStep1"])
            
        wasStartedBy(run:c42dc36e-6dfd-11e7-bc24-0242ac110002, -, -, 
                     run:4305467e-6dfb-11e7-885d-0242ac110002, 2017-10-27T15:01:00Z)
        // ...

Identifying intermediate data

Output 1B file is also Input 2C and Input 3D downstream

Simple filenames -> duplications

./data/step1/outputB.txt 
./data/step2/inputC.txt
./data/step3/inputD.txt

Content-adressable

SHA-256 hash of bytes as filename:

./data/51/51fb8af0c4ae0422fbe88340d91880ecb9d7537cf57339c1cf1256b7ca58f32d

RFC6920 URI as global identifier:

nih:sha-256;51fb8af0c4ae0422fbe88340d91880ecb9d7537cf57339c1cf1256b7ca58f32d

All-in-one prov trace -> messy

identifiers mismatch (e.g. "step1" both in #main and #nested)

Multiple wasGeneratedBy for same entity

Do everyone need to understand execution hierarchy?

Multiple workflow PROV profiles combined?

prov:alternateOf

Relating global identifier to local paths

used(run:2e1287e0-6dfb-11e7-8acf-0242ac110002, 
     data:5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03, 
     2017-10-27T14:29:00+01:00, [prov:role='wf:main/input1']))

entity(data:5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03)

   // which we have stored a copy of within the research object
   specializationOf(./data/58/5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03,
                    data:5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03)

. but what about multiple workflows creating the same bytes?

e.g. when was the empty string generated?

used(run:2e1287e0-6dfb-11e7-8acf-0242ac110002, 
     urn:uuid:f940c301-46fd-4a6b-808d-d6beed700f3a, 
     2017-10-27T14:29:00+01:00, [prov:role='wf:main/input1']))

used(run:2e1287e0-6dfb-11e7-8acf-0242ac110002, 
     urn:uuid:63a0ff1b-45c6-41cb-97bf-2da7aa93ec0f, 
     2017-10-27T14:29:00+01:05, [prov:role='wf:main/input2']))

entity(data:5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03)
   // Different UUID for each occurrence
   specializationOf(urn:uuid:f940c301-46fd-4a6b-808d-d6beed700f3a,
                    data:5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03)
   specializationOf(urn:uuid:63a0ff1b-45c6-41cb-97bf-2da7aa93ec0f
                    data:5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03)
   // Also available as bytes in research Object
   specializationOf(./data/58/5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03,
                    data:5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03)

What about multiple workflows creating the same bytes?

When was the empty string generated?

Workaround: "virtual" entity for every role / activity occurance

Simplify:
multiple PROV files

Different "world views" of what happened

metadata/provenance/2e1287e0-6dfb-11e7-8acf-0242ac110002.prov.jsonld

metadata/provenance/4305467e-6dfb-11e7-885d-0242ac110002.prov.jsonld

metadata/provenance/c42dc36e-6dfd-11e7-bc24-0242ac110002.prov.jsonld
     Bonus: Obvious slot for tool-specific provenance

Prospective provenance? UUIDv5 hash of permalink

metadata/prospective/39ab126a-e0c9-5cac-a67e-2b7fdb8ad25f/cwl.ttl

metadata/prospective/39ab126a-e0c9-5cac-a67e-2b7fdb8ad25f/wfdesc.ttl

metadata/prospective/39ab126a-e0c9-5cac-a67e-2b7fdb8ad25f/pplan.ttl

Workflow provenance profiles

https://w3id.org/ro/2016-01-28/wfprov/

http://www.opmw.org/model/OPMW

https://www.slideshare.net/dgarijo/opmw

http://www.opmw.org/model/p-plan/

https://www.slideshare.net/dgarijo/pplan

https://github.com/KNMI/s-provenance

https://purl.dataone.org/provone-v1-dev

https://xkcd.com/927/

How to tie it all together?

id:        doi:10.15490/seek.1.investigation.56
createdOn: 2015-07-10T16:46:00Z
createdBy: http://orcid.org/0000-0001-9842-9718

aggregates:
 - id:         data/sequence/specimen5.bam
   conformsTo: http://gemrb.org/iesdp/file_formats/ie_formats/bam_v1.htm

 - id:         http://example.com/blog/about-specimen5
   authoredBy: http://orcid.org/0000-0001-7066-3350

 - id:         http://www.myexperiment.org/workflows/3355
   history:    provenance/workflow-evolution.ttl

annotations:
 - about:       data/sequence/specimen5.bam
   content:     annotations/specimen5-properties.jsonld
   createdBy:   http://orcid.org/0000-0001-7066-3350

 - about:       data/sequence/specimen5.bam
   content:     http://example.com/blog/about-specimen5
   motivatedBy: oa:questioning

Research Object manifest

(simplified)

Reuse standards:
OAI-ORE, BagIt, W3C JSON-LD, PROV, Web Annotation Model

metadata/manifest.json​
data/sequence/specimen5.bam
provenance/workflow-evolution.ttl
http://example.com/blog/about-specimen5
http://www.myexperiment.org/workflows/335

http://orcid.org/0000-0001-7066-3350
http://gemrb.org/iesdb/
   file_formats_ie_formats_bam_v1.html

https://w3id.org/ro/bagit
doi:10.1109/BigData.2016.7840618

Who is using Research Objects?

Structure of CWL run Research Object:

• data: content-adressable by sha256 hash
• workflow: input object (json file) with relativised paths, packed.cwl
  executable workflow containing the workflow specification and tool specifications with relativised paths to re-run inside an RO.
• snapshot: This directory contains copies of the original workflow and tool specifications files as-is (warning: might contain absolute paths or be host-specific).
• metadata: provenance about the workflow run, its data products and manifest for this Research Object.

https://github.com/common-workflow-language/common-workflow-language/wiki