Open, continuous and collaborative benchmarking
Workshop BC²
Basel, 11.09.23
Why benchmarking and why open, collaborative and continuous benchmarking?
https://www.scrna-tools.org
1. cms_default 2. cms_kmin 3. lisi
Lütge A. et al, 2020
Norel et al, 2011
Luecken et al., 2021
-
Meta-analysis of 62 method benchmarks in the field of single cell omics
Sonrel et al, 2023
Sonrel et al, 2023
Data
inputs
simulations
results
Reproducibility
versions
environments
workflows
scale
comprehensive
continuous
Code
available
extensible
reusable
currently part of most benchmarks
not part of current standards
Design and basic concepts
Method developer/
Benchmarker
Method user
Methods
Datasets
Metrics
Omnibenchmark
Goals:
standardized datasets
= 1 "module" (renku project )
method results
metric results
interactive result exploration
Method user
Method developer/
Benchmarker
Module coordination and user roles
GitLab
Docker
Workflow
Module:
Template code
Module code
= 1 "data bundle" (data files + meta data)
= 1 "module" (renku project )
Orchestrator