work in open

Open Science

The six principles of Open Science

... more than open access

Document: Why?

• Organise ideas
• Understanding code and steps in the future for you and others
• Fixing errors
• Help in future publication

Document: Where?

• File System (e.g. README or TODO files)
• Control Version System
  • Git, SVN, etc.
• Content Management System
  • Wiki CMS, Drupal, etc.
• Electronic Lab Notebook (ELN)

Document: How?

• Plain text
• Format
  • Unstructured
    • Free
    • Markdown
    • Wikitext

Document: How?

• Format
  • Structured
    • Config files
      • XML, JSON, INI, YAML
    • Templates (e.g. in wikis)
    • Database Management Systems (Relation or NoSQL)

Document: How?

• When text is not possible
  • Ensure open formats
    • Interoperability
    • Avoid vendor lock-in
    • e.g., for images: PNG, TIFF
  • Whenever possible, favor lossless formats (e.g., JPEG < TIFF)

Document: visibility

• Open / private
  • The more accessible, the easier for third parties to collaborate
  • Important to define the moment of disclosure
    • Publishing strategy
    • Engagement
    • Handling 3rd-party issues

Document: license

• Check with your institution!
• Copyleft or not? (e. g., GPL vs MIT)
• Some licenses may be more suitable for some contents
  • Documents, articles and most works such as images: Creative Commons
  • Code (GPL, MIT, Apache, etc.)
  • Others (e.g., databases)
    • Reference for datasets

Tag and track: Why?

• Convenient backup
• Error tracking and reversion
• Checking history
• Allowing collaboration on different time points
• Publication of specific snapshots

Tag and track: Where?

• Code, documentation:
  • Control Version System (Git, SVN, etc.)
    • Interfaces:
      • Github
      • Gitlab (allows local installation)
  • Wiki CMS (e.g. [Semantic] MediaWiki)
• Data, files
  • Plain Git (small files) or Git with large files
  • Document Management Systems

Tag and track: Concepts

• Revision, Version, Commit
• Branch
• Tag, Release
• Fork, Pull request

Tag and track: Concepts

https://commons.wikimedia.org/wiki/File:Git-branching-model.pdf

Tag and track: Publish

• Working and executable code
  • Dockerhub, Quay.io, etc.
• Identify Content & Code (DOI)
  • Figshare
  • Zenodo (with Github)
• Bio specific repositories
  • Sequence Read Archive (SRA)
  • GEO Archive (Genome Expression Data)
  • ENA, EGA and others. Detail
• Figures and multimedia for the general public
  • Wikimedia Commons

Reproduce: Why?

• Nowadays not only textual statements but also code and data
• Peers and collaborators should be able to reproduce by themselves
  • Check errors
  • Improve code, data
  • Test in different conditions

Standing on the shoulders of giants

Reproduce: How?

• Code requirements, recipes
  • Scripts
  • Test frameworks
  • Package managers (e.g. Conda)
  • Jupyter
• Virtualisation
  • Hypervisor: VirtualBox, VMWare, etc.
  • Containers: Docker, Singularity

Reproduce: Note on python

• pyenv & pyenv-virtualenv

  • pyenv install x.y.z

  • pyenv virtualenv x.y.x myvenv

• pip

  • pip freeze > requirements.txt

  • pip install -r requirements.txt

Reproduce: Other languages

• Perl: perlbrew
• PHP: phpbrew
• Java: jenv
• NodeJS: nvm
• etc.

Reproduce: Conda

• Popular package manager
  • Takes also care of binaries and libraries
• Bioconda: specific Bioinformatics recipes

Reproduce: Jupyter

• Former IPython Notebook
• Combines in a single notebook documentation (Markdown), comments and executable code with its output
• Underlying notebook format is a JSON text file
  • Can be exported into PDF, HTML, etc.

Reproduce: Jupyter

• Apart from Python (2 or 3), now also different languages with Kernels:
  • R, Perl5, Perl6, Javascript, more...
• Additional widgets (e.g. for charts)
• Convenient for sharing code and training
• Jupyter gallery in Github

Reproduce: Docker

• Allows shareable Linux systems that can be run in any machine where Docker is installed
• Build images with a script file (Dockerfile), very similar to a Linux command-line script
• Repository of Docker images
  • You can reuse, adapt, extend
  • Don't reinvent the wheel

Reproduce: Docker

• Microservices principle
  • 1 Image -> n Containers -> n Services
  • n Services -> 1 full application
• Example: BLAST Web application
  • Web server container
  • Database container
  • BLAST application running container
• Making it work together:
  • System scripts
  • Docker compose
  • Kubernetes
  • etc.

Reproduce: Singularity

• Like Docker but more suitable for HPC environments
• No need of a Docker daemon running / less problematic for security
• Docker images convertible into Singularity ones
• Container images are files by default so they can be archived and moved more easily

Pipelines & Workflows: Why?

• Write programs that do one thing and do it well.
• Write programs to work together.
• Write programs to handle text streams, because that is a universal interface.

Unix Philosophy

D. McIlroy, P.H.Salus

Pipelines & Workflows: How?

• Traditionally from Shell script files
• Frameworks or applications
  • Web-based
    • Galaxy
  • GUI and command-line
    • Apache Taverna
  • Command-line
    • Nextflow
    • Snakemake
• Common Workflow Language

Pipelines and Workflows: Nextflow

• Concepts
  • Processes
    • Any pipeline or program (in any language)
    • In local disk or in containers (Singularity, Docker)
  • Channels
    • FIFO queue
    • Normally files in a filesystem

Pipelines and Workflows: Nextflow

• Concepts
  • Config files
    • Different config files, calling one to another can be created for adapting to different scenarios
  • Executors
    • Local machine
    • HPC cluster: SGE, Univa, SLURM, etc.
    • Cloud systems: AWS, Azure, Google Cloud

Criteria

• Kind of tasks
• Team profiles
• Infrastructure and privacy
• Previous knowledge and time

Criteria: Tasks

• Data Analysis
• Interface / Web programming
• Teaching/Training

• Environment (where it can be achieved)
  • Interface/Web
  • HPC
  • etc.

Criteria: Profiles

• Wet lab scientists
• Statisticians, programmers
• Citizens

• Personal and working situations
  • Interns, PhD students, PostDocs
  • Technicians (full-time, temporary)
  • Project funding length

Criteria: Infrastructure, privacy

• Data transfer
  • Cluster vs Cloud
• Sysadmin or devops support
• Human or clinical data involved
• Funding vs time

Criteria: Knowledge

• Programming language(s)
  • Python, R, JavaScript, Java, Perl
• Availability of libraries / reusing
• Frameworks, platforms
  • Learning curve
  • Bus factor
  • Technical debt