GETTING STARTED WITH
COMPUTATIONAL REPRODUCIBILITY
Neha Moopen
Research Data Manager
2023-09-06
tinyurl.com/repco-gsns
THESE SLIDES ARE ADAPTED FROM
Best Practices in Writing Reproducible Code
@UtrechtUniversity
@NEONScience
&
LEARN MORE!
LEARN MORE!
LEARN MORE!
WHAT IS REPRODUCIBILITY?
Computational reproducibility is when detailed information is provided about code, software, hardware and implementation details (Victoria Stodden, 2014).
This image was created by Scriberia for The Turing Way community and is used under a CC-BY licence. The image was obtained from https://zenodo.org/record/3332808.
REPRODUCIBILITY SpECTRUM
A study may be more or less reproducible than another depending on what data and code are made available (Peng, 2011).
WHY IS THIS IMPORTANT?
- To show evidence of the correctness of your results.
- To enable others to make use of our methods and results.
The stakeholders involved:
- Collaborators
- Peer reviewers & journal editors
- Broad scientific community
- The public
- You!
If you need more convincing, see also: Five selfish reasons to work reproducibly (Florian Markowetz, 2015)
BEING REPRODUCIBLE
IS THIS ENOUGH?
- Access to the code
- Access to the data
-
(And let's assume we can replicate the environment)
How confident do you feel?
We need to do more: we need to inspire trust.
- The code is correct (and I have made it easy for you/someone to check);
- My workflow is robust;
- My workflow itself is accessible, and I will be guiding you through it.
THE FOUR FACETS OF REPRODUCIBILITY
ORGANIZATION
DOCUMENTATION
AUTOMATION
DISSEMINATION
ORGANIZATION
ORGANIZATION
WHY DO YOU NEED IT?
-
Your future self will be able to quickly find files.
-
Colleagues will be able to more quickly understand your workflow.
- Machine-readable names can be quickly and easily sorted and parsed.
ORGANIZATION
HOW CAN YOU DO IT?
FOLDER STRUCTURE
NAMING CONVENTION
VERSION CONTROL
ORGANIZATION
FOLDER STRUCTURE
-
Contain your project in a single recognizable folder.
-
Distinguish folder types, name them accordingly:
- Read-only: (raw) data, metadata
- Human-generated: code, paper, documentation
- Project-generated: clean data, figures, models...
Source: Wilson et al. (2017)
ORGANIZATION
Machine-readable and Human-readable
names ->
<- Names that support sorting
source: OSF's File naming Guide
NAMING CONVENTION
ORGANIZATION
File organization should:
- Reflect inputs, outputs and information flow
- Preserve raw data so it's not modified
- Carefully document & store intermediate & end outputs
- Carefully document & store data processing scripts
source: Intro to Reproducible Science @NEONScience
ORGANIZATION
File renaming tools:
-
Bulk Rename Utility (for Windows)
-
Renamer (for MacOS)
-
PSRenamer (for MacOS, Windows, Unix, Linux)
-
WildRename (for Windows)
ORGANIZATION
VERSION CONTROL
Git is a Version Control Software (VSC) that lives on your computer like any other program.
GitHub is a cloud-based hosting service that helps you manage Git repositories.
ORGANIZATION
WHY DO YOU NEED GIT & GITHUB?
source: phdcomics.com
ORGANIZATION
WHY DO YOU NEED GIT & GITHUB?
source: Turing Way
ORGANIZATION
WHY DO YOU NEED GIT & GITHUB?
source: Turing Way
ORGANIZATION
DO: Commits should be atomic: comprehensive 'units' of changes.
DON'T: edit for a full day and put this in a single commit (or worse: forget to...)
Commits should have informative messages so you (and others) can trace your steps
Track most files; .gitignore those files you don't.
Explore new ideas with branches, keep a stable version on master
HOW TO GIT: LAST TIPS
source: https://xkcd.com/1296/
ORGANIZATION: DISCUSSION
Are there changes you can make in your file & folder naming/organization?
Has anyone use Git/GitHub to version control, share, find code?
THE FOUR FACETS OF REPRODUCIBILITY
ORGANIZATION
DOCUMENTATION
AUTOMATION
DISSEMINATION
DOCUMENTATION
DOCUMENTATION
WHY DO YOU NEED IT?
-
You want yourself to understand how code written some time ago works
-
You want others to understand how to (re-)use your code
- Future 're-analysis' of your data is more efficient.
DOCUMENTATION
HOW CAN YOU DO IT?
COMMENTS
READMEs
DOCUMENTATION
Comments are annotations you write directly in the code source.
They:
-
are written for anyone who deals with your source code (including yourself)
-
explain parts that are not intuitive from the code itself
-
do not replace readable or structured code
-
can be used to directly generate documentation for users (if in a specific structure).
Comic source: Geek & Poke
DOCUMENTATION
Best practices for commenting:
DOCUMENTATION
The README page is the first thing your user will see!
The contents typically include one or more of the following:
- Configuration instructions
- Installation instructions
- Operating instructions
- A file manifest (list of files included)
- Copyright and licensing information
- Contact information for the distributor or programmer
- Known bugs
- Troubleshooting
- Credits and acknowledgments
Reference: Wikipedia's README page
DOCUMENTATION
An example README:
DOCUMENTATION
Check out Make a README:
DISCUSSION: DOCUMENTATION
Is your code well-annotated? Would future you, a direct colleague, an external colleague/peer understand it?
Do you have README files already?
THE FOUR FACETS OF REPRODUCIBILITY
ORGANIZATION
DOCUMENTATION
AUTOMATION
DISSEMINATION
AUTOMATION
AUTOMATION
WHY DO YOU NEED IT?
- More efficient to modify and repeat an analysis down the road.
- Easier for reviewers and colleagues to see every aspect of your methods.
- Self-documenting methods: your future self will forget small steps.
AUTOMATION
HOW DO YOU DO IT?
SCRIPTING
DRY
FUNCTIONALIZE
AUTOMATION
Everything should be done with a script. Pointing-and-clicking and/or copying-and-pasting, are not reproducible.
Think about:
- obtaining raw data
- converting data files
- editing data
- data cleaning
- data analyses
AUTOMATION
source: Best Practices in Writing Reproducible Code @UtrechtUniversity
DON'T REPEAT YOURSELF (DRY) AND...
AUTOMATION
Functions are smaller code units reponsible of one task.
-
Functions are meant to be reused
-
Functions accept arguments (though they may also be empty!)
-
What arguments a function accept is defined by its parameters
Functions do not necessarily make code shorter (at first)! Compare:
source: Best Practices in Writing Reproducible Code @UtrechtUniversity
AUTOMATION
It's better to think in building blocks:
source: Best Practices in Writing Reproducible Code @UtrechtUniversity
AUTOMATION
Among the many solutions you can utilize to avoid repeating yourself, you can consider implementing:
- a script for each 'step'
- use GNU Make to automate the workflow/build for your project
- write a reproducible report using R Markdown, Jupyter, Quarto
DISCUSSION: AUTOMATION
Do you have a tendency to rewrite or repeat code?
Do you see opportunities to write more functions?
THE FOUR FACETS OF REPRODUCIBILITY
ORGANIZATION
DOCUMENTATION
AUTOMATION
DISSEMINATION
DISSEMINATION
DISSEMINATION
WHY DO YOU NEED IT?
- Funding agency / journal requirement
- Community expects it
- Increased visibility / citation
- More efficient, less redundant science
DISSEMINATION
HOW DO YOU DO IT?
GITHUB x ZENODO
LICENSES
CITATION
DISSEMINATION
SOFTWARE LICENSES
-
Copyright is implicit; others cannot use your code without your permission.
-
Licensing gives that permission, and its boundaries and conditions.
-
Choosing a license early on means being aware of your license as the project proceeds (and not creating conflicts).
DISSEMINATION
There are over 80 OSI-approved licenses (and many, many others) to choose from. This is one that's often used:
DISSEMINATION
Archive your project on Zenodo, and get a DOI!
GitHub & Zenodo have a great integration that makes it easy to archive a whole repository.
DISSEMINATION
First, select your repository:
source: Best Practices in Writing Reproducible Code @UtrechtUniversity
DISSEMINATION
Second, release your project and follow the workflow:
source: Best Practices in Writing Reproducible Code @UtrechtUniversity
DISSEMINATION
Last,
- Add some final descriptions
- Click 'publish'
- Voilá!
source: Best Practices in Writing Reproducible Code @UtrechtUniversity
DISSEMINATION
As a final touch, take your DOI and place it as a badge in your GitHub README!
Links: GitHub / Workshop Materials / Zenodo
DISSEMINATION
How to cite code?
Links: GitHub / Workshop Materials / Zenodo
DISSEMINATION
How to cite code?
Links: GitHub / Workshop Materials / Zenodo
In addition to README.md & LICENSE.md, you can also include CITATION.cff in your GitHub repo.
Zenodo will read the CITATION.cff if available.
Learn more about CITATION.cff:
* Official Site:
DISSEMINATION
Use Binder to make your code immediately reproducible by anyone, anywhere!
DISSEMINATION
How does it work? Binder is a virtual, executable environment that runs the code in your GitHub repository.
DISCUSSION: DISSEMINATION
When do you think is the best time to publish your code / place it online? Why?
Have you already published your code? Where?
What software license did/would you choose? Why?
THE FOUR FACETS OF REPRODUCIBILITY
ORGANIZATION
DOCUMENTATION
AUTOMATION
DISSEMINATION
TIME TO WRAP UP!
TAKE-HOME MESSAGE!
You get more efficient, less redundant science: others can build upon our work!
ANOTHER WAY TO LOOK AT IT...
...YOUR REPRODUCIBILITY JOURNEY!
This image was created by Scriberia for The Turing Way community and is used under a CC-BY licence. The image was obtained from https://zenodo.org/record/3332808.