Reproducibility

using open data and open source 

 

Ann Gledson, Research Software Engineer

The University of Manchester
 

https://slides.com/anngledson/data-software-sustainability

Overview

  • Open Science
     
  • Open data, open source and open methodology
     
  • Environment data-set example

Open Science

  • Open Data
  • Open Source
  • Open Methodology
  • Open Peer Review
  • Open Access
  • Open Educational Resources

For overview: see NERC 'Constructing a Digital Environment' channel:                   Open Science webinar by Helen Glaves (British Geological Survey)

The nature of research

  • Research projects have short lifespans
  • Long-term ideas and goals
  • Non-continuous process
    • Funding
    • Loss of key developers / analysts
    • Changes in scope or delays
    • Changes in storage devices
  •  Difficult to plan resources
    • Data storage
    • Code re-use

Open Data

  • Reduce repetition of data cleaning and wrangling tasks
  • Focus more on key research questions
  • Reproducibility -> Trust!
  • Visibility:
    • E.g. Nature Scientific Data journal
    • Citations
  • Collaboration, communities and networking 
  • Funding opportunities

Data wrangling

FAIR data principles

Nature Scientific Data: principles

Credit: Scientists who share their data in a FAIR manner deserve appropriate credit. 

Re-use: Standardized and detailed descriptions make data easier to find and reuse. 

Quality: Critical evaluation is needed to verify experimental rigour.

Discovery: Scientists should be able to easily find datasets that are relevant. 

Open: Scientists work best when they can easily connect and collaborate.

Service: Committed to providing excellent service to both authors and readers.

(Full version: https://www.nature.com/sdata/about/principles)

Open Source

  • Reduce repetition of coding effort
  • Community of developers
    • Testing
    • Issue tracking
    • Pull requests
  • Reproducibility 
    • Github releases: Snapshots of code
  • Visibility:
    • Software Citation in Github
    • Findability
  • Master branch is public release code – each version on master is tagged with a semantic version number (https://semver.org/). Code in master works.
     
  • Release branch (if present) is temporary. Preparation for version release.  (Or use develop)
     
  • Commits on develop are stable and should work but some unanticipated unexpected behaviour may be present.
     
  • Feature branches are temporary*. Commits on these may be broken/unstable. 

Gitflow branch types

Gitflow release process

  • Tests run on develop branch – before making a release, make sure that all tests are passing (can be run as a CI job).
     
  • (optional) Create an issue that describes what changes have been made since last release version. Make sure everyone agrees that behaviour is as expected. Link to passing tests.
     
  • Merge develop into master and tag with the version number.

Open Methodology

  • Data analysis apps
    • R-Shiney Apps
    • Jupyter Notebook
    • Streamlit
    • Plotly Dash
    • Bokeh
       
  • Computational Workflows
    • FAIR workflows
    • Carole Goble UoM work (see links)

Example

Environment data

  • Automatic Urban and Rural Network (AURN)
    • NOx, SO2, O3, NO2, PM10, PM2.5
  • ​Medical and Environmental Data (Mash-up) Infrastructure (MEDMI)
    • Meteorological: temp, pressure, dewpoint temp, relative humidity
    • Pollens: alnus, ambrosia, artemesia, ..., urtica
  • ​​European Monitoring and Evaluation Programme (EMEP)
    • Model forecast data
    • ​NOx, NO2, SO2, O3, PM10, PM2.5
  • ​Complex extraction process:
    • Multiple data sources
    • Missing data (e.g. sensor down-time)
    • Variable UK area coverage

Cleaning and Imputation 

  • Remove duplicate/unphysical values​
  • Select sites by minimum temporal data coverage
  • scikit-learn (python) used to impute missing data using hourly time series
  • Imputation method
    • Bayesian Ridge
    • Quantile Transformer preprocessing
  • Final data: daily mean / maximum values (or simple daily count)

Regional estimations 

Diffusion method illustrated on fictional postcode regions

  • Regions where sensors exist: take mean
  • Regions with no sensors: take mean of surrounding regions
    • Working outwards until sensors found

Linking GitHub version to DOI

Shared Data

Links

  • FAIR Data: https://www.go-fair.org/fair-principles/

  • Git/GitHub Tutorial:  http://gcapes.github.io/git-course/

  • Gitflow Workflow: https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow

  • Semantic version numbers: https://semver.org/

  • Zenodo (DOIs) https://zenodo.org/

  • Linking GitHub to DOI:  https://guides.github.com/activities/citable-code/

  • Open data webinar - Helen Glaves (BGS):

    • https://www.youtube.com/channel/UCv8vRIuTxCP-DgNMCq9KxqA/videos

  • ​Computational Workflows

    • www.slideshare.net/carolegoble/fair-computational-workflows-249721518

Environment Dataset Links

  • 2016-2019 environment datasets:
    • measurements (original and imputed)
      • https://zenodo.org/record/4416028
      • includes link to extraction and imputation tool set
    • regional estimations (from original and imputed)
      • https://zenodo.org/record/4475652
      • includes link to region_estimators tool
    • Scientific Data paper:
      • https://www.nature.com/articles/s41597-022-01135-6​
  • Visualisation Tool:
    • http://minethegaps.manchester.ac.uk/
    • https://github.com/UoMResearchIT/mine-the-gaps
Made with Slides.com