Wet lab  Web lab

Open Science in Research

Open Science Symposium 2018

Anisha Keshavan

University of Washington

eScience Institute & Institute for Neuroengineering

The Brain.


Mental Health



We need a lot of data.

Because we are all so unique

Scaling up is easy, right?

We have computers

source: flickr

We have algorithms

A markov chain (wikimedia)

But we're human.

We have our strengths

45 MPH


We should combine our strengths


  • fast
  • sometimes stupid


  • slow
  • sometimes smart

By designing optimal interfaces

Design matters.

Open design matters.

we can overcome our limitations

and optimize for our strengths

  • we are slow: design for collaboration
  • we are smart: design for the human visual system

in science,

The internet.

Pediatric Mental Health

  • Healthy Brain Network Biobank = 10k MRI
  • How do brain tissue volumes change during development?
  • Children move = bad quality image
  • Visual inspection @ every stage of analysis

Lets go to the Web Lab


keshavan et al., 2017

Expert Feedback

Mindcontrol is nice, but its slow.

 I want to play on my phone"

Dr. Satra Ghosh, MIT


Brain Data Review


Keshavan, Anisha, Jason Yeatman, and Ariel Rokem. "Combining citizen science and deep learning to amplify expertise in neuroimaging." bioRxiv (2018): 363382. (under review)


Swipe Left

Swipe Right

Win Prizes

Many Ratings

But often, no agreement



Need to "down-weight" some raters

image id rater 0 rater 1
1 0 1
2 1 -

each rater is a feature

each image

is a sample

missing data

Rater importance vs # ratings

Corrected Ratings

Predict image quality

> 100,000 ratings
> 400 users
0.99 AUC = near perfect

Thanks eLife Innovation!


the app to annotate abstracts

Reproducibility crisis in neuroimaging

  • Results are hard to replicate
  • Neuroimaging datasets are small
  • Meta-analysis → more stable results?
    • we need metadata


  • Start small, annotate only sample size
  • Client-side NLP to highlight numbers
  • Tap on the numbers corresponding to sample size

# participants in neuroimaging studies on Autism

How do brain tissue volumes change during development?

back to our question:

A fine line


too long, didn't read in internet-speak

open science is honorable

of course,

open science is strategic

in the age of data-driven discovery

Thanks to my Web Lab!

Ariel Rokem

Jason Yeatman

Satra Ghosh

Dylan Nielson

Sharabesh Ramesh

Dave Kennedy

JB Poline

Neel Somani

Sharabesh Ramesh

Katie Bottenhorn

Arno Klein

Amanda Easson

Adam Richie-Halford

Josh Smith

Greg Kiar

Sook-Lei Liew

Chris Madan

Chris Markiewicz

Elizabeth Dupre

Kesshi Jordan

Rob Fatland

Shima Abadi

Naomi Penfold

Andreea Hrincu

Adam Thomas




Microsoft Azure

eLife Innovation

UW Institute for Neuroengineering

eScience Institute


All braindr raters


wet to web, longer

By Anisha Keshavan

wet to web, longer

  • 783