CHRIS - A Service for Collecting, Organizing, Processing, and Sharing Medical Image Data in the OpenCloud

 

Rudolph Pienaar, PhD

Jorge Bernal, PhD

Ata Turk, PhD

 

Boston Children's Hospital

Boston University

 

RedHat, Inc.


introduction

new technologies and "paradigms" are changing how we think of "data" and "compute".

both are becoming increasingly "mobile"

ChRIS

Research

Integration

Service

ChRIS

need to address the problem of "simple" distributed/cloud computing from medical compute perspective


overview / goals

as low as possible barrier of entry to developer

not be concerned about infrastructure details

write/develop as if running on a local (*nix) environment

be able to deploy the entire "cloud" system locally for testing/developing/deployment


overview / goals


overview / goals

how do I run this on the cloud?

how do I run this on "someone else's computer"?


overview / goals


overview / goals

containerization about the app itself

pervasive use of docker and docker hub

system handles I/O using simple FS abstractions

RESTful services -- multiple front ends to a single backend


overview / goals


overview / goals


containerization


CHRIS...

CHRIS is a bio-medical data workflow manage that allows easy and intuitive collection, analysis, and sharing of data between parties.

  • allows for advanced view rendering
  • allows for sharing of images
  • allows for immersive real time collaboration
  • allows for post-processing of images

many data sources... many compute sources...

via here

connect data here

to here...

and compute!


system topology

Main web server local data repository

remote compute nodes

data source (e.g. in hospital PACS)


distributed design...

web server

coordinator

data handler

compute handler


distributed design...

Red Hat collaboration


io

local web server

data descriptor

data handler


io

pfurl --verb POST --raw --http 172.17.0.2:5055/api/v1/cmd --msg \
'{  "action": "pushPath",
    "meta": {
        "remote": {
            "key":         "someKey"
        },
        "local": {
            "path":         "/home/data"
        },
        "transport": {
            "mechanism":    "compress",
            "compress": {
                "encoding": "base64",
                "archive":  "zip",
                "unpack":   true,
                "cleanup":  true
            }
        }
    }
}' --quiet --jsonpprintindent 4  

compute

local web server

exec descriptor

exec handler


compute

pfurl --verb POST --raw --http 172.17.0.2:5010/api/v1/cmd \
     --jsonwrapper 'payload' --msg \
'{  "action": "run",
        "meta": {
                "cmd":      "cal 7 1970",
                "auid":     "rudolphpienaar",
                "jid":      "cal-job-1234",
                "threaded": true
        }
}' --quiet --jsonpprintindent 4

compute... using a container

pfurl --verb POST --raw --http 10.17.24.163:5010/api/v1/cmd 
--jsonwrapper 'payload' --msg '
        {   "action": "run",
            "meta":  {
                "cmd":      "$execshell $selfpath/$selfexec --prefix test- --sleepLength 0 /share/incoming /share/outgoing",
                "auid":     "rudolphpienaar",
                "jid":      "simpledsapp-1",
                "threaded": true,
                "container":   {
                        "target": {
                            "image":            "fnndsc/pl-simpledsapp",
                            "cmdParse":         true
                        },
                        "manager": {
                            "image":            "fnndsc/swarm",
                            "app":              "swarm.py",
                            "env":  {
                                "shareDir":     "/home/tmp/share",
                                "serviceType":  "docker",
                                "serviceName":  "testService"
                            }
                        }
                }
            }
        }
'

coordination

timing can be complex...

PUSH data

separate coordinator service

WAIT!

EXEC on data

WAIT!

PULL data

WAIT!


coordination


coordination

            "meta-compute":  {
                "comment":  "The $ strings in the <cmd> are substituted with values found my running the plugin w/o any args. NB: the incoming and outgoing dirs need to be properly handled! In this example they are mapped from the <key> location in <pfioh> appropriately.",
                "cmd":      "$execshell $selfpath/$selfexec --prefix test- --sleepLength 0 /share/incoming /share/outgoing",
                "auid":     "rudolphpienaar",
                "jid":      "simpledsapp-1",
                "threaded": true,
                "container":   {
                        "target": {
                            "comment":          "The plugin to run. The <cmdParse>, if true, means substitute $ in the <cmd> string.",
                            "image":            "fnndsc/pl-simpledsapp",
                            "cmdParse":         true
                        },
                        "manager": {
                            "comment":          "The manager container to run, with relevant <env> settings.",
                            "image":            "fnndsc/swarm",
                            "app":              "swarm.py",
                            "env":  {
                                "meta-store":   "key",
                                "serviceType":  "docker",
                                "shareDir":     "%shareDir",
                                "serviceName":  "testService"
                            }
                        }
                },
                "comment":              "The <service> name is hardcoded in the pfcon source and denotes the IP and port of the <pfioh> service.",
                "service":              "pangea"
            }
        }

UI concepts


UI concepts


UI concepts


app to CHRIS...


BU partnership

Boston University Computer Science

Mentoring students on Cloud computing

Academic partner on the MOC


RedHat partnership

RedHat is active partner developing CHRIS infrastructure to the MOC

CHRIS as platform to help foster community and opensource development in the cloud using this paradigm

RedHat interested in helping use CHRIS to be a platform for "hack-a-thons" based on openswift, kubernetes, etc.


MOC


references... CHIPS/ChRIS

  • https://github.com/FNNDSC/ChRIS_ultron_backEnd
  • https://github.com/FNNDSC/ChRIS_ultron_frontEnd
    

references... services

 
  • https://github.com/FNNDSC/viewerjs
  • https://github.com/FNNDSC/gcjs
  • https://github.com/FNNDSC/fmjs
  • https://github.com/FNNDSC/toolbarjs
  • https://github.com/FNNDSC/rboxjs
  • https://github.com/FNNDSC/rendererjs
  • https://github.com/FNNDSC/thbarjs
  • https://github.com/FNNDSC/pman
  • https://github.com/FNNDSC/pfioh
  • https://github.com/FNNDSC/pfcon

references... viewers

  • https://goxtk.com
    
  • https://github.com/FNNDSC/ami

team

  • Jorge Bernal
  • Nicolas Rannou
  • Rudolph  Pienaar
  • Yangming Ou
  • Daniel Haehn
  • Daniel Ginsburg
  • Ellen Grant

BCH Team

BU Team

  • Orran Krieger
  • Ata Turk
  • Aditya Awalker

RedHat

  • Dan McPherson
  • and others!

fin!

Thank you!

NAMIC_2018

By Rudolph Pienaar

NAMIC_2018

NAMIC project week, MIT, Jan 2018

  • 1,403