Python in Life Sciences

@juliocesar_io

https://juliocesar.io

Federating Data for Science

About federation
Architecture
Tech Stack
- Ariadne GraphQL
- Apollo Gateway
- Nvidia Clara
Hands-on demo

Agenda

About Federation

It's been a challenge in bioinformatics.

Federation is the ability of multiple independent resources to act like a single resource. Cloud computing itself is a federation of resources, so the many assets, identities, configurations and other details of a cloud computing solution must be federated to make cloud computing practical’.

Architecture

High Level Components

The Stack

Gateway API
- Implementing Services Ariadne/GraphQL
- Federated Server Apollo
- DynamoDB Managed Schemas
Aggregation Service
- HDFS
- Async tasks
- Merge results
Computing Service
- Nvidia Clara SDK (Models, TensorRT Server, DICOM/PACS Interface)
- Kubernetes / Docker / Helm
- Management API
- Nvidia Render Server

Implementing Services

Ariadne GraphQL

Schema First SDL
Federated Schema
Apollo Support
The Python you know and love :)

Services Definition

Ariadne GraphQL

The Project structure

Ariadne GraphQL

type Query {
    models(id: ID, name: String, platform: String): [Model]
}

type Model @key(fields: "id"){
    id: ID!
    name: String!
    platform: String!
    data_type: String!
    kind: String!
    url: String!
    version: String!
}

Schema Definition

Ariadne GraphQL

type Query {
    medicalImage(id: ID): [MedicalImage]
}

type Model @key(fields: "id") @extends {
    id: ID! @external
    medicalImages: [MedicalImage]
}

type MedicalImage @key(fields: "id") {
    id: ID!
    type: String
    format: String
    url: String!
    model_id: Int!
}

AI Model Service

Medical Imaging Service

with the @extends and @external we reference the relationship with the AI Model Service

model = FederatedObjectType("Model")
query = QueryType()

@model.reference_resolver
def resolve_model_reference(_, _info, representation):
    return get_model_by_id(representation.get("id"))
  
  
schema = make_federated_schema("schema_file.graphql", [query, model])

Resolvers Reference

Ariadne GraphQL

AI Model Service

Ariadne GraphQL

 query = QueryType()
 model = FederatedObjectType("Model")
 medicalImage = FederatedObjectType("MedicalImage")
  
@model.resolve_reference
 def resolve_model(representation):
    model_id = representation.get('id')
    return Model(id=model_id)
  
  
@model.field("medicalImages")
def resolve_model_images(obj, info):
    kwargs = {
       "model_id": obj.id
    }

    return self.ds.get_medical_image(**kwargs)
  
schema = make_federated_schema("schema_file.graphql", [query, model, medicalImage])

Medical Imaging Service

Resolvers Reference

Making Queries on each service

Ariadne GraphQL

Federated Gateway on Apollo

Apollo Server

Single data graph that provides a unified interface for querying all of your backing data sources. This allows clients to fetch data from any number of sources simultaneously, without needing to know which data comes from which source.

Federated Gateway on Apollo

Apollo Server

Project Structure

Apollo Server

Registering Services

Apollo Server

const gateway = new ApolloGateway({
  serviceList: [
    { name: 'ai_model_service', url: 'http://localhost:4001' },
    { name: 'medical_imaging_service', url: 'http://localhost:4002' },
    { name: 'metrics_service', url: 'http://localhost:4003' }
  ]
});

const server = new ApolloServer({ gateway });
server.listen();

Schemas and resolvers live in the implementing services. The gateway serves only to plan and execute GraphQL operations across those implementing services

A managed way to register services with DynamoDB

Apollo Server

A managed way to register services with DynamoDB

Apollo Server

Making Federated Queries

Apollo Server

The Response

Apollo Server

{
    "data": {
        "models": [
            {
                "id": "100",
                "name": "segmentation_mri_brain_tumors_br16_t1c2tc_v1",
                "platform": "tensorflow_graphdef",
                "data_type": "TYPE_FP32",
                "kind": "KIND_GPU",
                "url": "https://api.ngc.nvidia.com/v2/resources/nvidia/clara/clara_ai_brain_tumor_pipeline/versions/0.6.0-2006.4/zip",
                "medicalImages": [
                    {
                        "id": "666",
                        "type": "MRI_T1c",
                        "format": "DICOM",
                        "url": "https://medical-imaging-service.s3.amazonaws.com/mock-sources/source-01/IMG0000.dcm"
                    },
                    {
                        "id": "667",
                        "type": "MRI_T1c",
                        "format": "DICOM",
                        "url": "https://medical-imaging-service.s3.amazonaws.com/mock-sources/source-01/IMG0001.dcm"
                    },
                    {
                        "id": "668",
                        "type": "MRI_T1c",
                        "format": "DICOM",
                        "url": "https://medical-imaging-service.s3.amazonaws.com/mock-sources/source-01/IMG0002.dcm"
                    }
                    .....
                    ...
                    ..
                    .

Computing Service

Nvidia Clara

Managing and scaling Imaging, Genomics, and Video Processing workloads. It uses Kubernetes under the hood to define a multi-staged container-based pipeline.

Architecture

Nvidia Clara

Service Interface Definition

Nvidia Clara

Project Deployment in AWS

Nvidia Clara

Using the services to run a model with Federated data

Nvidia Clara

{
    "pipeline_tag": "brain-tumor-pipeline",
    "input_tag": "dcm",
    "use_cache": true,
    "query": "query {\r\n  models(name:\"segmentation_mri_brain_tumors_br16_t1c2tc_v1\"){\r\n    id\r\n    name\r\n    platform\r\n    data_type\r\n    kind\r\n    url\r\n    medicalImages{\r\n      id\r\n      type\r\n      format\r\n      url\r\n    }\r\n  }\r\n}\r\n"
}

{
    "status": "RUNNIG",
    "jobs": "http://ec2-18-220-78-85.us-east-2.compute.amazonaws.com:32002/jobs/",
    "render": "http://ec2-18-220-78-85.us-east-2.compute.amazonaws.com:8080/renderserver"
}

Response

The Clara Process