Python in Life Sciences
@juliocesar_io
https://juliocesar.io
Federating Data for Science
- About federation
- Architecture
- Tech Stack
- Ariadne GraphQL
- Apollo Gateway
- Nvidia Clara
- Hands-on demo
Agenda
About Federation
It's been a challenge in bioinformatics.
Federation is the ability of multiple independent resources to act like a single resource. Cloud computing itself is a federation of resources, so the many assets, identities, configurations and other details of a cloud computing solution must be federated to make cloud computing practical’.
Architecture
High Level Components
The Stack
-
Gateway API
- Implementing Services Ariadne/GraphQL
- Federated Server Apollo
- DynamoDB Managed Schemas
-
Aggregation Service
- HDFS
- Async tasks
- Merge results
-
Computing Service
- Nvidia Clara SDK (Models, TensorRT Server, DICOM/PACS Interface)
- Kubernetes / Docker / Helm
- Management API
- Nvidia Render Server
Implementing Services
Ariadne GraphQL
- Schema First SDL
- Federated Schema
- Apollo Support
- The Python you know and love :)
Services Definition
Ariadne GraphQL
The Project structure
Ariadne GraphQL
type Query {
models(id: ID, name: String, platform: String): [Model]
}
type Model @key(fields: "id"){
id: ID!
name: String!
platform: String!
data_type: String!
kind: String!
url: String!
version: String!
}Schema Definition
Ariadne GraphQL
type Query {
medicalImage(id: ID): [MedicalImage]
}
type Model @key(fields: "id") @extends {
id: ID! @external
medicalImages: [MedicalImage]
}
type MedicalImage @key(fields: "id") {
id: ID!
type: String
format: String
url: String!
model_id: Int!
}AI Model Service
Medical Imaging Service
with the @extends and @external we reference the relationship with the AI Model Service
model = FederatedObjectType("Model")
query = QueryType()
@model.reference_resolver
def resolve_model_reference(_, _info, representation):
return get_model_by_id(representation.get("id"))
schema = make_federated_schema("schema_file.graphql", [query, model])
Resolvers Reference
Ariadne GraphQL
AI Model Service
Ariadne GraphQL
query = QueryType()
model = FederatedObjectType("Model")
medicalImage = FederatedObjectType("MedicalImage")
@model.resolve_reference
def resolve_model(representation):
model_id = representation.get('id')
return Model(id=model_id)
@model.field("medicalImages")
def resolve_model_images(obj, info):
kwargs = {
"model_id": obj.id
}
return self.ds.get_medical_image(**kwargs)
schema = make_federated_schema("schema_file.graphql", [query, model, medicalImage])Medical Imaging Service
Resolvers Reference
Making Queries on each service
Ariadne GraphQL
Federated Gateway on Apollo
Apollo Server
Single data graph that provides a unified interface for querying all of your backing data sources. This allows clients to fetch data from any number of sources simultaneously, without needing to know which data comes from which source.
Federated Gateway on Apollo
Apollo Server
Project Structure
Apollo Server
Registering Services
Apollo Server
const gateway = new ApolloGateway({
serviceList: [
{ name: 'ai_model_service', url: 'http://localhost:4001' },
{ name: 'medical_imaging_service', url: 'http://localhost:4002' },
{ name: 'metrics_service', url: 'http://localhost:4003' }
]
});
const server = new ApolloServer({ gateway });
server.listen();Schemas and resolvers live in the implementing services. The gateway serves only to plan and execute GraphQL operations across those implementing services
A managed way to register services with DynamoDB
Apollo Server
A managed way to register services with DynamoDB
Apollo Server
Making Federated Queries
Apollo Server
The Response
Apollo Server
{
"data": {
"models": [
{
"id": "100",
"name": "segmentation_mri_brain_tumors_br16_t1c2tc_v1",
"platform": "tensorflow_graphdef",
"data_type": "TYPE_FP32",
"kind": "KIND_GPU",
"url": "https://api.ngc.nvidia.com/v2/resources/nvidia/clara/clara_ai_brain_tumor_pipeline/versions/0.6.0-2006.4/zip",
"medicalImages": [
{
"id": "666",
"type": "MRI_T1c",
"format": "DICOM",
"url": "https://medical-imaging-service.s3.amazonaws.com/mock-sources/source-01/IMG0000.dcm"
},
{
"id": "667",
"type": "MRI_T1c",
"format": "DICOM",
"url": "https://medical-imaging-service.s3.amazonaws.com/mock-sources/source-01/IMG0001.dcm"
},
{
"id": "668",
"type": "MRI_T1c",
"format": "DICOM",
"url": "https://medical-imaging-service.s3.amazonaws.com/mock-sources/source-01/IMG0002.dcm"
}
.....
...
..
.Computing Service
Nvidia Clara
Managing and scaling Imaging, Genomics, and Video Processing workloads. It uses Kubernetes under the hood to define a multi-staged container-based pipeline.
Architecture
Nvidia Clara
Service Interface Definition
Nvidia Clara
Project Deployment in AWS
Nvidia Clara
Using the services to run a model with Federated data
Nvidia Clara
{
"pipeline_tag": "brain-tumor-pipeline",
"input_tag": "dcm",
"use_cache": true,
"query": "query {\r\n models(name:\"segmentation_mri_brain_tumors_br16_t1c2tc_v1\"){\r\n id\r\n name\r\n platform\r\n data_type\r\n kind\r\n url\r\n medicalImages{\r\n id\r\n type\r\n format\r\n url\r\n }\r\n }\r\n}\r\n"
}{
"status": "RUNNIG",
"jobs": "http://ec2-18-220-78-85.us-east-2.compute.amazonaws.com:32002/jobs/",
"render": "http://ec2-18-220-78-85.us-east-2.compute.amazonaws.com:8080/renderserver"
}Response
The Clara Process
Nvidia Clara
The Render Server
Nvidia Clara
Hands-on Demo
- Showcase several services of the architecture
- Use the gateway to get federated data
- Compute a model on a Nvidia Tesla Tensor Core GPU & Visualize the results
What's next?
- Federated Learning
- More models
- Add Business Logic
- Distributed Computing
- Build a scientific collaboration platform
on Github
@juliocesar_io
https://juliocesar.io
Python in life sciences, federating data for science
By Julio César
Python in life sciences, federating data for science
A data federated architecture for training AI models using Ariadne, Apollo and Nvidia Clara computing platform in Kubernetes for life sciences applications.
