#BuildStuffUA @hannelita
If you are bored, check my Assembly talk (BuildStuff LT) - lots of GIFs :)
http://bit.ly/2gxoIIF
https://github.com/neo4j-contrib/neo4j_doc_manager
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
"In 2006, Neal Ford coined the term
polyglot programming, to express the
idea that applications should be written in
a mix of languages to take advantage of
the fact that different languages are
suitable for tackling different problems.
Complex applications combine different
types of problems, so picking the right
language for each job may be more
productive than trying to fit all aspects into
a single language."
https://en.wikipedia.org/wiki/Polyglot_persistence
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
It might not be intuitive how to build some of these
structures with MongoDB
#BuildStuffUA @hannelita
Graphs are everywhere
TEAM, Neo4j
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
https://github.com/10gen-labs/mongo-connector
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
You
MC
Mongo Connector
#BuildStuffUA @hannelita
You
Call Mongo Connector
MC
#BuildStuffUA @hannelita
You
Call Mongo Connector
with Replica
MC
Hi!
#BuildStuffUA @hannelita
You
Points where's your Mongo
MC
#BuildStuffUA @hannelita
You
Points where's your Mongo
Points where is the other database
MC
DM
Elasticsearch
Solr
(Doc Manager)
MC
DM
Elasticsearch
Solr
(Doc Manager)
Mongo Replica watches Mongo Actions
Call actions on a Doc Manager (custom interface for the Mongo Connector)
We can translate these actions
into a Graph Structure
#BuildStuffUA @hannelita
mongo-connector (pip)
py2neo (neo4j)
#BuildStuffUA @hannelita
class DocManager(DocManagerBase):
def __init__(self, url, auto_commit_interval=DEFAULT_COMMIT_INTERVAL,
unique_key='_id', chunk_size=DEFAULT_MAX_BULK, **kwargs):
def upsert(self, doc, namespace, timestamp):
def bulk_upsert(self, docs, namespace, timestamp):
def update(self, document_id, update_spec, namespace, timestamp):
def remove(self, document_id, namespace, timestamp):
def search(self, start_ts, end_ts):
We can retrieve Mongo commands with this interface class
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
class DocManager(DocManagerBase):
def __init__(self, url, auto_commit_interval=DEFAULT_COMMIT_INTERVAL,
unique_key='_id', chunk_size=DEFAULT_MAX_BULK, **kwargs):
def upsert(self, doc, namespace, timestamp):
def bulk_upsert(self, docs, namespace, timestamp):
def update(self, document_id, update_spec, namespace, timestamp):
def remove(self, document_id, namespace, timestamp):
def search(self, start_ts, end_ts):
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
db.talks.insert( { "session":
#BuildStuffUA @hannelita
db.talks.insert( { "session":
#BuildStuffUA @hannelita
db.talks.insert( { "session": ...
Document:talks
Root node in Neo4j
#BuildStuffUA @hannelita
{ "session": { "title": "12 Years of Spring: An Open Source Journey", "topics": ["keynote", "spring"], "room": "Auditorium", "speaker": { "name": "Juergen Hoeller" } }, "venue": "Olympia Stadium" }
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
Keys which values are another JSON become nodes
{ "session": { "title": "12 Years of Spring: An Open Source Journey", ... }
#BuildStuffUA @hannelita
{ "session": { "title": "12 Years of Spring: An Open Source Journey", ... }
Keys which values are another JSON become nodes
#BuildStuffUA @hannelita
{ "session": { "title": "12 Years of Spring: An Open Source Journey", ... }
Document:session
Keys which values are another JSON become nodes
#BuildStuffUA @hannelita
{ "session": { "title": "12 Years of Spring: An Open Source Journey", ... }
Document:session
Keys which values are another JSON become nodes
#BuildStuffUA @hannelita
They get a composite label - :Document + key
{ "session": { "title": "12 Years of Spring: An Open Source Journey", ... }
Document:session
#BuildStuffUA @hannelita
The JSON value of that key is translated into node properties
{ "session": { "title": "12 Years of Spring: An Open Source Journey",
"topics": ["keynote", "spring"], "room": "Auditorium" }
#BuildStuffUA @hannelita
{ "session": { "title": "12 Years of Spring: An Open Source Journey",
"topics": ["keynote", "spring"], "room": "Auditorium" }
Document:session
title: "12 Years of Spring: An Open Source Journey"
topics: ["keynote", "spring"]
room: "Auditorium"
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
Document:session
title: "12 Years of Spring: An Open Source Journey"
topics: ["keynote", "spring"]
room: "Auditorium" _id: 324553ab342c324d7ff _ts: 621002135233213
#BuildStuffUA @hannelita
All the top level generated nodes will be connected to the root node
Document:session
Document:talks
#BuildStuffUA @hannelita
The relationship is a concatenation of the keys:
Document:session
Document:talks
talks_session
#BuildStuffUA @hannelita
Top level properties go the the root node:
{ "session": { ... }, "venue": "Olympia Stadium" }
#BuildStuffUA @hannelita
Top level properties go the the root node:
{ "session": { ... }, "venue": "Olympia Stadium" }
Document:talks
venue: "Olympia Stadium" _id: 324553ab342c324d7ff _ts: 621002135233213
#BuildStuffUA @hannelita
{ "session": { ... }, "venue": { "address": "...", "city": "Kiev" }, }
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
Document:session
Document:talks
talks_session
Document:venue
talks_venue
"session" : { "title" : "12 Years of Spring: An Open Source Journey", "abstract" : "History os Spring Framework", "speaker" : { "name" : "Josh Long", "company" : "A company" } }
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
"session" : { "title" : "12 Years of Spring: An Open Source Journey", "abstract" : "History os Spring Framework", "speaker" : { "name" : "Josh Long", "company" : "A company" } }
#BuildStuffUA @hannelita
Document:session
Document:speaker
Child node
Parent node
session_speaker
#BuildStuffUA @hannelita
Document:session
Document:speaker
Document:talks
#BuildStuffUA @hannelita
"session" : { "tracks": [{ "main":"Python" }, { "second":"Data" }] ... }
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
Document:session
Document:track0
talks_track0
talks_track1
Document:track1
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
"user": { "name": "Hanneli", "account_id": "32434ab2341192", "url": "medium.com/@hannelita" }
#BuildStuffUA @hannelita
"account" : { "number": "326708", "id": "32434ab2341192" }
user_account
Document:user
Document:account
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
"include": ["test.talks", "docs.info"] (config.json file)
#BuildStuffUA @hannelita
mongo-connector -m localhost:27017 -t http://localhost:7474/db/data -d neo4j_doc_manager -i room,timeslot,title
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
mongo-conenctor:
https://github.com/10gen-labs/mongo-connector
neo4j-doc-manager:
https://github.com/neo4j-contrib/neo4j_doc_manager
#BuildStuffUA @hannelita
Neo4j Cassandra connector :)
https://github.com/neo4j-contrib/neo4j-cassandra-connector
#BuildStuffUA @hannelita
#BuildStuffUA @hannelita
Questions?
hannelita@gmail.com
@hannelita