From documents

to graphs

#PyConIE

Hi!

  • Computer Engineer
  • Programming
  • Electronics
  • Math <3 <3
  • Physics
  • Lego
  • Meetups
  • Animals
  • Coffee
  • GIFs
  • Pokémon

Keynote at Python Brasil:

http://slides.com/hannelitavante-hannelita/rust-type-system-pybr12

Disclaimer

Views are on my own

Project from late 2015

Mostly for Neo4j 2.x

Agenda

  • Quick note about document oriented databases
  • Graph databases can help your data model
  • Creating connectors for MongoDB
  • neo4j_doc_manager general architecture
  • Data mapping
  • Challenges

"We need to restructure our data"

"Relational databases are not enough"

Document Oriented DB

  • Flexible data model
  • Easy to get started
  • Easy to represent the data

Store data as Documents!

Imagine that we have talks of a conference

Our Documents

Agenda

  • Quick note about document oriented databases
  • Graph databases can help your data model
  • Creating connectors for MongoDB
  • neo4j_doc_manager general architecture
  • Data mapping
  • Challenges

Sometimes we need to get some extra information

Possible questions

  • Which talks have a specific topic (ex: 'Databases')
  • Which speakers will also talk about this topic?
  • What are the sessions that will be hold into Auditorium and are about this topic?

These are common questions

More questions

  • Assuming that I do not want to change rooms, what is the best room to stay to get a higher number of sessions of a specific topic?

Further work

  • Recommendation system for the talks
  • Recommendation system for speakers
  • Build a tool to automatically build the sessions timetable based on topic distribution

Looks like we need some graphs!

Graphs are everywhere

TEAM, Neo4j

We can build graphs with information from Mongo

Agenda

  • Quick note about document oriented databases
  • Graph databases can help your data model
  • Creating connectors for MongoDB
  • neo4j_doc_manager general architecture
  • Data mapping
  • Challenges

From Documents to Graphs

Mongo Connetor

https://github.com/10gen-labs/mongo-connector

Mongo Connector

You

MC

Mongo Connector

Mongo Connector

You

Call Mongo Connector

MC

Mongo Connector

You

Call Mongo Connector

MC

Hi!

Mongo Connector

You

Points where's your Mongo

MC

Mongo Connector

You

Points where's your Mongo

Points where is the other database

MC

DM

Elasticsearch

Solr

(Doc Manager)

Mongo Connector

MC

DM

Elasticsearch

Solr

(Doc Manager)

Creates a thread to watch Mongo Actions (replica)

Mongo Connector

MC

DM

Elasticsearch

Solr

(Doc Manager)

Creates a thread to watch Mongo Actions

Call actions on a Doc Manager

We can translate these actions

into a Graph Structure

Agenda

  • Quick note about document oriented databases
  • Graph databases can help your data model
  • Creating connectors for MongoDB
  • neo4j_doc_manager general architecture
  • Data mapping
  • Challenges

Neo4j Doc Manager

mongo-connector (pip)

py2neo (neo4j)

class DocManager(DocManagerBase):

  def __init__(self, url, auto_commit_interval=DEFAULT_COMMIT_INTERVAL,
                 unique_key='_id', chunk_size=DEFAULT_MAX_BULK, **kwargs):
    

  def upsert(self, doc, namespace, timestamp):

  def bulk_upsert(self, docs, namespace, timestamp):

  def update(self, document_id, update_spec, namespace, timestamp):

  def remove(self, document_id, namespace, timestamp):
    
  def search(self, start_ts, end_ts):

We can retrieve Mongo commands with this interface class

We support Python 2 and Python 3

It will run like an auto importer. You just need to provide the database endpoints

Agenda

  • Quick note about document oriented databases
  • Graph databases can help your data model
  • Creating connectors for MongoDB
  • neo4j_doc_manager general architecture
  • Data mapping
  • Challenges

Sync Mongo with Neo4j

db.talks.insert(  { "session":
db.talks.insert(  { "session":
db.talks.insert(  { "session": ...

Document:talks

Root node in Neo4j

{
  "session": {
    "title": "12 Years of Spring: An Open Source Journey"
  },
  "topics":  ["keynote", "spring"],
  "room": "Auditorium",
  "speaker": {
    "name": "Juergen Hoeller"
  }
}
{
  "session": {
    "title": "12 Years of Spring: An Open Source Journey"
  },
  "topics":  ["keynote", "spring"],
  "room": "Auditorium",
  "speaker": {
    "name": "Juergen Hoeller"
  }
}

Document:session

Document:speaker

{
  "session": {
    "title": "12 Years of Spring: An Open Source Journey"
  },
  "topics":  ["keynote", "spring"],
  "room": "Auditorium",
  "speaker": {
    "name": "Juergen Hoeller"
  }
}

JSON properties become node properties

All the nodes are connected to the root node

Nested documents

"session" : {
    "title" : "12 Years of Spring: An Open Source Journey",
    "abstract" : "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015.",
    "conference" : {
      "city" : "London"
    }
  }

Nested documents

"session" : {
    "title" : "12 Years of Spring: An Open Source Journey",
    "abstract" : "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015.",
    "conference" : {
      "city" : "Dublin"
    }
  }

Nested documents

Document:session

Document:conference

Child node

Parent node

JSON array

"session" : { 
  "tracks": [{ "main":"Python" },
            { "second":"Data" }]
... }

JSON array

Document:session

Document:track0

talks_track0

talks_track1

Document:track1

We also support explicit ids to create a relationship

Explicit ids

{
  "name": "Hanneli",
  "account_id": "32434ab2341192",
  "url": "medium.com/@hannelita"
}

session_account

Document:session

Document:account

We also support a configuration file if you don't want to import all your data

We can specify the namespaces that we want to import:

"include": ["test.talks", "docs.info"] (config.json file)

It is also possible to specify the fields and collections via command line:

mongo-connector -m localhost:27017 -t http://localhost:7474/db/data -d neo4j_doc_manager -i room,timeslot,title

 

Agenda

  • Quick note about document oriented databases
  • Graph databases can help your data model
  • Creating connectors for MongoDB
  • neo4j_doc_manager general architecture
  • Data mapping
  • Challenges

1. Data model is a challenge.

Different representations (Documents -> Graphs)

2. Avoiding orphan nodes

3. Batching - maximum of 10k per batch

Projects

mongo-conenctor: 

https://github.com/10gen-labs/mongo-connector

neo4j-doc-manager: 

https://github.com/neo4j-contrib/neo4j_doc_manager

 

Next Projects

Neo4j Cassandra connector :) 

https://github.com/neo4j-contrib/neo4j-cassandra-connector

Lessons learned

  • Polyglot persistence is great
  • Graphs can be very useful for simplifying queries
  • Real applications: fraud detection
  • University (UK) is using it :)

Thank you :)

Questions?

 

hannelita@gmail.com

@hannelita