X

NEO4j

Uniiverse x neo4j

WHO AM I

WHY DO WE NEED A

GRAPH DATABASE?

Persist Social Data

AD-HOC QUERYING

DAILY EMAIL RECOMMENDATIONS

SOCIAL INVITE TOOLS

bUSINESs INTElLIGeNce

BUSINESS INTELLIGENCE

BUSINESS INTELLIGENCE

NUMbers

GRAPH API STACK

INTEGRATING WITH OUR RAILS API

INTEGRATING WITH OUR RAILS API

USER.rb

class User
  include Mongoid::Document
  field :graph_id, type: String
  after_create :insert_in_graph

  def insert_in_graph
    set :graph_id, create_vertex
  end

  def create_vertex
    attrs = {name: @user.name}
    raw_json = RestClient.post("http://graph.db/uniiverse/users", attrs)
    JSON.parse(raw_json)[:id]
  end
end

USER.rb

USER.RB + Sidekiq!

class User
  include Mongoid::Document
  field :graph_id, type: String
  after_create :insert_in_graph

  def insert_in_graph
    InsertUserVertexWorker.perform_async(id)
  end
end

SIDEKIQ VERTEX WORKER

class InsertUserVertexWorker
  include Sidekiq::Worker

  def perform(user_id)
    @user = User.find(user_id)
    @user.set :graph_id, create_vertex
  end

  def create_vertex
    attrs = {name: @user.name}
    raw_json = RestClient.post("http://graph.db/uniiverse/users", attrs)
    JSON.parse(raw_json)[:id]
  end
end         

ASYNC BENEFITS

EXAMPLE: UPSERT WORKER

class UpsertUserVertexWorker
  include Sidekiq::Worker
  sidekiq_options retry: true

  def perform(user_id)
    @user = User.find(user_id)
    graph_id = find_vertex || create_vertex
    @user.set :graph_id, graph_id
  end
  
  def find_vertex
    raw_json = RestClient.get("http://graph.db/uniiverse/users/find_by_mongo_id/#{@user.id}")
    JSON.parse(raw_json)[:id]
  rescue RestClient::ResourceNotFound
    # not found -- that's okay.
  end
  
  def create_vertex
    attrs = {name: @user.name}
    raw_json = RestClient.post("http://graph.db/uniiverse/users", attrs)
    JSON.parse(raw_json)[:id]
  end
end

LISTING.RB + SIDEKIQ!

class ListingCreateWorker
  include Sidekiq::Worker

  def perform(listing_id)
    @listing = Listing.find(listing_id)
    create_vertex # same as other slide
    create_edge
  end

  # user-[created]->listing
  def create_edge
    RestClient.post("http://graph.db/uniiverse/users/#{@listing.host.graph_id}/created_listing/#{@listing.graph_id}")
  endend    

FACEBOOK VERTEX

class FacebookAccount
  field :uid, type: String
  field :graph_id, type: String
  belongs_to :user
  after_create :upsert_in_graph
end

class FacebookVertexWorker
  def perform
    @facebook.set :graph_id, find_vertex || create_vertex
  end

  def find_vertex
    raw_json = RestClient.get("http://graph.db/facebook/users/find_by_uid/#{@facebook.uid}")
    JSON.parse(raw_json)[:id]
  end

  def create_vertex
    # do stuff
  end
end    

FACEBOOK FRIENDS

class FacebookAccount
  def update_friends

    fb = Koala::Facebook::API.new(token)

    friends = fb.get_connections("me", "friends", fields: "name,id,location").map |friend|
      { uid: friend[:uid], etc. }
    end

    RestClient.patch("http://graph.db/facebook/users/#{graph_id}/friends", friends)

  end
end   

PATCH FRIENDS API

patch "/facebook/users/:id/friends" do
  v1 = find_by_id(params[:id])
  params[:friends].each do |friend|
   v2 = find_or_create_vertex(friend)
   create_edge_between v1, v2, :friend, FACEBOOK_GRAPH
  end
end    

FRIENDS SORTED BY

MUTUAL FRIEND COUNTS

DENORMALIZING

MUTUAL FRIEND COUNTS

START a=node(#{id})
MATCH a-[:friend]-b-[:friend]-c, a-[r:friend]-c
WITH COUNT(distinct b) AS mutual_friend_count, r
SET r.mutual_friends = mutual_friend_count;
    

FRIENDS ORDERED BY

MUTUAL FRIEND COUNTS

START n=node(#{id})
MATCH n-[e:friend]-x
WITH x, e.mutual_friends? AS mutual_friends
ORDER BY mutual_friends desc
RETURN mutual_friends, x
    

SEE IT IN ACTION!

lessons learned

PROBLEM:

JVM HEAP out of memory

Neo4j Cache Settings

NEO4J CACHE SETTINGS

PROBLEM: 

Excessive disk usage

Neo4j logical logs

PROBLEM: 

MiGRATIONS

MIGRATIONS

SOLUTION!

get "/system/migrations/remove_attended_edges" do
  Thread.new do
    UNIIVERSE_GRAPH.e(:attended).delete!
  end
  200
end

Problem:

performance in the cloud

PERFORMANCE

AWS c1.xlarge

AWS m3.xlarge

DigitalOcean 48GB SSD

DIGITALOCEAN ROCKS.

PROBLEM:

Aws + DiGITALOCEAN

+ secure connections

DIGITALOCEAN LIMITATIONS?

SOLUTION: AUTOSSH!

AUTOSSH + MONIT:

check process graph_autossh_production with pidfile "/var/run/graph_autossh_production.pid"
    start program = "/etc/init.d/graph_autossh_production.sh start"
    stop program = "/etc/init.d/graph_autossh_production.sh stop"
    

AUTOSSH + INIT.d:

DAEMON=/usr/bin/autossh
DAEMON_OPTS="-2 -f -N -L $PORT:localhost:$PORT $SSH_HOSTNAME -p $SSH_PORT"

start-stop-daemon --start --quiet --exec $DAEMON -- $DAEMON_OPTS
    

problem:

MONITORING THE JVM

JCONSOLE

1. launch ssh tunneL

ssh -N -L9993:localhost:9993 -L9994:localhost:9994 graph.db

2. OPEN JCONSOLE

AGAINST PRODUCTION

wget http://archive.apache.org/dist/tomcat/tomcat-7/v7.0.32/bin/extras/catalina-jmx-remote.jar 1> /dev/null 2> /dev/null

jconsole -debug -J"-Djava.class.path=$$JAVA_HOME/lib/tools.jar:$$JAVA_HOME/lib/jconsole.jar:catalina-jmx-remote.jar" service:jmx:rmi://localhost:9994/jndi/rmi://localhost:9993/jmxrmi    

BACKUP

NEO4j + BACKUP

BACKUP USING NEO4j COMMUNITY

SOLUTION: LVM2

LVM2 Setup:

apt-get install -y lvm2

# tell LVM2 to use block device /dev/xvdf for its physical volume
pvcreate /dev/xvdf

# tell LVM2 to create a new volume group called "galileo_production" on that physical volume
vgcreate galileo_production /dev/xvdf 
# => creates /dev/galileo_production

# finally, create a usable logical volume for our new filesystem,
# but reserve 50% of the space for a realtime snapshot
lvcreate -n db --extents 50%FREE galileo_production
# => creates /dev/galileo_production/db

# format it and mount it:
mkfs.ext4 /dev/galileo_production/db
mkdir -p /mnt/db_production
echo "/dev/galileo_production/db /mnt/db_production   auto    defaults,nobootwait,comment=cloudconfig 0       2" >> /etc/fstab
mount /mnt/db_production   

TAKING A SNAPSHOT:

# immediately snapshot the production data
sudo lvcreate -L32G -s -n production_backup /dev/galileo/production

# mount the snapshot temporarily
sudo mkdir -p /mnt/snapshot
sudo mount /dev/galileo/production_backup /mnt/snapshot

# tgz it up!
sudo tar -zcvf /mnt/snapshot.tgz /mnt/snapshot

# unmount and destroys the snapshot
sudo umount /mnt/snapshot
sudo lvremove /dev/galileo/production_backup -f   

LET's AUTOMATE THIS!

OVERVIEW

1. lock_database (recommended):
  - POST "http://localhost:4242/system/shutdown"
    i.e. does 'graph.stop' in pacer

2. unmount_snapshot:
  - unmounts any stale snapshots via `umount`, then destroys via `lvremove`

3. mount_snapshot:
  - lvcreates the snapshot immediately
  - mounts it
     lock_database if @lock_url
 
4. unlock_database (needed if locked):
  - POST "http://localhost:4242/system/startup"
    i.e. does 'graph.start' in pacer

5. tar, gzip, split into 5G chunks, and upload!

OH OH!  DIGITALOCEAN ..

Loopback Filesystem FTW

# create a blank 120GB ISO file
dd if=/dev/zero of=/data/production.iso bs=1024 count=125829120 # 120GB * 1024*1024

# configure as a loopback device
losetup /dev/loop0 /data/production.iso
mkfs.ext4 /dev/loop0

# setup LVM2 physical volumes similar to above, but with loop0
pvcreate /dev/loop0
vgcreate galileo_production /dev/loop0    

QUESTIONS?

THANK YOU!