KVS

Abstract

Erlang Term 

Database

@5HT

How not to 

store a cat?



     SQL -- relational algebra for cats

     ORM -- trying to do OLAP in Java

     XML -- suck <and> blow waste

     JSON -- JavaScript database


How our fathers stored cats?



        BerkleyDB -- 1986 

        LDAP -- X.500 Interface to KV

CAP



CA -- SQL RDBMS, Mnesia (non-distr)
CP -- BDB,Redis,BigTable,MongoDB 
AP -- Dynamo,Riak,Couch,Cassandra

Consistency



GOSSIP AP -- Eventual

PAXOS CP -- Hadoop

RAFT CP -- etcd

ZAB CP -- ZooKeeper HBase Storm

2PC CP -- SQL RDBMS


Erlang AP KV Boom



CouchDB Couchbase Riak

Mnesia KAI LeoDB 

Hibari HanoiDB

Pitfalls

How to store a 

cat-to-cat subscriptions?


Secondary Indexes


1. LevelDB -- HanoiDB, Riak

2. DETS -- Mnesia, KAI

Pitfalls

How to put each cat in a box?


Consistency


1. Reconcile -- Editing History

2. Sequence Lock -- Erlang Process

REST

Riak/WebMachine 

CouchDB/Own

FarWest/Cowboy

Abstractions

Feuerlabs/kvdb 

project-fifo/fifo-db

KVS is...

Statically Typed

Mutable

Social Schema Based

Key-Value

Term Store

with Secondary Indexes

and REST ... Abstract DB

Social Schema




Users Products Reviews Feeds

Comments Groups Likes Acl

Teams Tags Messages

Payments Accounts

Transactions Purchases

KVS



Riak / Mnesia / KAI Backends

Cowboy/N2O REST

JSON/BERT/...

Records <-> Maps <-> Proplists

1500 LOC

KVS



#user { id, mail, name }

#product { id, name }

#review { id, body, product }





KVS



#user { id, next, prev, ... }

#product { id, next, prev, ... }

#review { id, next, prev, ... }

KVS core

  

     #container { id, top, count } 

         #iterator { id, container, 

                               prev, next

                               feeds = [] }


KVS Core



           #subscription { who,
                                            whom,
                                            what }

Macros


-define(CONTAINER, id, top, count=0).

-define(ITERATOR(Container),

id, container=Container,

feed_id, prev, next, feeds=[]).


-record(feed, {?CONTAINER, aclver}).

-record(user, {?ITERATOR(feed), name, email}).

-record(product, {?ITERATOR(feed), name}).


Schema



-define(CONTAINERS, [
    {feed, record_info(fields, feed)},
    {acl, record_info(fields, acl)},
    {transaction,  record_info(fields, transaction)},
    {payment,  record_info(fields, payment)} ]).


API


   POST/PUT kvs:add(#user{}).

        DELETE kvs:remove(user,2).

PUT/PATCH kvs:put(#user{id=2}).

               GET kvs:entries(Feed,user).

               GET kvs:get(user,2).

               GET kvs:all(user).

Feed Server



For each Container

we spawn Erlang Lock Process 

for Feed Write Operations 

ordering and consistensy


https://github.com/synrc/feeds

Console


> kvs:join('root@synrc.com').> kvs:initialize().> kvs:init_db().> kvs:dir().[{table,"user"}, {table,"feed"}, {table,"acl"}, {table,"transaction"}, ... ]

Console


> rr(kvs).> kvs:add(#user{id="mes@ua.fm"}).> kvs:entries(      kvs:get(feed,users),      user,undefined).[#user{id="mes@ua.fm"}]> kvs:get(user,"mes@ua.fm").{ok,#user{id="mes@ua.fm"}



1500 LOC


@5HT @doxtop


https://github.com/synrc/kvs

Made with Slides.com