Datalog

Beyond SQL

Relational DB

Data stored in relations (tables)

Fixed Schema

Foreign-key relationships

Columns or combination of columns as index

Document DB

Data stored as documents

Mostly schema-free

Some foreign-key relationships

Document properties as index

Graph DB

Data as Graph structures with nodes and annotated edges

Fixed Schema (everything is a graph)

Query-by-navigation

Node-based index

And many more...

Datalog

Declarative logic programming language

Subset of Prolog (1972)

Based on first-order logic

Not turing complete

Implementations in C/C++, Clojure, Erlang, Haskell, Java, Lua, OCaml, Prolog, Python, Racket, Ruby, Rust, Tcl

Datalog Example

jedi(luke).
sith_lord(anakin).
sith_lord(ben).

good(X) :- jedi(X).

mother(padme,luke).
mother(padme,leia).
mother(leia,ben).
mother(shmi,anakin).

father(anakin,luke).
father(anakin,leia).

parent(X,Y) :- mother(X,Y).
parent(X,Y) :- father(X,Y).

sibling(X,Y) :- parent(Z,X), parent(Z,Y), X \== Y.

grand_parent(X,Y) :- parent(X,Z), parent(Z,Y).

force_sensitive(X) :- parent(shmi,X).
force_sensitive(X) :- grand_parent(shmi,X).

ancestor(X,Y) :- parent(X,Y).
ancestor(X,Y) :- parent(X,Z), ancestor(Z,Y).

% force_sensitive(X) :- ancestor(shmi,X).

*Atoms are lower-case, variables are upper-case, % indicates a comment

Demo-Time!

Datalog DB

(in 6 easy steps)

Represent entities as sets of facts:

[attribute value]

[father "Vader"]

[name "Luke"]

[job "Jedi"]

Combine facts to entity, give it an ID:

[entity-id attribute value]

[12 father "Vader"]

[12 name "Luke"]

[12 job "Jedi"]

Datalog DB

Make it transactional:

[entity-id attribute value trx-id]

 

[11 father "Vader" 4300]

[11 name "Luke"    4300]

[11 job     "Jedi"      4300]

[12 father "Han"    4300]

[12 name "Ben"     4300]

[12 job "Sith Lord" 4300]

[12 alias "Kylo"       4300]

Datalog DB

Add/revoke flag

[entity-id attribute value trx-id add?]

 

[12 job "Sith Lord"               4301 false]

[12 job "Supreme Leader" 4301 true]

(* true is default and can be omitted)

Datalog DB

Refer to other entities via their id.

 

[13 name "Han"       4302]

[13 job "Smuggler"  4302]

[12 father "Han"      4302 false]

[12 father 13            4302]

 

[14 name "Anakin"  4302]

[14 job "Sith Lord"   4302]

[11 father "Vader"   4302 false]

[11 father 14            4302]

Aside: Data Structure Literals

;; <- semicolon introduces a comment

"foo" ;; a string (duh)

:bar  ;; a keyword (i.e. interned string)

:db/id ;; keyword id with a namespace

[ ] ;; vector

{ } ;; map from key->value

{ "key1" "value1"
  :key2 "value2"
}

#{ } ;; set

()   ;; list

What are our entities now?

As of Trx 4302

;; datoms
[11 :father 14]
[11 :name "Luke"]
[11 :job "Jedi"]


[12 :father 13]
[12 :name "Ben"]
[12 :job "Supreme Leader"]
[12 :alias "Kylo"]


[13 :name "Han"]
[13 :job "Smuggler"]

[14 :name "Anakin"]
[14 :job "Sith Lord"]

Let's add a DB system and some data...

;; Map form
{ :db/id 11 
  :father 14
  :name "Luke"
  :job "Jedi" }

{ :db/id 12
  :father 13
  :name "Ben"
  :job "Supreme Leader"
  :alias "Kylo" }

{:db/id 13
 :name "Han" :job "Smuggler"}

{:db/id 14
 :name "Anakin":job "Sith Lord"
}

Datahike: A datalog DB

(def uri "datahike:file:///tmp/tutorial")
(d/create-database uri)
(def conn (d/connect uri))
;; set up schema, which are also datoms. 
;; We use some helper functions to generate them
;; example: (schema/str-attr :test)
;; => #:db{:ident :test, :valueType :db.type/string, :cardinality :db.cardinality/one}
;; 
(d/transact conn [(schema/str-attr :name)
                  (schema/one-to-one :father)
                  (schema/str-attr :job)
                  (schema/many-str-attr :alias)
                  ])
;; add some data, propose Entity IDs via :db/id
(d/transact conn [{:db/id 11 :name "Luke"   :job "Jedi" :father 14}
                  {:db/id 12 :name "Ben"    :job "Supreme Leader" :father 13
                             :alias "Kylo" }
                  {:db/id 13 :name "Han"    :job "Smuggler"}
                  {:db/id 14 :name "Anakin" :job "Sith Lord"}
                  ])

Querying via Datalog

Find entity-ids, attributes, values by unification:

[:find ?name           ;; ?name is returned
 :where
   [?e :job "Jedi"]    ;; ?e is a free variable
   [?e :name ?name]    ;; "Jedi" is a constant
                       ;; unify statements to
] 
=> #{["Luke"]}         ;; find match

Querying

Navigating references

[:find ?name ?father
         :where
         [?e :name ?name]
         [?e :father ?f]
         [?f :name ?father]
]
> #{["Luke" "Anakin"] ["Ben" "Han"]}

Changing Schema

And adding data

#:db{:ident :born, :valueType :db.type/long, 
     :cardinality :db.cardinality/one}

;; birthdate relative to the Battle of Yavin (SW:ANH)
[{:db/id 11 :born  -19}    ;; Luke
 {:db/id 12 :born    5}    ;; Ben Solo
 {:db/id 13 :born  -29}    ;; Han Solo
 {:db/id 14 :born  -41}]   ;; Anakin

Querying with predicates

[:find ?name ?b
 :where
 [?e :name ?name]
 [?e :born ?b]
 [(< 0 ?b)]
]
=> #{["Ben" 5]}

(d/q 
'[:find ?name ?b
  :in $ ?year
  :where
  [?e :name ?name]
  [?e :born ?b]
  [(< ?year ?b)]] 
   (d/db conn) 0)

Aggregates

[:find (count ?e)
 :where
 [?e :name _]]
 => 4
 
[:find (min ?b)
 :where
 [_ :born ?b]]
 => [-41]
 
(min ?xs)
(max ?xs)
(count ?xs)
(count-distinct ?xs)
(sum ?xs)
(avg ?xs)
(median ?xs)
(variance ?xs)
(stddev ?xs)

Aggregates 2

[:find (sample 2 ?name)
 :where
 [_ :name ?name]]
 
[:find (min 2 ?born)
 :where
 [?e :name ?name]
 [?e :born ?born]]
 => [-41 -29]

 
(distinct ?xs)
(min n ?xs)
(max n ?xs)
(rand n ?xs)
(sample n ?xs)

 

Or use your own functions!

Pulling

[*] 11               ;; select all attributes
=> {:db/id 11, 
    :born -19, 
    :job "Jedi",
    :name "Luke",
    :father #:db{:id 14}}
    
[:name :job] 11           ;; one attribute
=> {:name "Luke", :job "Jedi"}


[:name {:father [:name]}] 11  ;; pull referenced entity
=> {:name "Luke", :father {:name "Anakin"}}

Retrieve entities (and referenced entities) via selector and entity-id

Pulling 2

[* {:_father [*]}] 14
=>
{:db/id 14,
 :born -41,
 :job "Sith Lord",
 :name "Anakin",
 :_father [{:db/id 11, 
           :born -19, 
           :father #:db{:id 14}, 
           :job "Jedi", 
           :name "Luke"}]}

References are bi-directional

Querying and Pulling

(d/q '[:find [pull ?e [:name :born]]
         :where
         [?e :job ?j]
         [(not= ?j "Jedi")]
       ] (d/db conn))
=> ([{:name "Han", :born -29}] 
    [{:name "Ben", :born 5}] 
    [{:name "Anakin", :born -41}])

Let's do both!

and much more (rules, computed values etc.)

=> queries themselves are just data

Immutable Datalog DBs

Append-only, mono or bi-temporal

  • Data is only appended, not deleted* or updated
  • Facts can be retracted, which is recorded as fact
  • Transactions are also facts
  • Transaction-metadata like timestamp is stored as well

=> an immutable series of timestamped transactions

 

=> the database can be viewed in different points in time

*Most vendors still provide excision to hard-delete facts

Database as value

Remember (d/db conn)?

Database at the current time!

 

Let's go back in time!

(d/as-of db timestamp!)

(def current-db (d/db conn))
(def db-of-2000 (d/as-of current-db #inst "2000-01-01"))

(d/pull current-db
 '[:name] 11
)
=> {:name "Luke"}

(d/pull db-of-2000
 '[:name] 11
)
=> nil

Transactions as value

Transactions are facts too

;; structure of a fact/datom
[entity attribute value trx]
;; added with every transaction
[trx :db/txInstant timestamp]

;;; we can add
;;; other metadata
[trx :tx/changed-by "beders"]

Transactions in queries

Find when and how facts where added or retracted

;; all transactions
[:find [[pull ?tx [*]] ...]
 :where [?tx :db/txInstant _]]
 =>
 [#:db{:id 536870918, :txInstant #inst"2020-01-09T17:48:25.381-00:00"}
 #:db{:id 536870916, :txInstant #inst"2020-01-09T09:53:50.720-00:00"}
 ....
 ]
 
 ;; all trxs where :born attribute was added
 [:find ?tx ?time
  :where [?tx :db/txInstant ?time]
         [_ :born _ ?tx true]
         ]
 =>
 #{[536870916 #inst"2020-01-09T09:53:50.720-00:00"] 
   [536870917 #inst"2020-01-09T10:00:17.160-00:00"]}
 
 ;; all facts added in a trx
 [:find ?e ?a ?v
  :in $ ?tx
  :where [?e ?a ?v ?tx]
 ] (d/db conn) 536870914
 =>
 {[11 :job "Jedi"] ....}

Implementations

Datomic

(Cognitect - 2012)

Storage backend:

DynamoDB, Cassandra, ORA, PostgreSQL

Used by:

ca, Netflix, FB, TimeInc, NuBank (80TB data, 2100 transactors, 13 shards)

 

 

Crux

(Juxt - 2019)

Bi-temporal, trx-time and valid-time

Storage backends:

RocksDB, Kafka (configurable)

Used by:

Oscaro (eCommerce)

Avisi (AtlasCRM)

 

 

Implementations

Datahike

(@Replikativ - 2019)

Uses hitchhiker trees (fractal, snapshottable, on-disk)

Good for single machines, millions of facts

DataScript

(@Tonsky - 2014)

In-memory, in-browser

Used by:

PrecursorApp (collab editing)

Cognician (coaching platform)

 

Introduction to Datalog for DBs

By Jochen Bedersdorfer

Made with Slides.com

Introduction to Datalog for DBs

Overview of Datalog (the prolog subset) and how it's used in databases like Datahike or Datomic

  • 887

More from Jochen Bedersdorfer