Datomic

"the most interesting database you've never heard of"

zilvinasu

A long time ago in a database far, far away....

  • General-purpose db
  • Value-oriented semantics
  • Multiple labels:
    • NoSQL
    • Graph db
    • Transactional store
    • Logic db
  • Targets 96% use cases of relational databases
Railsbank logo
“ANYONE WHO STOPS LEARNING IS OLD, WHETHER AT TWENTY OR EIGHTY.”
HENRY FORD

Who uses? 

Why? 🧐

  • Powerful alternative to SQL

    • Forget SQL injections

  • Flexible deployment

  • Read do not impact write

  • Database as a value

  • Entity modelling

  • Schema flexibility

  • Time Travel

  • Upsert everything

  • Learning is fun!

Components ⛓

 Tech details 🥚

Storage Services

  • Data is elsewhere
    • Dev: File system, in-memory
    • Production: PostgreSQL, DynamoDB, Cassandra
  • Data stored in the index (covered index)

Reads

  • Multi-threaded
  • Different APIs
  • Client connects to underlying storage
  • Infinitely scales
  • Independent of anything else
  • Failover using Load Balancer

Writes

  • Single-threaded
  • Different API
  • Vertical scaling
  • Failover support

Transactions

  • You must complete transaction
  • Queued executions
  • Requires no coordination
  • Executed when written to the backing storage
    • Peers update eventually

Structure

  • DB value in the client is just an immutable map
  • Queries are executed on the value
  • Value itself is lazy
  • Backed by Last Recently Used cache

Deployment ⚙️

AWS CloudFormation☁️

On-Premise 💻

Data model 𝌭

[e a v t added?]

Datoms

entity

attribute

value

transaction

SQL Example

id first_name last_name email
100 JSON McNugget mcnugget@mailinator.com
101 Jo Jenkins jenkins@mailinator.com

table: person

Datomic Example

[[100 :person/first-name "JSON"]
 [100 :person/last-name  "McNugget"]
 [100 :person/email      "mcnugget@mailinator.com"]

 [101 :person/first-name "Jo"]
 [101 :person/last-name  "Jenkins"]
 [101 :person/email      "jenkins@mailinator.com"]]

Evolving Schema 👷‍♀️

Main ideas

  • Defining attributes
  • Grouping attributes
  • "Transact" like any other data

Datoms again! 

{;; attribute's unique name
 :db/ident       :person/first-name

 ;; one or many
 :db/cardinality :db.cardinality/one

 ;; i.e. string, boolean, long, ref, uuid ...
 :db/valueType   :db.type/string     ;; value type

 ;; what's the purpose of your attribute?
 :db/doc         "Person's name"}
[{:db/ident       :person/first-name
  :db/cardinality :db.cardinality/one
  :db/valueType   :db.type/string
  :db/doc         "First name of the person"}

 {:db/ident       :person/last-name
  :db/cardinality :db.cardinality/one
  :db/valueType   :db.type/string
  :db/doc         "Last name of the person"}

 {:db/ident       :person/email
  :db/cardinality :db.cardinality/one
  :db/valueType   :db.type/string
  :db/doc         "Email of the person"}]
(d/transact
  conn
  [{:db/ident       :person/first-name
    :db/cardinality :db.cardinality/one
    :db/valueType   :db.type/string
    :db/doc         "First name of the person"}])

More 

{:db/unique      :db.unique/identity
 :db/fulltext    true
 :db/noHistory   true
 :db/isComponent true}

Things to keep in mind

  • Use migrations (i.e. conformity)
  • Leverage partitions
  • Schema is easily polluted
    • Attributes available for any entity

Sailing through the data ⛵️

Datalog

  • Query language used for Datomic
    • Syntactic subset of Prolog
  • Influenced by SPARQL (RDF query language)
  • Different APIs are available
    • Query API
    • Pull API
    • Entity API

Query API

;; pull all of the entity attributes
;; by matching the email
[:find ?email
 :where [?e :person/email ?email]]
;; actual example using the datomic api
(d/q
'[:find ?email
  :where [?e :person/email ?email]]
 db)

=> #{["jenkins@mailinator.com"] ["mcnugget@mailinator.com"]}

Pull API

  • Similar to GraphQL
  • Hierarchical selections for data
  • Lookup Refs
(d/pull
 (d/db conn)
 [:person/last-name :person/first-name]
 [:email "jenkins@mailinator.com"])

=> #:person{:last-name "Jenkins", :first-name "Jo"}

Entity API

;; getting lazy values
(d/entity
 (d/db conn)
 [:person/email "jenkins@mailinator.com"])
=> #:db{:id 17592186045419}

(d/entity (d/db conn) 17592186045419)
=> #:db{:id 17592186045419}

Writing data ✏️

  • Sent in form of transactions
  • Data structures following rules
  • List containing either data to be added or retracted

Two options

;; lower-level list-based
; List of format [command e a v]
; Command is either :db/add or :db/retract
[[:db/add "gm-tid" :person/first-name "JSON"]
 [:db/add "gm-tid" :person/last-name "McNugget"]
 [:db/add "gm-tid" :person/email "mcnugget@mailinator.com"]]


;; higher-level map-based
[{:db/id             100      
  :person/first-name "JSON"
  :person/last-name  "McNugget"
  :person/email      "mcnugget@mailinator.com"}]

Database Functions

  • Avoiding race conditions
  • Part of the transaction process
  • Functions are entities
  • Written in:
    • Clojure 😎
    • Java 💩

Example

;; in schema
 {:db/ident :person/normalize-last-name
  :db/fn    #db/fn
             {:lang   "clojure"
              :params [db email]
              :code
              (let [person (d/entity db [:person/email email])
                    normalized (clojure.string/upper-case (:person/last-name person))]
                [{:db/id            (:db/id person)
                  :person/last-name normalized}])}

;; make use of that
(d/transact conn [[:person/normalize-last-name "mcnugget@mailinator.com"]])

=>
#object[datomic.promise$settable_future$reify__4751
        0x230b0540
        {:status :ready,
         :val {:db-before datomic.db.Db, @e7f03ab :db-after, datomic.db.Db @106f37f,
               :tx-data [#datom[17592186045419 64 "MCNUGGET" 13194139534317 true]
                         #datom[17592186045419 64 "McNugget" 13194139534317 false]],
               :tempids {}}}]

Fancy stuff 💄

Peers cache the data

Query DB as if TX was applied

Query history for the entity changes

Log extraction

Fits good with Event-driven architectures

Not so Good

One must write selective enough queries

Only one writer transactor 🥺

Proprietary 🤯

Lack of answers in the holy bible

Party hard with Datomic 🎉

Forget the SQL

Add data about transactions themselves

It's not meant for

Big Strings

Export data to

AWS S3

Homework 🕵️‍♀️

  • Datomic docs
  • Chapter 6 in "Professional Clojure" book
  • Alternatives: JUXT CRUXT

The End

🏷 We're hiring!

Datomic: the most interesting database you've never heard of

By Žilvinas Urbonas

Made with Slides.com

Datomic: the most interesting database you've never heard of

Example project: https://github.com/zilvinasu/datomic-presentation

  • 403