A tale about the state

Stuart Sierra's Reloaded workflow & Component library

Karol Andrusieczko (Eyeota)

August 2017

State - necessary evil

• we want our programs to be stateless
• no real app is stateless
• we can tame state by limiting
  the places where it's created

What is global state?

(def state-a (atom {}))
(def state-b (atom 0))


; hidden dependencies coming from outside

(defn op1 []
  (swap! state-a ...))

(defn op2 []
  (swap! state-b ...))

(let [state-a (atom {})
      state-b (atom 0)]

      (defn op1 []
        (swap! state-a ...))
    
      (defn op2 []
        (swap! state-b ...)))

State created at the time of loading files,
not at runtime

How to avoid global state?

(defn constructor []
  {:a (atom {})
   :b (atom 0)})


; makes function dependencies clear

(defn op1 [state]
  (swap! (:a state) ...))

(defn op2 [state]
  (swap! (:b state) ...))

Much easier to test now!

(defn test-constructor []
  {:a (atom {:initial "awesome test!"})
   :b (atom 0)})

(deftest test-op1
  (let [state (test-constructor)]
    (is (= ... (op1 state)))))

Dynamic variables - same!

(def ^:dynamic *resource*)

(defn- internal-op1 []
  ... *resource* ...)

(defn op1 [arg]
  (internal-op1 ...))

Real world app: initialisation

(def config (load-config-file config-url))       ; loads only once 

(def db-connection (atom nil))                   ; holds global state


(defn connect-to-database
  []
  (reset! db-connection (db/connect! config)))


(defn init! []
  (connect-to-database!)
  (create-thread-pools!)
  (start-background-processes!)
  (start-web-server!))

Reloaded workflow

;; In src/com/example/my_project/system.clj

(ns com.example.my-project.system)

(defn system
  "Returns a new instance of the whole application."
  []
  ...)

{:db        {:uri "datomic:mem://dev"}
 :scheduler #<ScheduledThreadPoolExecutorService ...>
 :cache     #<Atom {}>
 :handler   #<Fn ...>
 :server    #<Jetty ...>}

Different constructors

(defrecord System
  [storage config web-service])

(defn dev-system []
  (map->System {:storage (mem-store)
                :config (local-config {:a 1 :b 2})
                :web-service (mock-web-service)}))

(defn prod-system []
  (map->System {:storage (sql-store config)
                :config (zookeeper-config)
                :web-service (web-srv config storage)}))

Start and stop

;; In src/com/example/my_project/system.clj

(defn start
  "Performs side effects to initialize the system, acquire resources,
  and start it running. Returns an updated instance of the system."
  [system]
  ...)

(defn stop
  "Performs side effects to shut down the system and release its
  resources. Returns an updated instance of the system."
  [system]
  ...)

Reloaded workflow

To be able to shut the application down, discard any transient state it might have built up, start it again, and return to a similar state (...) in less than a second.

The idea

Restarting the app

You don't want to restart JVM every time...

Reloaded workflow problem: system map

it starts growing

(defn start-db [system]
  (let [host (get-in system [:db :host])] ; one big complex map
    (assoc-in state [:db :conn]
              (db/connect host))))

(defn get-user [system name]
  (let [conn (get-in system [:db :conn])] ; everything sees everything
    (db/query conn "..." name)))

Reloaded workflow problem: system map

• order of initialising things matters
• each stateful object needs to be manually started and stopped

(defn start-all []
  (-> {}
      start-db
      start-queues        ; manual ordering
      start-thread-pool
      ...
      start-web-server))  ; hidden dependencies?

Solution: Stuart Sierra's Component

• Immutable data structure
• Public API
• Managed lifecycle
• Relationships to other components

Component: API, constructor

; definition, encapsulates state
(defrecord DB [host conn])



; public API (possibly with side effects)
(defn query [db & ...]
  (.doQuery (:conn db) ...))

(defn insert [db & ...]
  (.doStatement (:conn db) ...))



; constructor (no side effects)
(defn db [host]
  (map->DB {:host host}))

Component: lifecycle

(ns com.stuartsierra.component)

(defprotocol Lifecycle
  (start [component])
  (stop [component]))

; default implementation returns component itself

(defrecord DB [host conn]
  component/Lifecycle

  (start [this]
    (assoc this :conn (Driver/connect host)))

  (stop [this]
    (.stop conn)
    this))

Dependencies

(defrecord Customers [db email])



(defn notify-customer [customers name message]
  (let [{:keys [db email]} customers
        address (query db name)]
    (send email address message)))

System map with Component

(defn system []
  (component/system-map
    :customers (customers ...)
    :db (db ...)
    :email (email ...)
    ...))

Test components

(defrecord TestDB [host conn]
  component/Lifecycle

  (start [this]
    (let [conn (Driver/connect host)]
      (load-seed-data conn)              ; initialise test data
      (assoc this :conn conn)))

  (stop [this]
    (drop-database conn)                 ; drop db so that you start fresh
    (.stop conn)
    this))

Advantages

• Components decoupled
  • Easier to test and refactor
• Dependency sorting
• Replacing components with test implementation is easy
• All components kept in one flat map

Disadvantages

• It's a framework, not library (whole app buy-in)
  • if you don't like that, see https://github.com/tolitius/mount
• Can't start part of the system
• Use of defrecords feels a bit non-functional (?)

Testing the whole app

; test-system creates email using channels

(deftest test-notify-customer
  (let [system (component/start test-system)
        {:keys [customers email] system}]
    (try
      (notify customers "bob" "Hi, Bob!")
      (is (= "Hi, Bob!" (:message (<!! (:channel email)))))
      (finally
        (component/stop system)))))

Thank you!

We're hiring!

Thank you!

• Stuart Sierra's reloaded workflow article: http://thinkrelevance.com/blog/2013/06/04/clojure-workflow-reloaded
   
• Stuart Sierra's Component library explained:
  https://www.youtube.com/watch?v=13cmHf_kt-Q
   
• Stuart Sierra's talk: http://www.infoq.com/presentations/Clojure-Large-scale-patterns-techniques