Predictive Testing

https://github.com/bensu/facilier

Transform user actions into test cases

~Elm Architecture

~Elm Architecture

(view state)
;; => html

(step state action)
;; => state'

(event-handler state dom-event)
;; => action

~Elm Architecture: step

(defmulti step (fn [_ [k _]] k))

(defmethod step :session/load
  [state [_ {:keys [session/all]}]]
  (-> state
      (assoc :app/name (first (keys all)))
      (update :session/all #(merge % all))))

~Elm Architecture: event handlers

;; Handle DOM Events
{:onClick (handle [e]
            (f/raise! [:session/close nil]))}

;; Handle XHR Events
(http-request! (str test-url "/session/" id)
               (handle [e]
                 (let [{:keys [session]} e]
                   (cb {:session/full session}))))

Testing the ~Elm Architecture

• The app is a composition of functions
• Data flow can be recorded during user sessions
• The functions can be tested through invariants

;; Manual test cases
(is (= expected-result (tested-fn test-data)))

;; Generated test cases
(is (invariant? test-data (tested-fn test-data)))

Record States

• Only views can be tested
• Recorded data takes lots of space in disk unless a diffing mechanism is used (e.g., tx-data)

;; The current session is displayed
(when-let [current (:session/current state)]
  (let [n (tu/find-one-by-class html "session-title")]
    (is (= current (.-innerText n)))))

Record Actions

• Can reproduce the original States with step, so views and update can be tested.
• If the step function introduces randomness, the reproduced State diverges from the original State, leading to spurious errors.

;; Closing turns the current session nil
(let [[type load] action]
  (when (= :session/close type)
    (is (nil? (:session/current state)))))

Record Events

• Everything can be tested: event handlers, step, views.
• Need to monkey patch React's Event Dispatcher.
• Same randomness problems as with Actions and more.

;; A click event either closes or selects a session
(let [[type _] action]
  (when (= "click" (.-type dom-event))
    (is (contains? #{:session/close :session/select}
                   type))))

Test User Sessions Demo

Playback Sessions Demo

SauceLabs Playback Sessions

Problems: Too much data

• Is it possible to use all user sessions in each test run?
  • Keep representative samples
• Which samples do you keep?
  • Make up a metric and keep sessions that score substantially different for it (e.g., session length, diversity of actions)

Problems: Ops

• Who runs Facilier?
  • Probably you, (like Riemann).
• Does it have to be fast? Can it crash?
• Does it slow your fronted?
  • The act of recording and sending sessions over the wire might slow the frontend app down.

Problems: Privacy/Security

• Can the sessions be automatically sanitized?
  • Tested logic might depend on data matching the original
• Can the developer sanitize the data by providing a function?
  • More work, diminishing returns
• How can the tests access the sanitized data securely?
  • Add credentials to the service, harder to setup.

Problems: Randomness

• Does the update function introduce any randomness?
  • Random behavior makes the reproduce State diverge from the original one -> errors
• Is the tested logic time dependent?
  • original-time vs tested-time might matter.

Thanks!