Predictive Testing

https://github.com/bensu/facilier

Transform user actions into test cases

~Elm Architecture

~Elm Architecture

(view state)
;; => html

(step state action)
;; => state'

(event-handler state dom-event)
;; => action

~Elm Architecture: step

(defmulti step (fn [_ [k _]] k))

(defmethod step :session/load
  [state [_ {:keys [session/all]}]]
  (-> state
      (assoc :app/name (first (keys all)))
      (update :session/all #(merge % all))))

~Elm Architecture: event handlers

;; Handle DOM Events
{:onClick (handle [e]
            (f/raise! [:session/close nil]))}

;; Handle XHR Events
(http-request! (str test-url "/session/" id)
               (handle [e]
                 (let [{:keys [session]} e]
                   (cb {:session/full session}))))

Testing the ~Elm Architecture

  • The app is a composition of functions
  • Data flow can be recorded during user sessions
  • The functions can be tested through invariants
;; Manual test cases
(is (= expected-result (tested-fn test-data)))

;; Generated test cases
(is (invariant? test-data (tested-fn test-data)))

Record States

  • Only views can be tested
  • Recorded data takes lots of space in disk unless a diffing mechanism is used (e.g., tx-data)
;; The current session is displayed
(when-let [current (:session/current state)]
  (let [n (tu/find-one-by-class html "session-title")]
    (is (= current (.-innerText n)))))

Record Actions

  • Can reproduce the original States with step, so views and update can be tested.
  • If the step function introduces randomness, the reproduced State diverges from the original State, leading to spurious errors.
;; Closing turns the current session nil
(let [[type load] action]
  (when (= :session/close type)
    (is (nil? (:session/current state)))))

Record Events

  • Everything can be tested: event handlers, step, views.
  • Need to monkey patch React's Event Dispatcher.
  • Same randomness problems as with Actions and more.
;; A click event either closes or selects a session
(let [[type _] action]
  (when (= "click" (.-type dom-event))
    (is (contains? #{:session/close :session/select}
                   type))))

Test User Sessions Demo

Playback Sessions Demo

SauceLabs Playback Sessions

Problems: Too much data

  • Is it possible to use all user sessions in each test run?
    • Keep representative samples
  • Which samples do you keep?
    • Make up a metric and keep sessions that score substantially different for it (e.g., session length, diversity of actions)

Problems: Ops

  • Who runs Facilier?
    • Probably you, (like Riemann).
  • Does it have to be fast? Can it crash?
  • Does it slow your fronted?
    • The act of recording and sending sessions over the wire might slow the frontend app down.

Problems: Privacy/Security

  • Can the sessions be automatically sanitized?
    • Tested logic might depend on data matching the original
  • Can the developer sanitize the data by providing a function?
    • More work, diminishing returns
  • How can the tests access the sanitized data securely?
    • Add credentials to the service, harder to setup.

Problems: Randomness

  • Does the update function introduce any randomness?
    • Random behavior makes the reproduce State diverge from the original one -> errors
  • Is the tested logic time dependent?
    • original-time vs tested-time might matter.

Thanks!

 

 

Made with Slides.com