Predictive Testing
https://github.com/bensu/facilier
Transform user actions into test cases
~Elm Architecture
~Elm Architecture
(view state)
;; => html
(step state action)
;; => state'
(event-handler state dom-event)
;; => action
~Elm Architecture: step
(defmulti step (fn [_ [k _]] k))
(defmethod step :session/load
[state [_ {:keys [session/all]}]]
(-> state
(assoc :app/name (first (keys all)))
(update :session/all #(merge % all))))
~Elm Architecture: event handlers
;; Handle DOM Events
{:onClick (handle [e]
(f/raise! [:session/close nil]))}
;; Handle XHR Events
(http-request! (str test-url "/session/" id)
(handle [e]
(let [{:keys [session]} e]
(cb {:session/full session}))))
Testing the ~Elm Architecture
- The app is a composition of functions
- Data flow can be recorded during user sessions
- The functions can be tested through invariants
;; Manual test cases
(is (= expected-result (tested-fn test-data)))
;; Generated test cases
(is (invariant? test-data (tested-fn test-data)))
Record States
- Only views can be tested
- Recorded data takes lots of space in disk unless a diffing mechanism is used (e.g., tx-data)
;; The current session is displayed
(when-let [current (:session/current state)]
(let [n (tu/find-one-by-class html "session-title")]
(is (= current (.-innerText n)))))
Record Actions
- Can reproduce the original States with step, so views and update can be tested.
- If the step function introduces randomness, the reproduced State diverges from the original State, leading to spurious errors.
;; Closing turns the current session nil
(let [[type load] action]
(when (= :session/close type)
(is (nil? (:session/current state)))))
Record Events
- Everything can be tested: event handlers, step, views.
- Need to monkey patch React's Event Dispatcher.
- Same randomness problems as with Actions and more.
;; A click event either closes or selects a session
(let [[type _] action]
(when (= "click" (.-type dom-event))
(is (contains? #{:session/close :session/select}
type))))
Test User Sessions Demo
Playback Sessions Demo
SauceLabs Playback Sessions
Problems: Too much data
- Is it possible to use all user sessions in each test run?
- Keep representative samples
- Which samples do you keep?
- Make up a metric and keep sessions that score substantially different for it (e.g., session length, diversity of actions)
Problems: Ops
- Who runs Facilier?
- Probably you, (like Riemann).
- Does it have to be fast? Can it crash?
- Does it slow your fronted?
- The act of recording and sending sessions over the wire might slow the frontend app down.
Problems: Privacy/Security
- Can the sessions be automatically sanitized?
- Tested logic might depend on data matching the original
- Can the developer sanitize the data by providing a function?
- More work, diminishing returns
- How can the tests access the sanitized data securely?
- Add credentials to the service, harder to setup.
Problems: Randomness
- Does the update function introduce any randomness?
- Random behavior makes the reproduce State diverge from the original one -> errors
- Is the tested logic time dependent?
- original-time vs tested-time might matter.
Thanks!
Predictive Testing
By Sebastian Bensusan
Predictive Testing
Transform user actions into test cases with Facilier
- 1,923