Toward a grammar for behavioural experiments

Toward a grammar of behavioural experiments

Danielle Navarro, UNSW (@djnavarro)

djnavarro.link/rstudio2020

djnavarro.net

@djnavarro

Who is she?

A tool to build flexible behavioural experiments in R

@djnavarro

Behavioural experiments in the early 20th century

@djnavarro

Behavioural experiments in the early 21st century

@djnavarro

How does R look to a behavioural scientist?

Good statistical analysis
Good data wrangling
Good general programming

@djnavarro

Bad for experiments :-(

@djnavarro

Python has psychopy

https://www.psychopy.org/

Matlab has psychtoolbox

http://psychtoolbox.org/

What about R?

X11 graphics?

@djnavarro

X11 graphics?

  # start timing
  start <- Sys.time()
  
  # draw plot
  draw_plot(data$n_up[i], data$n_down[i])
  
  # wait for (then record) response
  data$response[i] <- getGraphicsEvent(
    prompt = "6 = up, b = down",
    consolePrompt = "",
    onKeybd = response
  )
  
  # record the response time
  data$rt[i] <- as.numeric(Sys.time() - start)
  
  # clear the plot
  clear_plot()

@djnavarro

X11 graphics?

Doesn't work online
Interactivity is limited
R doesn't "want" you to do it

Surprisingly easy to code
Few dependencies

@djnavarro

Shiny?

https://djnavarro.shinyapps.io/shiny/

@djnavarro

Shiny?

https://psyr.org/shiny.html

@djnavarro

Shiny?

Scaling to MTurk load is hard
Tricky for novices
Not tested for high-precision timing
Intensive at server side

Works online
Flexible interactivity

@djnavarro

???

Desiderata?

Written with R code
Relatively easy for novices
Experiments are brower based
Client side computation
Randomisation and flow control
Support richer designs than surveys

@djnavarro

jsPsych almost works!

Experiments are brower based
Client side computation
Randomisation and flow control
Support richer designs than surveys
Relatively easy for novices

... But it's javascript

@djnavarro

Code the experiment in R with jaysire

Build to jsPsych and R server

Deploy locally or to Google

using the package

@djnavarro

(... in brief!)

library(jaysire)

instructions <- trial_instructions(
  pages = c(
    "Welcome! Use the arrow buttons to browse these instructions",
    "Your task is to decide if an equation like '2 + 2 = 4' is true or false",
    "You will respond by clicking a button",
    "Press the 'Next' button to begin!"
  ),
  show_clickable_nav = TRUE,
  post_trial_gap = 1000
)

@djnavarro

library(jaysire)

instructions <- trial_instructions(
  pages = c(
    "Welcome! Use the arrow buttons to browse these instructions",
    "Your task is to decide if an equation like '2 + 2 = 4' is true or false",
    "You will respond by clicking a button",
    "Press the 'Next' button to begin!"
  ),
  show_clickable_nav = TRUE,
  post_trial_gap = 1000
)

trial1 <- trial_html_button_response(
  stimulus = "13 + 23 = 36",
  choices = c("true", "false"),
  post_trial_gap = 1000
)

trial2 <- trial_html_button_response(
  stimulus = "17 - 9 = 6",
  choices = c("true", "false")
)

@djnavarro

library(jaysire)

instructions <- trial_instructions(
  pages = c(
    "Welcome! Use the arrow buttons to browse these instructions",
    "Your task is to decide if an equation like '2 + 2 = 4' is true or false",
    "You will respond by clicking a button",
    "Press the 'Next' button to begin!"
  ),
  show_clickable_nav = TRUE,
  post_trial_gap = 1000
)

trial1 <- trial_html_button_response(
  stimulus = "13 + 23 = 36",
  choices = c("true", "false"),
  post_trial_gap = 1000
)

trial2 <- trial_html_button_response(
  stimulus = "17 - 9 = 6",
  choices = c("true", "false")
)

build_experiment(
  timeline = build_timeline(instructions, trial1, trial2),
  path = "path_to_folder",
  on_finish = fn_save_locally()
)

@djnavarro

djnavarro.github.io/jaysire

image_trial <- trial_image_button_response(
  stimulus = insert_resource("heart.png"), 
  stimulus_height = 400,
  stimulus_width = 400,
  choices = c("Unpleasant", "Neutral", "Pleasant") 
)

video_trial <- trial_video_button_response(
  sources = insert_resource(c("heart.mpg", "heart.webm")), 
  choices = c("Unpleasant", "Neutral", "Pleasant"), 
  trial_ends_after_video = FALSE,
  response_ends_trial = TRUE
)

build_experiment(
  timeline = build_timeline(image_trial, video_trial, audio_trial),
  path = temporary_folder(), 
  resources = build_resources("path_to_resource_folder"),
  use_webaudio = FALSE,
  on_finish = fn_save_locally()
)

@djnavarro

Use media files

@djnavarro

djnavarro.github.io/jaysire

@djnavarro

To images:
To HTML:
To video:
To audio:

Respond with a key press...

Etc...

Change response method

Respond with a slider bar...

To images:
To HTML:
To video:
To audio:

@djnavarro

djnavarro.github.io/jaysire

@djnavarro


confidence_scale <- c(
  "Very unconfident", 
  "Somewhat unconfident", 
  "Somewhat confident", 
  "Very confident"
)

page3 <- trial_survey_likert(
  preamble = "How confident in you R skills?",
  questions = list(
    question_likert("Data wrangling?", confidence_scale),
    question_likert("Data visualisation?", confidence_scale),
    question_likert("Statistical modelling?", confidence_scale),
    question_likert("Designing experiments?", confidence_scale),
    question_likert("R markdown documents?", confidence_scale)
  )
)

Create survey pages

@djnavarro

Data written to CSV

when deployed to Google App Engine responses are written to the Google Datastore

... when deployed locally

flow control

@djnavarro

# ------ everyone should see this ------
page1 <- trial_html_button_response(
  "Do you identify as LGBTIQ+?",
  c("Yes", "No", "Prefer not to say")
)

# ------ only some people should see this! ------
followup <- trial_survey_multi_select(
  question_multi(
    prompt = "Select all that apply",
    options = c(
      "Lesbian", "Gay", "Bisexual/Pansexual", "Transgender", 
      "Nonbinary", "Genderqueer", "Intersex", "Asexual", "Other"
    )
  )
)

Conditional execution

@djnavarro

# ------ everyone should see this ------
page1 <- trial_html_button_response(
  "Do you identify as LGBTIQ+?",
  c("Yes", "No", "Prefer not to say")
)

# ------ only some people should see this! ------
followup <- trial_survey_multi_select(
  question_multi(
    prompt = "Select all that apply",
    options = c(
      "Lesbian", "Gay", "Bisexual/Pansexual", "Transgender", 
      "Nonbinary", "Genderqueer", "Intersex", "Asexual", "Other"
    )
  )
)

# ------ a JS function to be evaluated at run time ------
condition_check <- fn_data_condition(button_pressed == "0")

# ------ a conditional timeline ------
page1a <- build_timeline(followup) %>% 
  tl_display_if(condition_check)

Conditional execution

@djnavarro

# ------ everyone should see this ------
page1 <- trial_html_button_response(
  "Do you identify as LGBTIQ+?",
  c("Yes", "No", "Prefer not to say")
)

# ------ only some people should see this! ------
followup <- trial_survey_multi_select(
  question_multi(
    prompt = "Select all that apply",
    options = c(
      "Lesbian", "Gay", "Bisexual/Pansexual", "Transgender", 
      "Nonbinary", "Genderqueer", "Intersex", "Asexual", "Other"
    )
  )
)

# ------ a JS function to be evaluated at run time ------
condition_check <- fn_data_condition(button_pressed == "0")

# ------ a conditional timeline ------
page1a <- build_timeline(followup) %>% 
  tl_display_if(condition_check)

Conditional execution

@djnavarro

# ------ question to be repeated until answered correctly ------
query <- trial_image_button_response(
  stimulus = insert_resource("heart.png"), 
  stimulus_height = 400,
  stimulus_width = 400,
  choices = c("Unpleasant", "Neutral", "Pleasant"),
  prompt = "You will not be allowed to continue unless you select 'Pleasant'"
)

Looped execution

@djnavarro

# ------ question to be repeated until answered correctly ------
query <- trial_image_button_response(
  stimulus = insert_resource("heart.png"), 
  stimulus_height = 400,
  stimulus_width = 400,
  choices = c("Unpleasant", "Neutral", "Pleasant"),
  prompt = "You will not be allowed to continue unless you select 'Pleasant'"
)

# ------ a looped timeline ------
page2 <- build_timeline(query) %>%
  tl_display_while(fn_data_condition(button_pressed != "2"))

Looped execution

@djnavarro

djnavarro.github.io/jaysire

why jaysire still sucks...

@djnavarro

it's a work in progress

the API is clunky
it should expose more of the jsPsych API
the use of JS callback is limited
... it's a long to-do list

@djnavarro

a deeper issue

@djnavarro

"trials" are not primitives

@djnavarro

cloze trials... very specific to linguistics
iat trials... very specific to social psychology
random dot motion tasks... maybe for monkeys?
vsl animate-occlusion trials... specific to Richard Aslin
vsl grid-scene trials... specific to Richard Aslin
etc...

For anything unusual you need to write a new jsPsych plugin

@djnavarro

The R code is used to specify a "trial" by building it from smaller parts

????????

This code is used to construct task-specific jspsych plugins at build time

@djnavarro

data +

aesthetics +

layers +

scales +

coordinate +

theme

decomposing plots

@djnavarro

??? +

???

decomposing experiments?

????????

what might it look like?

@djnavarro

????????

dx.doi.org/10.3758/s13414-011-0131-9

what might it look like?

@djnavarro

Elements:

image
audio
video
HTML

????????

Stimulus:

comprised of elements
has spatial layout
has temporal layout

what might it look like?

@djnavarro

Responses:

html button
key press
mouse move
slider
check box
text box

????????

Events:

record data
modify stimulus
provide feedback
end trial

Feedback:

specialised type of stimulus???

@djnavarro

A tool to build flexible behavioural experiments in R

djnavarro.github.io/jaysire

code

build

deploy

djnavarro.link/rstudio2020

templates & randomisation

@djnavarro

# ----------- template -----------
trial_template <- trial_html_button_response(
  stimulus = insert_variable("stimulus"),
  choices = c("true", "false"),
  post_trial_gap = 1000
)

@djnavarro

use templates & variables...

# ----------- template -----------
trial_template <- trial_html_button_response(
  stimulus = insert_variable("stimulus"),
  choices = c("true", "false"),
  post_trial_gap = 1000
)

# ----------- timeline -----------
trials <- build_timeline(trial_template) %>%
  tl_add_variables(stimulus = c("13 + 23 = 36", "17 - 9 = 6", "etc")) %>%
  tl_add_parameters(randomize_order = TRUE, repetitions = 2)

@djnavarro

... to randomise & repeat

other options?

@djnavarro

Shiny?

@djnavarro

Works online
Flexible interactivity
Simpler with psychTestR

Still inherits some of the other issues :-(

formr?

formr.org looks awesome, but targeted at surveys

@djnavarro

a demo experiment

@djnavarro

djnavarro.github.io/jaysire