### Danielle Navarro

I am a computational cognitive scientist at the University of New South Wales. My research focuses on human concept learning, reasoning and decision making.

13 July 2018, useR! conference, Brisbane

slides.com/djnavarro/user2018

(Part 1)

(Part 2)

(Part 3)

compcogscisydney.org/learning-statistics-with-r/

- It is a grim, terrible world without snapchat
- I am asked to teach intro stats to 300 psych students
- I make a huge mistake

- Reasoning about data
- Working with data
- Using "real" tools

- Memorising commands
- Rote learning tests
- Strict rules for analysis
**Fear!**

Frequentist statistics

Bayesian statistics

We teach the sampling distribution of the mean

...but we don't talk about sampling distributions in any other context

Sampling distribution for the maximum of 5 measurements?

Easy to simulate, if you have the tools!

- We'll lose professional accreditation
- IT department won't allow it
- Interface is too confusing
- SPSS is the academic standard
- Our other classes use SPSS
- Other universities don't use R
- Employers value SPSS
- Academics won't learn it
- The students will hate it

* I wasn't at UNSW at the time, but I've heard these sentiments from all over

**it's 2010?**

**A lot of my students are priced out**

```
my_data <- attach(iris)
```

I attached the data, and it doesn't work...

... ah, right. I can't reasonably expect to take a text written for a different audience and think it will work in my class.

http://compcogscisydney.org/

learning-statistics-with-r/

**I tried to address student fears**

**I aimed for a relaxed style**

**I covered the usual topics for psych methods**

**Extended to cover Bayesian and orthodox**

**I tried to show complexity where I could**

**I discussed some of R's quirks**

**I covered basic data visualisation**

**I covered basic programming**

**I even touched on regular expressions for some reason**

**... I may have gone a little overboard**

- My student evaluations were excellent 💯
- I won a teaching award 🙄
- I ended up with a book, apparently 📕

**Well, it certainly helped me**

**The students?**

- They learned the basics of R nicely 🎉
- They grasped the statistical concepts better 🎉
- They needed more exercises ⏱️
- They struggled to generalise 😢

- Written in LaTeX
- Predates tidyverse
- Needs editing
- Etc

**The book has limitations**

**Also this...**

*"An introductory textbook, well-written though it is, does not constitute a scholarly work"*

- CC-BY-SA 4.0
- Source on GitHub
- Links:

http://compcogscisydney.org/

learning-statistics-with-r/

compcogscisydney.org/projects/

Photo by Tachina Lee on Unsplash

Sharks have plaxium blood

Seagulls do not

Training set:

Test set:

- Sharks
- Seagulls
- Goldfish
- Penguins

Rather a lot of methodological details here

Sharks have plaxium blood

Seagulls do not

Training set:

```
model {
# mean and covariance matrix defining the Gaussian process
for(i in 1:ncat) {
mean_gp[i] <- m
cov_gp[i,i] <- (sigma^2) + (tau^2)
for(j in (i+1):ncat) {
cov_gp[i,j] <- (tau^2) * exp(-rho * (test[i] - test[j])^2)
cov_gp[j,i] <- cov_gp[i,j]
}
}
# sample a function from the Gaussian process
cov_gp_inv <- inverse(cov_gp)
f ~ dmnorm(mean_gp, cov_gp_inv)
# pass f through logit function to get a function on [0,1]
for(i in 1:ncat) {
phi[i] <- 1/(1+exp(-f[i]))
}
#
# [SNIP]
#
}
```

**Gaussian process prior**

**Classification function**

**Data**

**Gaussian process prior**

**Classification function**

**Data**

**Learning and classification and are viewed as an inverse probability problem ... i.e. Bayesian reasoning**

- 2 sampling types
- 2 evidence types
- 6 test points

- 2 sampling types
- 3 sample sizes
- 7 test points

- 2 sampling types
- 2 base rates
- 7 test points

**Some experiments...**

**... and some model fits**

(there's a modest amount of parameter tuning required to get quantitative fits but the ordinal effects are invariant)

**It works surprisingly well**

- Data tidying
- Data summary
- Drawing figures
- Linear modelling
- R Markdown
- Interface to JAGS

The usual things...

The less usual...

- String edits on JAGS models
- Robustness analysis for GP

image source: flaticon

- Four different slot machines?
- Four unknown restaurants?
- Four conference sessions?

M1

M2

M3

M4

`lose $5`

`win $2`

`lose $1`

`lose $1`

`win $2`

`lose $3`

I've not used these machines recently, and someone else has taken them

I've concentrated recent bets on these machines

`win $2`

`lose $5`

1

2

3

4

5

6

7

8

chosen

viable option not chosen

someone takes the machine

someone takes the machine

\color{purple}{E_{j,t-1}}

$\color{purple}{E_{j,t-1}}$

\color{blue}{K_{jt}}

$\color{blue}{K_{jt}}$

\color{orange}{S_{j,t-1}}

$\color{orange}{S_{j,t-1}}$

\color{purple}{E_{jt}}

$\color{purple}{E_{jt}}$

\color{orange}{S_{jt}}

$\color{orange}{S_{jt}}$

Expected reward for option \(j\) on this trial

Uncertainty about reward for option \(j\) on this trial

Expected reward for option \(j\) on last trial

Uncertainty about reward for option \(j\) on last trial

Uncertainty drives Kalman gain

Kalman gain influences beliefs about expected reward and uncertainty

```
# Update the expected utility of the arm by applying a Kalman
# filtering update rule. The model here does not learn any underlying
# trend for the arm, nor does it learn the volatility of the environment
mod$utilityUpdate <- function(current, chosen, gain, outcome) {
current + chosen * gain * (outcome - current)
}
# Compute the Kalman gain
mod$kalmanGain <- function(uncertainty, mod) {
(uncertainty + mod$param["sigma_xi"]^2) /
(uncertainty + mod$param["sigma_xi"]^2 + mod$param["sigma_eps"]^2)
}
# Update the posterior uncertainty associated with the arm, again
# using the Kalman rule
mod$uncertaintyUpdate <- function(current, chosen, gain, mod) {
(1 - chosen * gain) * (current + mod$param["sigma_xi"]^2)
}
```

In the standard reinforcement learning task, human behaviour is closely approximated by the Kalman filter model

When the threat of option loss is added, systematic differences appear

(* Yes, there's a data point missing. I'm hiding something)

- Data tidying
- Data summary
- Drawing figures
- Linear modelling
- R Markdown

The usual...

- Simulations for study design
- Model implementation in R

The others...

smallworldofwords.org

(cue)

(response)

- 90,000 participants
- 12,000 words
- 4 million responses

**https://smallworldofwords.org/en/project/visualize**

(Collins & Loftus 1975)

**similarity(rose, car)**

**similarity(cloud, house)**

**>**

Useful at the margins

- Drawing figures
- Linear modelling
- Helpful with the small studies

My skill limitations?

- I don't know much about working with big data sets in R
- Suggestions appreciated!

y_tho()

A cognitive psychologist with some experience with R

A grumpy psychometrician with a limited time budget

A social psychologist who wants to believe

Hello? Was it something I said?

Photo by Sven Scheuermeier on Unsplash

**Act I**

**Act II**

Hey, how would you feel about posting research code to the web?

Oh hell no. People would call me stupid

What? You literally wrote a textbook on latent variable modelling and your Ph.D. research won a best paper award at *Multivariate Behavior Research. *__You__ are not stupid

Yeah but my code has to be written in 20 minute consults and it's ugly. People are unkind about code

- Our code works (mostly), but it's ugly
- We have limited time & money
- It's difficult for us to make nice code, but at least we have some training

- You want me to put myself in the firing line too? Pfft
- Why should I do that????
- Byeeee....

- "Everyone you meet is fighting a hard battle"
- The code you see was written under unknown constraints
- Behavioural scientists aren't trained coders (any more than you are a behavioural scientist)

https://quoteinvestigator.com/2010/06/29/be-kind/

base R

tidyverse

jsPsych

Matlab

JASP

BayesFactor

.Rmd

SQL

lme

JAGS

Stan

base R

tidyverse

jsPsych

Matlab

JASP

BayesFactor

.Rmd

SQL

lme

JAGS

Stan

Even after 2.5 years at UNSW, integration of tools across different contexts is modest

Model selection via MDL principle, imposing a complexity penalty similar to a Bayesian approach with Jeffreys priors

You should use AIC

Hypothesis testing used a priori mapping of theoretically meaningful patterns onto order constrained inference problems

You need to run an ANOVA to test for interactions

We used package A

__ Everybody__ knows that package B is better

(seriously, you are **terrible** at sales)

**tidyverse**

**lme**

**git**

**BayesFactor**

**lavaan**

**car**

**Shiny**

**R markdown**

**knowing all the R**

**It can be hard to find a team within a single applied field...**

**#rstats**

**But there are people who will support you**

**Yay!**

By Danielle Navarro

Traditionally, R has been viewed as a language for data science and statistics. In the social sciences it has been extremely popular with researchers at the more quantitative end of the spectrum - but uptake has been less widespread outside of the more statistically inclined. I don't think the R language needs to be limited in this way. Since 2011 I've been teaching introductory research methods classes for undergraduates using R, running programming classes for R with postgraduate students, doing my own data analysis with R, implementing cognitive models with R and occasionally even running behavioural experiments in R. In this talk I reflect on some of these experiences - the good, the bad and the ugly - and discuss prospects and challenges for wider adoption R as a tool within the psychological sciences.

- 7,863

I am a computational cognitive scientist at the University of New South Wales. My research focuses on human concept learning, reasoning and decision making.