The work split 

• 20% preliminary research 

• 60% web development

• 20% sampling and analysis

The presentation will reflect this!

Why research video games?

a) they're a culturally topical phenomenon involving millions of people

b) they're good for non-game research

• genuine teams for research on team processes
  • ​different levels of 'teamedness'
• intrinsically motivated experts in their native niche for computerized testing 
• quantified level of expertise (rankings/ratings)
• easily accessible for web-based experiments

c)  e-sports psychology

• professional gaming is a billion dollar business​​
• LoL - $8.000.000 price money/season plus yearly salary
  • rigorous training regimes, competitions, sponsorship...
  • pro gamers can benefit from psychological techniques 

Problems with existing research

• Irrelevance
• Small sample sizes (N=11?)
• How to define novices and experts
• Fuzziness in genres and instruments
• Gender comparisons with biased samples (e.g. Boot et al. (2008) - male experts and novices, female controls)
• Very little research on some of the most popular genres! (looking at MOBAs)

 

League of Legends

• Competitive free to play PvP MOBA / action real-time strategy (ARTS)
• 67 million monthly active players, 27 million active daily

How does it work?

• From level 1 to 30 with normal games
  • 'unranked' games against other players or AI
  • award XP and IP
• Level 30 and access to 16 champions -> ranked games
  • solo, 3v3, 5v5
  • first 10 games for placement, then MMR matched with similar opponents and adjusted for wins/losses

League System

The battery

A total of ~20-30 minutes of participation:

1. Landing page - informed consent
2. Demographic items
3. Group Environment Questionnaire - cohesion in sports teams (task vs. social)
4. Nasa Task Load Index - mental/physical/temporal/effort/performance/frustration
5. Eriksen Flanker task - executive func/conflict adaptation/inhibition
6. 2D mental rotation
7. Spatial Span - visuospatial working memory
8. Tower of London - planning
9. Finish page

Sampling

Online forums and communities:

• MOBA fire
• Riot Gaming boards and forums
• Reddit League of Legends community
• League of Legends streamers on Twitch.tv
• Twitter
• Personal contact with players
• An e-sports community in Singapore

 

Unfortunately, Riot was non-responsive

Participants

• N = 278, eligible N = 218, finished N = 146

• between ages 15 and 45 (M = 21.42, SD = 4.15)

Geographics by game regions:

• North America (N=143)

• EU-West (N=94)

• EU-North and East (N=21)

• Latin America (N=7)

• New Zealand (N=8)

• Turkey (N=2)

Play style (ranked):

• solo queue N=206
• 5v5 N= 51
• 3v3 N=18

Structure in the GEQ

• Promax rotation, r = 0.36.
• PC1 = task-oriented, a = 0.83
• PC2 = social, a = 0.79
• Distribution of the scores is not great

Dimension reduction for cognitive tasks

PCA, 4 - component varimax

Results

Regression...

Note - no imputation!

• Step-wise analysis (naughty...)
• A few working preliminary models
  • solo rating ~ RC3 + RC4 + frustration:
    • R^2 =.16, F(3,57)=3.52, p<.05); 
      • b(RC3) = 0.22, p = 0.08
      • b(frustration) = -0.24, p = 0.06
      • b(RC4) = 0.18, p < 0.05
    • Either just RC3 or RC4 works, too -> stepwise returns RC3 + frustration
  • Interesting (happens due to low sample size, but still): 
    • 5v5 rating ~ task load + RC2:
      • R^2 = 1. , F(2,20)= 2.464e+3, p<.0001
        • b(task load) = 0.99, p < 0.0001
        • b(RC2) = 0.05, p < 0.0001
          • I get a similar model with RC3 and RC2
          • TLX is overpowering
          • Resample or come up with unbiased imputation

 

Conclusion

1. TLX works remarkably well
2. Spatial span works remarkably unwell

As to the intentions...

1. See how cognitive skills and team cohesion are implied in player performance in LoL

• Partly.
  • Cognitive tasks appear important - better test selection.
  • Check out communication style instead of team cohesion, try different conative factors 

2. Identify cognitive skills relevant to gameplay in MOBAs ( + gauge the usability of a sports team cohesion inventory for League of Legends teams)

• Partly.
  • Explore other cognitive skills - 3D rotation, task switching, ToL with CAT, go/no go
  • GEQ lacks discriminativeness - maybe for professional teams

3. Get experience building JavaScript-based web experiments

• Yes.

4. Gauge the usability of web-based methods and data for this kind of research

• Yes - with certain reservations

• ... and more!
  •  invalid response times
  •  bugs (and poor bug reports)
  •  different hardware
  •  mobile access (!!)
  •  confounding environmental variables
  •  batteries are longer and strenuous than in survey research -> question of motivation

How to make your participants stick around

​MOTIVATE

• make a bug-free product
• inform
• be available, responsive, respectful
• feedback
• CAT - less time for smaller standard error
• Get creative - try some gamification
• add mechanisms for virality

some of the references

• Karle J. W., Watter S., Shedden J. M. (2010). Task switching in video game players: benefits of selective attention but not resistance to proactive interference. Acta Psychologica (Amst.) 134(1).
• Green, C. S., & Bavelier, D. (2003). Action video-game modifies visual selective attention. Nature, 423, 534-537.
• Corsi, P.M.(1972), Human memory and the medial temporal region of the brain, Unpublished doctoral dissertation. McGill University.
• Eriksen, B. A., & Eriksen, C. W. (1974). Effects of noise leters upon the identification of a target letter in a nonsearch task. Perception & Psychophysics, 16, 143-149.
• Hart, S. G. & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Human Mental Workload (P. A. Hancock and N. Meshkati (Eds.)), 139-183. North-Holland: Elsevier Science. 
• Fletcher, R. B. & Whitton, S. M. (2014). The Group Environment Questionnaire: A Multilevel Confirmatory Factor Analysis. Small Group Research, 45(1), 68-88. doi: 10.1177/1046496413511121
• de Leeuw, J.R. (2014). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior Research Methods. Advance online publication. doi:10.3758/s13428-014-0458-y
• Hart, S. (2006). Nasa-Task Load Index (Nasa-TLX); 20 Years Later. Human Factors and Ergonomics Society Annual Meeting Proceedings, 50, 904-908.
• Pöllänen, T. (2014). Test battery of JavaScript cognitive tasks written using jsPsych for my master's thesis. Github repository. Retrieved from https://github.com/tuuleh/masters-battery.
• Owen, A. M. & Hampshire, A. (n.d.). Spatial span ladder [software for online web-based testing]. Cambridge Brain Sciences Inc, University of Western Ontario, Canada. Retrieved from http://www.cambridgebrainsciences.com/browse/memory/test/spatial-span-ladder. 
• Feng, J., Spence, I. & Pratt, J. (2007). Playing an action video game reduces gender differences in spatial cognition. Psychological science, 18(10), 850-855.
• Maillot, P., Perrot, A. & Hartley, A. (2012). Effects of interactive physical-activity video-game training on physical and cognitive function in older adults. Psychology and Aging, 27(3), 589-600.
• Anderson, C. A., Bailey, K. & West, R. (2010). A negative association between video game experience and proactive cognitive control. Psychophysiology, 47, 34/42.
• Dye, M.W.G., Green, C.S., & Bavelier, D. (2009). The development of attention skills in action video game players. Neuropsychologia, 47(8-9), 1780-1789. doi:10.1016/j.neuropsychologia.2009.02.002.
• Boot, W. R., Framer, A. F., Simons, D. J., Fabiani, M. & Gratton, G. (2008). The effects of video game playing on attention, memory and executive control. Acta Psychologica, 129, 387-398.
• Okagaki, L., & Frensch, P. A. (1994). Effects of video game playing on measures of spatial performance: Gender effects in late adolescence. Journal of Applied Developmental Psychology, 15(1), 33-58.
• Stins, J. F., Polderman, T. J. C., Boomsma, D. I., & de Geus, E. J. C. (2007). Conditional accuracy in response interference tasks: Evidence from the Eriksen flanker task and the spatial conflict task. Advances in Cognitive Psychology 3 (3), 389–396.
• Mueller, S. T., & Piper, B. J. (2014). The Psychology Experiment Building Language (PEBL) and PEBL Test Battery. Journal of neuroscience methods (222), 250–259.
• Best, J. R. (2012). Exergaming immediately enhances children's executive function. Developmental Psychology, 48(5), 1501-1510.
• Guha, A., Jereen, A., DeGutis, J. & Wilmer, J. (2014). No action video game training effects for multiple object tracking or mental rotation. Journal of Vision, 22(14).
• Colzato L. S., van Leeuwen P. J. A., van den Wildenberg W. P. M., Hommel B. (2010). DOOM’d to switch: superior cognitive flexibility in players of first person shooter games. Frontiers of psychology, 1(8).
• Green C. S., Bavelier D. (2007). Action video game experience alters the spatial resolution of attention. Psychological Science, 18(1), 88-94.
• Carron, A. V., Widmeyer, W. N., & Brawley, L. R. (1985). The development of an instrument to assess cohesion in sport teams: The Group Environment Questionnaire. Journal of Sport Psychology, 7, 244-266.
• N. Pobiedina, J. Neidhardt, M. d. C. Calatrava Moreno & H. Werthner (2013). Ranking factors of team success in Proc. of the 22nd international conference on World Wide Web companion. International World Wide Web conferences Steering Committee, 2013, pp. 1185-1194.
• Drachen, A., Yancey, M., Maguire, J., Chu, D., Yuhui Wang, I., Mahlmann, T., Schubert, M. & Klabajan, D. (2014). Skill-Based Differences in Spatio-Temporal Team Behaviour in Defence of The Ancients 2 (DotA 2) (Inproceeding), Proceedings of the IEEE Games, Entertainment, and Media (GEM) Conference 2014.