Cognitive skills and team cohesion in League of Legends players

Tuuli Pöllänen, 28.11.2014

Mentor: izr. prof. dr. Grega Repovš

My intentions

1. see how cognitive skills and team cohesion are implied in player performance in League of Legends

2. identify cognitive skills relevant to gameplay in MOBAs ( + gauge the usability of a sports team cohesion inventory for League of Legends teams)

  • Implications for:
    • instrument selection for training studies
    • contribute to the tiny body of academic literature on MOBAs
    • help players and their teams understand success factors

3. Get experience building JavaScript-based web experiments

4. Gauge the usability of web-based methods and data for this kind of research

 

The work split 

  • 20% preliminary research 

  • 60% web development

  • 20% sampling and analysis

The presentation will reflect this!

Why research video games?

a) they're a culturally topical phenomenon involving millions of people

b) they're good for non-game research

  • genuine teams for research on team processes
    • ​different levels of 'teamedness'
  • intrinsically motivated experts in their native niche for computerized testing 
  • quantified level of expertise (rankings/ratings)
  • easily accessible for web-based experiments

c)  e-sports psychology

  • professional gaming is a billion dollar business
  • LoL - $8.000.000 price money/season plus yearly salary
    • rigorous training regimes, competitions, sponsorship...
    • pro gamers can benefit from psychological techniques 

Problems with existing research

  • Irrelevance
  • Small sample sizes (N=11?)
  • How to define novices and experts
  • Fuzziness in genres and instruments
  • Gender comparisons with biased samples (e.g. Boot et al. (2008) - male experts and novices, female controls)
  • Very little research on some of the most popular genres! (looking at MOBAs)

 

League of Legends

  • Competitive free to play PvP MOBA / action real-time strategy (ARTS)
  • 67 million monthly active players, 27 million active daily

How does it work?

  • From level 1 to 30 with normal games
    • 'unranked' games against other players or AI
    • award XP and IP
  • Level 30 and access to 16 champions -> ranked games
    • solo, 3v3, 5v5
    • first 10 games for placement, then MMR matched with similar opponents and adjusted for wins/losses

League System

The development process

 

1. Exploring existing solutions

  • Three options for web experiments: Java, Flash or JavaScript
  • Flash: nope.

2. Java:

  • plenty of existing research
  • premade (albeit inactive) platforms
  • I got to work and...
    • Abandoned the idea of having animated tasks (for now)
    • Client side Java?
      • Your Java version is out-of-date. 
      • Application Blocked
      • Application Blocked by Security Settings 
      • Your security settings have blocked an application from running with an out-of-date or expired version of Java. 
      • Your security settings have blocked an untrusted application from running.

3. JavaScript...

Creating web-based cognitive tasks in JavaScript

  • No prior research on RT accuracy
  • Only one pre-existing non-commercial solution for creating web experiments
    • JsPsych (de Leeuw, 2014)
      • flexible plugins to create experiments (jQuery)
      • front end
      • so I forked and since June I ...
        • Altered some of the features in the core
        • Wrote two plugins for my own use
        • Added some pages without jsPsych

And then I had everything running locally! Huzzah!

Now what?

Back end architecture...

  • Node.js + Express 4.0
  • Sequelize ORM for MySQL database
  • ... on a DigitalOceans virtual server in San Francisco

And then I had a launch party!

 

... and then I realized I hadn't even started with my research yet...

... and I had less than three weeks until the end of the season.

The battery

A total of ~20-30 minutes of participation:

  1. Landing page - informed consent
  2. Demographic items
  3. Group Environment Questionnaire - cohesion in sports teams (task vs. social)
  4. Nasa Task Load Index - mental/physical/temporal/effort/performance/frustration
  5. Eriksen Flanker task - executive func/conflict adaptation/inhibition
  6. 2D mental rotation
  7. Spatial Span - visuospatial working memory
  8. Tower of London - planning
  9. Finish page

Sampling

Online forums and communities:

  • MOBA fire
  • Riot Gaming boards and forums
  • Reddit League of Legends community
  • League of Legends streamers on Twitch.tv
  • Twitter
  • Personal contact with players
  • An e-sports community in Singapore

 

Unfortunately, Riot was non-responsive

Participants

  • = 278, eligible = 218, finished N = 146

  • between ages 15 and 45 (M = 21.42, SD = 4.15)

Geographics by game regions:

  • North America (N=143)

  • EU-West (N=94)

  • EU-North and East (N=21)

  • Latin America (N=7)

  • New Zealand (N=8)

  • Turkey (N=2)

Play style (ranked):

  • solo queue N=206
  • 5v5 N= 51
  • 3v3 N=18

Structure in the GEQ

  • Promax rotation, = 0.36.
  • PC1 = task-oriented, = 0.83
  • PC2 = social, = 0.79
  • Distribution of the scores is not great

Dimension reduction for cognitive tasks

PCA, 4 - component varimax

Results

Regression...

Note - no imputation!

  • Step-wise analysis (naughty...)
  • A few working preliminary models
    • solo rating ~ RC3 + RC4 + frustration:
      • R^2 =.16, F(3,57)=3.52, p<.05); 
        • b(RC3) = 0.22, p = 0.08
        • b(frustration) = -0.24, p = 0.06
        • b(RC4) = 0.18, p < 0.05
      • Either just RC3 or RC4 works, too -> stepwise returns RC3 + frustration
    • Interesting (happens due to low sample size, but still): 
      • 5v5 rating ~ task load + RC2:
        • R^2 = 1. , F(2,20)= 2.464e+3, p<.0001
          • b(task load) = 0.99, p < 0.0001
          • b(RC2) = 0.05, < 0.0001
            • I get a similar model with RC3 and RC2
            • TLX is overpowering
            • Resample or come up with unbiased imputation

 

Conclusion

  1. TLX works remarkably well
  2. Spatial span works remarkably unwell

As to the intentions...

1. See how cognitive skills and team cohesion are implied in player performance in LoL

  • Partly.
    • Cognitive tasks appear important - better test selection.
    • Check out communication style instead of team cohesion, try different conative factors 

2. Identify cognitive skills relevant to gameplay in MOBAs ( + gauge the usability of a sports team cohesion inventory for League of Legends teams)

  • Partly.
    • Explore other cognitive skills - 3D rotation, task switching, ToL with CAT, go/no go
    • GEQ lacks discriminativeness - maybe for professional teams

3. Get experience building JavaScript-based web experiments

  • Yes.

4. Gauge the usability of web-based methods and data for this kind of research

  • Yes - with certain reservations

Last things about the web experiments...

  •  'some' back-end considerations
  •  combining data from web and lab studies
  •  All the shortcomings of web-based survey research:
    •  participant drop-out
    •  invalid responses
    •  submitting false data
    •  multiple entries
    •  self-selective sample
    •  ...
  • ... and more!
    •  invalid response times
    •  bugs (and poor bug reports)
    •  different hardware
    •  mobile access (!!)
    •  confounding environmental variables
    •  batteries are longer and strenuous than in survey research -> question of motivation
  • AWESOME for prototyping
  • OVERSAMPLE!
  • Combine with APIs

How to make your participants stick around

​MOTIVATE

  • make a bug-free product
  • inform
  • be available, responsive, respectful
  • feedback
  • CAT - less time for smaller standard error
  • Get creative - try some gamification
  • add mechanisms for virality

some of the references

  • Karle J. W., Watter S., Shedden J. M. (2010). Task switching in video game players: benefits of selective attention but not resistance to proactive interference. Acta Psychologica (Amst.) 134(1).
  • Green, C. S., & Bavelier, D. (2003). Action video-game modifies visual selective attention. Nature, 423, 534-537.
  • Corsi, P.M.(1972), Human memory and the medial temporal region of the brain, Unpublished doctoral dissertation. McGill University.
  • Eriksen, B. A., & Eriksen, C. W. (1974). Effects of noise leters upon the identification of a target letter in a nonsearch task. Perception & Psychophysics, 16, 143-149.
  • Hart, S. G. & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Human Mental Workload (P. A. Hancock and N. Meshkati (Eds.)), 139-183. North-Holland: Elsevier Science. 
  • Fletcher, R. B. & Whitton, S. M. (2014). The Group Environment Questionnaire: A Multilevel Confirmatory Factor Analysis. Small Group Research, 45(1), 68-88. doi: 10.1177/1046496413511121
  • de Leeuw, J.R. (2014). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior Research Methods. Advance online publication. doi:10.3758/s13428-014-0458-y
  • Hart, S. (2006). Nasa-Task Load Index (Nasa-TLX); 20 Years Later. Human Factors and Ergonomics Society Annual Meeting Proceedings, 50, 904-908.
  • Pöllänen, T. (2014). Test battery of JavaScript cognitive tasks written using jsPsych for my master's thesis. Github repository. Retrieved from https://github.com/tuuleh/masters-battery.
  • Owen, A. M. & Hampshire, A. (n.d.). Spatial span ladder [software for online web-based testing]. Cambridge Brain Sciences Inc, University of Western Ontario, Canada. Retrieved from http://www.cambridgebrainsciences.com/browse/memory/test/spatial-span-ladder. 
  • Feng, J., Spence, I. & Pratt, J. (2007). Playing an action video game reduces gender differences in spatial cognition. Psychological science, 18(10), 850-855.
  • Maillot, P., Perrot, A. & Hartley, A. (2012). Effects of interactive physical-activity video-game training on physical and cognitive function in older adults. Psychology and Aging, 27(3), 589-600.
  • Anderson, C. A., Bailey, K. & West, R. (2010). A negative association between video game experience and proactive cognitive control. Psychophysiology, 47, 34/42.
  • Dye, M.W.G., Green, C.S., & Bavelier, D. (2009). The development of attention skills in action video game players. Neuropsychologia, 47(8-9), 1780-1789. doi:10.1016/j.neuropsychologia.2009.02.002.
  • Boot, W. R., Framer, A. F., Simons, D. J., Fabiani, M. & Gratton, G. (2008). The effects of video game playing on attention, memory and executive control. Acta Psychologica, 129, 387-398.
  • Okagaki, L., & Frensch, P. A. (1994). Effects of video game playing on measures of spatial performance: Gender effects in late adolescence. Journal of Applied Developmental Psychology, 15(1), 33-58.
  • Stins, J. F., Polderman, T. J. C., Boomsma, D. I., & de Geus, E. J. C. (2007). Conditional accuracy in response interference tasks: Evidence from the Eriksen flanker task and the spatial conflict task. Advances in Cognitive Psychology 3 (3), 389–396.
  • Mueller, S. T., & Piper, B. J. (2014). The Psychology Experiment Building Language (PEBL) and PEBL Test Battery. Journal of neuroscience methods (222), 250–259.
  • Best, J. R. (2012). Exergaming immediately enhances children's executive function. Developmental Psychology, 48(5), 1501-1510.
  • Guha, A., Jereen, A., DeGutis, J. & Wilmer, J. (2014). No action video game training effects for multiple object tracking or mental rotation. Journal of Vision, 22(14).
  • Colzato L. S., van Leeuwen P. J. A., van den Wildenberg W. P. M., Hommel B. (2010). DOOM’d to switch: superior cognitive flexibility in players of first person shooter games. Frontiers of psychology, 1(8).
  • Green C. S., Bavelier D. (2007). Action video game experience alters the spatial resolution of attention. Psychological Science, 18(1), 88-94.
  • Carron, A. V., Widmeyer, W. N., & Brawley, L. R. (1985). The development of an instrument to assess cohesion in sport teams: The Group Environment Questionnaire. Journal of Sport Psychology, 7, 244-266.
  • N. Pobiedina, J. Neidhardt, M. d. C. Calatrava Moreno & H. Werthner (2013). Ranking factors of team success in Proc. of the 22nd international conference on World Wide Web companion. International World Wide Web conferences Steering Committee, 2013, pp. 1185-1194.
  • Drachen, A., Yancey, M., Maguire, J., Chu, D., Yuhui Wang, I., Mahlmann, T., Schubert, M. & Klabajan, D. (2014). Skill-Based Differences in Spatio-Temporal Team Behaviour in Defence of The Ancients 2 (DotA 2) (Inproceeding), Proceedings of the IEEE Games, Entertainment, and Media (GEM) Conference 2014.

Cognitive skills and team cohesion in League of Legends players​

By tuuli

Cognitive skills and team cohesion in League of Legends players​

  • 2,305