The work split 

• 20% preliminary research 

• 60% web development

• 20% sampling and analysis

Technical bits on GitHub:

http://www.gihtub.com/tuuleh/masters-battery

Why research video games?

a) they involve millions of people

b) also for non-game research

• genuine teams for research on team processes
  • ​different levels of 'teamedness'
• intrinsically motivated experts in their native niche for computerized testing 
• quantified level of expertise (rankings/ratings)
• easily accessible for web-based experiments

c)  e-sports psychology

• professional gaming is a billion dollar business​​
• LoL - $8.000.000 price money/season plus yearly salary
  • rigorous training regimes, competitions, sponsorship...
  • pro gamers can benefit from psychological techniques 

League of Legends

• Competitive free to play PvP MOBA / action real-time strategy (ARTS)
• 67 million monthly active players, 27 million active daily

How does it work?

• From level 1 to 30 with normal games
  • 'unranked' games against other players or AI
  • award XP and IP
• Level 30 and access to 16 champions -> ranked games
  • solo, 3v3, 5v5
  • first 10 games for placement
    • MMR used to match with similar opponents

1. Two teams, 3 or 5 players each
2. 1 "champion" per player
3. different roles:

• carry
• support
• tank
• bruiser
• jungler

4.  kill towers, minions, monsters (jungle) and enemy champions to collect resources

5.  overpower the enemy and destroy their nexus (base)

League System

Spatial span

• 4x4 grey grid with flashing rectangles
• reproduce the sequence with clicking
• decrement after two mistakes, incremented after a success
• over after four mistakes

Tower of London

• planning
• how many moves (minimum) from image A to image B
• 24 stimuli
  • between 2 and 6 difficulty

Sampling

Online forums and communities:

• MOBA fire
• Riot Gaming boards and forums
• Reddit League of Legends community
• League of Legends streamers on Twitch.tv
• Twitter
• Personal contact with players
• An e-sports community in Singapore
• No response from Riot

Participants

• N = 278, eligible N = 218, finished N = 146

• between ages 15 and 45 (M = 21.42, SD = 4.15)

Geographics by game regions:

• North America (N=143)

• EU-West (N=94)

• EU-North and East (N=21)

• Latin America (N=7)

• New Zealand (N=8)

• Turkey (N=2)

Play style (ranked):

• solo queue N=206
• 5v5 N= 51
• 3v3 N=18

Structure in the GEQ

• R - psych (Revelle, 2014)
• PCA, Promax rotation, r = 0.36.
• PC1 = task-oriented, a = 0.83
• PC2 = social, a = 0.79
• Distribution of the scores is not great

PCA for cognitive tasks

• R - psych (Revelle, 2014)
• 4 - component varimax
• Scoring with regression
• No missing value imputation

Spearman's correlations, solo

Spearman correlations, team

Subtitle

Regression...

Note - no imputation!

• Step-wise analysis with stepAIC function in MASS R package (Venables & Ripley, 2002)
• Best functioning models:
  • Solo rating: 
    • Flanker accuracy, planning and frustration:
      • R^2 =.16, F(3,57)=3.52, p<.05); 
        • b(RC3) = 0.22, p = 0.08
        • b(frustration) = -0.24, p = 0.06
        • b(RC4) = 0.18, p < 0.05
  • Team rating: 
    • Impossible models with TLX
    • 5v5 rating ~ task load + mental rotation:
      • R^2 = 1. , F(2,20)= 2.464e+3, p <.0001
        • b(task load) = 0.99, p < 0.0001
        • b(RC2) = 0.05, p < 0.0001
          • Similar model with RC3 and RC2
          • TLX is overpowering but non-significant on its own?

Conclusion

1. TLX works remarkably well - effort/frustration
2. Spatial span works remarkably unwell
3. Participant motivation and response styles are a huge challenge

As to the intentions...

1. Do cognitve skills and team cohesion relate to performance in League of Legends?

• Cognitive tasks appear important, but better test selection is needed
• Check out communication style instead of team cohesion, try different conative factors 

2. Which cognitive skills are relevant to gameplay in MOBAs ( + usability of GEQ and NASA TLX for game studies)

• Explore other cognitive skills - 3D rotation, task switching, ToL with CAT, go/no go
• GEQ somewhat lacks discriminativeness
• NASA TLX works great

3. Get experience building JavaScript-based web experiments ( + contribute something to jsPsych)

• Yes. Working on the plugins.

4. Gauge the usability of web-based methods and data for cognitive psychology game studies

• Yes - with care and certain reservations

• ... and more!
  •  invalid response times
  •  bugs (and poor bug reports)
  •  different hardware
  •  mobile access (!!)
  •  confounding environmental variables
  •  batteries are longer and strenuous than in survey research -> question of motivation

Question of motivation

• Bugs and poor usability:
  • drop outs
  • bad data
• Information and feedback
• Use adaptive testing
• Try some gamification
• Implement motivating features
  • login systems
  • data from API
  • spreadsheets with feedback
  • ...

Unreliable/malicious use?

• use an ORM
• aggregate AFTER cleaning up invalid trials
• discard data from 'non-serious' participants
• social network login, data from APIs
  • more reliable data
    • and more of it
  • less work for participants
  • reduces number of non-serious participants
  • anonymity?

• Anderson, C. A., Bailey, K. & West, R. (2010). A negative association between video game experience and proactive cognitive control. Psychophysiology, 47, 34/42.
• Andrews G., Murphy K. (2006). “Does video-game playing improve executive function?” in Frontiers in Cognitive Sciences, ed. Vanchevsky M. A., editor. (New York, NY: Nova Science Publishers, Inc.), 145–161. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3171788/
• Bainbridge, W. S. (2007). The scientific research potential of virtual worlds. Science, 317, 472-476.
• Balicer, R. (2005). Modeling Infectious Diseases Dissemination Through Online Role-Playing Games. Epidemiology 18 (2) pp. 260–261
• Best, J. R. (2012). Exergaming immediately enhances children's executive function. Developmental Psychology, 48(5), 1501-1510.
• Blizzard Entertainment. (2010, October 7). World of Warcraft® subscriber base reaches 12 million worldwide. Retrieved from: http://eu.blizzard.com/en-gb/company/press/pressreleases.html?101007
• Boot, W. R., Framer, A. F., Simons, D. J., Fabiani, M. & Gratton, G. (2008). The effects of video game playing on attention, memory and executive control. Acta Psychologica, 129, 387-398.
• Carron, A. V., Widmeyer, W. N., & Brawley, L. R. (1985). The development of an instrument to assess cohesion in sport teams: The Group Environment Questionnaire. Journal of Sport Psychology, 7, 244-266.
• Colzato L. S., van Leeuwen P. J. A., van den Wildenberg W. P. M., Hommel B. (2010). DOOM’d to switch: superior cognitive flexibility in players of first person shooter games. Frontiers of psychology, 1(8).
• Cooke, N.J., Gorman, J.C. & Winner, J.L. (2006). Team cognition. In F. Durso, R. Nickerson, S. Dumais, S. Lewandowsky, and T. Perfect, Handbook of Applied Cognition, 2nd Edition Wiley.
• Corsi, P.M.(1972), Human memory and the medial temporal region of the brain, Unpublished doctoral dissertation. McGill University.
• de Leeuw, J. R. (2014). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior Research Methods. Advance online publication. doi:10.3758/s13428-014-0458-y
• Drachen, A., Yancey, M., Maguire, J., Chu, D., Yuhui Wang, I., Mahlmann, T., Schubert, M. & Klabajan, D. (2014) Skill-Based Differences in Spatio-Temporal Team Behaviour in Defence of The Ancients 2 (DotA 2) (In proceeding) Proceedings of the IEEE Games, Entertainment, and Media (GEM) Conference 2014.
• Ducheneaut, N., Dmitri, W., Nickell, E., Xiog, L., Yee, N. & Zhang, Y. (2006). From tree house to barracks: The social life of guilds in World of Warcraft. Games and culture 1(4), 338-361
• Dye, M.W.G., Green, C.S., & Bavelier, D. (2009). The development of attention skills in action video game players. Neuropsychologia, 47(8-9), 1780-1789. doi:10.1016/j.neuropsychologia.2009.02.002.
• Eichstaedt, J. (2001). An inaccurate-timing filter for reaction-time measurement by JAVA-applets imple-menting internet-based experiments. Behavior Research Methods, Instruments, and Computers, 33(1), 179–186.
• El-Shagi, M. & von Schweinitz, G. (2014). The Diablo 3 Economy: An Agent Based Approach [PDF document]. Retrieved from: http://econ.henu.edu.cn/upload/20141016/14134680789594.pdf
• Entertainment Software Association (ESA) (2011). The 2011 Essential Facts About the Computer and Video Game Industry. Retrieved 20.12.2011 from http://www.theesa.com/facts/pdfs/ESA_EF_2011.pdf.
• Eriksen, B. A., & Eriksen, C. W. (1974). Effects of noise leters upon the identification of a target letter in a nonsearch task. Perception & Psychophysics, 16, 143-149.
• Feng, J., Spence, I. & Pratt, J. (2007). Playing an action video game reduces gender differences in spatial cognition. Psychological science, 18(10), 850-855.
• Ferrari, S. (2013) From Generative to Conventional Play: MOBA and League of Legends, Proceedings of DiGRA 2013: DeFragging Game Studies, Volume 1 (1), 1-17.
• Fletcher, R. B. & Whitton, S. M. (2014). The Group Environment Questionnaire: A Multilevel Confirmatory Factor Analysis. Small Group Research, 45(1), 68-88. doi: 10.1177/1046496413511121
• Green C. S., Bavelier D. (2007). Action video game experience alters the spatial resolution of attention. Psychological Science, 18(1), 88-94.
• Green, C. S., & Bavelier, D. (2003). Action video-game modifies visual selective attention. Nature, 423, 534-537.
• Guha, A., Jereen, A., DeGutis, J. & Wilmer, J. (2014). No action video game training effects for multiple object tracking or mental rotation. Journal of Vision, 22(14).
• Hart, S. (2006). Nasa-Task Load Index (Nasa-TLX); 20 Years Later. Human Factors and Ergonomics Society Annual Meeting Proceedings, 50, 904-908.
• Hart, S. G. & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Human Mental Workload (P. A. Hancock and N. Meshkati (Eds.)), 139-183. North-Holland: Elsevier Science.
• Heslin, R. (1964). Predicting group task effectiveness from member characteristics. Psychological Bulletin, 62, 248-256.
• Hollenbeck, J.R., Moon, H., Ellis, A.P., West, B.J., Ilgen, D.R., Sheppard, L., Porter, C.O., & Wagner J.A. (2002). Structural contingency theory and individual differences: Examination of external and internal person-team fit. Journal of Applied Psychology, 87, 599-606.
• Karle J. W., Watter S., Shedden J. M. (2010). Task switching in video game players: benefits of selective attention but not resistance to proactive interference. Acta Psychologica (Amst.) 134(1).
• LePine, J.A., Hollenbeck, J.R., Ilgen, D. R., & Hedlund, J. (1997). Effects of individual differences on the performance of hierarchical decision-making teams: Much more than g. Journal of Applied Psychology, 82, 803-811.
• Maillot, P., Perrot, A. & Hartley, A. (2012). Effects of interactive physical-activity video-game training on physical and cognitive function in older adults. Psychology and Aging, 27(3), 589-600.
• Mueller, S. T., & Piper, B. J. (2014). The Psychology Experiment Building Language (PEBL) and PEBL Test Battery. Journal of neuroscience methods (222), 250–259.
• N. Pobiedina, J. Neidhardt, M. d. C. Calatrava Moreno & H. Werthner (2013). Ranking factors of team success in Proc. of the 22nd international conference on World Wide Web companion. International World Wide Web conferences Steering Committee, 2013, pp. 1185-1194.
• Narishm. (2013). A map of the Summoner's Rift map on League of Legends with an overlay dividing it into easy to remember regions. Retrieved 29.11.2014 from http://narishm.deviantart.com/art/League-of-Legends-Summoner-s-Rift-Regions-Map-413345149
• Okagaki, L., & Frensch, P. A. (1994). Effects of video game playing on measures of spatial performance: Gender effects in late adolescence. Journal of Applied Developmental Psychology, 15(1), 33-58.
• Owen, A. M. & Hampshire, A. (n.d.). Spatial span ladder [software for online web-based testing]. Cambridge Brain Sciences Inc, University of Western Ontario, Canada. Retrieved from http://www.cambridgebrainsciences.com/browse/memory/test/spatial-span-ladder.
• Perlroth, N. (2013). Department of Homeland Security: Disable Java ‘Unless It Is Absolutely Necessary’. Bits – the business of technology, a New York Times blog. Retrived from: http://bits.blogs.nytimes.com/2013/01/14/department-of-homeland-security-disable-java-unless-it-is-absolutely-necessary/
• Pöllänen, T. I. (2014). Test battery of JavaScript cognitive tasks written using jsPsych for my master's thesis. Github repository. Retrieved from https://github.com/tuuleh/masters-battery.
• Revelle, W. (2014) psych: Procedures for Personality and Psychological Research, Northwestern University, Evanston, Illinois, USA, http://CRAN.R-project.org/package=psych Version = 1.4.8.
• Riot Games. (2014). “Our Games”. Retrieved from: http://www.riotgames.com/our-games
• Salas, E., Dickinson, T. L., Converse, S.A. & Tannenbaum, S. I. (1992). Toward an understanding of team performance and training. In R.W. Swezey & E. Salas (Eds.), Teams: Their training and performance (pp. 3-20). Norwood, NJ: Ablex.
• Shalice, T. (1982). Specific impairments of planning. Philosophical Transactions of the Royal Society B: Biological Sciences, 298, 199-209.
• Stins, J. F., Polderman, T. J. C., Boomsma, D. I., & de Geus, E. J. C. (2007). Conditional accuracy in response interference tasks: Evidence from the Eriksen flanker task and the spatial conflict task. Advances in Cognitive Psychology 3 (3), 389–396.
• Venables, W. N. & Ripley, B. D. (2002) Modern Applied Statistics with S. Fourth Edition. Springer, New York. ISBN 0-387-95457-0
• W3Techs (2014). Usage of client-side programming languages for websites. Retrieved 2.12.2014 from http://w3techs.com/technologies/overview/client_side_language/all

Let's talk!