The influence of cognitive skills and team cohesion on player performance in Multiplayer Online Battle Arena
Tuuli Pöllänen, 15.12.2014
My intentions
1. Do cognitve skills and team cohesion relate to performance in League of Legends?
2. Which cognitive skills are relevant to gameplay in MOBAs ( + usability of GEQ and NASA TLX for game studies)
-
Implications for:
- instrument selection for training studies
- contribute to the tiny body of academic literature on MOBAs
- help players and their teams understand success factors
3. Get experience building JavaScript-based web experiments ( + contribute something to jsPsych)
4. Gauge the usability of web-based methods and data for cognitive psychology game studies
The work split
-
20% preliminary research
-
60% web development
-
20% sampling and analysis
Technical bits on GitHub:
http://www.gihtub.com/tuuleh/masters-battery
Why research video games?
a) they involve millions of people
b) also for non-game research
-
genuine teams for research on team processes
- different levels of 'teamedness'
- intrinsically motivated experts in their native niche for computerized testing
- quantified level of expertise (rankings/ratings)
- easily accessible for web-based experiments
c) e-sports psychology
- professional gaming is a billion dollar business
- LoL - $8.000.000 price money/season plus yearly salary
- rigorous training regimes, competitions, sponsorship...
- pro gamers can benefit from psychological techniques
League of Legends
- Competitive free to play PvP MOBA / action real-time strategy (ARTS)
- 67 million monthly active players, 27 million active daily
How does it work?
- From level 1 to 30 with normal games
- 'unranked' games against other players or AI
- award XP and IP
- Level 30 and access to 16 champions -> ranked games
- solo, 3v3, 5v5
- first 10 games for placement
- MMR used to match with similar opponents
- Two teams, 3 or 5 players each
- 1 "champion" per player
- different roles:
- carry
- support
- tank
- bruiser
- jungler
4. kill towers, minions, monsters (jungle) and enemy champions to collect resources
5. overpower the enemy and destroy their nexus (base)
League System
The development process
1. Exploring existing solutions
- Java, Flash or JavaScript
2. Flash: nope.
3. Java:
- Well researched, existing platforms for creating experiments
- Client side Java?
- Your Java version is out-of-date.
- Application Blocked
- Application Blocked by Security Settings
- Your security settings have blocked an application from running with an out-of-date or expired version of Java.
- Your security settings have blocked an untrusted application from running.
- Client side Java?
4. JavaScript...
Creating web-based cognitive tasks in JavaScript
- No prior research on RT accuracy
- Only one open source platform
- JsPsych (de Leeuw, 2014)
- flexible plugins to create experiments (jQuery)
- ! styling and back end are all up to you
- I forked my own version:
- Altered some of the features in the core
- Wrote two plugins for my own use
- Added some pages without jsPsych
- NOT COMPATIBLE with recent changes to jsPsych
- I'm working on it...
- JsPsych (de Leeuw, 2014)
Back end
- Node.js + Express 4.0
- modular, easy to build on
- Nginx as proxy
- Sequelize for MySQL database
- jsPsych spits out CSV or JSON
- maybe experiment with MongoDB
- jsPsych spits out CSV or JSON
- ... on a DigitalOceans virtual server in San Francisco
- SSDs!
...with very little time for sampling before the end of season 2014.
The battery
A total of ~20-30 minutes of participation:
- Informed consent
- Demographic items
- Group Environment Questionnaire
- Nasa Task Load Index
- Eriksen Flanker task
- 2D mental rotation
- Spatial Span
- Tower of London
- Feedback page
live at http://leagueoflegends.web-psychometrics.com
GitHub: http://github.com/tuuleh/masters-battery
GEQ
Carron, Widmeyer & Brawley, 1985) - group cohesion in sports psychology
- group integration - social
- group integration - task
- individual attractions to the group - social
- individual attractions to the group - task
Promax PCA - task + social
NASA TLX
Hart & Staveland, 1988 - task load inventory
- mental demand
- physical demand
- temporal demand
- performance
- frustration
- effort
Conflict adaptation
Processing speed
Interference
Implementation similar to PEBL
- fast pace, constant time between stimuli
- 12 x 4
Mental rotation
Two-dimensional version
- 2AFC
- mirrored (P) or identical (Q)
- three stimuli
- at 0, 60, 120, 180, 240 and 300 degrees
- feedback between trials
Eriksen flanker
Spatial span
- 4x4 grey grid with flashing rectangles
- reproduce the sequence with clicking
- decrement after two mistakes, incremented after a success
- over after four mistakes
Tower of London
- planning
- how many moves (minimum) from image A to image B
- 24 stimuli
- between 2 and 6 difficulty
Sampling
Online forums and communities:
- MOBA fire
- Riot Gaming boards and forums
- Reddit League of Legends community
- League of Legends streamers on Twitch.tv
- Personal contact with players
- An e-sports community in Singapore
- No response from Riot
Participants
-
N = 278, eligible N = 218, finished N = 146
-
between ages 15 and 45 (M = 21.42, SD = 4.15)
Geographics by game regions:
-
North America (N=143)
-
EU-West (N=94)
-
EU-North and East (N=21)
-
Latin America (N=7)
-
New Zealand (N=8)
-
Turkey (N=2)
Play style (ranked):
- solo queue N=206
- 5v5 N= 51
- 3v3 N=18
Structure in the GEQ
- R - psych (Revelle, 2014)
- PCA, Promax rotation, r = 0.36.
- PC1 = task-oriented, a = 0.83
- PC2 = social, a = 0.79
- Distribution of the scores is not great
PCA for cognitive tasks
- R - psych (Revelle, 2014)
- 4 - component varimax
- Scoring with regression
- No missing value imputation
Spearman's correlations, solo
Spearman correlations, team
Subtitle
Regression...
Note - no imputation!
- Step-wise analysis with stepAIC function in MASS R package (Venables & Ripley, 2002)
- Best functioning models:
- Solo rating:
- Flanker accuracy, planning and frustration:
-
R^2 =.16, F(3,57)=3.52, p<.05);
- b(RC3) = 0.22, p = 0.08
- b(frustration) = -0.24, p = 0.06
- b(RC4) = 0.18, p < 0.05
-
R^2 =.16, F(3,57)=3.52, p<.05);
- Flanker accuracy, planning and frustration:
- Team rating:
- Impossible models with TLX
- 5v5 rating ~ task load + mental rotation:
-
R^2 = 1. , F(2,20)= 2.464e+3, p <.0001
- b(task load) = 0.99, p < 0.0001
- b(RC2) = 0.05, p < 0.0001
- Similar model with RC3 and RC2
- TLX is overpowering but non-significant on its own?
-
R^2 = 1. , F(2,20)= 2.464e+3, p <.0001
- Solo rating:
Conclusion
- TLX works remarkably well - effort/frustration
- Spatial span works remarkably unwell
- Participant motivation and response styles are a huge challenge
As to the intentions...
1. Do cognitve skills and team cohesion relate to performance in League of Legends?
- Cognitive tasks appear important, but better test selection is needed
- Check out communication style instead of team cohesion, try different conative factors
2. Which cognitive skills are relevant to gameplay in MOBAs ( + usability of GEQ and NASA TLX for game studies)
- Explore other cognitive skills - 3D rotation, task switching, ToL with CAT, go/no go
- GEQ somewhat lacks discriminativeness
- NASA TLX works great
3. Get experience building JavaScript-based web experiments ( + contribute something to jsPsych)
- Yes. Working on the plugins.
4. Gauge the usability of web-based methods and data for cognitive psychology game studies
- Yes - with care and certain reservations
Last things about the web experiments...
- All the shortcomings of web-based survey research:
- participant drop-out
- invalid responses
- submitting false data
- multiple entries
- self-selective sample
- ...
- ... and more!
- invalid response times
- bugs (and poor bug reports)
- different hardware
- mobile access (!!)
- confounding environmental variables
- batteries are longer and strenuous than in survey research -> question of motivation
- Great for prototyping
- Oversample
- Triangulate with APIs
Question of motivation
- Bugs and poor usability:
- drop outs
- bad data
- Information and feedback
- Use adaptive testing
- Try some gamification
- Implement motivating features
- login systems
- data from API
- spreadsheets with feedback
- ...
Unreliable/malicious use?
- use an ORM
- aggregate AFTER cleaning up invalid trials
- discard data from 'non-serious' participants
- social network login, data from APIs
- more reliable data
- and more of it
- less work for participants
- reduces number of non-serious participants
- anonymity?
- more reliable data
- Anderson, C. A., Bailey, K. & West, R. (2010). A negative association between video game experience and proactive cognitive control. Psychophysiology, 47, 34/42.
- Andrews G., Murphy K. (2006). “Does video-game playing improve executive function?” in Frontiers in Cognitive Sciences, ed. Vanchevsky M. A., editor. (New York, NY: Nova Science Publishers, Inc.), 145–161. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3171788/
- Bainbridge, W. S. (2007). The scientific research potential of virtual worlds. Science, 317, 472-476.
- Balicer, R. (2005). Modeling Infectious Diseases Dissemination Through Online Role-Playing Games. Epidemiology 18 (2) pp. 260–261
- Best, J. R. (2012). Exergaming immediately enhances children's executive function. Developmental Psychology, 48(5), 1501-1510.
- Blizzard Entertainment. (2010, October 7). World of Warcraft® subscriber base reaches 12 million worldwide. Retrieved from: http://eu.blizzard.com/en-gb/company/press/pressreleases.html?101007
- Boot, W. R., Framer, A. F., Simons, D. J., Fabiani, M. & Gratton, G. (2008). The effects of video game playing on attention, memory and executive control. Acta Psychologica, 129, 387-398.
- Carron, A. V., Widmeyer, W. N., & Brawley, L. R. (1985). The development of an instrument to assess cohesion in sport teams: The Group Environment Questionnaire. Journal of Sport Psychology, 7, 244-266.
- Colzato L. S., van Leeuwen P. J. A., van den Wildenberg W. P. M., Hommel B. (2010). DOOM’d to switch: superior cognitive flexibility in players of first person shooter games. Frontiers of psychology, 1(8).
- Cooke, N.J., Gorman, J.C. & Winner, J.L. (2006). Team cognition. In F. Durso, R. Nickerson, S. Dumais, S. Lewandowsky, and T. Perfect, Handbook of Applied Cognition, 2nd Edition Wiley.
- Corsi, P.M.(1972), Human memory and the medial temporal region of the brain, Unpublished doctoral dissertation. McGill University.
- de Leeuw, J. R. (2014). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior Research Methods. Advance online publication. doi:10.3758/s13428-014-0458-y
- Drachen, A., Yancey, M., Maguire, J., Chu, D., Yuhui Wang, I., Mahlmann, T., Schubert, M. & Klabajan, D. (2014) Skill-Based Differences in Spatio-Temporal Team Behaviour in Defence of The Ancients 2 (DotA 2) (In proceeding) Proceedings of the IEEE Games, Entertainment, and Media (GEM) Conference 2014.
- Ducheneaut, N., Dmitri, W., Nickell, E., Xiog, L., Yee, N. & Zhang, Y. (2006). From tree house to barracks: The social life of guilds in World of Warcraft. Games and culture 1(4), 338-361
- Dye, M.W.G., Green, C.S., & Bavelier, D. (2009). The development of attention skills in action video game players. Neuropsychologia, 47(8-9), 1780-1789. doi:10.1016/j.neuropsychologia.2009.02.002.
- Eichstaedt, J. (2001). An inaccurate-timing filter for reaction-time measurement by JAVA-applets imple-menting internet-based experiments. Behavior Research Methods, Instruments, and Computers, 33(1), 179–186.
- El-Shagi, M. & von Schweinitz, G. (2014). The Diablo 3 Economy: An Agent Based Approach [PDF document]. Retrieved from: http://econ.henu.edu.cn/upload/20141016/14134680789594.pdf
- Entertainment Software Association (ESA) (2011). The 2011 Essential Facts About the Computer and Video Game Industry. Retrieved 20.12.2011 from http://www.theesa.com/facts/pdfs/ESA_EF_2011.pdf.
- Eriksen, B. A., & Eriksen, C. W. (1974). Effects of noise leters upon the identification of a target letter in a nonsearch task. Perception & Psychophysics, 16, 143-149.
- Feng, J., Spence, I. & Pratt, J. (2007). Playing an action video game reduces gender differences in spatial cognition. Psychological science, 18(10), 850-855.
- Ferrari, S. (2013) From Generative to Conventional Play: MOBA and League of Legends, Proceedings of DiGRA 2013: DeFragging Game Studies, Volume 1 (1), 1-17.
- Fletcher, R. B. & Whitton, S. M. (2014). The Group Environment Questionnaire: A Multilevel Confirmatory Factor Analysis. Small Group Research, 45(1), 68-88. doi: 10.1177/1046496413511121
- Green C. S., Bavelier D. (2007). Action video game experience alters the spatial resolution of attention. Psychological Science, 18(1), 88-94.
- Green, C. S., & Bavelier, D. (2003). Action video-game modifies visual selective attention. Nature, 423, 534-537.
- Guha, A., Jereen, A., DeGutis, J. & Wilmer, J. (2014). No action video game training effects for multiple object tracking or mental rotation. Journal of Vision, 22(14).
- Hart, S. (2006). Nasa-Task Load Index (Nasa-TLX); 20 Years Later. Human Factors and Ergonomics Society Annual Meeting Proceedings, 50, 904-908.
- Hart, S. G. & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Human Mental Workload (P. A. Hancock and N. Meshkati (Eds.)), 139-183. North-Holland: Elsevier Science.
- Heslin, R. (1964). Predicting group task effectiveness from member characteristics. Psychological Bulletin, 62, 248-256.
- Hollenbeck, J.R., Moon, H., Ellis, A.P., West, B.J., Ilgen, D.R., Sheppard, L., Porter, C.O., & Wagner J.A. (2002). Structural contingency theory and individual differences: Examination of external and internal person-team fit. Journal of Applied Psychology, 87, 599-606.
- Karle J. W., Watter S., Shedden J. M. (2010). Task switching in video game players: benefits of selective attention but not resistance to proactive interference. Acta Psychologica (Amst.) 134(1).
- LePine, J.A., Hollenbeck, J.R., Ilgen, D. R., & Hedlund, J. (1997). Effects of individual differences on the performance of hierarchical decision-making teams: Much more than g. Journal of Applied Psychology, 82, 803-811.
- Maillot, P., Perrot, A. & Hartley, A. (2012). Effects of interactive physical-activity video-game training on physical and cognitive function in older adults. Psychology and Aging, 27(3), 589-600.
- Mueller, S. T., & Piper, B. J. (2014). The Psychology Experiment Building Language (PEBL) and PEBL Test Battery. Journal of neuroscience methods (222), 250–259.
- N. Pobiedina, J. Neidhardt, M. d. C. Calatrava Moreno & H. Werthner (2013). Ranking factors of team success in Proc. of the 22nd international conference on World Wide Web companion. International World Wide Web conferences Steering Committee, 2013, pp. 1185-1194.
- Narishm. (2013). A map of the Summoner's Rift map on League of Legends with an overlay dividing it into easy to remember regions. Retrieved 29.11.2014 from http://narishm.deviantart.com/art/League-of-Legends-Summoner-s-Rift-Regions-Map-413345149
- Okagaki, L., & Frensch, P. A. (1994). Effects of video game playing on measures of spatial performance: Gender effects in late adolescence. Journal of Applied Developmental Psychology, 15(1), 33-58.
- Owen, A. M. & Hampshire, A. (n.d.). Spatial span ladder [software for online web-based testing]. Cambridge Brain Sciences Inc, University of Western Ontario, Canada. Retrieved from http://www.cambridgebrainsciences.com/browse/memory/test/spatial-span-ladder.
- Perlroth, N. (2013). Department of Homeland Security: Disable Java ‘Unless It Is Absolutely Necessary’. Bits – the business of technology, a New York Times blog. Retrived from: http://bits.blogs.nytimes.com/2013/01/14/department-of-homeland-security-disable-java-unless-it-is-absolutely-necessary/
- Pöllänen, T. I. (2014). Test battery of JavaScript cognitive tasks written using jsPsych for my master's thesis. Github repository. Retrieved from https://github.com/tuuleh/masters-battery.
- Revelle, W. (2014) psych: Procedures for Personality and Psychological Research, Northwestern University, Evanston, Illinois, USA, http://CRAN.R-project.org/package=psych Version = 1.4.8.
- Riot Games. (2014). “Our Games”. Retrieved from: http://www.riotgames.com/our-games
- Salas, E., Dickinson, T. L., Converse, S.A. & Tannenbaum, S. I. (1992). Toward an understanding of team performance and training. In R.W. Swezey & E. Salas (Eds.), Teams: Their training and performance (pp. 3-20). Norwood, NJ: Ablex.
- Shalice, T. (1982). Specific impairments of planning. Philosophical Transactions of the Royal Society B: Biological Sciences, 298, 199-209.
- Stins, J. F., Polderman, T. J. C., Boomsma, D. I., & de Geus, E. J. C. (2007). Conditional accuracy in response interference tasks: Evidence from the Eriksen flanker task and the spatial conflict task. Advances in Cognitive Psychology 3 (3), 389–396.
- Venables, W. N. & Ripley, B. D. (2002) Modern Applied Statistics with S. Fourth Edition. Springer, New York. ISBN 0-387-95457-0
- W3Techs (2014). Usage of client-side programming languages for websites. Retrieved 2.12.2014 from http://w3techs.com/technologies/overview/client_side_language/all
Let's talk!
The influence of cognitive skills and team cohesion on player performance in Multiplayer Online Battle Arena Tuuli Pöllänen, 15.12.2014