I built a robot to cheat at Pokémon!!

Sophie Déziel

Rails dev @ Acquisio

Twitch Streamer

@sophiedeziel

Guiz de Pessemier

Co-founder of Outerminds

Youtuber and Twitch Streamer

@jeuserieux

TEAM ROCKET

Automation to the rescue!!

T. eam

R. ocket

A. utomated

S. hiny

H. unter

Two challenges

Controlling the game
Have input to make decisions

Control the game options

Open an modify a Joy Con
Linear actuators
Servo-motors

Servo-motors

Cheap and easy to find
You can position them precisely
Easy to control
Slow

Image input options

HDMI capture card
Color sensor
Webcam

Webcam

Easy to position close to the screen
Lots of Open-Source Software to control them
Plug and Play (most of the time)

We can go forward

We need to make it:

Reliable
Precise
Easy to code
Configurable

Make it reliable

Work for several hours unsupervised
Perfect timing
Must not move

Precise with "Computer vision"

Easy to setup
Grab a screenshot on demand
Isolate a single pixel
Check hue inclusion in a range
False positives vs false negatives

Easy to code and test

Use Arduino + Firmata
Script can be in any general purpose programming language
ImageMagick to pick a single pixel's hue

loop do
  press(:a, @alolan[:delay_1])
  press(:a, @alolan[:delay_2])
  press(:a, @alolan[:delay_3])
  press(:a, @alolan[:delay_4])
  press(:a, @alolan[:delay_5])
  press(:a, @alolan[:delay_6])
  press(:a, @alolan[:delay_7])
  press(:a, @alolan[:delay_8])

  fire('shiny') if shiny?

  light(:not_shiny, true)
  sleep @alolan[:delay_9]

  press(:a, @alolan[:delay_10])
  press(:a, @alolan[:delay_11])
  press(:a, @alolan[:delay_12])
  light(:not_shiny, false)


  fire('reset')
end