I built a robot to cheat at Pokémon!!

Sophie Déziel

Rails dev @ Acquisio

Twitch Streamer

@sophiedeziel

Guiz de Pessemier

Co-founder of Outerminds

Youtuber and Twitch Streamer

@jeuserieux

TEAM ROCKET

Automation to the rescue!!

T. eam

R. ocket

A. utomated

S. hiny

H. unter

Two challenges

  • Controlling the game
     
  • Have input to make decisions

Control the game options

  • Open an modify a Joy Con
     
  • Linear actuators
     
  • Servo-motors

Servo-motors

  • Cheap and easy to find
     
  • You can position them precisely
     
  • Easy to control
     
  • Slow

Image input options

  • HDMI capture card
     
  • Color sensor
     
  • Webcam

Webcam

  • Easy to position close to the screen
     
  • Lots of Open-Source Software to control them
     
  • Plug and Play (most of the time)

We can go forward

We need to make it:

  • Reliable
     
  • Precise
     
  • Easy to code
     
  • Configurable

Make it reliable

  • Work for several hours unsupervised
     
  • Perfect timing
     
  • Must not move

Precise with "Computer vision"

  • Easy to setup
     
  • Grab a screenshot on demand
     
  • Isolate a single pixel
     
  • Check hue inclusion in a range
     
  • False positives vs false negatives

Easy to code and test

  • Use Arduino + Firmata
     
  • Script can be in any general purpose programming language
     
  • ImageMagick to pick a single pixel's hue
loop do
  press(:a, @alolan[:delay_1])
  press(:a, @alolan[:delay_2])
  press(:a, @alolan[:delay_3])
  press(:a, @alolan[:delay_4])
  press(:a, @alolan[:delay_5])
  press(:a, @alolan[:delay_6])
  press(:a, @alolan[:delay_7])
  press(:a, @alolan[:delay_8])

  fire('shiny') if shiny?

  light(:not_shiny, true)
  sleep @alolan[:delay_9]

  press(:a, @alolan[:delay_10])
  press(:a, @alolan[:delay_11])
  press(:a, @alolan[:delay_12])
  light(:not_shiny, false)


  fire('reset')
end

Configurable

  • Control remotely
     
  • Edit timing, connections and color ranges
     
  • Visualise the webcam

Putting it all together

The working prototype!

Ooops!

Unplanned features

  • Controls improvements
     
  • Easy updates
     
  • Communication features (SMS and Tweets)

And now?

Want to build it?

So cool!

github.com/sophiedeziel/robo-catcher

You can modify it:

  • Add Lights and lasers
     
  • Remove the Arduino and use the RP GPIO
     
  • Improve the interface
     
  • Adapt the codebase for another game

Thank you so much

Question?

OS Hardware

  • Raspberry Pi
     
  • Arduino

OSS

  • Raspbian
  • Motion
  • Firmata
  • Ruby
  • Rails
  • RMagick and Image Magick
  • Nginx
  • Git

Copy of I built a robot to cheat at Pokémon!!

By Sophie Déziel

Made with Slides.com

Copy of I built a robot to cheat at Pokémon!!

!!Con version

  • 1,449

More from Sophie Déziel