Machine Learning and drones and stuff
Tyler Graf
Senior Web Dev
FamilySearch
How does a neural network work?
Hello World
Recognizing written digits
28px
28px
Imagine writing a program to detect the number
1.0
0.6
0.2
Activation
784
0
1
2
3
4
5
6
7
8
9
1.0
1.0
0.6
0.2
const node = (nodes, bias) =>
nodes
.map(({ activation, weight }) => activation * weight)
.reduce((aggregator, current) => aggregator + current, 0) + biasEach Node is a function
1.0
0.2
0.8
784
0
1
2
3
4
5
6
7
8
9
1.0
0.8
0.7
0.8
0.8
0.8
0.8
0.6
0.5
0.4
0.1
0.1
0.2
0.7
0.2
0.3
0.2
0.1
0.1
Step 1: Try Stuff
Training
Hey Neural Network, what's this?
Uhh....2? No, 7! No, 1!
784
0
1
2
3
4
5
6
7
8
9
1.0
0.8
0.7
0.8
0.8
0.8
0.8
0.6
0.5
0.9
0.1
0.6
0.2
0.7
0.2
0.3
0.5
0.1
0.1
4
0
1
2
3
4
5
6
7
8
9
0.9
0.1
0.6
0.2
0.4
0.2
0.3
0.5
0.1
0.1
(0.4 - 1)
2
(0.2 - 0)
2
(0.3 - 0)
2
(0.5 - 0)
2
(0.1 - 0)
2
(0.1 - 0)
2
(0.2 - 0)
2
(0.6 - 0)
2
(0.9 - 0)
2
(0.1 - 0)
2
+
+
+
+
+
+
+
+
+
Cost
2.03
Average Cost for all 10,000
Cost: 3.05
0
1
2
3
4
5
6
7
8
9
0.8
0.8
0.6
0.5
0.9
0.1
0.6
0.2
0.7
0.2
0.3
0.5
0.1
0.1
Backpropogation
Local Minimum
Global
Minimum
Goal
Control a drone by moving my hand in front of my laptop
Step 1: Control a Drone
© 2012
Parrot Bebop 2
npm i ar-drone
npm i drone-node
npm i parrot-drone-node
npm i node-bebop
const bebop = require('node-bebop')
const drone = bebop.createClient()
drone.connect(()=>{
drone.takeoff()
drone.forward(50)
drone.back(50)
drone.land()
})Drone API
const bebop = require('node-bebop')
const io = require("socket.io")(http);
const drone = bebop.createClient()
drone.connect(() => {
io.on("connection", function(socket) {
socket.on("command", ({ command, speed }) => {
drone[command](speed);
});
});
});
http.listen(3000, function() {
console.log("listening on *:3000");
});Socket IO Server
document.addEventListener("keydown", ({ key, shiftKey }) => {
switch (key) {
case "u":
command("up", 50);
break;
case "d":
command("down", 50);
break;
case "e":
command("counterClockwise", 100);
break;
case "r":
command("clockwise", 100);
break;
case "ArrowUp":
command("forward", 50);
break;
case "ArrowDown":
command("backward", 50);
break;
case "ArrowLeft":
command("left", 50);
break;
case "ArrowRight":
command("right", 50);
break;
default:
}
});Socket IO Client
Demo
Step 2: Machine Learn Myself
The YouTubes
The Twitters
The Githubs
Step 3: Hand Signals
First Try
First Try Results
Second Try Results
Fourth Try
> 1000 images
Fifth Try
Ninth Try
Step 4: Labelling
Record myself with QuickTime
Child in tub
👖ON!👍🏻
Use ffmpeg to capture every 10th frame
ffmpeg -i videos/flat.mov -vf "select=not(mod(n\,10))" -vsync vfr -q:v 2 done/img_%03d.jpg
Label Images
Step 4: Training
IBM
- Create IBM Account
- Create some service account things
- Get some key
- Solve for y
- npm i cloud-annotations
Run the training
$ cacli train
Takes about 20 min for 250 photos
(Some amount of training time per month is free)
35 min for 1000 photos
Step 5: Control Drone
