or
Let’s build a real-time macaron detection robot!
@AdrienneTacke
👋🏻 All Things Open!
Macar n
Macar n
???
What is a Robot?
The Idea
Let's build it!
Moar actuation!
AGENDA
SENSORS, CONTROLLERS, ACTUATION
THIS SOUNDS FAMILIAR...
WHAT DOES IT DO?
SIMPLIFIED ROBOT ARCHITECTURE
WHAT WILL WE USE?
CAN WE ADD AN ARM?!
TEST THE ARM
1
2
3
4
VIAM MACHINE
CUSTOM ML MODEL
SET UP SENSORS
PUT IT ALL TOGETHER
What is a robot?
Sensors
Controller
Actuators
Sensors
Controller
Actuators
Input
Processing
Output
Input
Processing
Output
Input
How the system gets information
Input
How the system gets information
Sensors
How a robot gets information
SOFTWARE
ROBOTICS
Sensors
How a robot gets information
Ultrasonic sensor
PROXIMITY/DISTANCE
Inertial Measurement Unit
ORIENTATION AND MOVEMENT
Camera
IMAGE DATA (JPEG, VIDEO STREAM)
Processing
The logic that decides what to do
Processing
The logic that decides what to do
Controller
The robot's decision-making logic
SOFTWARE
ROBOTICS
Controller
The robot's decision-making logic
Raspberry Pi
SINGLE BOARD COMPUTER
Output
The actions a system takes
Output
The actions a system takes
Actuators
How a robot affects its environment
SOFTWARE
ROBOTICS
Actuators
How a robot affects its environment
Wheel (Motor)
MOVEMENT
Servo
PRECISE ANGULAR MOVEMENT
Arm
PRECISE MOVEMENT/MANIPULATION
Gripper
MANIPULATION
Actuators
Sensors
Hardware
Driver
Middleware/
Platform
MOOS (Mission Oriented Operating Suite)
OROCOS (Open Robot Control Software)
YARP (Yet Another Robot Platform)
low-level software that communicates with specific hardware components
Application
your custom code; task logic; decision-making logic
ARCHITECTURAL LAYERS OF A ROBOT
The Idea
The Idea
Use computer-vision to determine whether something is a macaron or a macaroon
?
The Idea
Use computer-vision to determine whether something is a macaron or a macaroon
?
?
What will we use?
?
What will we use?
Raspberry Pi
USB Webcam
?
What will we use?
Raspberry Pi
USB Webcam
Dataset of labeled macarons/macaroons
What will we use?
Raspberry Pi
USB Webcam
Dataset of labeled macarons/macaroons
LiteRT custom model
It's a macaroon!
It's a macaron!
What will we use?
Raspberry Pi
USB Webcam
Dataset of labeled macarons/macaroons
LiteRT custom model
It's a macaroon!
It's a macaron!
Let's build it!
But, what if we want to use the arm?
The Idea
Use computer-vision to determine whether something is a macaron or a macaroon
?
New
The Idea
Use computer-vision to determine whether something is a macaron or a macaroon
?
New
,then have a
robotic arm tell us by waving a labeled flag!
MACAROON
MACARON
Iteration!
Visual
Proximity Alert
Use an RGB LED to indicate safe and dangerous distances and an ultrasonic sensor to determine those distances.
Sense
Control
Act
Visual Proximity Alert
Ultrasonic sensor
distance >= threshold ? setLed(green)
distance <= threshold ? setLed(red)
RGB LED
CV-powered Beverage Detector
Create a custom model trained on your favorite drinks and combine with a vision service. Create a Typescript app as a "kiosk".
Sense
Control
Act
Webcam
Custom Beverage Model
confidence >= threshold ?
setBoundBox(probableDrink)
Display
CV-powered Beverage Detector
1
2
3
Robotics isn't magic (nor are they only fancy humanoids), it's sensors > controllers > actuation
Hardware interaction can be abstracted away by the right platform.
(Also, robotics development doesn't have to be the wild wild west)
Your experience in software is directly applicable to robotics
TL;DR
Your next great project
What will YOU build?
CV-powered Beverage Detector
Visual Proximity Alert
Build it yourself!
Step-by-step tutorials and workshops
codelabs.viam.com
Thank you !
Stop by the booth (#3) for swag and an interactive robotic arm demo!
Liked this talk? Tell me! @AdrienneBraganzaTacke on LinkedIn and tag @ViamRobotics!
(@abt.bsky.social on Bluesky!)
