Neuromorphic Algorithms and Hardware for Event-based Processing
Gregor Lenz
Jury
Alejandro Linares-Barranco, reviewer
Bruno Gas, examiner
Sylvain Saïghi, reviewer
Sio-Hoi Ieng, advisor
July 6th 2021
Machine learning is ubiquitous
Speech recognition
Computer vision
Recommender systems
image from openai.com/blog/ai-and-compute
Current demand for computation scales at an unprecedented rate
GPT-3:
- 175bn parameters
- 355 GPU-years
- $4.6m cost
- 1 GWh of energy for training
Can we learn from the brain?
- At 10^15 connections and 100bn neurons, computes extremely efficiently (20 W)
- Asynchronous communication
- Neuromorphic engineering copies basic components such as artificial neurons and spike-based communication
Components of a neuromorphic vision system
Sensor
Algorithm
Hardware
Different hardware backends
Neuromorphic hardware
Conventional low-power hardware
Conventional hardware
1. Event-based Cameras and Algorithms
Event-based face detection
Lenz et al., Frontiers of Neuroscience 2020
The dynamics of eye blinks
The dynamics of eye blinks
Lenz et al., Frontiers of Neuroscience 2020
- tracking with μs precision and in difficult lighting situations
- lower power than gold standard methods
- robust to multiple faces and partial occlusions
Lenz et al., Frontiers of Neuroscience 2020
Event-based face detection using the dynamics of eye blinks
Event-based face detection using the dynamics of eye blinks
Event-based face detection using the dynamics of eye blinks
Perspectives on event-based processing
- Event-based cameras enable fine-grained spatio-temporal features
- Event-based algorithms can save power when used in conjunction with event-based cameras
- we need low-power hardware
2. A Mobile Framework for Event-based Computer Vision
Mobile phone Android framework
Mobile phone Android framework
NavGesture Database
Maro et al., Frontiers of Neuroscience 2020
Event-based gesture recognition
Maro et al., ICAG 2019, best demo award
Event-based aperture-robust
optical flow
Akolkar et al., 2020
events
Event-based image reconstruction
Scheerlinck et al., 2020
RNN
Buffer / latency trade-off
Buffer / latency trade-off
Buffer / latency trade-off
Buffer / latency trade-off
Summary of our event-based mobile framework
- Event-based computer vision and mobile devices can benefit from each other
- ARM-based architecture saves power consumption
- Pre-processing is still required on conventional hardware
von Neumann architecture
- von Neumann hardware separates logic from memory
- Applications are increasingly data hungry
- 62.5% of energy consumption is data movement: Boroumand et al., 2018
- Solution: in-memory computing
3. Neural Computation on Neuromorphic
Hardware Using Precise Timing
Dedicated neuromorphic hardware
- Basic unit is the spiking neuron
- Loihi has 128 cores with 1k neurons each
Two examples of neural encoding schemes
-
Rate coding the most dominant encoding scheme today
- Temporal encoding:
Time To First Spike (TTFS)
much more efficient
Spike Time Computation Kernel (STICK)
Lagorce and Benosman, 2016
General-purpose computation building blocks
Memory
Router
Subtractor
Rate vs temporal coding when calculating dynamic systems on Loihi
dynamic 2nd order system
Lorenz attractor
Performance compared to rate-based networks implemented in Nengo
MNIST ANN / SNN conversion
MNIST SNN classification performance
[58] Massa et al., 2020
[36] Rueckauer et al., 2021
MNIST SNN classification performance
Energy delay product on GPU: 222
Conclusions
- Neuromorphic algorithms can reduce power consumption
- New computing principles for low-power devices
- Neuromorphic computing is efficient (not accurate)
Perspectives
- Need to identify the right tasks
- Ideally we want the full neuromorphic pipeline
Contributions
Journals
Lenz, Ieng and Benosman, High Speed Event-based Face Detection and Tracking Using the Dynamics of Eye Blinks, Frontiers of Neurscience 2020.
Lenz, Oubari, Orchard and Ieng, Neural Computation Using Precise Timing on Loihi, submitted 2021.
Oubari, Exarchakis, Lenz, Benosman and Ieng, Efficient Spatio-temporal Feature Clustering for Large Event-based Datasets, submitted 2021.
Conferences
Maro, Lenz, Reeves and Benosman, Event-based Visual Gesture Recognition with Background Suppression running on a smart-phone, 14th ICAG 2019.
Haessig, Lesta, Lenz, Benosman and Dudek, A Mixed-Signal Spatio-Temporal Signal Classifier for On-Sensor Spike Sorting, ISCAS 2020.
Open Source Software
Frog: An Android framework for event-based vision.
Loris: Python library to handle files from neuromorphic cameras.
Tonic: Event-based datasets and transformations based on PyTorch.
Quartz: ANN to SNN conversion using temporal coding.
Latency measurements
small batch
large batch
Event-based Gesture Recognition
Frame Reconstruction From Events
thesis defence
By Gregor Lenz
thesis defence
- 168