Neuromorphic Algorithms and Hardware for Event-based Processing

Gregor Lenz

Jury

Alejandro Linares-Barranco, reviewer

Bruno Gas, examiner

Sylvain Saïghi, reviewer

Sio-Hoi Ieng, advisor

July 6th 2021

Machine learning is ubiquitous

Speech recognition

Computer vision

Recommender systems

image from openai.com/blog/ai-and-compute

Current demand for computation scales at an unprecedented rate

GPT-3:

• 175bn parameters
• 355 GPU-years
• $4.6m cost
• 1 GWh of energy for training

• At 10^15 connections and 100bn neurons, computes extremely efficiently (20 W)
• Asynchronous communication
• Neuromorphic engineering copies basic components such as artificial neurons and spike-based communication

Components of a neuromorphic vision system

Sensor

Algorithm

Hardware

Different hardware backends

Neuromorphic hardware

Conventional low-power hardware

Conventional hardware

1. Event-based Cameras and Algorithms

Event-based face detection

Lenz et al., Frontiers of Neuroscience 2020

The dynamics of eye blinks

The dynamics of eye blinks

Lenz et al., Frontiers of Neuroscience 2020

• tracking with μs precision and in difficult lighting situations
   
• lower power than gold standard methods
   
• robust to multiple faces and partial occlusions

Lenz et al., Frontiers of Neuroscience 2020

Event-based face detection using the dynamics of eye blinks

Event-based face detection using the dynamics of eye blinks

Event-based face detection using the dynamics of eye blinks

Perspectives on event-based processing

• Event-based cameras enable fine-grained spatio-temporal features
   
• Event-based algorithms can save power when used in conjunction with event-based cameras
   
• we need low-power hardware

2. A Mobile Framework for Event-based Computer Vision

Mobile phone Android framework

Mobile phone Android framework

NavGesture Database

Maro et al., Frontiers of Neuroscience 2020

Event-based gesture recognition

Maro et al., ICAG 2019, best demo award

Event-based aperture-robust

optical flow

Akolkar et al., 2020

events

Event-based image reconstruction

Scheerlinck et al., 2020

RNN

Buffer / latency trade-off

Summary of our event-based mobile framework

• Event-based computer vision and mobile devices can benefit from each other
   
• ARM-based architecture saves power consumption
   
• Pre-processing is still required on conventional hardware

von Neumann architecture

• von Neumann hardware separates logic from memory
   
• Applications are increasingly data hungry
   
• 62.5% of energy consumption is data movement: Boroumand et al., 2018
   
• Solution: in-memory computing

3. Neural Computation on Neuromorphic
Hardware Using Precise Timing

Dedicated neuromorphic hardware

• Basic unit is the spiking neuron
  
  
  
   
• Loihi has 128 cores with 1k neurons each

Two examples of neural encoding schemes

• Rate coding the most dominant encoding scheme today
   
• Temporal encoding:
  Time To First Spike (TTFS)
  much more efficient

Spike Time Computation Kernel (STICK)

Lagorce and Benosman, 2016

General-purpose computation building blocks

Memory

Router

Subtractor

Rate vs temporal coding when calculating dynamic systems on Loihi

dynamic 2nd order system

Lorenz attractor

Performance compared to rate-based networks implemented in Nengo

MNIST SNN classification performance

[58] Massa et al., 2020

[36] Rueckauer et al., 2021

MNIST SNN classification performance

Energy delay product on GPU: 222

Conclusions

• Neuromorphic algorithms can reduce power consumption
   
• New computing principles for low-power devices
   
• Neuromorphic computing is efficient (not accurate)

Perspectives

• Need to identify the right tasks
   
• Ideally we want the full neuromorphic pipeline

Contributions

Journals

Lenz, Ieng and Benosman, High Speed Event-based Face Detection and Tracking Using the Dynamics of Eye Blinks, Frontiers of Neurscience 2020.

Lenz, Oubari, Orchard and Ieng, Neural Computation Using Precise Timing on Loihi, submitted 2021.

Oubari, Exarchakis, Lenz, Benosman and Ieng, Efficient Spatio-temporal Feature Clustering for Large Event-based Datasets, submitted 2021.

Conferences

Maro, Lenz, Reeves and Benosman, Event-based Visual Gesture Recognition with Background Suppression running on a smart-phone, 14th ICAG 2019.

Haessig, Lesta, Lenz, Benosman and Dudek, A Mixed-Signal Spatio-Temporal Signal Classifier for On-Sensor Spike Sorting, ISCAS 2020.

Open Source Software

Frog: An Android framework for event-based vision.

Loris: Python library to handle files from neuromorphic cameras.

Tonic: Event-based datasets and transformations based on PyTorch.

Quartz: ANN to SNN conversion using temporal coding.

Latency measurements

small batch

large batch

Event-based Gesture Recognition

Frame Reconstruction From Events