interview

Neuromorphic Systems for Mobile Computing

Gregor Lenz

My background

Master's degree in Biomedical Engineering in Vienna, Austria
2.5 years working in a tech start-up, IT consultancy, Imperial College London, Prophesee
About to finish PhD in neuromorphic engineering in Paris, France

Neuromorphic System

1. Sensor

2. Algorithm

3. Hardware

Goal: low power system for mobile devices

Advantage: temporal and spatial sparsity

1. Connecting a Neuromorphic Camera to a Mobile Device

Event-based Visual Recognition on Mobile Phone

Motivation: assist elderly and visually impaired people with hand gestures and voice commands
Prototype that connects a small
event camera
Event-based always on
processing

integrates different modalities
(gestures / speech)
displays event camera output
in real time
detects 6 different gestures: 4
directions, home, select

Event-based Visual Recognition on Mobile Phone

Maro, Lenz, Reeves and Benosman, Event-based Visual Gesture Recognition with Background Suppression running on a smart-phone, 14th ICAG 2019. Best demo award.

Event-based Visual Recognition on Mobile Phone

Maro, Lenz, Reeves and Benosman, Event-based Visual Gesture Recognition with Background Suppression running on a smart-phone, 14th ICAG 2019. Best demo award.

Event-based Visual Recognition on Mobile Phone

Maro, Lenz, Reeves and Benosman, Event-based Visual Gesture Recognition with Background Suppression running on a smart-phone, 14th ICAG 2019. Best demo award.

Extended Mobile Phone Android framework

Extended framework for other visual tasks: event-based flow, image reconstruction
variable frame rates save power when
no new input
grey-level images allow downstream
classical computer vision pipelines

2. Efficient neuromorphic algorithm for camera data

Event-based Face Detection Using the Dynamics of Eye Blinks

Lenz, Ieng and Benosman, Event-based Face Detection and Tracking Using the Dynamics of Eye Blinks, Frontiers of Neuroscience 2020.

tracking with μs precision and in difficult light situations
lower power than gold standard methods
robust to multiple faces and partial occlusions

Event-based Face Detection Using the Dynamics of Eye Blinks

Lenz, Ieng and Benosman, Event-based Face Detection and Tracking Using the Dynamics of Eye Blinks, Frontiers of Neuroscience 2020.

Event-based Face Detection Using the Dynamics of Eye Blinks

Lenz, Ieng and Benosman, Event-based Face Detection and Tracking Using the Dynamics of Eye Blinks, Frontiers of Neuroscience 2020.

Event-based Face Detection Using the Dynamics of Eye Blinks

3. Neural computation on neuromorphic hardware

Neural Encoding Schemes

Many spiking neural
networks use rate coding
Temporal encoding exists:
Time To First Spike (TTFS)
fairly inaccurate
We use alternative
encoding scheme based
on inter spike intervals (ISI)

Spike Time Computation Kernel (STICK)

Values are encoded in Inter Spike Intervals
4 different synapses provide 3 different current accumulation methods

Lagorce & Benosman, 2015

Spike Time Computation Kernel (STICK)

Mathematical operations are cast into handcrafted spiking neural networks
Networks for value storage, linear, nonlinear and differential computation

Lagorce & Benosman, 2015

Logarithm network

Neural Computation on Loihi

Composable networks for general purpose computing using artificial neurons

Neural Computation on Loihi

More efficient when calculating dynamic systems in comparison to population-coded framework on same hardware

Neural Computation on Loihi

Conversion of networks trained on GPUs for efficient inference on Loihi

Neural Computation on Loihi

Our method uses one spike per neuron at similar classification accuracy, which is significantly less than any rate-coded method

Conclusions

Event cameras are suitable for sporadic, sparse signals, but need tight integration
Mobile devices can already benefit from low-power event-by-event approaches or even variable frame rates, but could do so even more using spiking neural networks
Temporal coding on neuromorphic hardware looks promising, opens up possibility for spiking computer
ANN/SNN conversion using temporal coding has low EDP for low batch size

Why am I the right candidate?

about to complete PhD in neuromorphic enginering
familiar with taking orthogonal approaches due to the nature of events
worked on Loihi for the past 1.5 years
authored several software packages
like writing software that is easy to use

Topics to explore

Temporal augmentation: can we make algorithms more robust by changing timings, maybe even learn representations in an unsupervised manner
Online learning: continual model updates
biologically plausible learning rules

picture adapted from Lillicrap et al. 2020

Conclusion

Event cameras are suitable for sporadic, sparse signals, but need tight integration
Mobile devices can benefit from low-power event-by-event approaches or even variable frame rates
ideally makes use of spiking hardware