10x/year

Source: OpenAI Blog

Moore's law

There is a huge demand gap

Solution

Domain-Specific Accelerators and Efficient DNN operators

Systolic Arrays in TPUs

25-29x more performant than GPUs 

Domain-Specific Hardware Accelerators

Efficient DNN Operator

Depthwise Separable Convolutions
Computationally Efficient and ubiquitous

Source: Once For All

Comparing MobileNet-V2 with ResNet-50

> What can we do to make DW convolution     faster on systolic arrays?

> How good is the inference                                 performance of our proposed solution?

Incommensurate

Scaling

> Why are depthwise convolutions inefficient on systolic arrays?

Formal analysis using Systolic-Algorithms

FuSeConv: Fully Separable 1D Convolutions, our hardware/software co-design solution

FuSeConv is 3x-7x more faster on 64x64 Systolic Arrays with minimum overhead

> What can we do to make DW convolution faster on systolic arrays?

> How good is the inference performance of our proposed solution?

Standard Convolution

FLOPS = N x M x K x K x C\( _{in}\) x C\( _{out}\)

FLOPS = N x M x C\( _{in}\) x K x K + N x M x C\( _{in}\) x C\( _{out}\)

> A class of algorithms that runs efficiently on systolic architectures

> Computational loops are transformed into Regular Iterative                 Algorithm (RIA)

> RIAs that have constant offsets only can be synthesized on systolic     arrays

> i, j indices maps to Spatial dimensions      of array

> k index maps to Temporal dimension

Time

Dim1

> Non-Constant Offset indices in RIA

> 2D Convolution is not a Systolic Algorithm

After im2col transformation

Under Utilization

> More filters -> Data reuse -> High Utilization

> Extendable for convolution with channels too

2D Convolution with Multiple Filters

> Equivalent to C\( _{in}\) 2D single channel Convolutions

> No Filter Reuse availble in Depthwise Convolution

Each channel-wise 2D convolution is sequential!!

Fully Separable Convolution (FuSeConv) : Composes of 1D depthwise convolutions and pointwise convolution

Depthwise Separable Convolution

Full Variant (D = 1)

Half Variant (D = 2)

> FuSeConv composes of only 1D convolutions.

> 1 x K & K x 1depthwise convolutions

> 1 x 1 Pointwise Convolution

Matrix Multiplication

1D Convolutions

> Constant Offset => Systolic Algorithm

On 32 x 32 Systolics

Channel 0

Channel 1

> Channelwise 1D convolutions can be computed in parallel

> FuSeConv + our dataflow mapping exploits parallelism which were absent     in depthwise convolutions mapping natively on Systolic Arrays

> Evaluate 4 FuSe variants and compare with baseline

> Analytical latency model based on SCALE-SIM

> MobileNets (V1, V2, V3-Small, V3-Large) and MnasNet-B1 

Choose half of layers greedily to maximize speedup of layer

>  Full (D = 1) FuSe

>  Half (D = 2) FuSe

>  50% Full FuSe

>  50% Half FuSe

> Latency = Load Values + Compute +  Communicate PSums + Flush Outputs

All layers are replaced with FuSeConv

Average drop ~1% in Half Variants, drop ~0.3% in Full Variants

Inference Latency and Speedup on 64 x 64 Systolic Array

Scaling up of Speedup of Full FuSe variant wrt baseline

Multiple Trade-Offs solutions

Newer networks scales differently

Operator-wise Latency Distribution of Full FuSe Variants

More Speedup from Initial layers

Inference latency dominated by Depthwise -> Point-wise

> Efficient Inference is still a crucial problem for DNN deployment. Currently           proposed solutions are Hardware Accelerators and Efficient DNN Operator

> However, Depthwise Separable Convolutions are inefficient on Systolic Arrays     and lacks data reuse to exploit parallelism

> We propose FuSeConv (Fully Separable 1D Convolutions) as a drop-in                     replacement for Depthwise Separable Convolutions

> Our Co-Design solution is atleast 4X faster than efficient mobile networks such as       MobileNets and MnasNet with negligible accuracy-drop

> Motivates hardware-aware Neural Operator Search (NOS)

> We also proposed a 1D systolic dataflow for efficiently executing FuSeConv layers