Outline

• Review of Convolution Network
• Transferred/Compact Filters
  • Compact Filters
    • Bottleneck Skill
    • Group Convolution
    • Depthwise Separable Convolution and 1x1 Convolution 
    • Dilated Convolution
  • Transferred Filters
    • Group Equivariant
    • Ghost Module
• Classical Models
  • SqueezeNet
  • MobileNet
  • ShuffleNet
  • ShiftNet

Convolution Network Save Parameters

Dense Network

Conv Network

Complexity of Convolution Network

Feature Map

MxNxC

A Filter

KxKxC (+1:bias)

Filter 1

A New Channel

(M/s)x(N/s)x1

Complexity of Convolution Network

Feature Map

MxNxC

A Filter

KxKxC (+1:bias)

Filter 1

Filter 2, 3, ..., D-1

Filter D

New Feature Map

(M/s)x(N/s)xD

shift s pixel

Using cheaper mehtods to get information.

Saving parameter numbers or computation cost.

https://www.facebook.com/Lowcostcosplay/

Compact Filters

• Bottleneck Skill
• Group Convolution
• Depthwise Separable Convolution 
• Dilated Convolution

Generate feature maps with cheaper method.

Bottleneck Skill

(NxN) : Original

(NxM+MxN = 2xNxM) : Bottleneck

2N/M : Ratio (Less than 1 if M < 0.5N)

Group Convolution

(Input Channels, Output Channles, Kerenl Size)
(C, N, (K x K))

Parameters:

K x K x C x N : Traditional Convolution

K x K x (C/M) x (N/M) x M : M-Groups Group Convolution (=KxKxCxN/M)

1/M : Ratio

https://www.cnblogs.com/shine-lee/p/10243114.html

Depthwise Seperable Convolution and 1x1 Convolution

https://medium.com/@zurister/depth-wise-convolution-and-depth-wise-separable-convolution-37346565d4ec

(Input Channels, Output Channles, Kerenl Size):
(C, C, (K x K))

Parameters:

Traditional Convolution

     K x K x C x N
============================

Depthwise Seperable Convolution
(C-Groups Group Convolution)
     K x K x (C/C) x (C/C) x C

1 x 1 Convolution

     1 x 1 x C x N
============================

Ratio (D.S.C. cost ratio+ 1x1 C. cost ratio)

     1/N + 1/(K x K)

Dilated Convolution

Use dilate method to skip some nodes. If we do this, we can get greater respective fields compare to traditinal convolution.
And each nodes' contribute times are equal.

Oord, Aaron van den, et al. "Wavenet: A generative model for raw audio." arXiv preprint arXiv:1609.03499 (2016).

Convolution

Dilated

Convolution

Transferred Filters

Transfer the existed feature map/filters to get new feature map/filters.

• Group Equivariant
• Ghost Module

Group Equivariant

Apply rotate, flip on filters to get more feature map.

Ghost Module(2019 Nov)

Ghost Module(2019 Nov)

Use a very cheap transform to generate a new feature map base on old features.

• SqueezeNet
• MobileNet
• ShuffleNet
• ShiftNet

SqueezeNet(2016)

Fire Module

Take an example

input channels : 128

output channels : 128

S_1x1 : 16 ((1x1x128+1)x16)
E_1x1 : 64 ((1x1x16+1)x64)
E_3x3 : 64 ((3x3x16+1)x64)
 

12432 (Block's params)

Traditional Conv

147584 (((3x3x128+1)x128))

Ratio : 0.084

MobileNet(2017)

Take an example

input channels : 128

output channels : 128

3x3 DWConv : (3x3+1)x128
1x1 Conv : (1x1x128+1)x128

Block's Params
1280+16512=17792

Traditional Conv
147584 (((3x3x128+1)x128))
Ratio : 0.12

ShuffleNet(2017)

ShuffleNet(2017)

Take an example

input channels : 240

output channels : 240

Bottleneck channels : 240/4=80

Groups : 2

1x1 GConv : (240+1)x80/2
3x3 DWConv : (3x3+1)x80
1x1 GConv : (80+1)x240/2

Block's Params
9640+800+9720=20160

Traditional Conv
518640 ((9x240+1)x240)
Ratio : 0.04

ShiftNet(2017)

Different from use K x K kernel to increse respective fields, this work shift feature maps to different direction to make differnt location's features can communicate to each other.