X  - frame

An attempt on frame interpolation with GAN

\infin

What is frame interpolation?

A brief review on frames

Image from Lecture slides

\alpha
1-\alpha

Generate in-between frames

Synthetic video dataset

We have this claw and ball...

Sample Frames

\alpha = 0.7
\alpha = 0.6
\alpha = 0.2
\alpha = 0.9

WGAN-GP with U-Net

WGAN-GP with U-Net

Results

Start_Frame            GT           Predicted      End_Frame

Forced learning?

Generator Pyramid Loss

...

s

real

...

s

fake

Discriminator

py_gen

Results

Start_Frame            GT           Predicted      End_Frame

Discriminator Pyramid Loss

...

s

B-frame

...

s

A-frame

Discriminator

α

1-α

...

s

M-frame

py_dis

Results

Start_Frame            GT           Predicted      End_Frame

The real stuff

movie <Zathura>

Sample Frames

\alpha = 0.94
\alpha = 0.40
\alpha = 0.55

WGAN-GP with U-Net

WGAN-GP with U-Net

Results

Start_Frame            GT              Predicted      End_Frame

Conclusion

  • Forced learning may be helpful at times
  • Phase interpolation is learnable by a simple GAN
  • Balanced WGAN loss + L2 loss

Inf-Frames

By james01

Inf-Frames

frame interpolation with pyramid structured GAN

  • 495