X  - frame

An attempt on frame interpolation with GAN

\infin
\infin

What is frame interpolation?

A brief review on frames

Image from Lecture slides

\alpha
α\alpha
1-\alpha
1α1-\alpha

Generate in-between frames

Synthetic video dataset

We have this claw and ball...

Sample Frames

\alpha = 0.7
α=0.7\alpha = 0.7
\alpha = 0.6
α=0.6\alpha = 0.6
\alpha = 0.2
α=0.2\alpha = 0.2
\alpha = 0.9
α=0.9\alpha = 0.9

WGAN-GP with U-Net

WGAN-GP with U-Net

Results

Start_Frame            GT           Predicted      End_Frame

Forced learning?

Generator Pyramid Loss

...

s

real

...

s

fake

Discriminator

py_gen

Results

Start_Frame            GT           Predicted      End_Frame

Discriminator Pyramid Loss

...

s

B-frame

...

s

A-frame

Discriminator

α

1-α

...

s

M-frame

py_dis

Results

Start_Frame            GT           Predicted      End_Frame

The real stuff

movie <Zathura>

Sample Frames

\alpha = 0.94
α=0.94\alpha = 0.94
\alpha = 0.40
α=0.40\alpha = 0.40
\alpha = 0.55
α=0.55\alpha = 0.55

WGAN-GP with U-Net

WGAN-GP with U-Net

Results

Start_Frame            GT              Predicted      End_Frame

Conclusion

  • Forced learning may be helpful at times
  • Phase interpolation is learnable by a simple GAN
  • Balanced WGAN loss + L2 loss
Made with Slides.com