Video Super-Resolution: Approaches to modelling and evaluation

Andrey Lukyanenko

CV R&D Lead, MTS AI

Content

• What is Video Super-Resolution

• Approaches to Single Image Super-Resolution

• Approaches to Video Super-Resolution

• Real-Time Video Super-Resolution

• Real-World Video Super-Resolution

• Evaluation: metrics and datasets

About me

• Economist
• ~4 years as ERP-consultant
• ~8 months self-study
• ~5 years as Data Scientist
• Tech Lead in R&D in Computer Vision @ MTS AI
• Kaggle Competition Master, Notebook top-1, Discussion Grandmaster

VSR: definition

• Video super-resolution is the task of upscaling a video from a low-resolution to a high-resolution.
• Multiple solutions for the same LR image

VSR: definition

VSR: depiction

VSR: usage examples

VSR: usage examples

VSR: usage examples

VSR: usage examples

VSR: usage examples

SR, IR, IE

SISR

SISR

SISR

SISR

Nearest-neighbor Interpolation

Bilinear interpolation

Bicubic interpolation

SRCNN

SRCNN

VDSR

FSRCNN

ESPCN

EDSR

Recursive networks

DRCN

LapSRN

SRGAN

ESRGAN

Diffusion models

Diffusion models

SwinIR

SwinIR

VSR

VSR

MEMC

Deep-DE

TecoGAN

Deformable Convolution

Deformable Convolution

EDVR

EDVR

FFCVSR

3DSRnet

3DSRnet

STCN

PFNL

CARN

CARN

FAST

RSISR

RSISR

BSR

RSISR: self-supervised

Datasets

Datasets

Metrics

• Pairwise metrics
• Image quality metrics
• Human evaluation

Pairwise preference

• Choose what is better: Original and SR
• No external distractions
• The same setup
• Check the participants

Conclusions

• Many approaches, no free lunch
• No best metric
• Worse performance in real-world
• Multiple degradations
• Low performance for extreme SR (x16 and higher)
• Requires a lot of VRAM

References

• https://bitmovin.com/super-resolution-machine-learning-p1/

• https://blog.adobe.com/en/publish/2021/03/10/from-the-acr-team-super-resolution#gs.ptzvbh

• https://www.analyticsvidhya.com/blog/2021/05/deep-learning-for-image-super-resolution/

• https://www.ijert.org/video-super-resolution-techniques-a-survey

• https://arxiv.org/abs/2007.12928

• https://github.com/xinntao/BasicSR

• https://github.com/ChaofWang/AWSRN

• https://github.com/idealo/image-super-resolution/

References

• https://github.com/xinntao/Real-ESRGAN

• https://github.com/cszn/BSRGAN

• https://ai.googleblog.com/2021/07/high-fidelity-image-generation-using.html

• https://habr.com/ru/post/439766/

• https://arxiv.org/pdf/1904.07523.pdf

• https://arxiv.org/pdf/1501.00092.pdf

• https://paperswithcode.com/task/super-resolution

• https://blog.paperspace.com/image-super-resolution/

• https://arxiv.org/abs/1803.08664

• https://andlukyane.com/blog/paper-review-rwsrfisc

• https://andlukyane.com/blog/paper-review-swinir

References

• https://arxiv.org/abs/2103.14006
• https://arxiv.org/pdf/2007.12928.pdf
• https://www.analyticsvidhya.com/blog/2021/05/deep-learning-for-image-super-resolution/
• https://github.com/ChaofWang/Awesome-Super-Resolution
• https://github.com/LoSealL/VideoSuperResolution
• https://arxiv.org/pdf/2107.03055.pdf
• https://github.com/Thmen/EGVSR
• https://arxiv.org/ftp/arxiv/papers/2107/2107.05307.pdf
• https://ai.googleblog.com/2021/07/high-fidelity-image-generation-using.html
• https://developer.huawei.com/consumer/en/hiai/engine/image-super-resolution/

References

• https://www.techeblog.com/nvidia-ai-research-maxine-video-calls-conference/
• https://www.theverge.com/2019/12/17/21025811/ai-super-resolution-zoom-enhance-pixelmator-pro
• https://bitmovin.com/wp-content/uploads/2020/05/Super-Resolution.006-1-e1590143447660.jpeg
• https://dmitryulyanov.github.io/deep_image_prior
• http://www.mit.edu/~sze/fast.html
• https://www.imageeprocessing.com/2017/11/nearest-neighbor-interpolation.html
• https://theailearner.com/2018/12/29/image-processing-bilinear-interpolation/
• https://towardsdatascience.com/deformable-convolutions-demystified-2a77498699e8

Contacts

• https://andlukyane.com/

• https://twitter.com/AndLukyane

• ods.ai @artgor