Notes of

State of the Art on Neural Rendering

cited by 115 (2021 Sep)

  • First survey paper of this topic
  • Some of them mix the physical knowledge from CV
  • Focus on find controllable photorealistic output
  • Cases
    • Novel view synthesis
    • Semantic photo manipulation
    • Facial and Body Reenactment(重現)
    • Relighting
    • Free-Viewpoint Video
    • Photo-realistic avatars for AR/VR
  • ​Open Research Problem

First publications that used the term neural rendering

Neural scene representation and rendering

Use arbitrary numbers of 

(viewpoint, image) as observation

synthesis the view from other viewpoint

Survey of surveys :)

Theoretical Fundamentals(Sec 4)

4.1 Physical Image Formation

  • 4.1.1 Scene Representations
  • 4.1.2 Camera Models
  • 4.1.3 Classical Rendering
  • 4.1.4. Light Transport
  • 4.1.5. Image-based Rendering

=======================================

4.2 Deep Generative Models

  • 4.2.1. Learning a Generator
  • 4.2.2. Learning using Perceptual Distance
  • 4.2.3. Learning with Conditional GANs
  • 4.2.4. Learning without Paired Dat

Neural Rendering (Sec 5)

Control

Open research problem

Current (2020)

Novel view synthesis

Relighting under novel lighting

animating faces/bodies

guide with semantic mask

Computer Graphics Modules

Some different channels can be feed as DL inputs

such as a depth map, normal map, camera/world space position maps, albedo(反照率) map, a diffuse(漫射) rendering of the scene, and many more

Explicit vs. Implicit Control

Implicit

Explicit

Multi-modal Synthesis

Generality

 For example, if the method operates on human heads, it will aim to be applicable to all humans.

 

(non general)

For many tasks, object specific approaches are currently producing higher quality results, at the cost of lengthy training times for each object instance.

 

For real-world applications such training times are prohibitive improving general models is an open problem and an exciting research direction.

Applications of Neural Rendering  (Sec 6)

  • Required Data
  • Network Inputs
  • Network Outputs
  • Contents
  • Controllable Parameters
  • Explicit control
  • CG module
  • (6.1) Semantic Photo Synthesis and Manipulation
  • (6.2) Novel View Synthesis
  • (6.3) Free Viewpoint Video
  • (6.4) Relighting
  • (6.5) Facial/Body Reenactment

Table 1 : Selected methods presented in this survey

Semantic Photo Synthesis and Manipulation

Semantic Photo Synthesis and Manipulation

Novel View Synthesis

Free Viewpoint Video

Relighting

Facial Reenactment

Body Reenactment

Open Challenges (Sec 7)

  • Generalization
    Adequate for unseen pose/scenario
  • Scalability
    So many constrainants for complexity and size of the scenes
  • Editability
    Alougth something such as mesh can be edit out of the model.
    Some of neural texture is embedding in the model, and cannot be edit
    explicitly.
  • Multimodal Neural Scene Representations
    This paper mainly focus on image and videos as I/O, some other projects can use other I/O such the composition of Voice & Video

Social Implications (Sec 8)

Conclusion (Sec 9)

Made with Slides.com