Notes of
State of the Art on Neural Rendering
cited by 115 (2021 Sep)
- First survey paper of this topic
- Some of them mix the physical knowledge from CV
- Focus on find controllable photorealistic output
-
Cases
- Novel view synthesis
- Semantic photo manipulation
- Facial and Body Reenactment(重現)
- Relighting
- Free-Viewpoint Video
- Photo-realistic avatars for AR/VR
- Open Research Problem
First publications that used the term neural rendering
Neural scene representation and rendering
Use arbitrary numbers of
(viewpoint, image) as observation
synthesis the view from other viewpoint
Survey of surveys :)
Theoretical Fundamentals(Sec 4)
4.1 Physical Image Formation
- 4.1.1 Scene Representations
- 4.1.2 Camera Models
- 4.1.3 Classical Rendering
- 4.1.4. Light Transport
- 4.1.5. Image-based Rendering
=======================================
4.2 Deep Generative Models
- 4.2.1. Learning a Generator
- 4.2.2. Learning using Perceptual Distance
- 4.2.3. Learning with Conditional GANs
- 4.2.4. Learning without Paired Dat
Neural Rendering (Sec 5)
Control
Open research problem
Current (2020)
Novel view synthesis
Relighting under novel lighting
animating faces/bodies
guide with semantic mask
Computer Graphics Modules
Some different channels can be feed as DL inputs
such as a depth map, normal map, camera/world space position maps, albedo(反照率) map, a diffuse(漫射) rendering of the scene, and many more
Explicit vs. Implicit Control
Implicit
Explicit
Multi-modal Synthesis
Generality
For example, if the method operates on human heads, it will aim to be applicable to all humans.
(non general)
For many tasks, object specific approaches are currently producing higher quality results, at the cost of lengthy training times for each object instance.
For real-world applications such training times are prohibitive improving general models is an open problem and an exciting research direction.
Applications of Neural Rendering (Sec 6)
- Required Data
- Network Inputs
- Network Outputs
- Contents
- Controllable Parameters
- Explicit control
- CG module
- (6.1) Semantic Photo Synthesis and Manipulation
- (6.2) Novel View Synthesis
- (6.3) Free Viewpoint Video
- (6.4) Relighting
- (6.5) Facial/Body Reenactment
Table 1 : Selected methods presented in this survey
Semantic Photo Synthesis and Manipulation
Semantic Photo Synthesis and Manipulation
Novel View Synthesis
Free Viewpoint Video
Relighting
Facial Reenactment
Body Reenactment
Open Challenges (Sec 7)
- Generalization
Adequate for unseen pose/scenario - Scalability
So many constrainants for complexity and size of the scenes - Editability
Alougth something such as mesh can be edit out of the model.
Some of neural texture is embedding in the model, and cannot be edit explicitly. - Multimodal Neural Scene Representations
This paper mainly focus on image and videos as I/O, some other projects can use other I/O such the composition of Voice & Video
Social Implications (Sec 8)
Conclusion (Sec 9)
