A Generative Model for Volume Rendering

Matthew Berger, Jixian Li, and Joshua A. Levine

Introduction

Similar to InSituNet

The general idea is to take a "view" and a discretized transfer function, and produce a volume rendered image.

Learns representation of volume

Introduction

Volume Rendering

The General Idea

Generative Adversarial Networks (GANs) again

\(\displaystyle\min_G\displaystyle\max_DL_{adv}(G,D)\)

A Tale of Two GANs

Opacity GAN

Color GAN

Composing these gives the full generator

input:

output:

A Tale of Two GANs

Why two gans?

  • The complex relationship between parameters makes it hard to stabilize training --- this is a common problem for GANs of this structure (largely 'fixed' by Progressive GANs)
  • It is relatively easier to train a GAN to generate a low-resolution image (which is often stable)
  • Image-to-image upscaling DCGANs are also quite stable

A Tale of Two GANs

Opacity GAN

A Tale of Two GANs

Color GAN

Training

Training Data

Acquiring data is relatively easy.

  • Fix the model
  • A view is randomly chosen
  • A color image and opacity image are rendered

view: (azimuth sin, azimuth cos, elevation, in plane rotation, distance to camera)

200,000 samples generated per model

Training

Procedure

Not much interesting here

Opacity GAN:

  • Learning rate = \(2\times10^{-4}\), halved every 5 epochs
  • Typical Wasserstein loss
  • Unsure about discriminator-generator ratio

Color GAN:

  • Learning rate = \(8\times10^{-6}\), halved every 8 epochs
  • Color transfer function represented in LAB space
  • Typical DCGAN loss
  • Unsure about discriminator-generator ratio

Results

Other Experiments

Transfer function sensitivity

Other Experiments

Transfer function sensitivity

Other Experiments

Transfer function sensitivity

Other Experiments

Latent space exploration

Other Experiments

Latent space exploration

High dimensional points, use tSNE to reduce dimensionality (to 2D)

Other Experiments

Latent space exploration

Other Experiments

Latent space exploration

Text

A Generative Model for Volume Rendering

By Joshua Horacsek

A Generative Model for Volume Rendering

  • 665