From Style Transfer to Text-Driven Image Manipulation

Before Style Transfer

How to summarize texture?

https://paperswithcode.com/dataset/psu-near-regular-texture-database

Before Style Transfer

How to summarize texture?

Define some handmade feature representation
like color, gradient, frequency...
Then use statistics

Image => Feature Extract => Summarize => Distance

https://paperswithcode.com/dataset/psu-near-regular-texture-database

How to summarize texture?

Image

Feature
Extract

Summarize

Distance

Distance measure for vector/distribution

Manjunath , B. S., & Ma, W. Y. (1996). Texture features for browsing and retrieval of image data. pattern analysis and machine intelligence , 18 (8), 837IEEE Transactions on 52 842.

Ojala, Timo, Matti Pietikainen, and Topi Maenpaa. "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns." IEEE Transactions on pattern analysis and machine intelligence 24.7 (2002): 971-987.

How to summarize texture?

Gabor Filter Bank

$(H, W, 1) \rightarrow (H, W, f \cdot \theta)$

Image

Feature
Extract

Summarize

Distance

use $\mu, \sigma$

$(H, W, f \cdot \theta) \rightarrow (2\cdot f \cdot \theta)$

Distance measure for vector/distribution

Manjunath , B. S., & Ma, W. Y. (1996). Texture features for browsing and retrieval of image data. pattern analysis and machine intelligence , 18 (8), 837IEEE Transactions on 52 842.

Ojala, Timo, Matti Pietikainen, and Topi Maenpaa. "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns." IEEE Transactions on pattern analysis and machine intelligence 24.7 (2002): 971-987.

How to summarize texture?

Gabor Filter Bank

$(H, W, 1) \rightarrow (H, W, f \cdot \theta)$

Image

Feature
Extract

Summarize

Distance

use $\mu, \sigma$

$(H, W, f \cdot \theta) \rightarrow (2\cdot f \cdot \theta)$

Use Histogram

$(H, W, 1)_{\in \{0,1,...,9\}} \rightarrow$

Distance measure for vector/distribution

Manjunath , B. S., & Ma, W. Y. (1996). Texture features for browsing and retrieval of image data. pattern analysis and machine intelligence , 18 (8), 837IEEE Transactions on 52 842.

Ojala, Timo, Matti Pietikainen, and Topi Maenpaa. "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns." IEEE Transactions on pattern analysis and machine intelligence 24.7 (2002): 971-987.

9

Rotation Invariant Local Binary Pattern

$(H, W, 1)_{\in \{0,1,...,255\}} \rightarrow (H, W, 1)_{\in \{0,1,...,9\}}$

How to summarize texture?

Gabor Filter Bank

$(H, W, 1) \rightarrow (H, W, f \cdot \theta)$

Image

Feature
Extract

Summarize

Distance

use $\mu, \sigma$

$(H, W, f \cdot \theta) \rightarrow (2\cdot f \cdot \theta)$

Use Histogram

$(H, W, 1)_{\in \{0,1,...,9\}} \rightarrow$

Distance measure for vector/distribution

Manjunath , B. S., & Ma, W. Y. (1996). Texture features for browsing and retrieval of image data. pattern analysis and machine intelligence , 18 (8), 837IEEE Transactions on 52 842.

Ojala, Timo, Matti Pietikainen, and Topi Maenpaa. "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns." IEEE Transactions on pattern analysis and machine intelligence 24.7 (2002): 971-987.

9

<

$\geq$

Thresholding

Rotation Invariant Local Binary Pattern

$(H, W, 1)_{\in \{0,1,...,255\}} \rightarrow (H, W, 1)_{\in \{0,1,...,9\}}$

11000000

How to summarize texture?

Gabor Filter Bank

$(H, W, 1) \rightarrow (H, W, f \cdot \theta)$

Rotation Invariant Local Binary Pattern

$(H, W, 1)_{\in \{0,1,...,255\}} \rightarrow (H, W, 1)_{\in \{0,1,...,9\}}$

Image

Feature
Extract

Summarize

Distance

use $\mu, \sigma$

$(H, W, f \cdot \theta) \rightarrow (2\cdot f \cdot \theta)$

Use Histogram

$(H, W, 1)_{\in \{0,1,...,9\}} \rightarrow$

Distance measure for vector/distribution

Manjunath , B. S., & Ma, W. Y. (1996). Texture features for browsing and retrieval of image data. pattern analysis and machine intelligence , 18 (8), 837IEEE Transactions on 52 842.

Ojala, Timo, Matti Pietikainen, and Topi Maenpaa. "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns." IEEE Transactions on pattern analysis and machine intelligence 24.7 (2002): 971-987.

9

rotation invariant

<

$\geq$

Thresholding

How to summarize data?

Data

Feature
Extract

Describe

Distance

If we have good feature extractor...

Distance measure for vector/distribution

Describe data w/ or w/o statistic...

A Cute Dog Staring You

Feature
Extract

Distance

If we have good feature extractor...

Use Other Task's Pretrained Weight
Create It By Yourself

Distance measure for vector/distribution

Describe

Describe data w/ or w/o statistic...

A Cute Dog Staring You

How to summarize data?

Data

Style Transfer

Lin, Tianwei, et al. "Drafting and Revision: Laplacian Pyramid Network for Fast High-Quality Artistic Style Transfer." arXiv preprint arXiv:2104.05376 (2021).

Content

Style

Styllized

Objective

Find a stylized image, which has

Content image's content
Style image's style

Style Transfer

Jing, Yongcheng, et al. "Neural Style Transfer: A Review." arXiv preprint arXiv:1705.04058 (2017).

Style Transfer

Jing, Yongcheng, et al. "Neural Style Transfer: A Review." arXiv preprint arXiv:1705.04058 (2017).

Style Transfer

Jing, Yongcheng, et al. "Neural Style Transfer: A Review." arXiv preprint arXiv:1705.04058 (2017).

Style Transfer

Jing, Yongcheng, et al. "Neural Style Transfer: A Review." arXiv preprint arXiv:1705.04058 (2017).

Image Optimization (Inference $\equiv$ train:minutes)
Find an image
Model Optimization (Inference:real time; Train:hours)
Find a model can transfer image
1. Per-Style-Per-Model (PSPM)
  Model contain 1 style
2. Multiple-Style-Per-Model (MSPM)
  Model contain n style
3. Arbitrary-Style-Per-Model (ASPM)
  Model contain any style

Style Transfer

Jing, Yongcheng, et al. "Neural Style Transfer: A Review." arXiv preprint arXiv:1705.04058 (2017).

Image Optimization (Inference $\equiv$ train:minutes)
Find an image
Model Optimization (Inference:real time; Train:hours)
Find a model can transfer image
1. Per-Style-Per-Model (PSPM)
  Model contain 1 style
2. Multiple-Style-Per-Model (MSPM)
  Model contain n style
3. Arbitrary-Style-Per-Model (ASPM)
  Model contain any style

Style Transfer

Jing, Yongcheng, et al. "Neural Style Transfer: A Review." arXiv preprint arXiv:1705.04058 (2017).

Image Optimization (Inference $\equiv$ train:minutes)
Find an image
Model Optimization (Inference:real time; Train:hours)
Find a model can transfer image
1. Per-Style-Per-Model (PSPM)
  1 style
2. Multiple-Style-Per-Model (MSPM)
  n styles
3. Arbitrary-Style-Per-Model (ASPM)
  any style

Image style transfer using convolutional neural networks

Notes

Arxiv : 1508.06576
First paper for "Neural" Style Transfer
Get the result by optimize the image
Plenty of later papers use their loss function
Cost minutes to generate an image
Cited by 3267 at 2021 Nov

Feature
Extract

Distance

If we have good feature extractor...

Use Other Task's Pretrained Weight
Create It By Yourself

Distance measure for vector/distribution

Describe

Describe data w/ or w/o statistic...

A Cute Dog Staring You

Data

VGG

(A Pretrained Model)

$\mathcal{L}_{content}$ : Feature tensor close to content image's feature tensor

$\mathcal{L}_{style}$ : Stat(feature) close to style image's stat(feature)

$\mathcal{L}_{total} = \mathcal{L}_{content} + \lambda \mathcal{L}_{style}$

$\hat{I}$

VGG

(A Pretrained Model)

$\mathcal{L}_{content}$ =( - ) $^2$

$\mathcal{L}_{content}$ : Feature tensor close to content image's feature tensor

$\mathcal{L}_{style}$ : Stat(feature) close to style image's stat(feature)

$\mathcal{L}_{total} = \mathcal{L}_{content} + \lambda \mathcal{L}_{style}$

$\hat{I}$

VGG

(A Pretrained Model)

$\mathcal{L}_{style}$ =(G( )-G( )) $^2$

$\mathcal{L}_{content}$ =( - ) $^2$

$\mathcal{L}_{content}$ : Feature tensor close to content image's feature tensor

$\mathcal{L}_{style}$ : Stat(feature) close to style image's stat(feature)

$\mathcal{L}_{total} = \mathcal{L}_{content} + \lambda \mathcal{L}_{style}$

$\hat{I}$

VGG

(A Pretrained Model)

$\mathcal{L}_{style}$ =(G( )-G( )) $^2$

$\mathcal{L}_{content}$ =( - ) $^2$

$\mathcal{L}_{content}$ : Feature tensor close to content image's feature tensor

$\mathcal{L}_{style}$ : Stat(feature) close to style image's stat(feature)

$\mathcal{L}_{total} = \mathcal{L}_{content} + \lambda \mathcal{L}_{style}$

result $\leftarrow argmin_{\color{red}\hat{I}}\mathcal{L}_{total}({\color{red}\hat{I}})$

$\hat{I}$

VGG

(A Pretrained Model)

$\mathcal{L}_{style}$ =(G( )-G( )) $^2$

$\mathcal{L}_{content}$ =( - ) $^2$

$\mathcal{L}_{content}$ : Feature tensor close to content image's feature tensor

$\mathcal{L}_{style}$ : Stat(feature) close to style image's stat(feature)

$\mathcal{L}_{total} = \mathcal{L}_{content} + \lambda \mathcal{L}_{style}$

result $\leftarrow argmin_{\color{red}\hat{I}}\mathcal{L}_{total}({\color{red}\hat{I}})$

$\hat{I} = \hat{I} - \alpha\frac{\partial \mathcal{L}_{total}}{\partial \hat{I}}$

$\hat{I}$

VGG

(A Pretrained Model)

$\mathcal{L}_{style}$ =(G( )-G( )) $^2$

$\mathcal{L}_{content}$ =( - ) $^2$

$\mathcal{L}_{content}$ : Feature tensor close to content image's feature tensor

$\mathcal{L}_{style}$ : Stat(feature) close to style image's stat(feature)

$\mathcal{L}_{total} = \mathcal{L}_{content} + \lambda \mathcal{L}_{style}$

$F \in R^{C \times X \times Y}, G(F) \in R^{C \times C}$

$G(F)_{c1, c2} = \frac{1}{X\cdot Y}\sum_{x, y}[F_{c1,x,y} \cdot F_{c2,x,y}]$

$G$ : gram matrix

result $\leftarrow argmin_{\color{red}\hat{I}}\mathcal{L}_{total}({\color{red}\hat{I}})$

$\hat{I} = \hat{I} - \alpha\frac{\partial \mathcal{L}_{total}}{\partial \hat{I}}$

Get more abstract result while use deeper layer for content loss

Their Result

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Johnson, Justin, Alexandre Alahi, and Li Fei-Fei. “Perceptual Losses for Real-Time Style Transfer and Super-Resolution.” arXiv preprint arXiv:1603.08155 (2016).

Notes

Arxiv : 1603.08155
Get the result by optimize a model
Thier research has 2 branch :
1. style transfer
2. super resolution
Plenty of later papers use the term "Perceptual"
Per-Style-Per-Model(PSPM)
Real time while inference
Cited by 5962 at 2021 Nov

$\hat{I}$

Perceptual

$\hat{I}$

prev : $argmin_{{\color{red}\hat{I}}}\mathcal{L}_{total}({\color{red}\hat{I}})$

Perceptual

$f_W({\color{blue}I})$

Model $f_W$

prev : $argmin_{{\color{red}\hat{I}}}\mathcal{L}_{total}({\color{red}\hat{I}})$

this: $argmin_{f_W}\sum_{{\color{blue}I}\in dataset}\mathcal{L}_{total}(f_W({\color{blue}I}))$

A learned representation for artistic style

Notes

Arxiv : 1610.07629
Get the result by optimizing a model
Multiple-Style-Per-Model (MSPM)
Use conditional instance normalization(CIN) for multiple style transfer
The standard setting contain 32 styles, each style contain about 0.2% total parameters.
Real time while inference
Cited by 727 at 2021 Nov

Before A Learned Representation For Artistic Style

Color Distrbution Matching

https://www.dreamstime.com/different-times-day-night-morning-noon-evening-city-landscape-buildings-skyscrapers-urban-concept-background-isolated-image189577922

Source

Stat(R)

Stat(G)

Stat(B)

Target

Stat(R)

Stat(G)

Stat(B)

Before A Learned Representation For Artistic Style

Color Distrbution Matching

https://www.dreamstime.com/different-times-day-night-morning-noon-evening-city-landscape-buildings-skyscrapers-urban-concept-background-isolated-image189577922

Before A Learned Representation For Artistic Style

https://www.dreamstime.com/different-times-day-night-morning-noon-evening-city-landscape-buildings-skyscrapers-urban-concept-background-isolated-image189577922

Color Distrbution Matching

Target

Stat(R)

Stat(G)

Stat(B)

Normalize
( $\mu=0, \sigma=1$ )
( $\mu=0, \sigma=1$ )
( $\mu=0, \sigma=1$ )

Source

Stat(R)

Stat(G)

Stat(B)

A Learned Representation For Artistic Style

Each Style Use a $(\gamma, \beta)$ pair

Target

Stat(R)

Stat(G)

Stat(B)

Normalize
( $\mu=0, \sigma=1$ )
( $\mu=0, \sigma=1$ )
( $\mu=0, \sigma=1$ )

Source

Stat(R)

Stat(G)

Stat(B)

$f_W({\color{blue}I})$

Model $f_W$

$n \times$

Act

Conv

$n \times$

Act

$f_W({\color{blue}I})$

Model $f_W$

$n \times$

Act

This

$S_2$

$S_1$

Interpolate

$S$ =

$\alpha S_1+(1-\alpha)S_2$

$S_2$

$S_1$

Interpolate

$S_3$

$S_4$

Exploring the structure of a real-time, arbitrary neural artistic stylization network

Notes

Arxiv : 1705.06830
Get the result by optimizing a model
Arbitrary-Style-Per-Model (ASPM)
It generalized CIN for adaptive to arbitrary Style
Real time to generate an image with gpu
Cited by 180 at 2021 Nov

Prev Work

Style Prediction Network

This Work

Architecture

Small Recap

Papers:

1508.06576 Image Optimization
1603.08155 Per-Style-Per-Model (PSPM)
1610.07629 Multiple-Style-Per-Model (MSPM)
1705.06830 Arbitrary-Style-Per-Model (ASPM)

About half year a big improve

Small Recap

Papers:

1508.06576 Image Optimization
1603.08155 Per-Style-Per-Model (PSPM)
1610.07629 Multiple-Style-Per-Model (MSPM)
1705.06830 Arbitrary-Style-Per-Model (ASPM)

About half year a big improve

Not Enough?

(Methods up to March 2018, Cited by 335 at Nov 2021)

Jing, Yongcheng, et al. "Neural Style Transfer: A Review." arXiv preprint arXiv:1705.04058 (2017).

	# https://github.com/openai/CLIP#usage
	import torch
	import clip
	from PIL import Image

	device = "cuda" if torch.cuda.is_available() else "cpu"
	model, preprocess = clip.load("ViT-B/32", device=device)

	image = preprocess(Image.open("CLIP.png")).unsqueeze(0).to(device)
	text = clip.tokenize(["a diagram", "a dog", "a cat"]).to(device)

	with torch.no_grad():
	image_features = model.encode_image(image)
	text_features = model.encode_text(text)

	logits_per_image, logits_per_text = model(image, text)
	probs = logits_per_image.softmax(dim=-1).cpu().numpy()

	print("Label probs:", probs) # prints: [[0.9927937 0.00421068 0.00299572]]

	# https://github.com/huggingface/tokenizers
	output = tokenizer.encode("Hello, y'all! How are you 😁 ?")
	print(output.tokens)
	# ["Hello", ",", "y", "'", "all", "!", "How", "are", "you", "[UNK]", "?"]
	# string => tokens
	# token => idx => embedding

	# https://github.com/huggingface/tokenizers
	output = tokenizer.encode("Hello, y'all! How are you 😁 ?")
	print(output.tokens)
	# ["Hello", ",", "y", "'", "all", "!", "How", "are", "you", "[UNK]", "?"]
	# string => tokens
	# token => idx => embedding

	# https://github.com/huggingface/tokenizers
	output = tokenizer.encode("Hello, y'all! How are you 😁 ?")
	print(output.tokens)
	# ["Hello", ",", "y", "'", "all", "!", "How", "are", "you", "[UNK]", "?"]
	# string => tokens
	# token => idx => embedding