Tuesday, May 15, 2018

Non Photorealistic Rendering - Image Abstraction by Structure Adaptive Filtering

This post describes all the parameters that impact the rendered image in "Image Abstraction by Structure Adaptive Filtering" by Jan Eric Kyprianidis and Jürgen Döllner. Note that this paper was seriously influenced by "Real-Time Video Abstraction" by Holger Winnemöller, Sven C. Olsen, and Bruce Gooch. IF you are a bit confused by the title of the paper, "Image Abstraction by Structure Adaptive Filtering", you are not the only one. In layman terms, it's simply a cartoon filter.


Overview of the method (picture comes from the paper cited above).

There are 2 parameters to control the number of iterations in "Separated OABF":
- n_e. That's the number of iterations before edges are detected. Kyprianidis uses n_e = 1 while Winnemöller uses n_e = 1 or 2.

- n_a. That's the total number of iterations.

Step 1: Local Orientation Estimation

This is to establish the Edge Tangent Flow (ETF) vector field. See Non Photorealistic Rendering - Edge Tangent Flow (ETF).

Parameters used:
- Variance of the Gaussian used to blur the structure tensors. What value should be used is not really discussed in the paper.

Step 2: Separated Orientation-aligned Bilateral Filter

See Non Photorealistic Rendering - Separated Orientation-Aligned Bilateral Filter (Separated OABF).

Parameters used:
- sigma_d. That's the variance of the spatial Gaussian function that's part of the bilateral filter. Both Kyprianidis and Winnemöller use sigma_d = 3.0.
- sigma_r. That's the variance of the color Gaussian function that's part of the bilateral filter. Both Kyprianidis and Winnemöller use sigma_r = 4.25.

For this step, the number of iterations used is the number of iterations before the edges are detected, that is, n_e.

Step 3: Separated Flow-based Difference-of-Gaussians Filter (Separated FDoG)

See Non Photorealistic Rendering - Separated Flow-based Difference of Gaussians (Separated FDoG).

Parameters used for the DoG filter that is applied in the gradient direction:
- sigma_e. That's the variance of the spatial Gaussian. The variance of the other spatial Gaussian is set to 1.6*sigma_e so that DoG approximates LoG (Laplacian of Gaussians). Kyprianidis uses sigma_e = 1.0. Don't know about Winnemöller.
- tau. That's the sensitivity of edge detection. Kyprianidis uses tau = 0.99 while Winnemöller uses tau = 0.98.

Parameters used to smooth the edges in the direction of the flow curves:
- sigma_m. That's the variance of the Gaussian used to smooth the edges. Kyprianidis uses sigma_m = 3.0. Don't know about Winnemöller.

Parameters used to threshold the edges:
- phi_e. Kyprianidis uses phi_e = 2.0 while Winnemöller uses phi_e between 0.75 and 5.0.

There is another parameter that can be used:
- n. That's the number of iterations of "Separated FDoG". Kyprianidis uses n = 1 most of the times. Don't know about Winnemöller.

Step 4: Separated Orientation-aligned Bilateral Filter

Parameters used:
- sigma_d. Same as before.
- sigma_r. Same as before.

For this step, the number of iterations used is the number of iterations that remain, that is, n_a-n_e.

Step 5: Color Quantization

See Non Photorealistic Rendering - Pseudo-Quantization.

Parameters used:
- quant_levels. That's the number of levels used to quantize the luminance. Winnemöller uses nbins = 8 to 10.
- phi_q. Controls the softness of the quantization. Winnemöller uses phi_q = 3.0 to 14.0.

Here's an example:


Input RGB image.


Output blended image.

Parameters used:
tensor_sigma = 3
n_e = 1
n_a = 1
sigma_d = 3
sigma_r = 4.25
fdog_n = 2
fdog_sigma_e = 1
fdog_tau = 0.99
fdog_sigma_m = 3
fdog_phi = 2
phi_q = 3
quant_levels = 8

Here's a quick video:


At the moment, the software is sitting on my linux box but it is not available for download. If you like this cartoon rendering, feel free to send me your photographs and it will be my pleasure to cartoonify them for you.

2 comments:

  1. Replies
    1. haha thanks Pepe. I really like this effect as it is quite convincing as a cell shader. Kudos to the people that came up with this approach.

      Delete