Non Photorealistic Rendering - Stroke-Based Rendering (SBR)

I have combined elements from two academic papers to create my own Stroke-Based Rendering (SBR) software. I used "Painterly Rendering with Curved Brush Strokes of Multiple Sizes" by Aaron Hertzmann for the general framework but I didn't particularly like his curved brush strokes. I prefer straight strokes with an oil paint texture. So, I turned to "An Algorithm For Automatic Painterly Rendering Based On Local Source Image Approximation" by Michio Shiraishi and Yasushi Yamaguchi to handle the brush strokes.

The following 2 images show the pseudo-code for the framework. They come straight from the Hertzmann paper.

The input is an RGB image (sourceImage) and a sequence of brush radii of decreasing size (R1 to Rn). The output is an RGB image (canvas) which is initialized to some middle gray color. For each brush radius, the source image is convolved by a Gaussain blur of variance f_sigma * Ri where Ri is the current brush radius. The parameter f_sigma is some constant factor that enables to increase or decrease the blurring. If f_sigma is set to a very small value, no blurring takes place. A layer of paint is then laid on the reference image (blurred source image) using the paintLayer function described below.

A grid is virtually constructed with a grid cell size equal to f_grid * R where f_grid is some constant factor and R is the current brush radius. Then, for each grid cell, he computes the average error within the grid cell, the error being defined as the difference in color between the current canvas color and the reference image color. If the average difference in color is greater than T (error_threshold), the pixel with the largest difference in color is chosen as the center of the brush stroke and that brush stroke is added to the list of brush strokes (the color for the brush stroke comes from the color of the pixel in the reference image). Once all the brush strokes have been created, they are randomized, and applied to the canvas.

As mentioned previously, I don't like curved brush strokes as I don't think it reflects particularly well what most fine art painters do. Straight brush strokes are in my opinion better and they are much easier to implement. Let's turn now our attention to the Shiraishi paper to make and paint the brush strokes.

The pixel with the largest error in the grid cell (its color is the brush stroke color) and the radius define a square window in the reference image. What Shiraishi does is create a grayscale difference image considering the brush stroke color as the reference color. He then uses image moments to define the equivalent rectangle of that square difference image. The center of the equivalent rectangle defines the brush stroke center. The angle theta between the longer edge of the equivalent rectangle and the x-axis defines the angle of the brush stroke. The width and length of the equivalent rectangle define the width and length of the brush stroke. This completely defines the brush stroke. Recall though that all brush strokes are made before they are applied onto the canvas.

To paint a given brush stroke, a rectangular grayscale texture image (where white means fully opaque and black means fully transparent) is scaled so that it matches the equivalent rectangle in terms of width and height, rotated by theta, translated so that its center matches the brush stroke center, and then painted onto the canvas using alpha blending. If you want to be real fancy and somehow simulate the impasto technique where thick layers of oil paint are applied, you may also use a rectangular grayscale bump map image alongside the texture image and a bump map alongside the canvas.

Here's an example:

Input RGB image.

Output canvas without bump mapping.

Output canvas image with bump mapping.

Parameters used:
brush radius = 32 16 8 4 2
f_sigma = 1e-05
error_threshold = 60

A few notes:
- I use a very small f_sigma so that the reference image never gets blurred. Because of that, the input image needs to be slightly blurred as high frequency artifacts could be a problem when evaluating the image moments.
- I always use f_grid = 1.0. That's why it's not a parameter.
- To render the bump mapping, I use Gimp as it has a very convenient bump map filter.

Here's a quick video:


At the moment, the software is sitting on my linux box but it is not available for download. If you like this type of painterly rendering, feel free to send me your photographs and it will be my pleasure to "paint" them for you.