Tuesday, May 21, 2019

A quick guide on using StereoPhoto Maker (SPM) to generate depth maps

I am assuming that you have installed StereoPhoto Maker and the combo DMAG5/DMAG9b on your computer. If you haven't done so yet, follow the instructions in How to make Facebook 3D Photo from Stereo pair. I am assuming you have downloaded and extracted the DMAG5/DMAG9b stuff in a directory called dmag5_9. That directory should look like so:


Contents of the dmag5_9 directory.

Because alignment of the stereo pair is of prime importance in depth map generation, I recommend going into "Edit->Preferences->Adjustment" and checking the box that says "Better Precision (slow)". As I don't particularly like large images, the first thing I do after loading the stereo pair is to resize the images to something smaller. In this guide, the stereo pair is an mpo coming from a Fuji W3 camera. The initial dimensions are 3441x2016. You could generate the depth maps using those dimensions but everything is gonna take longer. Clicking on "Edit->Resize", I change the width to 1200. I guess I could have resized to something larger but I wouldn't resize to anything larger than 3000 pixels. Once the stereo pair has been resized, I click on "Adjust->Auto-alignment" to align/rectify the images.

To generate the depth map, I click on "Edit->Depth Map->Create depth map from stereo pair" where I am presented with this window:


Default "Create depth map from stereo pair" window.

What I recommend doing is getting the background and foreground disparity values manually first. To get the background value, use the arrow keys so that the two red/cyan views come into focus (merge) for a background point. Same idea for the foreground value. I keep track of those two values and then click on "Get Values (automatic)". If the automatic values make sense when compared to the ones obtained manually, I just leave them be. If they don't, I edit them and put back the values obtained manually. Since my image width is less than 3000 pixels, I don't bother with the "maximum image width" box. I don't like the idea of SPM resizing the images automatically so I always make sure that my image width is less than what it's in the "maximum image width" box. I only want the left depth map so I keep the "Create left/right depth maps" unchecked. Since I want white to be for the foreground and black for the background (I always use white for the foreground in my depth maps), I click on "Create depth map (front: white, back: black)". Note that because I clicked on "Create depth map (front: white, back: black)", when it is time to save the depth map by going into "Edit->Depth map->Save as Facebook 3d photo (2d photo+depth)", I will need to click on "white" for the "Displayed depth map front side->white" so that the depth map does not get inverted when saved by SPM. If you don't mind a depth map where the foreground is black, then leave the "Create depth map (front: white, back: black)" radio button alone and don't click on it.

I recommend using the default parameters in DMAG5 (on the left) and DMAG9b (on the right) to get the first depth map (as you may need to tweak the parameters to get the best possible depth map). To be sure I have the default settings (SPM stores the latest used settings), I click on "Default settings". To get the depth map, I click on "Create depth map". This is the result:


Left image and depth map produced by SPM.

To get the left image on the left and the depth map on the right, click on the "Side-by-side" icon in the taskbar. For more info on DMAG5, check Depth Map Automatic Generator 5 (DMAG5). For more info on DMAG9b, check Depth Map Automatic Generator 9b (DMAG9b) and/or have a look at dmag9b_manual.pdf in the dmag5_9 directory.

Now, if you go into the dmag5_9 directory, you will find some very interesting intermediate images:
- 000_l.tif. That's the left image as used by DMAG5.
- 000_r.tif. That's the right image as used by DMAG5.
- con_map.tiff. That's the confidence map computed by DMAG9b. White means high confidence, black means low confidence. If you clicked on "Create left/right depth maps", the confidence map gets overwritten when DMAG9b is called to optimize the right depth map.
- dps_l.tif. That's the left depth map produced by DMAG5.
- dps_r.tif. That's the right depth map produced by DMAG5. Even if the "Create left/right depth maps" box is unchecked, the right depth map is always generated by DMAG5 in order to detect left/right inconsistencies in the left depth map.
- out.tif. That's the depth map produced by DMAG9b. DMAG9b uses the depth map generated by DMAG5 as input and improves it. If you clicked on "Create left/right depth maps", there should be out.tif (left depth map) and out_r.tif (right depth map).

The depth map "out.tif" is basically the same as the depth map that you get by clicking on "Edit->Depth map->Save as Facebook 3D photo (image+depth)".

It should be noted that once you have created a depth map, you cannot re-click on "Edit->Depth map->Create depth map from stereo pair", possibly change parameters, and create another depth map. If you do that, the new depth map is basically going to be garbage because the right image has been replaced by the previously created depth map (take a look at 000_r.tif in the dmag5_9 directory to convince yourself). To re-generate the depth map, you need to undo or reload the stereo pair.

At this point, you may want to play with the DMAG5/DMAG9b parameters to see if you can improve the depth map. I think it's a good idea but be aware there may be areas in your image where the depth map cannot be improved upon. For instance, if you have an area that has no texture (think of a blue sky or a white wall) or an area that has a repeated texture, it is quite likely the depth map is gonna be wrong no matter what you do. So, improving the depth map is not always easy as, usually, some areas will get better while others will get worse. Some parameters like DMAG5 radius or DMAG9b sample rate spatial depend on the image dimensions and should kind of be tailored to the image width. I think it's best to change one parameter at a time and see the effect it has on the depth map. If things get better, keep changing that parameter (in the same direction) until things get worse. Then, you tweak the next parameter. Of course, it is up to you whether or not you want to spend the time tinkering with parameters. If you always shoot with the same camera setup, this tinkering may only need to be done once.

Let's see which parameters used by DMAG5 and DMAG9b are worth tinkering with in the quest for a better depth map. I believe changing the sample rate spatial used by DMAG9b from the default 32 to 16 or even 8 should be the first thing to change when trying to improve the depth map. I think the default 32 is probably too large especially if the image width is not that large (like here for our test case).

What DMAG5 parameters (on the left side of the window) give you the most bang for your bucks when trying to improve the mesh quality?

- radius. The larger the radius, the smoother the depth map generated by DMAG5 is going to be but the less accurate. As the radius goes down, more noise gets introduced into the depth map.
- downsampling factor. The larger the downsampling factor, the faster DMAG5 will run but the less accurate. Running DMAG5 using a downsampling factor of 2 is four times faster than running DMAG5 using a downsampling factor of 1 (no downsampling).

Note that those observations concern DMAG5 only. So, you should look at the dps_l.tif file to see the effects of changing DMAG5 parameters. Also, you may find that sometimes the depth map generated by DMAG5 is actually better than the one generated by the combo DMAG5/DMAG9b. Note that DMAG9b can be so aggressive that variations in the depth map produced by DMAG5 do not matter much.


Left depth map generated by DMAG5 (dps_l.tif) using default values.

What happens to the left depth map generated by DMAG5 if you change the radius from 16 to 8 (every other parameter set to default)? Let's find out!


"Create depth map from stereo pair" window. Changed DMAG5 radius from 16 to 8 (every other parameter set to default).


Left depth map generated by DMAG5 (dps_l.tif). Changed DMAG5 radius from 16 to 8 (every other parameter set to default).

What happens to the left depth map generated by DMAG5 if you change the downsampling factor from 2 to 1 (every other parameter set to default)? Let's find out!


"Create depth map from stereo pair" window. Changed DMAG5 downsampling factor from 2 to 1 (every other parameter set to default).


Left depth map generated by DMAG5 (dps_l.tif). Changed DMAG5 downsampling factor from 2 to 1 (every other parameter set to default).

What DMAG9b parameters (on the right side of the window) give you the most bang for your bucks when trying to improve the mesh quality?

- sample rate spatial. The larger the sample rate spatial, the more aggressive DMAG9b will be. I recommend going from 32 (default value) to 16, 8, and even 4. If you can clearly see "blocks" in your depth map, the sample rate spatial is probably too large and should be reduced (by a factor of 2).
- sample rate range. The larger the sample rate range, the more aggressive DMAG9b will be. The default value is 8 but you can try 4 or 16 and see if it's any better.
- lambda. The larger the lambda, the smoother the depth map is going to be (in other words, the more aggressive DMAG9b will be). The default value is 0.25 but you can certainly try larger or smaller values. If you don't want the output depth map to be too different from the depth map generated by DMAG5 (dps_l.tiff), use smaller values for lambda (you will probably also need to use smaller values for the sample rate spatial and the sample rate range).


Depth map generated by DMAG9b (out.tif) using default values.

What happens to the depth map generated by DMAG9b if you change the sample rate spatial from 32 to 16 (every other parameter set to default)? Let's find out!


"Create depth map from stereo pair" window. Changed DMAG9b sample rate spatial from 32 to 16 (every other parameter set to default).


Depth map generated by DMAG9b (out.tif). Changed DMAG9b sample rate spatial from 32 to 16 (every other parameter set to default).

What happens to the depth map generated by DMAG9b if you change the sample rate range from 8 to 4 (every other parameter set to default)? Let's find out!


"Create depth map from stereo pair" window. Changed DMAG9b sample rate range from 8 to 4 (every other parameter set to default).


Depth map generated by DMAG9b (out.tif). Changed DMAG9b sample rate range from 8 to 4 (every other parameter set to default).

What happens to the depth map generated by DMAG9b if you change lambda from 0.25 to 0.5 (every other parameter set to default)? Let's find out!


"Create depth map from stereo pair" window. Changed DMAG9b lambda from 0.25 to 0.5 (every other parameter set to default).


Depth map generated by DMAG9b (out.tif). Changed DMAG9b lambda from 0.25 to 0.5 (every other parameter set to default).

Now, if you want to manually edit the generated depth map, you can do so in SPM by clicking on "Edit->Depth map->Correct depth map". If you want to do edit the generated depth map semi-automatically, you can use the techniques centered around DMAG11 or DMAG4 that are described in Case Study - How to get depth maps from old stereocards using ER9c, DMAG5, DMAG9b, and DMAG11 and Case Study - How to improve depth map quality with DMAG9b and DMAG4.

For the ultimate experience in editing depth maps semi-automatically, I recommend using 2d to 3d Image Conversion Software - The 3d Converter: load up the left image and the depth map (add an alpha channel if it does not have one), use the eraser tool to delete parts in the depth map you do not like (that becomes sparse_depthmap_rgba_image.png needed by the3dconverter), create a new layer and trace where you don't want the depths to bleed through (that becomes edge_rgba_image.png needed by the3dconverter), and run the3dconverter to get the new depth map called dense_depthmap_image.png. You do not to worry or care about gimp_paths.svg, ignored_gradient_rgba_image.png, and emphasized_gradient_rgba_image.png as they are not needed.

Sunday, May 12, 2019

Case Study - DMAG5/DMAG9b vs DMAG5b/DMAG9b

This post kinda compares a depth map produced by the combo DMAG5/DMAG9b vs the combo DMAG5b/DMAG9b. Thanks to my good friend Katsuhiko Inoue for providing the stereo pair (taken in portrait mode with an iphone X).


Left image of stereo pair after rectification by ER9b.


Right image of stereo pair after rectification by ER9b.

I do not know how the right image was obtained. It certainly was not obtained from a portrait mode stereo photo using the dual lens as it's not possible to extract the right image from an iphone X stereo photo. Even if you could extract the right image, it would not be the same focal length as the left image, meaning you would need specialized depth map generation software to get the depth map. Here, I am talking about the dual lens iphone X (back-facing camera system), not the TrueDepth sensor (front-facing camera system). I think the depth map produced by the iphone X was used here to create a synthetic right image using 3dsteroid pro or stereophoto maker. Basically, what I am gonna be doing here is see if I can recover the original depth map from the left image and a synthetic right image.

The dimensions of the original stereo pair are 3024x4032. I reduced the dimensions to 1800x2400 so that DMAG9b would run faster. The only reason I ran ER9b was to get the min and max disparities. It looks like the original stereo pair was very well aligned. Note that because the baseline is so small, you don't want to reduce the image size too much otherwise you are going to get a depth map with few depth levels (shades of gray) as far as DMAG5 and DMAG5b are concerned. Note that the number of depth levels is equal to the difference between the min and max disparities. So, for example, if the min disparity is -44 and the max disparity is 10, you are gonna get 55 depth levels (shades of gray) in the depth map produced by DMAG5 or DMAG5b. Something to consider.

Now, let's run DMAG5 using the following input file:

image 1 = ../er9b/image_l.png
image 2 = ../er9b/image_r.png
min disparity for image 1 = -44
max disparity for image 1 = 10
disparity map for image 1 = depthmap_l.png
disparity map for image 2 = depthmap_r.png
occluded pixel map for image 1 = occmap_l.png
occluded pixel map for image 2 = occmap_r.png
radius = 16
alpha = 0.9
truncation (color) = 30
truncation (gradient) = 10
epsilon = 255^2*10^-4
disparity tolerance = 0
radius to smooth occlusions = 9
sigma_space = 9
sigma_color = 25.5
downsampling factor = 2

I believe those are the default values in StereoPhoto Maker.


Left depth map obtained by DMAG5.

If you want to experiment, you could change the value for the radius. Maybe try 8 or 32 instead of 16 and see what happens. Also, you may want to change the downsampling factor to 1 instead of 2. It will take longer but you will get more levels of depth in the depth map (shades of gray).

Let's run DMAG9b using the following input file:

reference image = ../../er9b/image_l.png
input disparity map = ../depthmap_l.png
sample_rate_spatial = 32
sample_rate_range = 8
lambda = 0.25
hash_table_size = 100000
nbr of iterations (linear solver) = 25
sigma_gm = 1
nbr of iterations (irls) = 32
radius (confidence map) = 12
gamma proximity (confidence map) = 12
gamma color similarity (confidence map) = 12
sigma (confidence map) = 2
output depth map image = depthmap_l_dmag9b.png

I believe those are the default in StereoPhoto Maker except for sigma. Here, I am using sigma = 2.0, SPM uses 32.0. I don't think it matters much to be honest. Recall that the lower the sigma, the less confidence is given to the depth in the input depth map.


Confidence map. White means very confident in input depth, black means little confidence. Since sigma is relative low, the black streaks (poor confidence) are quite prominent.


Depth map produced by DMAG9b.

Let's change sigma from 2.0 to 32.0 and run DMAG9b using the following input file:

reference image = ../../er9b/image_l.png
input disparity map = ../depthmap_l.png
sample_rate_spatial = 32
sample_rate_range = 8
lambda = 0.25
hash_table_size = 100000
nbr of iterations (linear solver) = 25
sigma_gm = 1
nbr of iterations (irls) = 32
radius (confidence map) = 12
gamma proximity (confidence map) = 12
gamma color similarity (confidence map) = 12
sigma (confidence map) = 32
output depth map image = depthmap_l_dmag9b.png


Confidence map. Since sigma is relatively high, the black streaks (poor confidence) are pretty narrow.


Depth map produced by DMAG9b.

Not a whole lot of difference so I am gonna continue with sigma = 2.0. Let's change sample_rate_spatial from 32 to 16 and run DMAG9b using the following input file:

reference image = ../../er9b/image_l.png
input disparity map = ../depthmap_l.png
sample_rate_spatial = 16
sample_rate_range = 8
lambda = 0.25
hash_table_size = 100000
nbr of iterations (linear solver) = 25
sigma_gm = 1
nbr of iterations (irls) = 32
radius (confidence map) = 12
gamma proximity (confidence map) = 12
gamma color similarity (confidence map) = 12
sigma (confidence map) = 2
output depth map image = depthmap_l_dmag9b.png


Depth map produced by DMAG9b.

I think it's a bit better so let's continue the trend and change sample_rate_spatial from 16 to 8. Let's run DMAG9b using the following input file:

reference image = ../../er9b/image_l.png
input disparity map = ../depthmap_l.png
sample_rate_spatial = 8
sample_rate_range = 8
lambda = 0.25
hash_table_size = 100000
nbr of iterations (linear solver) = 25
sigma_gm = 1
nbr of iterations (irls) = 32
radius (confidence map) = 12
gamma proximity (confidence map) = 12
gamma color similarity (confidence map) = 12
sigma (confidence map) = 2
output depth map image = depthmap_l_dmag9b.png


Depth map produced by DMAG9b.

I think I hit the sweet spot so I am gonna stop here. Note that as sample_rate_spatial goes down, the cpu time for DMAG9b goes up.

Because the interocular distance is small, it can be worthwhile to use DMAG5b instead of DMAG5 to get the initial depth map. DMAG5b is a very simple algorithm but it will not perform well at object boundaries if the baseline used to take the stereo pair was (relatively) large. Here, it should perform ok since the pair was taken with an iphone with dual cameras.

Let's run DMAG5b using the following input file:


Depth map produced by DMAG5b.

The depth map produced by DMAG5b is actually better (I think) than the depth map produced by DMAG5. In this particular case. Personally, I would stop here and not even bother with DMAG9b but let's see how the best DMAG5/DMAG9b combo (as seen right above) compares with DMAG5b/DMAG9b.

Let's try to improve this depth map using DMAG9b and the following input file (same as the one right above):

reference image = ../../er9b/image_l.png
input disparity map = ../depthmap_l.png
sample_rate_spatial = 8
sample_rate_range = 8
lambda = 0.25
hash_table_size = 100000
nbr of iterations (linear solver) = 25
sigma_gm = 1
nbr of iterations (irls) = 32
radius (confidence map) = 12
gamma proximity (confidence map) = 12
gamma color similarity (confidence map) = 12
sigma (confidence map) = 2
output depth map image = depthmap_l_dmag9b.png


Depth map produced by DMAG9b.

Here, it does not really matter how the initial depth map was obtained as DMAG9b is quite aggressive. To make DMAG9b less aggressive, lambda is probably the parameter to change. The lower lambda is, the less aggressive DMAG9b is going to be.

Thursday, May 9, 2019

2d to 3d conversion - Great White

This post is an example that shows how to use 2d to 3d Image Conversion - The 3d Converter to create a depth map semi-automatically. I have uploaded on dropbox the gimp file which contains all the layers and the paths: great_white.xcf. See 2d to 3d Image Conversion - The 3d Converter for how to use "The 3d Converter".

This is what the3dconverter_input.txt looks like:
reference_rgb_image.png
sparse_depthmap_rgba_image.png
dense_depthmap_image.png
gimp_paths.svg
ignored_gradient_rgba_image.png
emphasized_gradient_rgba_image.png
edge_rgba_image.png
0.0

All you need is a reference image (layer reference_rgb_image in great_white.xcf saved as reference_rgb_image.png), an "edge image" (layer edge_rgba_image in great_white.xcf saved as edge_rgba_image.png), a sparse depth map (layer sparse_depthmap_rgba_image in great_white.xcf saved as sparse_depthmap_rgba_image.png), and a bunch of equal_depth and relative_depth paths (saved as gimp_paths.svg). This is all done in Gimp and you can get to those by downloading great_white.xcf.

Don't worry about ignored_gradient_rgba_image.png and emphasized_gradient_rgba_image.png as those are not used and therefore don't need to exist.


Reference image aka reference_rgb_image.png.


Reference image, sparse depth map, edge image, and gimp paths as seen in great_white.xcf.

The sparse depth map consists of a white blob for the tip of the nose and a black scribble to denote the background. I know it is a bit weird to relegate the water to the background but I don't see any other way to do it. Still, it kinda places the shark in front of the water. Weird, right. For the edge image, I simply traced the outline of the shark using the pencil tool (keeping the shift key pressed so that the line segments are always straight) with the smallest possible hard brush. The purpose of the edge image is to prevent the depths to bleed across object boundaries. The gimp paths are shown in blue. The ones that kinda look like half circles are the equal_depth paths. The ones that kinda connect the equal_depth paths together are the relative_depth paths. What is cool about using gimp paths is that it is very easy to modify them, in particular, it is very easy to change the relative depths between equal_depth paths as all you have to do is rename the relative_depth paths.


Gimp paths: equal_depth and relative_depth paths.

The +XX in the name of a relative_depth path indicates the relative depth between the beginning and end of the path. So, if you need to change the relative depth, you just need to change the XX in the name of the path.


Dense depth map produced by The 3d Converter.


Wiggle/wobble created using depthy.me.


Wiggle/wobble created using wigglemaker.

Check the 3D effect on Facebook (no need to be registered or logged in into Facebook): Facebook 3D photo. I have got to say that Facebook did an excellent job rendering depth maps with their 3D photos. Clap clap!

This is kinda a trial and error process. So, to help me in visualizing the dense depth map, I use Depth Player. I think it is a great tool. Note that the depth map doesn't have to be too accurate if you just want to post it as a Facebook 3D photo.

Video that kinda explains how to get the input files needed by The3dConverter:

Tuesday, April 23, 2019

Epipolar Rectification 9c (ER9c)

ER9c is a variant of Epipolar Rectification 9b (ER9b). ER9c is far less aggressive as ER9b in rectifying images, so if you use ER9b and see a lot of distortion in the rectified images (when compared to the original images), I invite you to use ER9c instead.

In the ugosoft3d-9-x64.rar which you can download at the 3D Software page, you will find the manual for ER9c (er9c_manual.pdf) and a test case for ER9c under er9c_test.

Here is an example:


Input left and right images.

You can clearly see in this animated gif that switches between left and right image that the pair is not aligned. There is quite a bit of rotation, which is not good for automatic depth map generation.


Output left and right images.

The output left and right images are now aligned. This is good news for the automatic depth map generator that will generate the depth maps.

It should be noted that ER9c like ER9b gives the minimum and maximum disparity of the rectified stereo pair in the console window printout. These disparities can be used as input to the automatic depth map generators that are available here for download. Unlike ER9, there's no need to manipulate those values.

The windows executable (guaranteed to be virus free) is available for free via the 3D Software Page.

Monday, April 22, 2019

2d to 3d conversion - Bruce Lee

This post is an example that shows how to use 2d to 3d Image Conversion - The 3d Converter to create a depth map semi-automatically. I have uploaded on dropbox the gimp file which contains all the layers and the paths: bruce_lee.xcf. See 2d to 3d Image Conversion - The 3d Converter for how to use "The 3d Converter".

This is what the3dconverter_input.txt looks like:
reference_rgb_image.png
sparse_depthmap_rgba_image.png
dense_depthmap_image.png
gimp_paths.svg
ignored_gradient_rgba_image.png
emphasized_gradient_rgba_image.png
edge_rgba_image.png
0.0

All you need is a reference image (layer reference_rgb_image saved as reference_rgb_image.png), an "edge image" (layer edge_rgba_image saved as edge_rgba_image.png), a sparse depth map (layer sparse_depthmap_rgba_image saved as sparse_depthmap_rgba_image.png), and a bunch of equal_depth and relative_depth paths (saved as gimp_paths.svg).

Don't worry about ignored_gradient_rgba_image.png and emphasized_gradient_rgba_image.png as those are not used and therefore don't need to exist.


Reference image.


Dense depth map produced by The 3d Converter.


Wiggle/wobble created by Wiggle Maker.

Sunday, April 21, 2019

2d to 3d conversion - The Shawshank Redemption

This post is an example that shows how to use 2d to 3d Image Conversion - The 3d Converter to create a depth map semi-automatically. I have uploaded on dropbox the gimp file which contains all the layers and the paths: shawshank.xcf. See 2d to 3d Image Conversion - The 3d Converter for how to use "The 3d Converter".

Step 1:

Create an "edge image" (as a layer) using the "Pencil Tool" (using smallest possible hard-edge brush). All you have to do is trace along object borders (depth discontinuities). Depths in the dense depth map will not propagate across depth discontinuities symbolized by the edges in the "edge image".

Step 2:

Create a sparse depth map. Here, I simply put one white dot to indicate where the depth of my scene begins (going from foreground to background). You could put more depth clues in the sparse depth map but I find it easier to control depth clues using gimp paths.

Step 3:

Create equal_depth and relative_depth paths. In the case of an equal_depth path, all points in the path will have the same depth. Note that you can have a single path made up of multiple components. In the case of a relative_depth path, if the depth variation is positive, the 2nd point is farther in the scene by an amount corresponding to the depth variation. If the depth variation is negative, the 2nd point is closer. Note that a relative_depth path can only be defined by 2 points. I routinely run "The 3d Converter" with what I currently have and use Depth Player to check the produced depth map. If it's not too good, I go back to my gimp paths, fix them up, and re-run "The 3d Converter". I do this back and forth dance until the depth map looks good enough (doesn't have to be perfect). Note that I did some modeling for the prison warden that's peeking into the hole in the wall but I am not even sure you need to do that (for a wiggle/wobble). Do not spend any time modeling stuff that's not in the foreground (the other 2 guys peeking in), just model them as if they were cardboard props (meaning, give them a single uniform depth). That's what view-master did with their cartoon reels.

Step 4:

Once you are happy with how the 3d scene looks in "Depth Player" (doesn't have to be perfect), you can use Wiggle Maker to create a wiggle/wobble for sharing on social media.


Reference image.


Dense depth map obtained with "The 3d Converter".


Wiggle/wobble obtained with "Wiggle Maker".

Saturday, April 20, 2019

Painting Software - The Scribbler

"The Scribbler" simulates the scribbling effect where one kinda draws circles following a tracing path. This is an implementation of "Tone- and Feature-Aware Circular Scribble Art" by Chun-Chia Chiu, Yi-Hsiang Lo, Ruen-Rone Lee and Hung-Kuo Chu.

The software (for the windows 64 bit operating system) is available to download on dropbox at thescribbler-x64.rar.

After having extracted the archive, I invite you to go to the "jimi_hendrix" folder. There you will see a file named "thescribbler.bat" which needs to be edited (using notepad or wordpad or whatever editor you like) so that the path to the executable, thescribbler.exe, is the correct one for your installation. The software is run by double-clicking on "thescribbler.bat". The output from The Scribbler is an image showing black scribbles over a white background.

The Scribbler's behavior is controlled by the input file named "thescribbler_input.txt". The following shows what's in "thescribbler_input.txt" in the "jimi_hendrix" directory:

source image = source_image.png
tracing path points = sample.tsp
tracing path segments = sample.cyc
minimum disk radius = 2
maximum disk radius = 16
minimum center velocity = 0.2
maximum center velocity = 8
scribble rotation increment = 80
output image = output_image.png

The source image is the reference photograph. It can be either tiff, png, or jpg. It can be either grayscale or full color.

The file containing the tracing path points (sample.tsp) is the output from Painting Software - The Stippler.

The file containing the tracing path segments (sample.cyc) is the output from linkern. See Painting Software - The Path Maker.

The minimum disk radius, maximum disk radius, minimum center velocity, and maximum center velocity control the density of the scribbles. Imagine following the tracing path (the one generated by Painting Software - The Path Maker) and drawing circles without lifting the pen. Clearly, you can draw small or large circles. The minimum disk radius and maximum disk radius control that. About as clearly, you can go slow or fast drawing those circles. The minimum center velocity and maximum center velocity control that.

The scribble rotation increment is a bit hard to explain, to be honest. It's an angle in degrees that should be between 0 and 360. I guess you'll just have to try changing it and see what happens. I think that values between 20 and 80 are good values to try out.

The output image contains the scribbles in image form. It can be either tiff, png, or jpg.


Source image.


Output image.

Painting Software - The Path Maker

"The Path Maker" takes in a stippled image (produced by Painting Software - The Stippler) and outputs an image showing the shortest path that connects all these stipples. The path solves (approximately) the Traveling Salesman Problem (TSP), that is, it is the shortest possible route that visits each city (stipple point) and returns to the origin city (stipple point). At some point somebody figured out that solving the TSP on a stippled image creates a cool effect, mainly, because there is supposed to be no intersection. That's how "TSP art" was born.

Now, the crux of the problem is the (approximate) solution of the Traveling Salesman Problem (TSP) given a set of points (in our case, a whole bunch of stipples). I didn't code that part (it's a tough one) but the guys at Concorde TSP Solver did, and very well. Linkern is an implementation of the Chained-Lin-Kernighan heuristic for the TSP. The Path Maker uses the cycle/tour generated by linkern to produce the output image.

The software (for the windows 64 bit operating system) is available to download on dropbox at thepathmaker-x64.rar. It contains both linkern and The Path Maker. Concorde's linkern is used to get the tracing path as a cycle/tour (in text file form) and The Path Maker is used only to get the output image representing that tracing path.

After having extracted the archive, I invite you to go to the "jimi_hendrix" folder. There you will see a file named "linkern.bat" which needs to be edited (using notepad or wordpad or whatever editor you like) so that the path to the executable, linkern.exe, is the correct one for your installation. The software is run by double-clicking on "linkern.bat". The output from linkern (sample.cyc) is a tour/cycle of the points contained in the file sample.tsp. The file sample.tsp is the output from Painting Software - The Stippler. In the "jimi_hendrix" directory, there is also a file named "thepathmaker.bat" which needs to be edited (using notepad or wordpad or whatever editor you like) so that the path to the executable, thepathmaker.exe, is the correct one for your installation. The software is run by double-clicking on "thepathmaker.bat". The output from The Path Maker is an image showing the cycle/tour in image form.

The Path Maker's behavior is controlled by the input file named "thepathmaker_input.txt". The following shows what's in "thepathmaker_input.txt" in the "jimi_hendrix" directory:

source image = source_image.png
tracing path points = sample.tsp
tracing path segments = sample.cyc
output image = output_image.png
dot radius = 1
dot radius 2 (antialiasing) = 3

The source image is the reference photograph. It can be either tiff, png, or jpg. It can be either grayscale or full color. It should be the same one you used when running Painting Software - The Stippler.

The file containing the tracing path points (sample.tsp) is the output from Painting Software - The Stippler.

The file containing the tracing path segments (sample.cyc) is the output from linkern.

The output image contains the tracing path (black over white). It can be either tiff, png, or jpg.

Dot radius controls the size of the stipple dots. Dot radius 2 controls the amount of antialiasing. Dot radius 2 has to be greater than or equal to dot radius. Note that if dot radius 2 is equal to dot radius, there is no antialiasing in the output image.


Source image.


Output image.

Because Concorde can not be used for commercial purposes (actually, it's supposed to be used for academic research only), The Path Maker cannot be used for commercial purposes either. At some point, if there is a lot of interest for TSP art (which I doubt very much), I might implement my own version of an approximate TSP solver (based on the Lin-Kernighan TSP heuristic) but not at the moment as linkern does an excellent job.

Friday, April 19, 2019

Painting Software - The Stippler

"The Stippler" simulates the stippling painting effect where only relatively small black dots are used to render a grayscale image which hopefully is a realistic representation of something, in our case, a photograph. This is an implementation of "Adaptive Incremental Stippling using the Poisson-Disk Distribution" by Ignacio Ascencio-Lopez, Oscar Meruvia-Pastor, and Hugo Hidalgo-Silva.

The software (for the windows 64 bit operating system) is available to download on dropbox at thestippler-x64.rar.

After having extracted the archive, I invite you to go to the "jimi_hendrix" folder. There you will see a file named "thestippler.bat" which needs to be edited (using notepad or wordpad or whatever editor you like) so that the path to the executable, thestippler.exe, is the correct one for your installation. The software is run by double-clicking on "thestippler.bat". The output from The Stippler is an image showing the stipples as black dots on a white background.

The Stippler's behavior is controlled by the input file named "thestippler_input.txt". The following shows what's in "thestippler_input.txt" in the "jimi_hendrix" directory:

source image = source_image.png
minimum disk radius = 4
maximum disk radius = 16
output image = sample.png
output file = sample.tsp
dot radius = 1
dot radius 2 (antialiasing) = 3

The source image is the reference photograph. It can be either tiff, png, or jpg. It can be either grayscale or full color.

The minimum and maximum disk radii control the density of the stipples.

The output image contains the stipples in image form (black dots over white background). It can be either tiff, png, or jpg.

The output file contains the stipples in text form. This output file is the one that's used by "The Path Maker".

Dot radius controls the size of the stipple dots. Dot radius 2 controls the amount of antialiasing. Dot radius 2 has to be greater than or equal to dot radius. Note that if dot radius 2 is equal to dot radius, there is no antialiasing in the output image.


Source image.


Output image.

Sunday, February 24, 2019

2d to 3d Image Conversion Software - The 3d Converter

"The 3d Converter" aka the3dconverter is an attempt at making easier the conversion of a 2d image into a 3d image via the creation of a dense depthmap. I know there is already dmag4 and dmag11 that can do that but the process is not ideal as it is quite difficult to create/edit the sparse depthmap. I think it is much easier to limit the depth clues given in the sparse depthmap to a strict minimum and use equal depth constraints as well as relative depth constraints to control the depths. It is much easier to modify equal depth constraints and relative depth constraints than to modify the sparse depthmap.

The idea for the3dconverter came from "Deep Annotations: Designing Depth of a Single Image for Depth-based Effects" by Jingtang Liao, Shuheng Shen, Elmar Eisemann. I liked some of their ideas but not all. I added my own as well based on my experience converting 2d images into 3d using dmag4 and dmag11.

Now, I don't have the time nor the know-how to create a GUI for the3dconverter so I am gonna rely on gimp to handle the equal depth constraints and relative depth constraint using gimp paths.

After you have extracted the3dconverter-x64.rar (windows 64 bit), you should see a directory called "blade_runner". It contains everything you need to check that the software is running correctly on your machine (windows 64 bit). The file called "the3dconverter.bat" is the one you double-click to run the software. It tells windows where the executable is. You need to edit the file so that the location of the executable is the correct one for your installation. I use notepad++ to edit text files but I am sure you can use notepad or wordpad with equal success. If you want to use the3dconverter on your own photograph, create a directory alongside "blade_runner" (in another words, at the same level as "blade_runner"), copy "the3dconverter.bat" and "the3dconverter_input.txt" from "blade_runner" to that new directory and double-click "the3dconverter.bat" after you have properly created all the input files the3dconverter needs.

The file "no-2--blade-runner-1982_640.xcf" is a gimp file that contains the reference rgb image, the sparse depthmap rgba image, the gimp paths for equal depth constraints and relative depth constraints, and the edge rgba image. It contains all the information that is needed by the3dconverter.

The input file "the3dconverter_input.txt" contains the following lines:
no-2--blade-runner-1982_640.jpg
sparse_depthmap_rgba_image.png
dense_depthmap_image.png
gimp_paths.svg
ignored_gradient_rgba_image.png
emphasized_gradient_rgba_image.png
edge_rgba_image.png
0.0

Line 1 contains the name of the reference rgb image. The file must exist.

Line 2 contains the name of the sparse depthmap rgba image. The file must exist.

Line 3 contains the name of the dense depthmap image. The file will be created by the3dconverter.

Line 4 contains the name of the gimp paths for equal depth constraints and relative depth constraints. If the file does not exist, the3dconverter assumes there are no equal depth constraints and no relative depth constraints.

Line 5 contains the name of the ignored-gradient rgba image. If the file doesn't exist, the3dconverter assumes there are no regions where the reference image gradients should be ignored.

Line 6 contains the name of the emphasized-gradient rgba image. If the file doesn't exist, the3dconverter assumes there are no regions where the reference image gradients should be emphasized.

Line 7 contains the name of the edge rgba image. If the file doesn't exist, the3dconverter assumes there is no edge image.

Line 8 contains the value for beta.

The reference rgb image is of course the image for which a dense depthmap is to be created. It is recommended not to use mega-large images as it may put an heavy strain on the solver (memory-wise). I always recommend starting small. Here, the reference rgb image is only 640 pixels wide. If the goal is to create 3d wiggles/wobbles, it is more than enough. If you see that your machine can handle it (it doesn't take too long to solve and the memory usage is reasonable), then you can switch to larger reference rgb images.


Reference rgb image.

The sparse depthmap rgba image is created just like with dmag4 or dmag11. You use the "Pencil Tool" (no anti-aliasing) to draw depth clues on a transparent layer. Depths can vary from pure black (deepest background) to pure white (closest foreground). When you zoom, you should see either fully transparent pixels or fully opaque pixels because, hopefully, all the tools you have used do not create anti-aliasing. If, for some reason, there are semi-transparent pixels in your sparse depthmap, click on Layer->Transparency->Threshold Alpha and then press Ok.


Sparse depthmap rgba image.


Sparse depthmap rgba image (zoom).

Gimp paths are used to impose soft constraints on pixel depths. You can use gimp paths to indicate that one pixel should have the same depth as another pixel (equal_depth paths) or that one pixel should have the same depth as another pixel plus or minus some (integer) value (relative_depth paths). If you don't give gimp paths (no gimp paths file present), the3dconverter relies solely on the sparse depth map (and possibly the edge image if you provide one) to generate the dense depth map and behaves just like Depth Map Automatic Generator 4 (DMAG4).

What I like to do when converting an image from 2d is to 3d is create a whole bunch of equal_depth paths which I then connect to each other with relative_depth paths. If you consider two equal_depth paths connected by a relative_depth path, the3dconverter will more or less create a gradient fill between the two equal_depth paths. If you look at 2d to 3d conversion - Great White, I created a bunch of equal_depth paths along the shark's body which I then connected with relative_depth paths. Because you can control the depth between each equal_depth path (via the name of the relative_depth path that connects them), you can simulate non-linear gradient fills, here, along the shark's body.

In the case of an equal_depth gimp path, for each path segment, the depth at the segment starting pixel should be equal to the depth at the segment ending pixel. In other words, we want the depth at the two segment ends to be equal. An equal_depth gimp path may have as many segments as desired. The order in which you create the anchor points does not matter at all. The name given to an equal_depth gimp path must contain "equal_depth".

If an equal_depth gimp path is made up of several components (you used the shift + left click combo or you merged gimp paths), the components are considered (by The 3d Converter) to be independent of each other. For example, if an equal_depth gimp path has two components, The 3d Converter does as if the two components were created as separate equal_depth gimp paths. So, for the two components to be actually constrained to be at the same depth, an additional equal_depth gimp path should be created to link the two. Of course, if you don't intend to create equal_depth gimp paths with multiple components, you do not have to worry about this.

In the case of a relative_depth gimp path, the depth at the starting pixel should be equal to the depth at the ending pixel plus or minus the depth variation. A relative_depth gimp path should only have one segment. The order in which the anchor points are created is important since the sign of the depth variation ("+" or "-") depends on it. The name given to a relative_depth gimp path must contain "relative_depth". After "relative_depth", there should be a space and then either "+" or "-" giving the sign of the depth variation. After "+" or "-", there should be an integer value indicating the depth variation (in grayscale value, that is, between 1 and 255). If the depth variation sign is "+", the starting pixel is supposed to be in front of the ending pixel. If the depth variation sign is "-", the starting pixel is supposed to be behind the ending pixel. Recall that grayscale values in a depthmap vary from 0 (pure black), furthest in the background, to 255 (pure white), closest in the foreground. Keep that in mind when you select the depth variations of your relative_depth gimp paths. What is cool about using the name of the relative_depth gimp path to indicate the depth variation is that it is very easy to change: double-click on the name and change the depth variation.


Gimp paths.

In the picture above, you can see the reference image, the sparse depthmap image, the edge image, and all the gimp paths used for the equal depth constraints and the relative depth constraints. The gimp paths are blue except the selected one which is shown in red. Any gimp path with more than one segment is always an equal_depth path.


Gimp paths as seen in the Paths dialog window.

The picture above shows the naming conventions for the gimp paths. It should be noted that gimp automatically gives different numbers to paths that are given the same name. That's where the numbers after the "#" sign come from. The path name "relative_depth +15 #1" indicates that the path is a relative_depth path and that the depth at the starting segment pixel should be equal to the depth at the ending segment pixel plus 15 (the starting segment pixel should be more in the foreground than the ending segment pixel by an amount of 15). The suffix "#1" was added automatically by gimp because there is already another path called "relative_depth +15". After I created that particular path (using the " Paths Tool"), gimp automatically named it "Unnamed". I double-clicked the path name, changed it to "relative_depth +15" and pressed return. Gimp automatically changed the name to "relative_depth +15 #1".

When you are working with gimp paths, you should always have the Paths dialog window within easy reach. To get to it, click on Windows->Dockable Dialogs->Paths. Once you have created a gimp path and are done with it, it will show up in the Paths dialog window as "Unnamed" but will not be visible. To make it visible, just do the same thing you would do for layers (click to the left of the name until the eye shows up).

To save the gimp paths, right-click the active path and select "Export Gimp Path". Make sure to select "Export all paths from this image" in the drop-down menu before saving. This will save all gimp paths into a single SVG file, eg, "gimp_paths.svg".

It's probably a good idea at this point to explain how to create/edit paths in gimp:

To create a new path, click on "Paths Tool". In the " Tool Options", check the " Polygonal" box as we will only need to create paths that are polygonal (no Bezier curves needed here). Click (left) to create the first anchor.

To add an anchor, click (left). To close a path (you really don't need to do that here but it doesn't hurt to know), press CTRL and hover the cursor over the starting anchor and click (left).

To start a new path component (you really don't need to do that here but it doesn't hurt to know), pres SHIFT and click (left).

To move an anchor, hover the cursor over an existing anchor, click (left) and drag.

To insert an anchor, press CTRL and click (left) anywhere on the segment betwen two existing anchors.

To delete an anchor, press SHIFT+CTRL and click (left) on the anchor you want to delete.

The ignored-gradient rgba image tells the3dconverter not to look at the reference image gradients (when diffusing the depths) for any pixel that is not transparent in the ignored-gradient image. If you use an edge image, you do not need to provide an ignored-gradient rgba image.

The emphasized-gradient rgba image tells the3dconverter to emphasize the reference image gradients (when diffusing the depths) for any pixel that is not transparent in the emphasized-gradient image. If you use and edge image, you do not need to provide an emphasize-gradient rgba image.

The edge rgba image tells the3dconverted not to diffuse the depths past the pixels that are not transparent in the edge image. It is similar to the edge image used in dmag4. To create an edge rgba image, create a new layer and trace along the object boundaries in the reference image. An object boundary is basically a boundary between two objects at different depths. To trace the edge image, I recommend using the " Pencil Tool" with a hard brush (no anti-aliasing) of size 1 (smallest possible). Press SHIFT and click (left) to draw line segments. I usually use the color "red" for my edge images but you can use whatever color you want. Make sure to always use tools that are not anti-aliased! If you need to use the "Eraser Tool", check the "Hard Edge" in the tool options. When you zoom on your edge image, it should look like crisp staircases.


Edge rgba image.


Edge rgba image (zoom).

The parameter beta controls the diffusion process. The larger the beta, the more difficult it will be for the depths to diffuse when encountering variations in the reference image gradients. If beta is set to 0.0, the depths will diffuse no matter what the reference image gradients are. If beta is relatively large, the depths will have a hard time diffusing whenever there are color changes in the reference image even if those color changes are minute. The parameter beta can be set to 0.0 if an edge image is used as diffusion will always be stopped at the edge image.

I don't generally use ignored-gradient or emphasized-gradient rgba images since I prefer using an edge rgba image. This is why ignored_gradient_rgba_image.png and emphasized_gradient_rgba_image.png do not actually exist in the "blade_runner" directory.

Once you are done, simply save the layer containing the sparse depthmap as sparse_depthmap_rgba_image.png, the layer containing the edge image as edge_rgba_image.png, and the gimp paths as gimp_paths.svg. Double-click on "the3dconverter.bat" and the3dconverter should spit out the dense depthmap in no time as dense_depthmap_image.png assuming nothing went wrong.


Dense depthmap image created by the3dconverter.

To see if the dense depthmap is good, you can use Depth Player. To create a wiggle/wobble, you can use Wiggle Maker. If your goal is solely to create wiggles, the depth map really doesn't need to be too accurate and depths not in the immediate foreground really do not have to be too accurate (only worry about the foreground objects).


Wiggle/wobble created by Wiggle Maker.

Any problems? Send me your .xcf gimp file (zipped)!

Video that kinda explains how to get the input files needed by The3dConverter: