2d to 3d Image Conversion Software - the3dconverter

The3dconverter is an attempt at making easier the conversion of a 2d image into a 3d image via the creation of a dense depthmap. I know there is already dmag4 and dmag11 that can do that but the process is not ideal as it is quite difficult to create/edit the sparse depthmap. I think it is much easier to limit the depth clues given in the sparse depthmap to a strict minimum and use equal depth constraints as well as relative depth constraints to control the depths. It is much easier to modify equal depth constraints and relative depth constraints than to modify the sparse depthmap.

The idea for the3dconverter came from "Deep Annotations: Designing Depth of a Single Image for Depth-based Effects" by Jingtang Liao, Shuheng Shen, Elmar Eisemann. I liked some of their ideas but not all. I added my own as well based on my experience converting 2d images into 3d using dmag4 and dmag11.

Now, I don't have the time nor the know-how to create a GUI for the3dconverter so I am gonna rely on gimp to handle the equal depth constraints and relative depth constraint using gimp paths.

After you have extracted the3dconverter-x64.rar (windows 64 bit), you should see a directory called "blade_runner". It contains everything you need to check that the software is running correctly on your machine (windows 64 bit). The file called "the3dconverter.bat" is the one you double-click to run the software. It tells windows where the executable is. You need to edit the file so that the location of the executable is the correct one for your installation. I use notepad++ to edit text files but I am sure you can use notepad or wordpad with equal success. If you want to use the3dconverter on your own photograph, create a directory alongside "blade_runner" (in another words, at the same level as "blade_runner"), copy "the3dconverter.bat" and "the3dconverter_input.txt" from "blade_runner" to that new directory and double-click "the3dconverter.bat" after you have properly created all the input files the3dconverter needs.

The file "no-2--blade-runner-1982_640.xcf" is a gimp file that contains the reference rgb image, the sparse depthmap rgba image, the gimp paths for equal depth constraints and relative depth constraints, and the edge rgba image. It contains all the information that is needed by the3dconverter.

The input file "the3dconverter_input.txt" contains the following lines:
no-2--blade-runner-1982_640.jpg
sparse_depthmap_rgba_image.png
dense_depthmap_image.png
gimp_paths.svg
ignored_gradient_rgba_image.png
emphasized_gradient_rgba_image.png
edge_rgba_image.png
0.0

Line 1 contains the name of the reference rgb image. The file must exist.

Line 2 contains the name of the sparse depthmap rgba image. The file must exist.

Line 3 contains the name of the dense depthmap image. The file will be created by the3dconverter.

Line 4 contains the name of the gimp paths for equal depth constraints and relative depth constraints. If the file does not exist, the3dconverter assumes there are no equal depth constraints and no relative depth constraints.

Line 5 contains the name of the ignored-gradient rgba image. If the file doesn't exist, the3dconverter assumes there are no regions where the reference image gradients should be ignored.

Line 6 contains the name of the emphasized-gradient rgba image. If the file doesn't exist, the3dconverter assumes there are no regions where the reference image gradients should be emphasized.

Line 7 contains the name of the edge rgba image. If the file doesn't exist, the3dconverter assumes there is no edge image.

Line 8 contains the value for beta.

The reference rgb image is of course the image for which a dense depthmap is to be created. It is recommended not to use mega-large images as it may put an heavy strain on the solver (memory-wise). I always recommend starting small. Here, the reference rgb image is only 640 pixels wide. If the goal is to create 3d wiggles/wobbles, it is more than enough. If you see that your machine can handle it (it doesn't take too long to solve and the memory usage is reasonable), then you can switch to larger reference rgb images.

Reference rgb image.

The sparse depthmap rgba image is created just like with dmag4 or dmag11. You use the "Pencil Tool" (no anti-aliasing) to draw depth clues on a transparent layer. Depths can vary from pure black (deepest background) to pure white (closest foreground). When you zoom, you should see either fully transparent pixels or fully opaque pixels because, hopefully, all the tools you have used do not create anti-aliasing. If, for some reason, there are semi-transparent pixels in your sparse depthmap, click on Layer->Transparency->Threshold Alpha and then press Ok.

Sparse depthmap rgba image.

Sparse depthmap rgba image (zoom).

Gimp paths are used to impose soft constraints on pixel depths. You can use gimp paths to indicate that one pixel should have the same depth as another pixel (equal_depth paths) or that one pixel should have the same depth as another pixel plus or minus some (integer) value (relative_depth paths). If you don't give gimp paths (no gimp paths file present), the3dconverter relies solely on the sparse depth map (and possibly the edge image if you provide one) to generate the dense depth map and behaves just like Depth Map Automatic Generator 4 (DMAG4).

What I like to do when converting an image from 2d is to 3d is create a whole bunch of equal_depth paths which I then connect to each other with relative_depth paths. If you consider two equal_depth paths connected by a relative_depth path, the3dconverter will more or less create a gradient fill between the two equal_depth paths. If you look at 2d to 3d conversion - Great White, I created a bunch of equal_depth paths along the shark's body which I then connected with relative_depth paths. Because you can control the depth between each equal_depth path (via the name of the relative_depth path that connects them), you can simulate non-linear gradient fills, here, along the shark's body.

In the case of an equal_depth gimp path, for each path segment, the depth at the segment starting pixel should be equal to the depth at the segment ending pixel. In other words, we want the depth at the two segment ends to be equal. An equal_depth gimp path may have as many segments as desired. The order in which you create the anchor points does not matter at all. The name given to an equal_depth gimp path must contain "equal_depth".

If an equal_depth gimp path is made up of several components (you used the shift + left click combo or you merged gimp paths), the components are considered (by the3dconverter) to be independent of each other. For example, if an equal_depth gimp path has two components, The3dconverter does as if the two components were created as separate equal_depth gimp paths. So, for the two components to be actually constrained to be at the same depth, an additional equal_depth gimp path should be created to link the two. Of course, if you don't intend to create equal_depth gimp paths with multiple components, you do not have to worry about this.

In the case of a relative_depth gimp path, the depth at the starting pixel should be equal to the depth at the ending pixel plus or minus the depth variation. A relative_depth gimp path should only have one segment. The order in which the anchor points are created is important since the sign of the depth variation ("+" or "-") depends on it. The name given to a relative_depth gimp path must contain "relative_depth". After "relative_depth", there should be a space and then either "+" or "-" giving the sign of the depth variation. After "+" or "-", there should be an integer value indicating the depth variation (in grayscale value, that is, between 1 and 255). If the depth variation sign is "+", the starting pixel is supposed to be in front of the ending pixel. If the depth variation sign is "-", the starting pixel is supposed to be behind the ending pixel. Recall that grayscale values in a depthmap vary from 0 (pure black), furthest in the background, to 255 (pure white), closest in the foreground. Keep that in mind when you select the depth variations of your relative_depth gimp paths. What is cool about using the name of the relative_depth gimp path to indicate the depth variation is that it is very easy to change: double-click on the name and change the depth variation.

Gimp paths.

In the picture above, you can see the reference image, the sparse depthmap image, the edge image, and all the gimp paths used for the equal depth constraints and the relative depth constraints. The gimp paths are blue except the selected one which is shown in red. Any gimp path with more than one segment is always an equal_depth path.

Gimp paths as seen in the Paths dialog window.

The picture above shows the naming conventions for the gimp paths. It should be noted that gimp automatically gives different numbers to paths that are given the same name. That's where the numbers after the "#" sign come from. The path name "relative_depth +15 #1" indicates that the path is a relative_depth path and that the depth at the starting segment pixel should be equal to the depth at the ending segment pixel plus 15 (the starting segment pixel should be more in the foreground than the ending segment pixel by an amount of 15). The suffix "#1" was added automatically by gimp because there is already another path called "relative_depth +15". After I created that particular path (using the " Paths Tool"), gimp automatically named it "Unnamed". I double-clicked the path name, changed it to "relative_depth +15" and pressed return. Gimp automatically changed the name to "relative_depth +15 #1".

When you are working with gimp paths, you should always have the Paths dialog window within easy reach. To get to it, click on Windows->Dockable Dialogs->Paths. Once you have created a gimp path and are done with it, it will show up in the Paths dialog window as "Unnamed" but will not be visible. To make it visible, just do the same thing you would do for layers (click to the left of the name until the eye shows up).

To save the gimp paths, right-click the active path and select "Export Gimp Path". Make sure to select "Export all paths from this image" in the drop-down menu before saving. This will save all gimp paths into a single SVG file, eg, "gimp_paths.svg".

It's probably a good idea at this point to explain how to create/edit paths in gimp:

To create a new path, click on "Paths Tool". In the " Tool Options", check the " Polygonal" box as we will only need to create paths that are polygonal (no Bezier curves needed here). Click (left) to create the first anchor.

To add an anchor, click (left). To close a path (you really don't need to do that here but it doesn't hurt to know), press CTRL and hover the cursor over the starting anchor and click (left).

To start a new path component (you really don't need to do that here but it doesn't hurt to know), pres SHIFT and click (left).

To move an anchor, hover the cursor over an existing anchor, click (left) and drag.

To insert an anchor, press CTRL and click (left) anywhere on the segment betwen two existing anchors.

To delete an anchor, press SHIFT+CTRL and click (left) on the anchor you want to delete.

The ignored-gradient rgba image tells the3dconverter not to look at the reference image gradients (when diffusing the depths) for any pixel that is not transparent in the ignored-gradient image. If you use an edge image, you do not need to provide an ignored-gradient rgba image.

The emphasized-gradient rgba image tells the3dconverter to emphasize the reference image gradients (when diffusing the depths) for any pixel that is not transparent in the emphasized-gradient image. If you use and edge image, you do not need to provide an emphasize-gradient rgba image.

The edge rgba image tells the3dconverted not to diffuse the depths past the pixels that are not transparent in the edge image. It is similar to the edge image used in dmag4. To create an edge rgba image, create a new layer and trace along the object boundaries in the reference image. An object boundary is basically a boundary between two objects at different depths. To trace the edge image, I recommend using the " Pencil Tool" with a hard brush (no anti-aliasing) of size 1 (smallest possible). Press SHIFT and click (left) to draw line segments. I usually use the color "red" for my edge images but you can use whatever color you want. Make sure to always use tools that are not anti-aliased! If you need to use the "Eraser Tool", check the "Hard Edge" in the tool options. When you zoom on your edge image, it should look like crisp staircases.

Edge rgba image.

Edge rgba image (zoom).

The parameter beta controls the diffusion process. The larger the beta, the more difficult it will be for the depths to diffuse when encountering variations in the reference image gradients. If beta is set to 0.0, the depths will diffuse no matter what the reference image gradients are. If beta is relatively large, the depths will have a hard time diffusing whenever there are color changes in the reference image even if those color changes are minute. The parameter beta can be set to 0.0 if an edge image is used as diffusion will always be stopped at the edge image.

I don't generally use ignored-gradient or emphasized-gradient rgba images since I prefer using an edge rgba image. This is why ignored_gradient_rgba_image.png and emphasized_gradient_rgba_image.png do not actually exist in the "blade_runner" directory.

Once you are done, simply save the layer containing the sparse depthmap as sparse_depthmap_rgba_image.png, the layer containing the edge image as edge_rgba_image.png, and the gimp paths as gimp_paths.svg. Double-click on "the3dconverter.bat" and the3dconverter should spit out the dense depthmap in no time as dense_depthmap_image.png assuming nothing went wrong.

Dense depthmap image created by the3dconverter.

To see if the dense depthmap is good, you can use Depth Player. To create a wiggle/wobble, you can use Wiggle Maker. If your goal is solely to create wiggles, the depth map really doesn't need to be too accurate and depths not in the immediate foreground really do not have to be too accurate (only worry about the foreground objects).

Wiggle/wobble created by Wiggle Maker.

Any problems? Send me your .xcf gimp file (zipped)!

Video that kinda explains how to get the input files needed by The3dConverter: