Thursday, February 6, 2014

Depth Map Automatic Generator 4 (DMAG4)

DMAG4 is a semi-automatic 2D to 3D image converter. DMAG4 is an implementation of what's explained in Semi-Automatic 2D to 3D Image Conversion using Random Walks.

The sparse depth map (which the program requires) is a transparent image partially filled with brush strokes in various shades of gray to indicate depth. This is the non-automatic part. In Gimp or Photoshop, you open the reference image and create a new transparent layer on top of the reference image. This is where you are gonna draw the sparse depth map. The brush strokes should be done with the pencil tool using a 100% hard brush so that any painted pixel is fully opaque (no anti-aliasing). To verify that you are not generating semi-transparent pixels, you may want to zoom in on your painted brush strokes and look at the edges. They should be hard and look jagged (no anti-aliasing involved). Although semi-transparent pixels can be removed easily prior to exporting the sparse depth map, it is highly recommended to only use drawing tools with no anti-aliasing (no soft edges, only hard edges). After you are done scribbling with the crayon tool using various shades of gray to indicate approximate depths, save the layer (only the layer) in whatever format you want as long as it is lossless (png for example). To save the layer only, you can simply make the reference image not visible by clicking off the eye icon in the layer dialog box.

I think the default values for the parameters are quite good, but it does not hurt to understand how they behave.

The maximum number of iterations is a parameter used in the linear equation solver (which happens to be Lis, a Library of Iterative Solvers for Linear Systems). If you see that the depths don't seem to propagate correctly across areas of similar color (in the reference image), it's probably because the number of iterations is set too low.

The beta parameter is explained in Semi-Automatic 2D to 3D Image Conversion using Random Walks. The lower the beta, the more smooth the depth map is going to be although the more bleeding may occur across object boundaries.

The number of scales controls the size of the scale space. DMAG4 can build a scale space in order to mitigate the effects of noise or fine texture in the reference image (at the expense of respect of object boundaries). I think that 5 scales is probably the highest you will want to go. When the level of graph connection within a scale (con_level) is set to 1, each pixel is connected (by a graph edge) to four adjacent pixels (left, right, top, and bottom). If con_level is set to 2, it is connected to eight adjacent pixels (the four from level 1 and the four more situated diagonally). When the level of graph connection across scales (con-level2) is set to 1, each pixel is connected to one pixel on the previous and next scale. If it is set to 2, each pixel is connected to five pixels on the previous and next scale. If it is set to 3, each pixel is connected to nine pixels on the previous and next scale. The more connections, the more the depths get propagated. Using con_level = 1 and con_level2 = 1 is probably a good idea and there's really no need to change that.

In order to eliminate the bleeding of depths across objects of similar colors, it is possible to give a so-called edge image to DMAG4. An edge image is basically a trace over the object boundaries of the reference image. Just like for the sparse depth map, it is created by adding a transparent layer over the reference image and tracing (with the pencil tool using the smallest possible hard edge brush) over the object boundaries. The tracing on the edge image should be 1 pixel wide (in most places). The presence of an edge image enables to be less conservative regarding possible oversmoothing. Thus, beta can be lowered and the number of scales increased with less fear. Also, the sparse depth map probably doesn't need to be as dense. Note that the edge image does not need to be complete, that is, only areas where bleeding occurs can be traced over. If you do use one an edge image, make sure con_level and con_level2 are set to 1. To sum things up and be perfectly clear, DMAG4 does not need an edge image but if colors are similar between foreground and background, it is probably a good idea to create an edge image to avoid bleeding in some areas.

If the 32-bit version of the program crashes (which has been updated in a long while and may not have all the latest features), it's quite likely because it ran out of memory. In that case, the reference image should be reduced n size or you should really use the 64-bit version (always preferred in all cases).

If DMAG4 gives an error about the pixels being semi-transparent, it's because you probably used a tool that's anti-aliased (soft edges) at some point. To fix this in Gimp (Photoshop is probably similar), click on "Layer"->"Transparency"->"Threshold Alpha ...", and click OK. All your semi-transparent pixels are now either transparent or opaque and DMAG4 will thank you for it.

Here's an example without an edge image: 2D to 3D Conversion - Marilyn Monroe and Tom Ewell in The Seven Year Hitch.

Here's another example without an edge image: 2D to 3D Conversion - Marilyn Monroe in dance class.

Here's a video tutorial of the "Marilyn in dance class" conversion:


Here's an example with an edge image:


Edge image on top of reference image.


Sparse depth map on top of reference image.


Resulting dense depth map without using the edge image. Quite a bit of bleeding.


Resulting dense depth map using the edge image. No bleeding.

I think it might be a good idea to do a couple of experiments that demonstrate how DMAG4 works.

Experiment 1:


This shows both the reference 2d image (a yellowish ellipse in front of a bluish background) and the scribbles. Think of the yellow ellipse as an object in front of a blue wall.


This is the depth map that DMAG4 produces. This is the behavior that we want: propagation of the known depths (scribbled depths) only within similar colors. If objects in your 2d image are not clearly delineated (with a color change), there's a good chance there's gonna be some bleeding in the depth map. For example, between the dancers and the back wall in the "Marilyn in dance class" conversion video.

Experiment 2:


This show both the reference image (solid bluish color) and the scribbles (near white at the bottom and near black at the top).


This is the depth map that DMAG4 produces. This is the behavior that we want: smooth gradation between known depths (scribbled depths) within similar colors.

For more info on DMAG4, check 2D to 3D Image Conversion using DMAG4.

The windows executable (guaranteed to be virus free) is available for free via the 3D Software Page. Please, refer to the 'Help->About' page in the actual program for how to use it.

No comments:

Post a Comment