Thursday, February 6, 2014

Depth Map Automatic Generator 4 (DMAG4)

DMAG4 is a semi-automatic 2D to 3D image converter. DMAG4 is an implementation of what's explained in Semi-Automatic 2D to 3D Image Conversion using Random Walks.

The sparse depth map (which the program requires) is a transparent image partially filled with brush strokes in various shades of gray to indicate (approximate) depth. This is the non-automatic part. So, how do you create that sparse depth map? In Gimp or Photoshop, you open the reference image and create a new transparent layer on top of the reference image. This is where you are gonna draw the sparse depth map. The brush strokes should be done with the pencil tool using a 100% hard brush so that any painted pixel is fully opaque (no anti-aliasing). To verify that you are not generating semi-transparent pixels, you may want to zoom in on your painted brush strokes and look at the edges. They should be hard and look jagged (no anti-aliasing involved). Although semi-transparent pixels can be removed easily prior to exporting the sparse depth map, it is highly recommended to only use drawing tools with no anti-aliasing (no soft edges, only hard edges). Of course, you can use other tools to create the depth map, in particular, the selection and gradient tool. After you are done scribbling with the crayon tool using various shades of gray to indicate approximate depths, save the layer (only the layer) in whatever format you want as long as it is lossless (png for example). To save the layer only, you can simply make the reference image not visible by clicking off the eye icon in the layer dialog box.

I think the default values for the parameters are quite good, but it does not hurt to understand how they behave.

The maximum number of iterations is a parameter used in the linear equation solver (which happens to be Lis, a Library of Iterative Solvers for Linear Systems). If you see that the depths don't seem to propagate correctly across areas of similar color (in the reference image), it's probably because the number of iterations is set too low.

The beta parameter is explained in Semi-Automatic 2D to 3D Image Conversion using Random Walks. The lower the beta, the more smooth the depth map is going to be although the more bleeding may occur across object boundaries. In other words, the lower the beta, the easier it is for depths to propagate and the easier it is for depths to bleed across object boundaries. If an "edge image" is used, beta can be lowered quite a bit because bleeding is prevented by the existence of the "edge image". With a low beta, depths propagate very easily within an object as the bilateral filter (that's really what DMAG4 is) becomes a mere Gaussian filter.

The number of scales controls the size of the scale space. DMAG4 can build a scale space in order to mitigate the effects of noise or fine texture in the reference image (at the expense of respect of object boundaries). The more scales, the smoother the depth map is going to be, the more bleeding is going to occur, the longer it is going to take and the more memory is going to be used.

When the level of graph connection within a scale (con_level) is set to 1, each pixel is connected (by a graph edge) to four adjacent pixels (left, right, top, and bottom). If con_level is set to 2, it is connected to eight adjacent pixels (the four from level 1 and the four more situated diagonally). When the level of graph connection across scales (con-level2) is set to 1, each pixel is connected to one pixel on the previous and next scale. If it is set to 2, each pixel is connected to five pixels on the previous and next scale. If it is set to 3, each pixel is connected to nine pixels on the previous and next scale. The more connections, the more the depths get propagated. Using con_level = 1 and con_level2 = 1 is probably a good idea and there's really no need to change that.

In order to eliminate the bleeding of depths across objects of similar colors, it is possible to give a so-called "edge image" to DMAG4. An edge image is basically a trace over the object boundaries of the reference image. Just like for the sparse depth map, it is created by adding a transparent layer over the reference image and tracing over the object boundaries. The tracing on the edge image should be 1 pixel wide (in most places). The presence of an edge image enables to be less conservative regarding possible bleeding. Thus, beta can be lowered (say, from 90. to 10.) and the number of scales increased with less fear. Also, the sparse depth map probably doesn't need to be as dense. Note that the edge image does not need to be complete, that is, only areas where bleeding occurs can be traced over. If you do use one an edge image, make sure con_level and con_level2 are set to 1. To sum things up and be perfectly clear, DMAG4 does not need an edge image but if colors are similar between foreground and background, it is probably a good idea to create an edge image to avoid bleeding in some areas.

What is the best way to trace over object boundaries to create the "edge image"? In Gimp, I like to use the "Paths Tool" and draw a path along the object boundaries by left-clicking. When done drawing the path, click on "Stroke Path" in the "Tool Options" of the "Paths Tool", click on "Stroke line", click on "Solid color", uncheck the "Antialiasing" box, choose a "Line width" of 1.0 pixel, and click on "Stroke". When you zoom in on the stroke, it should 1 pixel wide with no antialiasing. The color of the stroke does not matter although I like to use red in most cases. It is very easy to do.

If the 32-bit version of the program crashes (which has been updated in a long while and may not have all the latest features), it's quite likely because it ran out of memory. In that case, the reference image should be reduced n size or you should really use the 64-bit version (always preferred in all cases).

If DMAG4 gives an error about the pixels being semi-transparent, it's because you probably used a tool that's anti-aliased (soft edges) at some point. To fix this in Gimp (Photoshop is probably similar), click on "Layer"->"Transparency"->"Threshold Alpha ...", and click OK. All your semi-transparent pixels are now either transparent or opaque and DMAG4 will thank you for it.

Here's an example without an "edge image":

Sparse depth map on top of reference image. This is a sparse depth map that would typically be used without an "edge image". If you use an edge image, you really don't need so many brush strokes in the sparse depth map (Check the "Interstellar" example below!)

Resulting dense depth map without using the "edge image". Quite a bit of bleeding, which is something the "edge image" helps avoid.

Here's an example with an "edge image":

Reference image.

Edge image.

To draw the edge image, I simply used the "Paths Tool" left-clicking all along the main subject.

Sparse depth map.

For the spare depth map, I used a gradient over the whole image which I then erased to reveal the main subject. Then, I drew as little as possible to suggest the depths on the main subject.

Dense depth map produced by DMAG4.

Here's a video tutorial for DMAG4 used without an "edge image":

Here's a video tutorial for DMAG4 used with an "edge image":

If you have any problem with DMAG4, feel free to send your reference image, sparse depth map, edge image (if any), and dense depth map (if you have gotten that far) to the email that should be somewhere in the right sidebar.

I think it might be a good idea to do a couple of experiments that demonstrate how DMAG4 works (no "edge image" involved).

Experiment 1:

This shows both the reference 2d image (a yellowish ellipse in front of a bluish background) and the scribbles. Think of the yellow ellipse as an object in front of a blue wall.

This is the depth map that DMAG4 produces. This is the behavior that we want: propagation of the known depths (scribbled depths) only within similar colors. If objects in your 2d image are not clearly delineated (with a color change), there's a good chance there's gonna be some bleeding in the depth map. For example, between the dancers and the back wall in the "Marilyn in dance class" conversion video.

Experiment 2:

This show both the reference image (solid bluish color) and the scribbles (near white at the bottom and near black at the top).

This is the depth map that DMAG4 produces. This is the behavior that we want: smooth gradation between known depths (scribbled depths) within similar colors.

For more info on DMAG4, check 2D to 3D Image Conversion using DMAG4.

The windows executable (guaranteed to be virus free) is available for free via the 3D Software Page. Please, refer to the 'Help->About' page in the actual program for how to use it.

No comments:

Post a Comment