Thursday, February 6, 2014

Depth Map Automatic Generator 4 (DMAG4)

DMAG4 is a semi-automatic 2D to 3D image converter. DMAG4 is an implementation of what's explained in Semi-Automatic 2D to 3D Image Conversion using Random Walks.

The sparse depth map (which the program requires) is a transparent image partially filled with brush strokes in various shades of gray to indicate (approximate) depth. This is the non-automatic part. So, how do you create that sparse depth map? In Gimp or Photoshop, you open the reference image and create a new transparent layer on top of the reference image. This is where you are gonna draw the sparse depth map. The brush strokes should be done with the pencil tool using a 100% hard brush so that any painted pixel is fully opaque (no anti-aliasing). To verify that you are not generating semi-transparent pixels, you may want to zoom in on your painted brush strokes and look at the edges. They should be hard and look jagged (no anti-aliasing involved). In case semi-transparent pixels get created for whatever reason (maybe you used a tool with anti-aliasing turned on), they can be easily removed prior to saving the sparse depth map by doing the following (in Gimp): click on "Layer"->"Transparency"->"Threshold Alpha ...", and click OK. After you are done scribbling on the sparse depth map layer, save the layer (only the layer) in the png format (lossless and preserves transparency). To save the layer only, you can simply make the reference image not visible by clicking off the eye icon in the layer dialog box.

Let's go through the parameters DMAG4 uses. Then, we will introduce the notion of an "edge image" to simplify the process.

The maximum number of iterations is a parameter used in the linear equation solver (which happens to be Lis, a Library of Iterative Solvers for Linear Systems). If you see that the depths don't seem to propagate correctly across areas of similar color (in the reference image), it's probably because the number of iterations is set too low.

The beta parameter is explained in Semi-Automatic 2D to 3D Image Conversion using Random Walks. The lower the beta, the more smooth the depth map is going to be although the more bleeding may occur across object boundaries (if no "edge image" is used). In other words, the lower the beta, the easier it is for depths to propagate and the easier it is for depths to bleed across object boundaries (if no "edge image" is used). If an "edge image" is used, beta can be lowered quite a bit because bleeding is prevented by the existence of the "edge image". With a low beta, depths propagate very easily within an object as the bilateral filter (that's really what DMAG4 is) becomes a mere Gaussian filter.

The number of scales controls the size of the scale space. DMAG4 can build a scale space in order to mitigate the effects of noise or fine texture in the reference image (at the expense of respect of object boundaries). The more scales, the smoother the depth map is going to be, the more bleeding is going to occur, the longer it is going to take and the more memory is going to be used. If you use an "edge image", setting this parameter to 1 should be just fine.

When the level of graph connection within a scale (con_level) is set to 1, each pixel is connected (by a graph edge) to four adjacent pixels (left, right, top, and bottom). If con_level is set to 2, it is connected to eight adjacent pixels (the four from level 1 and the four more situated diagonally). When the level of graph connection across scales (con-level2) is set to 1, each pixel is connected to one pixel on the previous and next scale. If it is set to 2, each pixel is connected to five pixels on the previous and next scale. If it is set to 3, each pixel is connected to nine pixels on the previous and next scale. The more connections, the more the depths get propagated. Using con_level = 1 and con_level2 = 1 is probably a good idea and there's really no need to change that.

Now, let's talk about the "edge image" which is really supposed to make everything easier at not much of a cost for the user.

In order to eliminate the bleeding of depths across objects of similar colors, it is possible to give a so-called "edge image" to DMAG4. An edge image is basically a trace over the object boundaries of the reference image. Just like for the sparse depth map, it is created by adding a transparent layer over the reference image and tracing over the object boundaries. The tracing on the edge image should be 1 pixel wide (in most places). The presence of an edge image enables to be less conservative regarding possible bleeding. Thus, beta can be lowered (say, from 90. to 10.) and the number of scales increased with less fear. Also, the sparse depth map probably doesn't need to be as dense. Note that the edge image does not need to be complete, that is, only areas where bleeding occurs can be traced over. If you do use one an edge image, make sure con_level and con_level2 are set to 1. To sum things up and be perfectly clear, DMAG4 does not need an edge image but if colors are similar between foreground and background, it is probably a good idea to create an edge image to avoid bleeding in some areas.

What is the best way to trace over object boundaries to create the "edge image"? In Gimp, I like to use the "Paths Tool" and draw a path along the object boundaries by left-clicking. When done drawing the path, click on "Stroke Path" in the "Tool Options" of the "Paths Tool", click on "Stroke line", click on "Solid color", uncheck the "Antialiasing" box, choose a "Line width" of 1.0 pixel, and click on "Stroke". When you zoom in on the stroke, it should 1 pixel wide with no antialiasing. The color of the stroke does not matter although I like to use red in most cases. It is very easy to do.

An alternative for creating the "edge image" is to simply use the pencil tool with a hard brush (no anti-aliasing) and set the brush size to the minimum. Press the shift key and trace along the object boundaries to create the edge image. Pressing the shift key while left-clicking allows you draw straight line segments with the pencil tool.

If DMAG4 gives an error about the pixels being semi-transparent, it's because you probably used a tool that's anti-aliased (soft edges) at some point. To fix this in Gimp (Photoshop is probably similar), click on "Layer"->"Transparency"->"Threshold Alpha ...", and click OK. All your semi-transparent pixels are now either transparent or opaque and DMAG4 will thank you for it.

Here's a post that shows the process: 3D Image Conversion - Top Gun.

Here's a video tutorial for DMAG4 used with an "edge image" (I am so sorry there is no sound):


A few words to the wise now ... It is recommended not to use large images. I would not use images that are bigger than 1 mega-pixels. If you do use large images, the max number of iterations should be increased. The cpu time will increase though as a result. So keep your images small! If you use an "edge image", you should use a low beta, like beta = 10.

If you have any problem with DMAG4, feel free to send your reference image, sparse depth map, edge image (if any), and dense depth map (if you have gotten that far) to the email that should be somewhere in the right sidebar.

The windows executable (guaranteed to be virus free) is available for free via the 3D Software Page. Please, refer to the 'Help->About' page in the actual program for how to use it.

Source code: DMAG4 on github.

UPDATE (03/03/2020): I now recommend using 2d to 3d Image Conversion Software - the3dconverter2 for 2d to 3d conversion over DMAG4.

17 comments:

  1. It's 2019 and I've just gotten around to your extremely helpful website. I've tried SPM and DMAG5_9 folder route to DM's. But I think DMAG_4, this tutorial and the YouTubes are where one needs to begin in order to demystify depthmap creations. Thanks for everything!

    ReplyDelete
    Replies
    1. Thanks Tony. You use dmag4, dmag11, or "The 3dConverter" when you have a single reference image and you use dmag5 or dmag6 + dmag9b when you have a pair of images taken either by a stereo camera or by a mono camera (using cha-cha method).

      Delete
  2. very apprecaite for your work..could you please tell me how I can generate single image source to depth image ? request to share step and your application version

    ReplyDelete
  3. Maybe look into https://www.youtube.com/watch?v=5E3ZdrbY6MI
    This is an older video but it has more explanation. Note that this older video does not make use of an "edge image".

    ReplyDelete
  4. I am trying to take 2d family photos and have a depth map auto generated to be able to make a 3d model so that it can be subsurface laser engraved into a 3d crystal. Do you know of the best software or way to do this? By the way this information alone was super helpful but I cant seem to figure out a way to do it.

    ReplyDelete
    Replies
    1. Better to use AI tools nowadays. Check my youtube videos in particular this one https://youtu.be/ON1oKs4k-A8 and this one https://youtu.be/jqcc8OaV6Bo. You may have to do some manual editing to the depth map but that should get you started.

      Delete
  5. I'm getting error while trying to compile it.

    make -f Makefile_O
    g++ -c -O -Wall -I../common compute_disparity_map.c
    In file included from compute_disparity_map.c:1:
    header.h:6:10: fatal error: io/proto.h: No such file or directory
    6 | #include "io/proto.h"
    | ^~~~~~~~~~~~
    compilation terminated.
    make: *** [Makefile_O:15: compute_disparity_map.o] Error 1

    ReplyDelete
    Replies
    1. You need to get the common stuff first and compile it. It's in this repo: https://github.com/ugocapeto/common

      Delete
    2. Thanks, I have compiled it without errors, but when I try to run it gives segmentation fault on dmag4. I'm using Ubuntu 22.04

      iffi@umac:~/PycharmProjects/dmag4/main$ ./dmag4
      Segmentation fault (core dumped)

      Delete
    3. Hmm, should work with ubuntu 22.04. It's segfaulting using the test cases that are provided in the repo under main/test? You may have to debug what's going on as it's more than likely happening when trying to read the dmag4_input.txt input file.

      Delete
    4. No, just simply testing dmag4 binary but getting this error. How can I run blade_runner from test???

      Delete
    5. cd to the directory where the blade runner example is located and run the executable there.

      Delete
  6. Tests looks fine.
    ./lis/test.sh
    === Running test test.sh

    checking linear solvers...

    number of processes = 1
    matrix size = 100 x 100 (460 nonzero entries)
    initial vector x = 0
    precision : double
    solver : BiCG 2
    precon : none
    storage : CSR
    lis_solve : normal end

    BiCG: number of iterations = 15 (double = 15, quad = 0)
    BiCG: elapsed time = 1.289845e-04 sec.
    BiCG: preconditioner = 5.960464e-06 sec.
    BiCG: matrix creation = 9.536743e-07 sec.
    BiCG: linear solver = 1.230240e-04 sec.
    BiCG: relative residual 2-norm = 2.553002e-16

    checking eigensolvers...

    number of processes = 1
    matrix size = 100 x 100 (460 nonzero entries)
    shift = 0.000000e+00
    initial vector x = 1
    precision : double
    esolver : Power 1
    storage : CSR
    lis_esolve : normal end

    Power: mode number = 0
    Power: eigenvalue = 7.837972e+00
    Power: number of iterations = 958
    Power: elapsed time = 3.081083e-03 sec.
    Power: preconditioner = 0.000000e+00 sec.
    Power: matrix creation = 0.000000e+00 sec.
    Power: linear solver = 0.000000e+00 sec.
    Power: relative residual 2-norm = 9.695325e-13

    ReplyDelete
  7. here is configure output truncated.


    Build with OpenMP library = no
    Build with MPI library = no
    Build Fortran interface = no
    Build SA-AMG preconditioner = no
    Build with double double precision support = no
    Build with long double precision support = no
    Build with 64bit integer support = no
    Enable dynamic linking = no
    Enable profiling = no

    C compiler = cc
    C flags = -O3 -fomit-frame-pointer -DHAVE_CONFIG_H
    C libraries = -lm -L/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/11/../../.. -lgcc_s

    ReplyDelete
  8. You are still having problems??? Note that the dmag4 that's on github is not a GUI. It reads the dmag4_input.txt and runs unattended. If you can run the test examples, then you are all set. Can you run the test examples?

    ReplyDelete
  9. no, getting this error

    ./dmag4 test/interstellar/dmag4_input.txt
    Segmentation fault (core dumped)

    ReplyDelete
    Replies
    1. you have to cd to the directory where the interstellar data is and run dmag4 without any argument. So something like "cd dmag4/main/dmag4_test/interstellar" and then you run the exec like this "../../dmag4".

      Delete