Monday, November 4, 2019

2d to 3d Image Conversion Software - the3dconverter2

The3dconverter2 is an attempt at making easier the conversion of a 2d image into a 3d image via the creation of a dense depthmap. I know there is already 2d to 3d Image Conversion Software - The 3d Converter but I think the idea of relative depth constraints is probably unnecessary, which is why the3dconverter2 does not have them. Unlike DMAG4 and DMAG11 which limit the propagation of assigned depths within to areas of similar colors, the3dconverter2 allows for totally free propagation of assigned depths. This means that the only way to prevent depths from propagating over object boundaries (at different depths) is to use a so-called "edge image" (also used by DMAG4).

After you have extracted the3dconverter2-x64.rar, you should see a directory called "joker". It contains everything you need to check that the software is running correctly on your machine (windows 64 bit). The file called "the3dconverter2.bat" is the one you double-click to run the software. It tells windows where the executable is. You need to edit the file so that the location of the executable is the correct one for your installation. I use notepad++ to edit text files but I am sure you can use notepad or wordpad with equal success. If you want to use the3dconverter2 on your own photograph, create a directory alongside "joker" (in another words, at the same level as "joker"), copy "the3dconverter2.bat" and "the3dconverter2_input.txt" from "joker" to that new directory and double-click "the3dconverter2.bat" after you have properly created all the input files the3dconverter2 needs.

The file "joker.xcf" is a gimp file that contains (as layers) content_image.png (the reference 2d image) and edge_rgba_image.png (the so-called edge image). It also contains the gimp paths for the depth constraints.

The input file "the3dconverter2_input.txt" contains the following lines:
content_image.png
sparse_depthmap_rgba_image.png
dense_depthmap_image.png
gimp_paths.svg
diffusion_direction_rgba_image.png
edge_rgba_image.png

Line 1 contains the name of the content rgb image. The file must exist.

Line 2 contains the name of the sparse depthmap rgba image. The file may or may not exist.

Line 3 contains the name of the dense depthmap image. The file will be created by the3dconverter2.

Line 4 contains the name of the gimp paths giving depth constraints. If the file does not exist, the3dconverter2 assumes there are no depth constraints given as gimp paths. In this case, there should be a sparse depthmap rgba image given.

Line 5 contains the name of the diffusion direction rgba image. If the file doesn't exist, the3dconverter2 assumes there are no regions where the diffusion direction is given. This is something I was working on but you can safely ignore this capability.

Line 6 contains the name of the edge rgba image. If the file doesn't exist, the3dconverter assumes there is no edge image.

The content rgb image is of course the image for which a dense depthmap is to be created. It is recommended not to use mega-large images as it may put an heavy strain on the solver (memory-wise). I always recommend starting small. Here, the reference rgb image is only 600 pixels wide. If the goal is to create 3d wiggles/wobbles or facebook 3d photos, it is more than enough. If you see that your machine can handle it (it doesn't take too long to solve and the memory usage is reasonable), then you can switch to larger reference rgb images.


Content rgb image.

The sparse depthmap rgba image is created just like with Depth Map Automatic Generator 4 (DMAG4) or Depth Map Automatic Generator 11 (DMAG11). You use the "Pencil Tool" (no anti-aliasing) to draw depth clues on a transparent layer. Depths can vary from pure black (deepest background) to pure white (closest foreground). When you zoom, you should see either fully transparent pixels or fully opaque pixels because, hopefully, all the tools you have used do not create anti-aliasing. If, for some reason, there are semi-transparent pixels in your sparse depthmap, click on Layer->Transparency->Threshold Alpha and then press Ok to remove the semi-transparent pixels. Note that you absolutely do not need to create a sparse depthmap rgba image to get the3dconverter2 going.

Gimp paths are used to impose constraints on pixel depths. You use gimp paths to indicate that one pixel should be at a specified depth. If you don't give gimp paths (no gimp paths file present), the3dconverter2 relies solely on the sparse depth map (and possibly the edge image if you provide one) to generate the dense depth map and behaves just like Depth Map Automatic Generator 4 (DMAG4). The name of a gimp path stores the depth at which the pixels making up the gimp path should be. The name of a gimp path must start with 3 digits ranging from 000 (pure black - furthest) to 255 (pure white - closest). For example, if the name of a gimp path is "146-blah blah blah", the pixels that make up the gimp path are supposed to be at a depth equal to 146. What is cool about this system is that it is quite simple to change the depth for a given gimp path: simply change the first 3 digits. To be perfectly clear, all pixels that make up a gimp path receive the depth assigned to the gimp path, not just the starting and ending pixel of each gimp path component.

When you are working with gimp paths, you should always have the Paths dialog window within easy reach. To get to it, click on Windows->Dockable Dialogs->Paths. Once you have created a gimp path and are done with it, it will show up in the Paths dialog window as "Unnamed" but will not be visible. To make it visible, just do the same thing you would do for layers (click to the left of the name until the eye shows up). After you create a gimp path (using the "Paths Tool"), gimp automatically names it "Unnamed". You can assign a depth to the gimp path by double-clicking on the path name and changing it to whatever depth you want the path to be.

To save the gimp paths, right-click the active path and select "Export Gimp Path". Make sure to select "Export all paths from this image" in the drop-down menu before saving. This will save all gimp paths into a single SVG file, eg, "gimp_paths.svg".

It's probably a good idea at this point to explain how to create/edit paths in gimp:

To create a new path, click on "Paths Tool". In the " Tool Options", check the " Polygonal" box as we will only need to create paths that are polygonal (no Bezier curves needed here). Click (left) to create the first anchor.

To add an anchor, click (left). To close a path (you really don't need to do that here but it doesn't hurt to know), press CTRL and hover the cursor over the starting anchor and click (left).

To start a new path component (you really don't need to do that here but it doesn't hurt to know), pres SHIFT and click (left).

To move an anchor, hover the cursor over an existing anchor, click (left) and drag.

To insert an anchor, press CTRL and click (left) anywhere on the segment betwen two existing anchors.

To delete an anchor, press SHIFT+CTRL and click (left) on the anchor you want to delete.

The edge rgba image tells the3dconverted not to diffuse the depths past the pixels that are not transparent in the edge image. It is similar to the edge image used in dmag4. To create an edge rgba image, create a new layer and trace along the object boundaries in the reference image. An object boundary is basically a boundary between two objects at different depths. To trace the edge image, I recommend using the " Pencil Tool" with a hard brush (no anti-aliasing) of size 1 (smallest possible). Press SHIFT and click (left) to draw line segments. I usually use the color "red" for my edge images but you can use whatever color you want. Make sure to always use tools that are not anti-aliased! If you need to use the "Eraser Tool", check the "Hard Edge" in the tool options. When you zoom on your edge image, it should look like crisp staircases.


Edge rgba image.

The white pixels are actually transparent (checkerboard pattern in gimp).

The following screen grab shows all the layers contained in content_image.xcf:


The following screen grab shows all the gimp paths contained in content_image.xcf:


As you can see, the names of the gimp paths all start with 3 digits ranging from 000 to 255. The name of the gimp path indicates its depth.

Once you are done with the editing in gimp, simply save the layer containing the sparse depthmap as sparse_depthmap_rgba_image.png (if you have one), the layer containing the edge image as edge_rgba_image.png, and the gimp paths as gimp_paths.svg. Double-click on "the3dconverter2.bat" and the3dconverter2 should spit out the dense depthmap in no time as dense_depthmap_image.png assuming nothing went wrong.

This is the dense depth map generated by the3dconverter2:


Dense depthmap image created by the3dconverter2.

To see if the dense depthmap is good, you can use Depth Player. To create a wiggle/wobble, you can use Wiggle Maker. If your goal is solely to create wiggles, the depth map really doesn't need to be too accurate and depths not in the immediate foreground really do not have to be too accurate (only worry about the foreground objects).


Wiggle/wobble created by Wiggle Maker.

Any problems? Send me your .xcf gimp file (zipped)! If the file is big, please use something like wetransfer to send it me or make it available via google drive.

I made a video on youtube that explains the whole 2d to 3d image conversion process from start to finish, kinda:


Here's another video tutorial that's much more in-depth but kinda assumes you already know how to install and run the3dconverter2:


Tips and tricks:
- I think it's a good idea to order the paths from foreground to background in the "Paths" dockable window. It's a little bit less confusing especially if you have tons of them.
- Use depthplayer to check your work often.

Download link (for windows 64 bit): the3dconverter2-x64.rar

Source code: the3dconverter2 on github.

39 comments:

  1. There is no directory "joker" in dropbox

    ReplyDelete
  2. Forgive me, but I do not see a download link. Would love to check this out.

    ReplyDelete
    Replies
    1. Hi:
      There is a download link at the end of the post, the last line actually. Cheers, Ugo-

      Delete
  3. When I load the png I get this error, "libpng warning: iCCP: known incorrect sRGB profile."

    ReplyDelete
  4. It appears the archive is corrupt. After Direct download it cannot be opened.

    ReplyDelete
    Replies
    1. Dear Theo:
      I have checked. I was able to download it and extract it on windows 7 (using winrar). On windows 8 or 10, you can extract directly by double clicking on the rar file.

      Delete
  5. I'm on Win 10 but direct download from dropbox fails. Not for some other rars of yours in dropbox (just checked to make sure it isn't just me or dropbox). But it fails.

    ReplyDelete
  6. Just tried again... "unknown format or damaged". File type is .rar and WinRar gives the message. Or is there an incompatibilty between WinRar versions? (mine is 3.62)

    ReplyDelete
  7. To make sure we're talking about the same rar: it's the the3dconverter2-x64.rar file

    ReplyDelete
    Replies
    1. If you put the following link in your browser (i use google chrome) https://www.dropbox.com/s/nq83x5km5a3ksx5/the3dconverter2-x64.rar?dl=0, dropbox should show you the content of the archive, which means that the archive is not corrupted. I have just tried and it shows the content of the archive.

      Delete
  8. I'm sorry, but it does not. Same error. Only for that rar. The others (for as far as I tried) work fine. I use Edge. I installed Firefox - same error.

    Sorry for that - I don't know what's wrong with whatever part.

    ReplyDelete
    Replies
    1. i have just tried on firefox and the rar file displays its content on dropbox. Maybe it's coming from antivirus. try to disable temporarily. if you still have problems, send me email (on main page, right sidebar is the addy).

      Delete
  9. And yes, I do see the archive contents - that seems OK. Somehow the transfer of the file doesn't work. Only for that rar which is odd.

    ReplyDelete
    Replies
    1. hmm, i don't know then. maybe antivirus prevents proper download. last resort : send me an email and i will send you the archive directly via wetransfer.

      Delete
  10. Seems like a ton more effort than DMAG, is it worth it?

    ReplyDelete
    Replies
    1. Are you talking about dmag11? I guess it depends on your level of pickyness. With dmag4, you kinda get depth bleeding if the objects are not well contrasted in terms of colour. Also, the depth clues are limited to various grayscale paint strokes. Sometimes, it's difficult to change things if you got the depth clues wrong. Using paths in the3dconverter2 allows you to easily make changes. To me, I think it's worth it.

      Delete
    2. is sparse depth required when using gimp paths? It seems like they're the same use, they both use values to suggest depth on part of an image.

      Delete
    3. No, a sparse depth map is not required. In the joker example, i don't use a sparse depth map.

      Delete
  11. I'm trying to use gimp_paths but it's saying "Format for gimp path id 255 #2 is not valid!" when i delete the path, the next path on top gets the same error. What am i doing wrong?

    ReplyDelete
    Replies
    1. You are using gimp right? Assuming you are, then i am not sure. i would have to look at your paths. Use the email address on the right sidebar, and send me your .xcf file. I will take a look.

      Delete
  12. You say that without an edge.png, it behaves the same way as DMAG11 or 4 but when I don't provide an edge.png, it creates like cloud-like dense_map. It doesn't detect it's own edges, why is that?

    ReplyDelete
    Replies
    1. With the3dconverter2, you absolutely need an edge image. But you don't need a sparse depth map (the one with the depth clues like in dmag4 or dmag11). If you don't provide a sparse depth map, all the depth clues come from the gimp paths.

      Delete
    2. Is there advantages to using gimp paths as opposed to sparse?

      Delete
    3. the advantage of using gimp paths is that it's very easy to change their depth. you just need to rename them. Example: you have a path called "186" which you think is at depth 186. You realize it's wrong and it should be at depth 190. You just need to rename the path "190" and you are done. With a sparse depth map, you need to recolor the area at the wrong depth. I thin gimp paths are easier to deal with this but if you are more comfortable with a sparse depth map and no gimp paths, you can certainly do that in the3dconverter2.

      Delete
  13. Is there also a Linux version of the3dconverter2?

    ReplyDelete
  14. I tried using this after a long hiatus. I get the following when trying to run the bat. file.

    reference rgb image = @╒╒o
    sparse depthmap rgba image = @╒╒o
    dense depthmap image = @╒╒o
    gimp paths = @╒╒o
    diffusion direction rgba image = @╒╒o
    edge rgba image = @╒╒o
    Filename @╒╒o does not exist!

    ReplyDelete
    Replies
    1. You will need to send me your input file. Note that nowadays it's relatively easy to get depth maps from 2d images using AI. Here are 1 video I made about LeiaPix Converter https://youtu.be/jqcc8OaV6Bo (now, you can save the depth map) and MiDaS v3 https://youtu.be/a_7GRsR4o_I Note that the LeiaPix Converter uses midas v2.1 but it's very easy to use.

      Delete
    2. Thanks for responding. As a test, I tried using the .bat file that came with the "Joker" folder and got the same message.

      The .bat is set to: cmd.exe /k "C:\Users\[My Username]\Desktop\the3dconverter2-x64\the3dconverter2.exe"

      I think that's how it's supposed to be done.

      Delete
    3. And you were running the .bat file in the joker directory?

      Delete
    4. Yes. The joker folder is in the the3dconverter2-x64 folder (default location).

      [I am using this porgram to test the depth map cleaning function.]

      Delete
    5. I have just downloaded the archive and tried running the3dconverter2 in the joker directory and it runs as it should. I really don't know what is happening in your case.

      You are double-clicking on threedconverter2.bat in the joker directory, not the .exe file in the directory above, right?

      Delete
  15. Can anyone help? I downloaded the source, trying to build for linux, the command in the readme.md does not work and I am novice at building from source. issue is with "make -f Makefile_g/Makefile_O" never works for me. Please help!

    ReplyDelete
    Replies
    1. Did you leave the same comment on github? As an issue. Probably easier if it's posted there. You use "make -f Makefile_g" to compile for debug version. You use "make -f Makefile_O" to compile for production version. When compiling libraries.

      Delete