Friday, January 18, 2019

depthplayer.ugocapeto.com - A web tool to display depth maps in 3d

Link to the web tool to display depth maps in 3d: Depth Player.

It all started when my good friend Gordon told me about depth-player, a web tool to display Google Lens Blur images in 3d coded by Jaume Sánchez (Clicktorelease). As you probably know, a Google Lens Blur image is a container for a reference image and a depth map (and some other parameters like focal length, near plane, far plane, and maybe more). It's cool and all but it would be real nice if you could simply input a reference image and a depth map instead of having to combine them into a Google Lens Blur image (using depthy.me).

Here comes Depth Player where the input is a reference image and its associated depth map. The depth map is assumed to be white for near objects and black for far objects, the exact opposite of a Google Lens Blur depth map. Not a big change from Jaume's version but I think it makes the depth map display tool more general and more useful.

The 3d model is displayed using central/perspective projection. The projection center is the camera center. The distance from the image plane to the camera center is equal to the focal length. The 3d points are obtained by considering, for each 2d point of the reference image in the image plane, a ray emanating from the camera center and passing through the 2d point. The position of the 3d point along the ray is determined by its depth as recorded in the depth map. To get the proper location (within scale), you need to know where the near and far planes are along the axis that goes through the camera center and the reference image center in the image plane (aka the principal axis). You also need to know the focal length. The math is very similar to what's happening in Point Cloud Maker 11 (PCM11) since I am basically doing the same thing. The 3d scene is basically encased in a pyramid whose four side edges pass through the camera center (the center of projection) and one corner of the reference image.

In Depth Player, you can choose between the following display modes:
- Solid,
- Point cloud, and
- Wireframe.

Personally, I find the "Point cloud" display to be the most useful. The problem with "Solid" and "Wireframe" render modes is that objects at depth discontinuities are connected in the triangular mesh that's used to display the 3d scene. As a result, you end up with bad color interpolation at depth discontinuities.

In Depth Player, you can adjust:
- Focal distance/length,
- Near plane,
- Far plane,
- Smooth mesh,
- Quad size,
- Point size, and
- Downsampling.

Any time you change a parameter, you need to click on "Create model" to see the changes. If you want to go back to the default settings, you have to reload the Depth Player page.

The focal distance, near plane, and far plane need to be adjusted so that the 3d scene that is displayed corresponds to reality (as best as possible).

The parameter "Smooth mesh" controls how smooth the triangular mesh is. The larger "Smooth mesh" is, the smoother the triangular mesh is, that is, the more smoothing has been applied to the depth map. As with all things, smoothing should be done in moderation. By default, there is no smoothing applied.

The parameter "Quad size" controls how fine the quadrangular mesh is. Note that the triangular mesh is obtained from the quadrangular mesh by splitting each quad into 2 triangles. The smaller "Quad size" is, the more refined the mesh is. You can clearly see that when you switch the display to "Wireframe". Bear in mind that the lower "Quad size" is set to, the longer it takes to create the 3d model. Note that "Quad size" has no effect when in "Point cloud" display mode. Personally, I would leave that parameter alone (set to 1.0) and use the "Downsampling" parameter instead to control the refinement of the 3d scene.

The "Point size" parameter controls the size of the points when the display is switched to "Point cloud".

The "Downsampling" parameter controls by how much the original reference image is downsampled (reduced in size) prior to generating the 3d scene. The larger "Downsampling" is set to, the coarse the mesh (and therefore the rendered scene) is gonna be. If "Downsampling" is set to 1, there is no downsampling performed on the reference image. This has an effect in all render modes. If you are in "Point cloud" mode, this is the only way to control the number of points in the 3d scene. Bear in mind that the lower "Downsampling" is set to, the longer it takes to create the 3d model.

The original depth-player (the one Jaume wrote) had an option to upload the 3d scene to sketchfab and an option to download an .obj file. I removed the direct upload option to sketchfab (don't like the idea of direct upload from one site to another). I left the option to save the 3d model as an .obj file which you can certainly upload to sketchfab if you want.

Here's a quick video showing Depth Player in action:



Here's another more in-depth video showing depthplayer in action:



All credit goes to Jaume Sánchez for his original depth-player. All I did was allow the loading of a reference image + depth map instead of a Google Lens Blur image. I also changed a few other things but nothing too involved. Because Jaume's software was released under an MIT License, I am releasing Depth Player under the same MIT License, which is reproduced below.

The MIT License (MIT)

Copyright (c) 2014 Jaume Sanchez

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

8 comments:

  1. Hi Ugo,
    Thank you so much for such a great enhancement, thank you for adding new adjustment settings and thank you for posting this article to have everything well explained!
    :D
    Gordon

    ReplyDelete
  2. Hello ugo brother. How can i upload 3d photo to facebook after creating depth map with dmag?

    ReplyDelete
    Replies
    1. i think you need an app like depthcam to merge image and depth map. And then, you need an iphone dual camera to upload to the 3d facebook group.

      Delete
  3. please create video for Point Cloud Maker 11

    ReplyDelete
    Replies
    1. PCM11? ok, I will look into making one. Cheers.

      Delete
    2. Note that depthplayer us very good at building and displaying point clouds from a reference image and depth map. Click on point cloud for the render option and it will display the cloud instead of the mesh. You can control the density of points using the downsample slider. I rarely use pcm11 because then you need to load it into something else like cloudcompare, meshlab, or sketchfab to see it.

      Delete
    3. Check my youtube channel. I have just created a video tutorial for pcm11. I even rewrote pcm11 so that it's easier to use.

      Delete