3D Photos - Skateboard jump

Left image.

Right image.

Depth map obtained with Depth Map Automatic Generator 7 (DMAG7).

What is likely to happen with DMAG7 is that you're gonna have different depths for areas that are of different colors even though they belong to the same object and should be at the same depth. For instance, the right shoe (his right) of the skateboarder: the white Nike swoosh and sole are not at the same depth as the black fabric that makes up most of the shoe. I guess one could play around with the parameters but an easier way albeit manual is to post-process the wrong depths in Gimp. Of course, that kinda defeats the purpose of an automatic depth map generator but I am not sure there's a more efficient way to do it.

Depth map smoothed by Edge Preserving Smoothing 7 (EPS7).

Depth map post-processed in Gimp.

Here, I kinda smoothed out the skateboarder and fixed up a few things: the head, the arms, and the space between the legs. Went back and forth between the depth map viewer and the wiggle maker until everything looked kinda reasonable.

Animated gif obtained using Depth Map Viewer.

Animated gif (stereo wiggle) obtained using Wiggle Maker.

A big thank you to Peter Simcoe for providing the stereo pair.

2D to 3D Conversion - London Calling

The Clash "London Calling" album cover. Pretty iconic, huh?

Depth map obtained with Depth Map Automatic Generator 4 (DMAG4) and a little of bit of post-processing in Gimp.

Actually, it was more than a little bit of post-processing in Gimp.

Animated gif obtained with Depth Map Viewer.

I did a lot of back and forth between Gimp and the depth map viewer. My favorite tools in Gimp are free select, gradient (blend tool), Gaussian blur, and smudge. I am sure there are other tools that are probably more appropriate for depth map painting but those are the ones I use. Dear reader, if you have a better way of doing it and want to share your technique, let me know!

Animated gif (stereo wiggle) obtained with Wiggle Maker.

Depth Map Viewer

I am proud to announce the release of Depth Map Viewer, a website where you can visualize the quality of a depth map (corresponding to a reference image) by rotating the corresponding 3d scene along the axis passing by the the center of the 3d scene and parallel to the height of the reference image. As a bonus, the animation can be saved as an animated gif. The depth map may come from anywhere as long as black means far and white means close.

I was quite inspired by Depth Player for Android Lens Blur images when making Depth Map Viewer. Can't say I use any of the source code although the revelation that three.js was used for the rendering made my job much easier.

I would like to thank whoever is responsible for jsgif because that's how Wiggle Maker creates the animated gifs. Works like a charm.

Manual is available here: Depth Map Viewer manual.

Here's the video that shows Depth Map Viewer in its full glory:


Depth Map Viewer has been fully tested on Google Chrome.

Depth Map Viewer 2 is a variant of Depth Map Viewer that outputs an animated PNG instead of an animated GIF. As you know, a GIF file only has 256 colors, which can be a problem if you are picky about colors, so it's quite an improvement to be able to get an animated PNG instead of a GIF. Absurde Photon took the code in Depth Map Viewer, modified it so that the output could be an animated PNG, and then gave the result back to me as Depth Map Viewer 2. The parameters are the same as Depth Map Viewer.

3D Photos - Hard at work

This article will demonstrate what Depth Map Automatic Generator 7 (DMAG7) and Edge Preserving Smoothing 7 (EPS7) can do. Recall that DMAG7 is based upon "Fast Bilateral-Space Stereo for Synthetic Defocus" by Jonathan T. Barron, Andrew Adams, YiChang Shih, and Carlos Hernández and that EPS7 is based upon "Domain transform for edge-aware image and video processing" by Eduardo Gastal and Manuel Oliveira.

The following stereo pair was provided by Jon Barron (and Google). The dimensions are 2432x1728. The min and max disparities are 0 and 25, respectively. This is not a large disparity range for a stereo pair of that size, meaning that there won't be a whole lot of occlusions to possibly worry about. It was probably taken with a small interaxial stereo camera.

Left image.

Right image.

Disparity map obtained with DMAG7 with spatial sample rate = 8, range sample rate = 32, radius = 12, lambda = 0.01, max number of iterations = 1000, hash table size = 10000.

Let's examine now the effects of applying edge preserving smoothing (with EPS7) on the depth map by playing with the filter spatial standard deviation and the filter range standard deviation. In the following, the spatial standard standard deviation is fixed (at 1000) and the range standard deviation is decreased logarithmically from 1000 to 10.

Filtered depth map with spatial standard deviation = 1000 and range standard deviation = 1000.

Filtered depth map with spatial standard deviation = 1000 and range standard deviation = 100.

Filtered depth map with spatial standard deviation = 1000 and range standard deviation = 10.

Now, let's fix the range standard deviation (at 1000) and decrease the spatial range deviation.

Filtered depth map with spatial standard deviation = 1000 and range standard deviation = 1000.

Filtered depth map with spatial standard deviation = 100 and range standard deviation = 1000.

Filtered depth map with spatial standard deviation = 10 and range standard deviation = 1000.

I personally kinda like the spatial standard deviation = 100 and range standard deviation = 1000 best, so this is the one we are gonna use to produce a wiggle using Wiggle Maker. Before we do that, it's probably a good idea to reduce the size of the images (left image and depth map) because you really don't want to create animated gifs from large images.

Animated gif (wiggle) obtained with Wiggle Maker.

To put things in perspective, below is the depth map obtained by Depth Map Automatic Generator 5 (DMAG5). I used to think stereo matching local methods (like DMAG5) could compete with global methods but not anymore. Up to now, the smoothness term used in global methods was really simplistic. You issue a penalty when the disparity of a given pixel differs from the disparities at immediately neighboring pixels but only if the disparity difference is small. A large disparity difference is assumed to be a legit depth change, which is a bit of stretch. Thanks to Fast Bilateral-Space Stereo, the smoothness term makes a lot more sense now since it is tied to a bilateral filter that takes into account location and color of neighboring pixels. This makes global methods really hard to beat now.

Depth map obtained with DMAG5 using radius = 12, alpha = 0.9, truncation (color) = 20, truncation (gradient) = 10, and epsilon = 1.

Wiggle Maker

I am proud to announce the release of Wiggle Maker, a website where you can make animations from an image and a depth map. The depth map may come from anywhere as long as black means far and white means close. For example, the depth map may have been generated by Depth Map Automatic Generator 7 (DMAG7) and smoothed by Edge Preserving Smoothing 7 (EPS7) from a stereo pair taken by a stereo camera.

I was quite inspired by depthy.me and Of Wiggle Photography [CoffeeScript]. Depthy.me relies solely on the WebGL renderer (shader) to create the wiggle and therefore doesn't actually create the animation frames from scratch like fulmicoton does. Wiggle Maker uses an approach similar to fulmicoton to create the frames, in other words, there's no WebGL rendering involved (maybe in a future release). I was also impressed by Depth Data Viewer but this is more like rendering a triangular mesh, which is cool to quickly see what's wrong with the depth map.

I would like to thank whoever is responsible for jsgif because that's how Wiggle Maker creates the animated gifs. Works like a charm.

Manual is available here: Wiggle Maker manual.

Here's the video that shows Wiggle Maker in its full glory:


Wiggle Maker has been fully tested on Google Chrome.

Wiggle Maker 2 is a variant of Wiggle Maker that outputs an animated PNG instead of an animated GIF. As you know, a GIF file only has 256 colors, which can be a problem if you are picky about colors, so it's quite an improvement to be able to get an animated PNG instead of a GIF. The parameters are the same as Wiggle Maker.

Edge Preserving Smoothing 7 (EPS7)

Edge Preserving Smoothing 7 (EPS7) smoothes an image (typically, a depth map) while preserving the edges of a joint image (typically, the left image of a stereo pair). It's an implementation of "Domain transform for edge-aware image and video processing" by Eduardo Gastal and Manuel Oliveira. Without going into much details, it's an approximation of the edge-preserving bilateral filter defined by the spatial standard deviation (controls amount of smoothing) and the range standard deviation (controld amount of edge preserving).

Here's an example:

Image to be smoothed (filtered). This could be a very noisy depth map.

Joint image. We want the previous image to be smoothed but we also want the smoothing to maintain the edges between the colored blocks.

Smoothed (filtered) image using spatial standard deviation = 60, range spatial deviation = 0.4, and number of iterations = 3. As can be seen, the noise was smoothed and the edges have been preserved, exactly what we wanted.

For an example that shows edge preserving smoothing applied to an actual depth map, you are invited to check 3D Photos - Hard at work.

The windows executable (guaranteed to be virus free) is available for free via the 3D Software Page.