FSG11 generates synthetic views given input images and depth maps. If a single image and depth map is provided, it behaves very much like FSG4 but a with a subtle difference: FSG11 gets rid of outliers prior to inpainting. If more than one image and depth map is provided, the additional images and depth maps are directly used to inpaint, avoiding the all too familiar blurring effect of most frame sequence generators (FSG4 included).

Generating the synthetic views using a single (main) image and depth map:

Yep, it's definitively blurry but I actually do think it looks pretty good (because it's silky smooth). Does inpainting the actual background gives better results? To do that, you need a background image and its associated depth map, which can be fed to FSG11.

Generating the synthetic views using an additional image and depth map (for the background):

I felt the need to write FSG11 in the context of 2d to 3d image conversion. Since you have to spend time to generate the (main) depth map, why not spend a tad more time generating a second image and a depth map for the background. I use the clone tool and eat away at the foreground to generate the background image. The associated background depth map is usually quite simple since the foreground objects are supposed to be gone. In some cases, it can even be pure black for a background at infinity.

The windows executable (guaranteed to be virus free) is available for free via the 3D Software Page.

## Tuesday, March 14, 2017

## Friday, March 10, 2017

### Lenticular Creation From Stereo Pairs Using Free Software

I have written a technical report which explains how to create a lenticular (assuming you have you already have the lenticular lenses at your disposal) when the starting point is either a stereo pair taken by a stereo camera, a couple of images of a static scene taken with a regular camera (using the very cool cha-cha method, for exanple), or an image and a depth map (perhaps resulting from a 2d to 3d image conversion).

Here's the link: Lenticular Creation From Stereo Pairs Using Free Software.

Bonus gif that goes with the paper:

Here's the link: Lenticular Creation From Stereo Pairs Using Free Software.

Bonus gif that goes with the paper:

## Sunday, March 5, 2017

### 3D Photos - Posing in front of the big column

The original stereo pair was 3603x2736 pixels (provided by my good friend Mike). I chose to reduce it by 50% (for convenience) to end up with a stereo pair of size 1802x1368 pixels. First step is to rectify the images in order to end up with matching pixels on horizontal lines, a requirement for most automatic depth map generators. Here, I am using ER9b but it's probably ok to rectify/align with StereoPhoto Maker.

ER9b gives:

min disparity = -53

max disparity = 1

We are gonna use those as input to the automatic depth map generator. The min and max disparities may also be obtained manually with DF2.

We are gonna use DMAG5 (first using a large radius and then using a small radius) followed by DMAG9b to get the depth map. I could have used other automatic depth map generators but I kinda like DMAG5 because it's fast and usually pretty good.

Let's start by using a large radius (equal to 32). Parameters used in DMAG5 (Note that I use a downsampling factor equal to 2 instead of 1 to speed things up.):

radius = 32

alpha = 0.9

truncation (color) = 30

truncation (gradient) = 10

epsilon = 255^2*10^-4

disparity tolerance = 0

radius to smooth occlusions = 9

sigma_space = 9

sigma_color = 25.5

downsampling factor = 2

Let's follow up with DMAG9b to improve the depth map. Parameters used in DMAG9b:

sample_rate_spatial = 16

sample_rate_range = 8

lambda = 0.25

hash_table_size = 100000

nbr of iterations (linear solver) = 25

sigma_gm = 1

nbr of iterations (irls) = 32

radius (confidence map) = 12

gamma proximity (confidence map) = 12

gamma color similarity (confidence map) = 12

sigma (confidence map) = 4

It's time now to use a small radius in DMAG5 (equal to 4). Parameters used in DMAG5:

radius = 4

alpha = 0.9

truncation (color) = 30

truncation (gradient) = 10

epsilon = 255^2*10^-4

disparity tolerance = 0

radius to smooth occlusions = 9

sigma_space = 9

sigma_color = 25.5

downsampling factor = 2

Let's follow up with DMAG9b to improve the depth map. Parameters used in DMAG9b (same as before):

sample_rate_spatial = 16

sample_rate_range = 8

lambda = 0.25

hash_table_size = 100000

nbr of iterations (linear solver) = 25

sigma_gm = 1

nbr of iterations (irls) = 32

radius (confidence map) = 12

gamma proximity (confidence map) = 12

gamma color similarity (confidence map) = 12

sigma (confidence map) = 4

I am gonna go with the depth map obtained using the small radius. Is it the best depth map that could be obtained automatically? Probably not because one could have tweaked further the parameters used in DMAG5 and DMAG9b. Also, one could have tried using DMAG2, DMAG3, DMAG5b, DMAG5c, DMAG6, or DMAG7 instead of DMAG5 to get the initial depth map. That's a whole lot of variables to worry about. Anyways, now is time to generate synthetic frames with FSG4 using the left image and the left depth map (and going on either side).

Parameters used for FSG4:

stereo window (grayscale value) = 128

stereo effect = 5

number of frames = 12

radius = 2

gamma proximity = 12

maximum number iterations = 200

Inpainting is typically done by applying a Gaussian blur, which explains why inpainted areas look blurry. FSG6 produces synthetic frames of better quality because the right image and depth map are also used to inpaint. However, with FSG6, the synthetic frames are limited to be between the left and right images.

Now, if the object of the game was to create a lenticular, those synthetic views would be now fed to either SuperFlip or LIC (Lenticular Image Creator) to create an interlaced image. The fun would not stop here however as this interlaced image would have to be printed on paper and then glued to a lenticular lens. Yes, it is indeed a whole lot of work!

ER9b gives:

min disparity = -53

max disparity = 1

We are gonna use those as input to the automatic depth map generator. The min and max disparities may also be obtained manually with DF2.

We are gonna use DMAG5 (first using a large radius and then using a small radius) followed by DMAG9b to get the depth map. I could have used other automatic depth map generators but I kinda like DMAG5 because it's fast and usually pretty good.

Let's start by using a large radius (equal to 32). Parameters used in DMAG5 (Note that I use a downsampling factor equal to 2 instead of 1 to speed things up.):

radius = 32

alpha = 0.9

truncation (color) = 30

truncation (gradient) = 10

epsilon = 255^2*10^-4

disparity tolerance = 0

radius to smooth occlusions = 9

sigma_space = 9

sigma_color = 25.5

downsampling factor = 2

Let's follow up with DMAG9b to improve the depth map. Parameters used in DMAG9b:

sample_rate_spatial = 16

sample_rate_range = 8

lambda = 0.25

hash_table_size = 100000

nbr of iterations (linear solver) = 25

sigma_gm = 1

nbr of iterations (irls) = 32

radius (confidence map) = 12

gamma proximity (confidence map) = 12

gamma color similarity (confidence map) = 12

sigma (confidence map) = 4

It's time now to use a small radius in DMAG5 (equal to 4). Parameters used in DMAG5:

radius = 4

alpha = 0.9

truncation (color) = 30

truncation (gradient) = 10

epsilon = 255^2*10^-4

disparity tolerance = 0

radius to smooth occlusions = 9

sigma_space = 9

sigma_color = 25.5

downsampling factor = 2

Let's follow up with DMAG9b to improve the depth map. Parameters used in DMAG9b (same as before):

sample_rate_spatial = 16

sample_rate_range = 8

lambda = 0.25

hash_table_size = 100000

nbr of iterations (linear solver) = 25

sigma_gm = 1

nbr of iterations (irls) = 32

radius (confidence map) = 12

gamma proximity (confidence map) = 12

gamma color similarity (confidence map) = 12

sigma (confidence map) = 4

I am gonna go with the depth map obtained using the small radius. Is it the best depth map that could be obtained automatically? Probably not because one could have tweaked further the parameters used in DMAG5 and DMAG9b. Also, one could have tried using DMAG2, DMAG3, DMAG5b, DMAG5c, DMAG6, or DMAG7 instead of DMAG5 to get the initial depth map. That's a whole lot of variables to worry about. Anyways, now is time to generate synthetic frames with FSG4 using the left image and the left depth map (and going on either side).

Parameters used for FSG4:

stereo window (grayscale value) = 128

stereo effect = 5

number of frames = 12

radius = 2

gamma proximity = 12

maximum number iterations = 200

Inpainting is typically done by applying a Gaussian blur, which explains why inpainted areas look blurry. FSG6 produces synthetic frames of better quality because the right image and depth map are also used to inpaint. However, with FSG6, the synthetic frames are limited to be between the left and right images.

Now, if the object of the game was to create a lenticular, those synthetic views would be now fed to either SuperFlip or LIC (Lenticular Image Creator) to create an interlaced image. The fun would not stop here however as this interlaced image would have to be printed on paper and then glued to a lenticular lens. Yes, it is indeed a whole lot of work!

### 3D Photos - Summer Palace

In this post, we are gonna try to get the best possible depth map for a stereo pair provided by my good friend Gordon. Size of the images is 1200x917 pixels, so about 1 mega pixels.

ER9b gives us:

min disparity = -82

max disparity = 7

Let's turn to our favorite automatic depth map generator, DMAG5, to get the depth map. Here, we are gonna use a downsampling factor of 2 to speed things up.

Let's start with the following parameters for DMAG5:

radius = 16

alpha = 0.9

truncation (color) = 30

truncation (gradient) = 10

epsilon = 255^2*10^-4

disparity tolerance = 0

radius to smooth occlusions = 9

sigma_space = 9

sigma_color = 25.5

downsampling factor = 2

Not a very good depth map! Unfortunately, we have occluded pixels on the right of Gordon and at the top of its head. The occluded pixels on the left are totally expected.

Let's call on DMAG9b to shake things up and improve the depth map.

Parameters we are gonna use in DMAG9b:

sample_rate_spatial = 16

sample_rate_range = 8

lambda = 0.25

hash_table_size = 100000

nbr of iterations (linear solver) = 25

sigma_gm = 1

nbr of iterations (irls) = 32

radius (confidence map) = 12

gamma proximity (confidence map) = 12

gamma color similarity (confidence map) = 12

sigma (confidence map) = 4

Better but it looks likes it is gonna be a tough one. Let's try something else by reducing the radius used in DMAG5 and post-process again with DMAG9b.

Let's use the following parameters for DMAG5:

radius = 4

alpha = 0.9

truncation (color) = 30

truncation (gradient) = 10

epsilon = 255^2*10^-4

disparity tolerance = 0

radius to smooth occlusions = 9

sigma_space = 9

sigma_color = 25.5

downsampling factor = 2

Clearly, there is a lot more noise but we are hoping the less smoothed and more accurate depths will give better results in DMAG9b.

Parameters we are gonna use in DMAG9b (same as before):

sample_rate_spatial = 16

sample_rate_range = 8

lambda = 0.25

hash_table_size = 100000

nbr of iterations (linear solver) = 25

sigma_gm = 1

nbr of iterations (irls) = 32

radius (confidence map) = 12

gamma proximity (confidence map) = 12

gamma color similarity (confidence map) = 12

sigma (confidence map) = 4

I think it might be possible to improve the depth map further either by tweaking further the parameters used in DMAG5 or by using another automatic depth map generator like DMAG2, DMAG3, DMAG5b, DMAG5c, or DMAG6.

ER9b gives us:

min disparity = -82

max disparity = 7

Let's turn to our favorite automatic depth map generator, DMAG5, to get the depth map. Here, we are gonna use a downsampling factor of 2 to speed things up.

Let's start with the following parameters for DMAG5:

radius = 16

alpha = 0.9

truncation (color) = 30

truncation (gradient) = 10

epsilon = 255^2*10^-4

disparity tolerance = 0

radius to smooth occlusions = 9

sigma_space = 9

sigma_color = 25.5

downsampling factor = 2

Not a very good depth map! Unfortunately, we have occluded pixels on the right of Gordon and at the top of its head. The occluded pixels on the left are totally expected.

Let's call on DMAG9b to shake things up and improve the depth map.

Parameters we are gonna use in DMAG9b:

sample_rate_spatial = 16

sample_rate_range = 8

lambda = 0.25

hash_table_size = 100000

nbr of iterations (linear solver) = 25

sigma_gm = 1

nbr of iterations (irls) = 32

radius (confidence map) = 12

gamma proximity (confidence map) = 12

gamma color similarity (confidence map) = 12

sigma (confidence map) = 4

Better but it looks likes it is gonna be a tough one. Let's try something else by reducing the radius used in DMAG5 and post-process again with DMAG9b.

Let's use the following parameters for DMAG5:

radius = 4

alpha = 0.9

truncation (color) = 30

truncation (gradient) = 10

epsilon = 255^2*10^-4

disparity tolerance = 0

radius to smooth occlusions = 9

sigma_space = 9

sigma_color = 25.5

downsampling factor = 2

Clearly, there is a lot more noise but we are hoping the less smoothed and more accurate depths will give better results in DMAG9b.

Parameters we are gonna use in DMAG9b (same as before):

sample_rate_spatial = 16

sample_rate_range = 8

lambda = 0.25

hash_table_size = 100000

nbr of iterations (linear solver) = 25

sigma_gm = 1

nbr of iterations (irls) = 32

radius (confidence map) = 12

gamma proximity (confidence map) = 12

gamma color similarity (confidence map) = 12

sigma (confidence map) = 4

I think it might be possible to improve the depth map further either by tweaking further the parameters used in DMAG5 or by using another automatic depth map generator like DMAG2, DMAG3, DMAG5b, DMAG5c, or DMAG6.

## Tuesday, February 21, 2017

### 2D to 3D Image Conversion using DMAG4

In order to better understand how Depth Map Automatic Generator 4 (DMAG4) (the semi-automatic 2d to 3d image conversion software) works, I wrote a little blurb about it, in particular, how I go about creating the sparse depth maps.

Here's the link: 2D to 3D Image Conversion using DMAG4.

Bonus wiggles/wobbles:

Here's the link: 2D to 3D Image Conversion using DMAG4.

Bonus wiggles/wobbles:

## Friday, February 17, 2017

### Depth Map Improvement Using The Fast Bilateral Solver

I have written a technical report about depth map improvement using the fast bilateral solver of Jonathan Barron. It basically shows what Depth Map Automatic Generator 9b (DMAG9b) can do. Now, in that paper, I tried to improve the quality of depth maps generated by Depth Map Automatic Generator (DMAG), a notorious (but cool) creator of "fuzz" near object boundaries, so I might have asked too much. Still, DMAG9b is very good at making object boundaries crisp.

Here's the link: Depth Map Improvement Using The Fast Bilateral Solver.

Here's the link: Depth Map Improvement Using The Fast Bilateral Solver.

### Depth Map Generation using Graph Cuts

I have written a technical report about depth map generation using graph cuts which basically explains how Depth Map Automatic Generator 3 (DMAG3). The cool thing about DMAG3 is that it explicitly handles occluded pixels. This means that there is no need to compute a right depth map and perform a left-right consistency check. You can compare the depths maps obtained by DMAG3 in this technical report with depth maps obtained by DMAG2, DMAG5, DMAG6, and DMAG7 in A Qualitative Comparison of Selected Automatic DepthMap Generators.

Here's the link: Depth Map Generation using Graph Cuts.

Here's the link: Depth Map Generation using Graph Cuts.

### Depth Map Generation using Optical Flow

I have written a technical report on depth map generation using optical flow which basically explains how Depth Map Automatic Generator (DMAG) works. As DMAG does not explicitly handle occluded pixels, it has a tendency to blur object boundaries. The cool thing about DMAG is that you don't have to give a minimum and maximum disparitiy. You can compare the depths maps obtained by DMAG in this technical report with depth maps obtained by DMAG2, DMAG5, DMAG6, and DMAG7 in A Qualitative Comparison of Selected Automatic DepthMap Generators.

Anyways, here's the link: Depth Map Generation using Optical Flow.

Anyways, here's the link: Depth Map Generation using Optical Flow.

## Thursday, February 16, 2017

### Depth Map Automatic Generator 11b (DMAG11b)

DMAG11b is a tool that can be used to improve the quality of depth maps. Given a reference rgb image and an input depth map, DMAG11b detects pixels where the depth is unreliable and makes the depth at that pixel transparent in the output depth map. The output depth map can be fed to DMAG4, DMAG9, or DMAG11 (the programs used for 2d to 3d image conversion) to fill the transparent pixels for hopefully a better depth map than the one we started with.

To determine if the depth (coming from the input depth map) at a given pixel is reliable, DMAG11b computes the depth variance using the joint bilateral filter. A low (high) depth variance translates to a high (low, respectively) confidence. The output depth map is generated by making transparent any pixel for which the depth (coming from the input depth map) is of low confidence.

This idea of using the variance of the depth to compute a confidence comes from Jon Barron and his paper entitled "The Fast Bilateral Solver".

Here's an example:

Depth confidence map. White (black) means high (low, respectively) confidence. This depth confidence map is computed internally by DMAG11b.

The windows executable (guaranteed to be virus free) is available for free via the 3D Software Page.

To determine if the depth (coming from the input depth map) at a given pixel is reliable, DMAG11b computes the depth variance using the joint bilateral filter. A low (high) depth variance translates to a high (low, respectively) confidence. The output depth map is generated by making transparent any pixel for which the depth (coming from the input depth map) is of low confidence.

This idea of using the variance of the depth to compute a confidence comes from Jon Barron and his paper entitled "The Fast Bilateral Solver".

Here's an example:

Depth confidence map. White (black) means high (low, respectively) confidence. This depth confidence map is computed internally by DMAG11b.

The windows executable (guaranteed to be virus free) is available for free via the 3D Software Page.

## Monday, February 13, 2017

### Depth Map Automatic Generator 11 (DMAG11)

DMAG11 is a joint bilateral filter which can be used to "densify" a sparse depth map in the context of 2d to 3d image conversion.

The input to DMAG11 (in terms of image files) is a reference image and a (sparse) depth map where where some depths have been scribbled upon. DMAG11 can be used to convert 2d images into 3d as an alternative to Depth Map Automatic Generator 4 (DMAG4) and Depth Map Automatic Generator 9 (DMAG9).

Here's an example:

The windows executable (guaranteed to be virus free) is available for free via the 3D Software Page.

The input to DMAG11 (in terms of image files) is a reference image and a (sparse) depth map where where some depths have been scribbled upon. DMAG11 can be used to convert 2d images into 3d as an alternative to Depth Map Automatic Generator 4 (DMAG4) and Depth Map Automatic Generator 9 (DMAG9).

Here's an example:

The windows executable (guaranteed to be virus free) is available for free via the 3D Software Page.

Subscribe to:
Posts (Atom)