Automatic depth map generation, stereo matching, multi-view stereo, Structure from Motion (SfM), photogrammetry, 2d to 3d conversion, etc. Check the "3D Software" tab for my free 3d software. Turn photos into paintings like impasto oil paintings, cel shaded cartoons, or watercolors. Check the "Painting Software" tab for my image-based painting software. Problems running my software? Send me your input data and I will do it for you.
This is the still I started with. I removed the background in Gimp and then imported the image into Blender using the import images as planes option. Then, it was sculpting time with the "Grab" brush and the "Smooth" brush. It took a while to get it the way I wanted and it was still not 100 percent right (doesn't really have to be anyway).
This is what it looked like when I was done sculpting. When I sculpt, I always lock the X and Y directions so that any movement is along the Z axis. This guarantees that when you look at the sculpted image straight on, it will always be correct (same as what you started with).
This is a quick animation to show the result. For the animation, I just rotated the plane of the image.
Now, I called this a "2d to 3d conversion" but in Blender lingo, it is usually referred to as "camera mapping". In an animation/film setting, it's called "matte painting".
A reader of this very blog sent me a 2D picture (shown right below) and asked me the best way to do a 2d to 3d conversion, and as automatically as possible (or as fast as possible). Well, to me, the only way to do a somewhat correct looking conversion is with Blender, the free 3d modeling program.
The first thing I did was to separate the objects in Gimp and export the images in the png file format.
This is the room with the kid removed from the picture. I duplicated the initial layer and happily painted all over the kid in order to recover the missing background. I used the clone tool and good old copy/pasting. It could obviously have been much better, in particular, I probably should have left the shadows under the kid's arm.
This is the kid all by himself. I duplicated the initial layer, used the lasso tool to make a selection around the kid and applied a layer mask to the layer using the selection. I always use a layer mask in case I need to go back to Gimp and alter the selection. Now, the crayon he's holding in the original 2d picture is a bit of a problem, so I decided to put the crayon in its own layer using the same technique.
Yep, that's the crayon all by itself. There's probably a better to do this but it works.
Now, the fun really begins with Blender. The first problem to solve is where to position the camera and how to orient it so that the 3d view matches the 2d image. This is not easy to do and it's often a trial and error process. There's an easy way though, and it's called BLAM. BLAM is an add-on that makes "camera matching" a total breeze. All you have to do is go into the "Movie Clip Editor", load up the 2d image, use the grease pencil to draw two sets (layers) of vanishing lines (Ctrl-D left mouse button to draw), and BLAM figures out where the camera is supposed to be. It even loads up the 2d image as background image for the camera and changes the resolution (dimensions) of the image so that it matches the camera perfectly. A true life saver!
This what you see after BLAM does its thing and you switch to "camera view" (0 on the NumPad). What you want to do now is build the 3d scene so that, when you are in camera view (0 in the NumPad), the 3d scene matches the background image. So, I created a cube and moved/scaled it so that the faces matched the room. the cube should not be rotated and the cube should not be deformed (the angles must remain at 90 degrees).
This shows the cube minus three of its faces and the texture from the room on the remaining faces. It looks jagged because I subdivided the mesh a few times and removed a bunch of faces. The way I applied the texture is to select everything (A) in edit mode, bring up UV mapping (U), and select "Project from View", making sure the view was camera view (0 on the NumPad). The projected mesh shows up in the "UV/Image Editor" viewport and there, you can open the texture file, that is, the image of the room. It is important to subdivide your mesh before mapping it to the texture otherwise you can get funny things near the edges of the camera view. Note that at this point, the texture will not show up when you attempt to render the 3d scene (F12). For the texture to render, you need to create a material and a texture. For the material, I checked the "Shadeless" box under "Shading" and checked the "Transparency" box (clicked on "Z Transparency" and set "Alpha" to 0). For the texture, I loaded the image of the room, made sure "UV" was selected for the "Coordinates" under "Mapping", and checked both Color (1.0) and Alpha (1.0) under "Diffuse"->"Influence".
At that point, you kinda have to wonder how you are gonna put the kid in the scene. Well, there are more than one way of doing it but it seems to me that sculpting a plane containing the image of the kid is probably the easiest. So, I imported the image of the kid (without the crayon) using "Images as Planes" and positioned the plane in the scene. I then went into "Sculpt Mode" and really went to town using the "Grab" brush. To help with sculpting, I used a Multiresolution modifier so that I could work at different scales. I locked the X and Y directions when sculpting so that everything moves perpendicular to the plane (assuming it's in "view plane" sculpting mode). Don't forget to uncheck the X in the mirror lock which is there by default! Sculpting is probably the only artistic part in all that process, so it's kind of rewarding when you get it right.
So, that's the 3d scene with the room and the kid without his crayon. The reason why I didn't sculpt the kid with his crayon is because it's impossible to do it correctly.
For the lone crayon, again, I imported the image as a plane and positioned the plane so that the crayon looks like it's in the kid's hand.
That's the 3d scene with the crayon added. You can see the plane that contains the crayon.
Now, if you try to render the scene (F12) and see that the plane is blocking what's behind it, it's probably because the transparency is not set right in the material. When you load an image as a plane, the transparency is gonna be set to "Mask" (assuming you ticked "Use Alpha" in the "Material Settings"). It's gotta to be changed to "Z Transparency".
At that point, the 3d scene is done and what's left to do is animate the thing. You really don't want to use the camera that was used to set up the 3d scene in case you need to go back to the 3d scene to change things around. It's recommended to duplicate the camera and use the extra camera for the animation work. For the animation, I generated all the frames in png format, went into the "Video Editing" screen layout to string them together, and then went back to the "Default" screen layout to generate the actual movie using the "AVI JPEG" file format.
Yep, that's the video alright. It's obviously far from being perfect but rendering the kid in 3d from a 2d picture is not exactly easy. If you simply want to produce a lenticular out of all this, you can simply pan the camera left to right, dump the in-between frames, and interlace the frames in something like Lenticular Image Creator (LIC) or Superflip.
Blender is a super powerful program but it has a pretty steep learning curve, so I recommend reading books about it like "Blender for Dummies" before even hitting the tutorials in Youtube. Check out the Blender Basics book to get you started, it's downloadable, free and nicely presented.
That's a pretty good tutorial on "camera mapping" or "matte painting" but it's in French. You can see the whole process of building the 3d scene always referring to the camera view with the background image. There are many other tutorials on youtube, maybe too many.
This is a stereo pair that was sent to me by a reader of this very blog. It's a scan of a Realist slide.
Left image.
Right image.
From the get go, you kinda know that it's likely the depth map for the background is going to be wrong (You need tecture to get accurate depth maps.) Not really a big deal.
For the disparities, I am gonna use minimum disparity = -16 and maximum disparity = 10. It's not a big deal if the min and max disparities are not accurate as long as they are overshot (The real max disparity is probably smaller than 10 but it's ok.)
Left depth map obtained using window radius = 12, alpha = 0.9, truncation value (color) = 7, truncation value (gradient) = 2, epsilon = 4, number of smoothing iterations = 1, and disparity tolerance = 0.
These are the default parameters. We can probably do better by playing around with the parameters.
Left depth map obtained using window radius = 24, alpha = 0.0, truncation value (color) = 128, truncation value (gradient) = 2, epsilon = 4, number of smoothing iterations = 1, and disparity tolerance = 1.
By setting alpha to 0.0, you are relying on color information only to make stereo matches (The color gradient is not involved at all.) By default, stereo matches are made by relying mostly on the color gradient (alpha = 0.9). Playing around with the parameters is no dark science and, in most cases, it's basically trying things out and see what works best.
That's about the best depth map I can get with DMAG5. To get the right depth map, the left and right images are flipped horizontally. The flipped right image is the new image1 and the flipped left image becomes image2. The depth map obtained using the exact same parameters (including min and max disparities) is flipped horizontally to give us the right depth map. From there, we can use Frame Sequence Generator 6 (FSG6) to generate the tweeners (interpolated views between the left and right image).
Animated gif obtained by animating the frames coming from FSG6.
