Structure from Motion - Case of two views/images

Structure from Motion (SfM) means recovery of the (sparse) 3D scene, that is the structure, and the camera poses, that is the motion. What follows is the Structure from Motion pipeline in the case of two (calibrated) images. Because the images are calibrated, the 3D scene can be reconstructed up to a scale factor. There are lots of pieces but each piece is rather easy to implement when you have at your disposal the book "Multiple View Geometry in computer vision" by Richard Hartley and Andrew Zisserman. This is fundamental reading material when doing anything that remotely relates to 3D scene reconstruction. Structure from Motion in the case of more than two views builds largely upon the two view case.

Once the outliers have been removed from the set of matches, the fundamental matrix F should be re-estimated by minimizing the reprojection error using the Levenberg-Marquardt algorithm.

The 3D structure and camera matrices may be improved by using what is called Bundle Adjustment. Please note that Bundle Adjustment needs good initial guesses, in other words, 3D scene reconstruction cannot rely solely on Bundle Adjustment.

It is possible to extract the camera matrices directly from the fundamental matrix F but, in this case, the 3D reconstruction is only projective, that is, it is known up to a projective transformation (aka homography or collineation). Knowing the calibration matrices makes the 3D reconstruction more useful (correct) as it is known up to scale.

Clearly, the 3D reconstruction obtained this way can only be sparse as the image matches are themselves sparse, usually obtained with SIFT and filtered by RANSAC or variants of those.

It should be noted that Epipolar Rectification 9 (ER9) and Epipolar Rectification 9b (ER9b) compute the fundamental matrix in order to get rid of outliers (bad matches). As a bonus (in the process of rectification), they also compute the focal length (the images are assumed to be taken by the same camera) from which one can easily get the (simplified) calibration matrix.