So it turns out this 3d reconstruction stuff is a lot more complicated than I initially thought. I am trying to get the essential matrix for my camera system, however I am having real trouble getting something that makes sense. I have code that produces an output, it’s just wrong.

It would help if I had a really good text that explained this stuff in baby language so a math retard like me could understand it but unfortunately I don’t at the moment so I am doing a lot of googling. I have ordered Multiple View Geometry in Computer Vision from Amazon as this seems to be the authoritative text, however its got a 4-6 week delivery time.

Looks like this could take a while.

Well, I was really hoping that my next post regarding the my vision project would be to announce that I have phase one working correctly. Alas that is not the case. After some hard debugging over the last couple of weeks I found and fixed a number of bugs and problems.

I am now much more confident in my matching between stereo images, and also in my interframe matching. I rewrote this stuff to be much more in line with the work of Se et al. During the testing process I found a major bug in my kd-tree range search implementation which must have been having a major impact on the number of good quality matches I was getting.

I find it quite difficult to test this stuff as I can’t think of a good way of automating the testing of the quality of matching between images. At the moment I simply annotate the images with the SIFT feature locations and then check for matches by eye.

I got really excited about a week ago when I thought I have finally cracked it and was getting reliable interframe matching and ego motion estimation. In fact I still think that for the most part it is working correctly. However, when I checked the 3d world coordinates the system was calculating for the observed features I realised they were completely wrong. Something that was in reality 1 meter away from the center of the camera system was registering as being nearly 3 meters away, and something that was in fact over 2m away came back less than a meter. I think the problem is in the way I am calculating the disparity and I am not taking all the camera intrinsic and extrinsic parameters into account properly.

To that end I am now working on calculating the essential matrix from the fundamental matrix, and then I should be able to much more accurately calculate the relative position of the observed features.

Time will tell.