The epipolar geometry turned out to be much harder than I anticipated. I now have enough of it done to be able to check the epipolar constraint successfully. I am now in the process of reworking my stereo matching algorithm to include the epipolar constraint along with disparity, orientation, scale and uniqueness constraints. I’m hoping this will improve my matching algorithm and reduce any false positives. That in turn should improve my localisation accuracy as there will be fewer outliers and less error to minimise.

It’s been pretty hard to find time to work on this stuff recently as we have been so busy socially. I guess it’s that time of year with Christmas parties, catching up with friends etc. We are going to Germany this weekend and are off to India for two weeks from the 25th of Dec so I won’t really get time to do any decent hacking on this for the next month or so. I am really hoping that I have enough of the pieces in place that I’ll be able to make some good progress in the new year and actually be able to report some reasonable results.

I have been doing a little bit of reading about FPGA design lately and am getting quite excited about that stage of the project. I expect that will be immensely difficult but should be quite interesting too.

I’m finally making progress on my Vision project again. After spending a couple of months trying to figure out how to do nonlinear least squares minimisation I think it is now working. I still have quite a bit of testing to do before I am sure it is giving sensible results but so far it looks good. I couldn’t have gotten this working without the tireless help of Robin Hewitt who (via the Seattle Robotics Society mailing list) coached me through the process.

I have now returned to the beginning of the project again. Before I go any further I have decided to rework what I have done so far. My understand of computer vision concepts has come a long way since I started this project and I am sure I can improve my original code.

To that end I am writing a camera calibration application that will more accurately calibrate my camera system. This program will be able to calculate the intrinsic parameters of each camera and save and load these parameters from file. It will also be able to calculate the essential matrix of the stereo system and hence will be enable me to far more accurately match SIFT features as I will be able to include the epipolar constraint which I am currently not using.

I am hoping this calibration work will have a drastic improvement on the accuracy of the system, but only time will tell I guess.

So close and yet so far. That is how it feels at the moment. I found the bug I mentioned in the last post, and had hoped would be the last piece of the puzzle before I had camera ego motion calculations working, but it wasn’t to be.

I now have a problem with the build up of error over consecutive frames. My sanity check to see if ego motion calculation is working is to run the system completely stationary ( the only option at the moment anyway ). Therefore the ego motion calculation between frames should render a 0, 0, 0, 0 result. i.e No movement in any of the 3 spatial dimensions and no rotation. Unfortunately I am getting results that look more like this after one iteration:
—————————————–
New position after ego motion estimation:
X = -0.044358336825996
Y = 0.025654321647698
Z = -0.003337081060976
Theta = 0.431281320335258

Obviously subsequent iterations compound this error and it quickly spirals out of control.

Now I realise in the real system there will be the odometery feeding into this and the odometery and ego motion will be combined in a kalman filter to get the actual estimated motion. In this way the accumulative error won’t matter so much as it will be dampened by the odometery each iteration.

My concern is that this error is far too big to start off with. I can accept (and was expecting) some error between frames, but in the above example the system believes it has moved 4cm to the left. 4cm is too much. What I have to do now is identify the source of this error. I suspect it may be something to do with excessive accuracy in the SIFT feature 3d position calculations. Any error between calculated feature locations could be being exacerbated by the least squares minimisation. That at least is my uneducated guess at this point. More investigation will hopefully shed some light on this issue.

At the risk of jinxing myself completely I think I am nearly at the point where my vision system is correctly calculating camera ego motion! Now if that didn’t make you excited it should because it is very very cool.

I managed to spend a bit of time hacking on the weekend and made a huge amount of progress. Now that I have identified what I suspect (but haven’t yet confirmed) are libsifts short comings and moved to using Lowe’s example binary to extract SIFT features things are starting to work. The matching algorithm I was using to match features between stereo images and between frames was also a bit broken, so I ported Lowe’s example matching algorithm from C to C# and that greatly improved matching performance. So now I have one bug to squash and ego motion calculations will be working. I’m pretty sure I know where the problem lies so hopefully I’ll get that nailed in the next day or two.

At that point I will have one remaining task before I can declare phase one complete. I still have to figure out how to use the camera extrinsic parameters to give more accuracy to the stereo triangulation calculations. At that point the system will be able to accurately build a map of SIFT features observed in it’s environment, and simultaneously localise itself within that environment.