I wanted to write a ROS node that will find edges or other common features in a scene (I'm using the checkerboard) to allow 3D range devices like 3D lidars or stereo cameras, to be registered with a color camera in order to produce colored blobs. I have done this manually in the past, but I want an automated solution.

The problem is that in order to do this correctly, you need to be able to freeze the internal parameters of one camera, while letting the others drift. This is because some devices have hardcoded internal parameters that they use in making the pointclouds. These are often quite different from what comes out of StereoCalibrate. [[StereoCalibrate]]. So, I'd like to fix these intrinsic parameters, but let the intrinsic parameters of the second camera be optimized to "absorb" any error that exists from the first camera's intrinsic parameters.

(Hopefully that made sense).

One can do this with the MATLAB stereo_calib toolbox, and it works great for registering a color camera with a SwissRanger [[SwissRanger]] lidar (the extrinsic parameters and the 2D color camera intrinsic parameters are optimized while the SwissRanger [[SwissRanger]] intrinsic parameters remain fixed).

Unfortunately, there is no way to do this in OpenCV [[OpenCV]] due to the rigidness of StereoCalibrate. [[StereoCalibrate]].