Update ROI of warp map using initUndistortRectifyMap()

,
1

I have a use case where I need to periodically change the warp maps for a camera (the extrinsics change). The operation is too expensive for me to update the full map all at one time - I need to be able to continue processing images as they arrive. I have attempted to update the map by by acting on an ROI of the map, but this appears to re-allocate / re-initialize the map (or otherwise do something that makes the original map change).

I looked over the code for initUndistortRectifyMap and Mat::create, and it appeared that it would not re-allocate the Mat if the type and dims were the same, but so far (using an ROI to index into the warp map, and by creating a warp map that uses the memory from the original (full size) warp map) and so far I haven’t been able to get it to work.

Here is a recent assertion failure (when calling initUndistortRectifyMap on a cv::Mat that uses the original map memory):


OpenCV Error: Assertion failed (!fixedType() || ((Mat*)obj)->type() == mtype) in create, file /home/steve/embedded-repo/opencv/modules/core/src/matrix.cpp, line 2386
terminate called after throwing an instance of 'cv::Exception'
  what():  /home/steve/embedded-repo/opencv/modules/core/src/matrix.cpp:2386: error: (-215) !fixedType() || ((Mat*)obj)->type() == mtype in function create

At this point my question is “should I be able to call initUndistortRectifyMap() on an ROI and update the original map without disrupting the rest of the map?”

For example:
Create the map like this:

cv::initUndistortRectifyMap(m_camMat, m_distCoeffs, cv::Mat(),
                            perspective*m_camMat, cv::Size(outputWidth, outputHeight), CV_16SC2,
                            ar_warpMap1, ar_warpMap2);

and then later update it something like:

cv::initUndistortRectifyMap(m_camMat, m_distCoeffs, cv::Mat(),
                            perspective*m_camMat, cv::Size(roiWidth, roiHeight), CV_16SC2,
                            ar_warpMap1(roi), ar_warpMap2(roi));

Alternatives I’m considering if this doesn’t work:

  1. Create a warp map for the ROI in a separate map, and then copy the resulting data to the correct location in the original map.
  2. Creating an array of warp maps (rectangular tiles) and update them (and apply them) individually.

Any guidance would be appreciated.

Thanks

-Steve

2

can you explain a bit more here ?

maybe you can precalculate / cache maps for expected changes ?

3

I’d spawn a worker thread for this, and when it’s done, it updates the map in use from the newly calculated map.

4

A little bit of context might help. The camera is more or less fixed, and has a view of a 2D work surface. The camera has a highly distorted WFOV lens (fixed focal length) and the intrinsics are accurately calibrated (0.15 pixel RMS reprojection error). The goal is to present an undistorted / perspective corrected image to a user. The user identifies a point feature in the image and selects it with the mouse. From the mouse position we are able to determine an accurate 2D location of that feature (so, for example, we could command a robot to go drill a hole “exactly” at that feature). The goal is for average positioning error to be <= 0.5 pixels (this corresponds to about 0.015" at the work surface).

What I have described is implemented and works well. The problem is that the camera is mounted in a way that isn’t perfectly fixed, so it experiences very slight translation and rotation changes. The camera movement can’t be solved mechanically. The changes are small, but big enough to exceed our total error if they aren’t accounted for. To account for this, the camera extrinsics are continually monitored / updated based on the image positions of known 3D features. We are able to detect and account for translations on the order of thousandths of an inch. This also is implemented and works well.

Unfortunately the extrinsics change is unpredictable, so pre-computing the warp maps won’t work. Even if that were an option in theory, the system is resource constrained and I don’t have enough memory to store more than one warp map.

Updating the entire warp map takes approximately one second of compute time. Even if we could stall the image pipeline for a full second (we can’t), the operation saturates one core (of two) on the CPU and causes system performance issues elsewhere.
The extrinsics change periodically - say every 5-15 seconds, and I need a way to update the warp map in place, and over time.

I was hoping that I could do that by calling initUndistortRectifyMap on an ROI of an existing map, but that doesn’t seem to work. If remap() took an additional parameter that altered the map lookup (a 2D homography would work great), that would be an option too.

For the moment I’m still committed to making the “update a warp map ROI” method work because it would fit in nicely with my current architecture, but I’m not opposed to bigger changes if that’s what’s needed. Additional memory or compute resources are not an option at this point.

Again, I’m open to ideas. I’m going to keep working on this and I’ll update with any news.

Thanks

Steve

5

I like this idea, but unfortunately I don’t have enough memory to double-buffer the maps. I could break the work in to partial map updates, and push the results to the full map with mat::copyTo(). (I think that would work). Breaking it into sub-jobs would also let me rate-limit the update process so I don’t hog the CPU.

The main downside is the additional copyTo() overhead, but I think that’s likely to be small. There is also some synchronization work / overhead, but again probably not a big issue. Architecturally it might actually be cleaner / more maintainable.

I’ll probably give this a shot next.

Thanks.

6

I see no reason why initUndistortRectifyMap would reallocate the map matrices you pass in, if they’re the correct size and type already. you just have to make sure the other parameters don’t change in any way that would make that necessary.

it should be able to update those maps in-place.

you could dig into the source code and modify it… it might be possible to add a ROI Rect parameter.

7

Yes, you are correct. It took me a while to track down my error, but once I did it started working as expected.

My error: I had created both maps as CV_16SC2 type, but the second map needs to be CV_16UC1.

So this is great news for me - I think I can do exactly what I want. Sorry for the confusion, and thanks for the help.

-Steve