OpenCV BackgroundSubtractor.getBackgroundImage crashes with invalid frame type

C++
,
1

I’m using OpenCV in an iOS app via Objective-C wrapper and it works generally fine but an attempt to read a background image from a background subtractor crashes the app:

cv::Ptr<BackgroundSubtractor> backSub = createBackgroundSubtractorMOG2();
for (int i = 0; i < arr.count; i++) { // arr - NSArray of UIImage
    Mat mask;
    Mat frame = [OpenCVWrapper _matFrom:arr[i]]; // converts UIImage to OpenCV.Mat
    backSub->apply(frame, mask); // this works great, I can read the mask and it's valid
}
    
Mat background;
backSub->getBackgroundImage(background); // this crashes

The error implies that my frame has an invalid format. But in my case, I’m just creating a variable to be used by the getBackgroundImage. I tried to create a new Mat with the expected type CV_8UC1 (and CV_8UC3) with no luck, the app is still crashing with the same error.

libc++abi: terminating with uncaught exception of type cv::Exception: OpenCV(4.3.0) /Volumes/build-storage/build/master_iOS-mac/opencv/modules/video/src/bgfg_gaussmix2.cpp:929: error: (-215:Assertion failed) frameType == CV_8UC1 || frameType == CV_8UC3 || frameType == CV_32FC1 || frameType == CV_32FC3 in function 'getBackgroundImage'

dyld4 config: DYLD_LIBRARY_PATH=/usr/lib/system/introspection DYLD_INSERT_LIBRARIES=/Developer/usr/lib/libBacktraceRecording.dylib:/Developer/usr/lib/libMainThreadChecker.dylib:/Developer/Library/PrivateFrameworks/DTDDISupport.framework/libViewDebuggerSupport.dylib

Same code works flawlessly in my python test (declare variable and store the background there):

background = backSub.getBackgroundImage()

How should I properly access a background image from createBackgroundSubtractorMOG2 in Objective-C?

2

it complains about the frame 's type, not about the backgroundimage.

so please check, what [OpenCVWrapper _matFrom:arr[i]] returns, exactly.

(any chance, it is CV_8UC4 ? apply() would work, but getBackgroundImage() will throw !)

also, crossposted here:

1 Like
3

Thanks for the reply!
_matFrom:arr[i] returns an CV_8UC4 image

+ (Mat)_matFrom:(UIImage *)source {
    CGImageRef image = CGImageCreateCopy(source.CGImage);
    CGFloat cols = CGImageGetWidth(image);
    CGFloat rows = CGImageGetHeight(image);
    Mat result(rows, cols, CV_8UC4);
    CGBitmapInfo bitmapFlags = kCGImageAlphaNoneSkipLast | kCGBitmapByteOrderDefault;
    size_t bitsPerComponent = 8;
    size_t bytesPerRow = result.step[0];
    CGColorSpaceRef colorSpace = CGImageGetColorSpace(image);
    CGContextRef context = CGBitmapContextCreate(result.data, cols, rows, bitsPerComponent, bytesPerRow, colorSpace, bitmapFlags);
    CGContextDrawImage(context, CGRectMake(0.0f, 0.0f, cols, rows), image);
    CGContextRelease(context);
    return result;
}
1 Like
4

ok, so, as it is now, your options are:

  • cvtConvert the frame back to BGR before apply() (expensive)
  • hack the resp. assert in the src code (it should work with CV_8UC4, imo)
  • seeing the bgimage is nice for debugging, but rarely nessecary in production (it’s an artificial, generated image) why not simply ditch it ?
5

Thanks for the options and the help!

Option #1: could you please elaborate here? I thought the Mat representation of the image was already in BGR format. If not, can I create it in this format because this is the only place I currently use it?

Option #2: hm… I can certainly change one line of the code (asset) but I’m probably not that proficient in rebuilding the iOS framework somehow after that, I have no idea how that framework was created and doubt that the process is documented.

Option #3: I specifically need a background image for my use case. I’m implementing a photo-creation app, which processes every frame of the video, strips moving objects (estimates the background), and then lets a user pick specific frames to apply to that estimated background (I’m using the frame and the frame mask to copy to background). I currently estimating the background manually by calculating the median frame but it’s very slow and I was hoping just to get the background from the background subtractor, in python it works very fast.