Using CAP_MSMF and CAP_DSHOW at same time with two cameras causes "can't grab frame. Error: -2147483638"

Hi guys,

I try to use two USB cams at the same time, one uses VideoCapture() with CAP_MSMF (thermal camera) and the other one with CAP_DSHOW (webcam).

System: Windows 10 (20H2)
Python: 3.9.4
opencv-python: 4.5.2.52

Both programs work fine if itself if started as single application, but if I start both at the same time, then the camera which uses CAP_MSMF ends with the following error message during read():

[ WARN:0] global C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-m8us58q4\opencv\modules\videoio\src\cap_msmf.cpp (1021) CvCapture_MSMF::grabFrame videoio(MSMF): can't grab frame. Error: -2147483638

This is how the call of the webcam with CAP_DSHOW looks like:

        # initialize the video stream
        print("[INFO] starting video stream...")
        vs = VideoStream(src=1)  # call inside WebcamVideoStream class: self.stream = cv2.VideoCapture(src, cv2.CAP_DSHOW)
        
        print("CAP_PROP_FRAME_WIDTH")
        vs.stream.stream.set(cv2.CAP_PROP_FRAME_WIDTH, 1280)
        print("CAP_PROP_FRAME_HEIGHT")
        vs.stream.stream.set(cv2.CAP_PROP_FRAME_HEIGHT, 720)
        print("CAP_PROP_FPS")
        vs.stream.stream.set(cv2.CAP_PROP_FPS, 25)

        vs.start()

        # loop over the frames from the video stream
        while True:
            # grab the frame from the threaded video stream
            frame = vs.read()

        ...
       # not relevant frame processing snipped

            cv2.imshow("webcam_tf", frame)
            if cv2.waitKey(20) == 27:
                break

This is how the call of the thermal cam with CAP_MSMF looks like:

        print("starting thermalcam...")
        cv2.namedWindow("thermalcam")
        cap = cv2.VideoCapture(0, cv2.CAP_MSMF)

        print("CAP_PROP_FPS")
        cap.set(cv2.CAP_PROP_FPS, 25)

        # request raw data from CV
        cap.set(cv2.CAP_PROP_CONVERT_RGB, 0)

        # request raw data from camera
        cap.set(cv2.CAP_PROP_ZOOM, 0x8004)

        # shutter
        cap.set(cv2.CAP_PROP_ZOOM, 32768)

        if cap.isOpened():  # try to get the first frame
            print("Read first frame")
            rval, frame = cap.read()
        else:
            print("Error opening webcam")
            rval = False

        if rval:

            print("Start cyclic reading")
            while True:
                # WEBCAM

                rval, frame = cap.read()
                if not rval:
                    break

		...
		# not relevant frame processing snipped
		
                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
                heatmap_gray = cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY)
                heatmap = cv2.applyColorMap(heatmap_gray, cv2.COLORMAP_HOT)


                heatmap = imutils.resize(heatmap, width=640)
                cv2.imshow("thermalcam", heatmap)

                if cv2.waitKey(20) == 27:
                    break

I actually load both cameras into separate processes, means they should not use the same ressources inside Python for sure.
This error even happens if I start the second camera in a separate Python (Miniconda) environment.

Any idea what could cause this problem on the CAP_MSMF device if a second device with CAP_DSHOW gets started?

do not use “VideoStream” from imutils. that library adds nothing of value here.

directly use OpenCV’s VideoCapture.

try accessing both cameras with apiPreference=cv.CAP_MSMF

Even if adds nothing of value here, could it cause any problems like this?

I am actually pretty new to Python as well as to OpenCV and I try to run the Face-Mask-Detection Python script which further uses the imutils stream in detect_mask_video.py
If there is no problem to expect with using “VideoStream” from imutils, then I would prefer to stay with it at the moment to prevent too much code changes there on the begin.

With CAP_MSMF on the Webcam I experienced a very long start up time on the Webcam, probably caused by this issue there.
This was the reason why I changed for testing purposes directly the call in the imutils library file webcamvideostream.py as followed to test it with CAP_DSHOW.

#		self.stream = cv2.VideoCapture(src)
		self.stream = cv2.VideoCapture(src, cv2.CAP_DSHOW)
		(self.grabbed, self.frame) = self.stream.read()

These set() functions also fail (set_ret == false) if I use CAP_MSMF instead of CAP_DSHOW as API backend.

        print("CAP_PROP_FRAME_WIDTH")
        set_ret = vs.stream.stream.set(cv2.CAP_PROP_FRAME_WIDTH, 1280)
        print("CAP_PROP_FRAME_HEIGHT")
        set_ret = vs.stream.stream.set(cv2.CAP_PROP_FRAME_HEIGHT, 720)
        print("CAP_PROP_FPS")
        set_ret = vs.stream.stream.set(cv2.CAP_PROP_FPS, 25)

So far as I have seen does imutils internally also use VideoCapture().
Would you think it could help to not use the “VideoStream” from imutils?

since you are aware of its implementation and how it uses VideoCapture, there’s no harm. it is however needless abstraction.

since MSMF seems to cause trouble, can you open both cameras using CAP_DSHOW instead?

1 Like

There could be problems caused by imuitils. For example, it could be programmed to use to one just camera configuration that it uses for multiple VideoCaptute objects. And anyway, any unnecessary code is a potential for unnecessary bugs.

To read out the thermal cam I use a modified version from the ht301_hacklib.

#        if video_dev == None:
#            video_dev = self.find_device()

#        self.cap = cv2.VideoCapture(video_dev)
#        self.cap = cv2.VideoCapture(0, cv2.CAP_DSHOW)
        self.cap = cv2.VideoCapture(0, cv2.CAP_MSMF)
        if not self.isHt301(self.cap):
            Exception('device ' + str(video_dev) + ": HT301 not found!")

        ret_val = self.cap.set(cv2.CAP_PROP_CONVERT_RGB, 0)

        # Use raw mode command to camera
        self.cap.set(cv2.CAP_PROP_ZOOM, 0x8004)

The data frame after read() looks like this:

ret, frame = self.cap.read()

2021-05-09_17-04-02_ht301_hacklib.py_-ht301_hacklib-_Visual_Studio_C

And it is possible to reshape it to get a 2 dimensional array of u16 with 256x196.

dt = np.dtype('<u2')
frame = frame.view(dtype=dt).reshape(196, 256) #(frame.shape[:2]))

So far as I understand is this then the CV_16UC1 frame format.

If I change the API backend to CAP_DSHOW, then the frame format changes like this (so far as I understand is it then CV_8UC3):

array([[[ 33, 255,  78],
        [ 32, 255,  77],
        [ 32, 255,  77],
        ...,
        [ 28, 255,  73],
        [ 29, 255,  74],
        [ 27, 255,  72]],

       [[ 36, 255,  81],
        [ 30, 255,  75],
        [ 35, 255,  80],
        ...,
        [ 29, 255,  74],
        [ 24, 255,  69],
        [ 25, 255,  70]],

       [[ 36, 255,  81],
        [ 33, 255,  78],
        [ 37, 255,  82],
        ...,
        [ 29, 255,  74],
        [ 27, 255,  72],
        [ 28, 255,  73]],

       ...,

       [[  0, 247,   0],
        [ 61, 255,  86],
        [ 31, 230, 255],
        ...,
        [255, 173,   0],
        [144,  81,   0],
        [164, 101,   0]],

       [[  0, 154,   0],
        [  0, 127,   0],
        [  0, 113,   0],
        ...,
        [207, 122,   0],
        [  0, 109,   0],
        [  0, 149,   0]],

       [[  0, 187,   0],
        [  0, 253,   0],
        [178, 115, 169],
        ...,
        [139, 112, 255],
        [255,  47, 171],
        [180,   0,  90]]], dtype=uint8)

(can only post 1 image as new user)
Its size is shown as 150528 bytes.

I am pretty sure that I have seen ret_val == False during my tests with CAP_DSHOW, but I cannot repeat that anymore. Right now it is True with both backends.

# request raw data from CV
ret_val = cap.set(cv2.CAP_PROP_CONVERT_RGB, 0)

Why does the output frame format change when I change the API backend and how can I get it from there to the CV_16UC1 format which is required to further process the frame?

So far as I can see decides its code only to us a piCamera or the cv2.VideoCapture().

How could imutils affect the camera operation even if I run the second camera in a separate Python Miniconda environment?

that should have an effect even with CAP_DSHOW.

if it does not, it’s a bug in OpenCV and worth opening an issue.

perhaps the functionality hasn’t been implemented… or it hasn’t been implemented for whatever pixel format the device reports.

either way, you’ve encountered some bugs I’d say are located in OpenCV.

  • using both MSMF and DSHOW at the same time should be working
  • DSHOW and CAP_PROP_CONVERT_RGB being false should work as expected, not return an RGB/BGR array

I understand, that if I set cv2.CAP_PROP_CONVERT_RGB = 0, then I should receive a frame as 1D u8 array, independent from the backend.

Some more tests I did with the two cameras do not reflect my understanding.

Thermal cam with CAP_MSMF:

import cv2
import numpy as np

# devices: 0 = thermal cam, 1 = webcam
#video = cv2.VideoCapture(0, cv2.CAP_DSHOW)
video = cv2.VideoCapture(0, cv2.CAP_MSMF)
#video.set(cv2.CAP_PROP_FOURCC, 0x32595559)

# request raw data from camera
ret_val = video.set(cv2.CAP_PROP_ZOOM, 0x8004)

# activate shutter at camera
ret_val = video.set(cv2.CAP_PROP_ZOOM, 32768)

# request RAW data from CV API
ret_val = video.set(cv2.CAP_PROP_CONVERT_RGB, 0)

print ("Video FOURCC")
print (hex(int(video.get(cv2.CAP_PROP_FOURCC)) & 0xffffffff))

if video.isOpened(): # try to get the first frame
    rval, frame = video.read()
else:
    rval = False
Video FOURCC
0x32595559

0x32595559 == “2YUY”
Returns 100352 bytes as shape:(1, 100352).

Thermal cam with CAP_DSHOW:

import cv2
import numpy as np

# devices: 0 = thermal cam, 1 = webcam
video = cv2.VideoCapture(0, cv2.CAP_DSHOW)
#video = cv2.VideoCapture(0, cv2.CAP_MSMF)
#video.set(cv2.CAP_PROP_FOURCC, 0x32595559)

# request raw data from camera
ret_val = video.set(cv2.CAP_PROP_ZOOM, 0x8004)

# activate shutter at camera
ret_val = video.set(cv2.CAP_PROP_ZOOM, 32768)

# request RAW data from CV API
ret_val = video.set(cv2.CAP_PROP_CONVERT_RGB, 0)

print ("Video FOURCC")
print (hex(int(video.get(cv2.CAP_PROP_FOURCC)) & 0xffffffff))

if video.isOpened(): # try to get the first frame
    rval, frame = video.read()
else:
    rval = False
Video FOURCC
0xe436eb7d

0xe436eb7d == “ä6ë}”

Returns 150528 bytes as shape:(196, 256, 3).

Webcam with CAP_MSMF:

import cv2
import numpy as np

# devices: 0 = thermal cam, 1 = webcam
#video = cv2.VideoCapture(1, cv2.CAP_DSHOW)
video = cv2.VideoCapture(1, cv2.CAP_MSMF)
#video.set(cv2.CAP_PROP_FOURCC, 0x32595559)

# request raw data from camera
ret_val = video.set(cv2.CAP_PROP_ZOOM, 0x8004)

# activate shutter at camera
#ret_val = video.set(cv2.CAP_PROP_ZOOM, 32768)

# request RAW data from CV API
#ret_val = video.set(cv2.CAP_PROP_CONVERT_RGB, 0)

print ("Video FOURCC")
print (hex(int(video.get(cv2.CAP_PROP_FOURCC)) & 0xffffffff))

if video.isOpened(): # try to get the first frame
    rval, frame = video.read()
else:
    rval = False
Video FOURCC
0x16

0x16 == not a readable ASCII code.
Returns 921600 bytes as shape:(480, 640, 3).

Webcam with CAP_DSHOW:

import cv2
import numpy as np

# devices: 0 = thermal cam, 1 = webcam
video = cv2.VideoCapture(1, cv2.CAP_DSHOW)
#video = cv2.VideoCapture(1, cv2.CAP_MSMF)
#video.set(cv2.CAP_PROP_FOURCC, 0x32595559)

# request raw data from camera
ret_val = video.set(cv2.CAP_PROP_ZOOM, 0x8004)

# activate shutter at camera
#ret_val = video.set(cv2.CAP_PROP_ZOOM, 32768)

# request RAW data from CV API
#ret_val = video.set(cv2.CAP_PROP_CONVERT_RGB, 0)

print ("Video FOURCC")
print (hex(int(video.get(cv2.CAP_PROP_FOURCC)) & 0xffffffff))

if video.isOpened(): # try to get the first frame
    rval, frame = video.read()
else:
    rval = False
Video FOURCC
0x32595559 == "2YUY"

Returns 921600 bytes as shape:(480, 640, 3).

Is this the way how it should be?

I did some tests to isolate this problem a bit more.
During my tests a had two webcams connected.

test1.py, which uses CAP_MSMF:

import cv2
import numpy as np

#video = cv2.VideoCapture(0, cv2.CAP_DSHOW)
video = cv2.VideoCapture(0, cv2.CAP_MSMF)

print ("Video FOURCC")
print (hex(int(video.get(cv2.CAP_PROP_FOURCC)) & 0xffffffff))

if video.isOpened(): # try to get the first frame
    rval, frame = video.read()
else:
    rval = False

while 1:
    cv2.imshow("preview1", frame)
    rval, frame = video.read()

    key = cv2.waitKey(1)
    if key == 27: # exit on ESC
        break

test2.py, which uses CAP_DSHOW:

import cv2
import numpy as np

video = cv2.VideoCapture(1, cv2.CAP_DSHOW)
#video = cv2.VideoCapture(1, cv2.CAP_MSMF)

print ("Video FOURCC")
print (hex(int(video.get(cv2.CAP_PROP_FOURCC)) & 0xffffffff))

if video.isOpened(): # try to get the first frame
    rval, frame = video.read()
else:
    rval = False

while 1:
    cv2.imshow("preview2", frame)
    rval, frame = video.read()

    key = cv2.waitKey(1)
    if key == 27: # exit on ESC
        break

If I start test2.py (CAP_DSHOW) first and after it test1.py (CAP_MSMF), then everything is fine and both videos are shown.
But if I do it the other way around, then video.isOpened() in test2.py returns false and it is not possible to read a frame.

The behavior is a bit different to the issue which happened during my test with the thermal cam, but it points out that there is a problem using CAP_DSHOW and CAP_MSMF at the same time.

1 Like

I did open an issue regarding the MSMF and DSHOW problem, it clearly looks to me like an unwanted behaviour.

Not sure about the CAP_PROP_CONVERT_RGB issue, maybe I do not understand completly how it should work.
Would you say that these results there are how it should be? Or should it return a 1D uint8 array in every situation if CAP_PROP_CONVERT_RGB is set to 0?

depends.

if the source is RGB, it should ideally return RGB (rather than BGR, which is OpenCV’s “native” format).
if the source is some YUV, it should return that… or “bytes”.
if the source is Y16 (grayscale) it should return that… or “bytes”.
if the source is some compressed data, it should return that as “bytes”.

if it gives you some 3-channel data for the thermal camera, even though you asked for disabled CONVERT_RGB, it’s a bug… or the camera has a “capture pin” that emits a false-color picture in addition to physical quantities (temperature).

The camera has several modes (e.g. colored RGB) and one of them is the raw mode which gets activated via this command, using the zoom channel.

The commands work fine in general with CAP_MSMF as well as with CAP_DSHOW, because the shutter command works without problems with both APIs and it also uses the zoom channel.

If I set the RAW data command to the camera, then the camera stays in RAW mode until it looses its power. I can see this by opening any other camera application accessing the thermal camera after I have sent a command for RAW or any other modes.

If the camera provides RAW data but it only works via CAP_MSMF, then I asume a problem with CAP_DSHOW.
Is the raw mode operation just rarely used in opencv that no one would notice such a problem?

it’s a fairly new feature and not implemented uniformly across all the backends… because nobody knows all the backends, or can test them.

most people who notice a problem either live with it or work around it. you’re the first of the remaining few who actually need it to work and can’t use a workaround. someone is always first. the world can be very small sometimes.

the open issue about failure to open the device is a good step.

the “raw” mode (CONVERT_RGB false) for DSHOW not working right… I see some issues that look related, so I guess someone’s aware of the problem at least: Issues · opencv/opencv · GitHub

feel free to add any of your investigative findings to these issues, or link back to this thread. it’d help anyone willing to implement a fix.

1 Like