I see, this makes sense. So letting Python decide and switch threads when the current thread is possibly waiting for a frame removes the throttling issue.
I also tested this:
cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)
For each VideoCapture and after testing for two days it looks like the issue is gone. I’m even doing a lot of processing with PyTorch now while grabbing these frames. But I’m going to go ahead and implement your suggestion too.