You can first use the bounding rect intersections for a fast estimate whether contours might intersect.
You can even compare the boundingRect sizes as a first indication of the possible-best IoU those contours can have. For example, if A is very big and B ist very small, the IoU can’t be high.
Once you precomputed the contour masks you can run the IoU computation on subimages of region boundingRectA | boundingRectB.
To save memory you can even draw “local masks” which are only as big as the bounding rect of that contour, but that makes IoU computation on those masks more complicated and a tiny bit slower, so maybe dont think about that unless you are working with very big images.