The detection script has been moved here: opencv/detect_msdk.cmake at master · opencv/opencv · GitHub You can add debug messages to this file to find out where the problem is.
And I remembered that you also need libva-dev package for it to work, perhaps this is your issue.
MFX: Can't initialize session
means that there was a problem with MediaSDK initialization, perhaps it is not compatible with your system (MSDK or iHD driver).
Try to use GStreamer backend:
- install gstreamer1.0-vaapi package and maybe gstreamer1.0-plugins-good and gstreamer1.0-plugins-ugly
- create VideoCapture with GStreamer pipeline instead of filename:
VideoCapture("filesrc location=<filename.mp4> ! qtdemux ! vaapidecodebin ! videoconvert ! appsink sync=false")