VideoWriter_IntelMFX: Written file is not playable

schorsch_76 · September 13, 2022, 1:52pm

Hello,

i run OpenCV 4.5.4 on Windows 10 x64 21H2. I compiled it from source with the Intel Media SDK MSDK2021R1.exe.

In my application i grab files and write them to HDD as a video file. I open it as following:

			p_writer->open("video.mp4",
						   cv::VideoWriter::fourcc('A', 'V', 'C', ' '),
						   file_fps, frame_size,
				{
					cv::VIDEOWRITER_PROP_HW_ACCELERATION,  cv::VIDEO_ACCELERATION_MFX,
					cv::VIDEOWRITER_PROP_HW_DEVICE, -1
				});

I write as following:

		if (p_writer && p_writer->isOpened())
		{
			*p_writer << current_frame.GetMat();
		}

When i step into the write function i can see that the VideoWriter_IntelMFX::write_one() function is used and the file is extended. But the video file is not playable. With ffmpeg i can check it, there is the codec indicated but it reports no duration… Visual Studios Binary Editor shows that there is “random data” in the file…

ffmpeg version 2022-09-12-git-3ce6fa6b6d-full_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 12.1.0 (Rev2, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libjxl --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-liblensfun --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      57. 36.101 / 57. 36.101
  libavcodec     59. 43.100 / 59. 43.100
  libavformat    59. 31.100 / 59. 31.100
  libavdevice    59.  8.101 / 59.  8.101
  libavfilter     8. 48.100 /  8. 48.100
  libswscale      6.  8.112 /  6.  8.112
  libswresample   4.  9.100 /  4.  9.100
  libpostproc    56.  7.100 / 56.  7.100
Input #0, h264, from 'video.mp4':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x940 [SAR 1:1 DAR 64:47], 29 fps, 29 tbr, 1200k tbn

For completeness the cmake configuration output. If i use the ffmpeg backend it works and the video is playable. I just played aroud with the container format and the codec. The MFX writer seem to support MPG2, H264, H265, AVC , HEVC codec (from the source).

Any help would be appreceated …

Selecting Windows SDK version 10.0.18362.0 to target Windows 10.0.19044.
Detected processor: AMD64
Could NOT find PythonInterp (missing: PYTHON_EXECUTABLE) (Required is at least version "2.7")
Could NOT find PythonInterp (missing: PYTHON_EXECUTABLE) (Required is at least version "3.2")
libjpeg-turbo: VERSION = 2.1.0, BUILD = opencv-4.5.4-libjpeg-turbo
Could NOT find OpenJPEG (minimal suitable version: 2.0, recommended version >= 2.3.1). OpenJPEG will be built from sources
OpenJPEG: VERSION = 2.4.0, BUILD = opencv-4.5.4-openjp2-2.4.0
OpenJPEG libraries will be built from sources: libopenjp2 (version "2.4.0")
found Intel IPP (ICV version): 2020.0.0 [2020.0.0 Gold]
at: C:/Daten/Entwicklung/3rdParty/OpenCV/build/3rdparty/ippicv/ippicv_win/icv
found Intel IPP Integration Wrappers sources: 2020.0.0
at: C:/Daten/Entwicklung/3rdParty/OpenCV/build/3rdparty/ippicv/ippicv_win/iw
Could not find OpenBLAS include. Turning OpenBLAS_FOUND off
Could not find OpenBLAS lib. Turning OpenBLAS_FOUND off
Could NOT find BLAS (missing: BLAS_LIBRARIES) 
Could NOT find LAPACK (missing: LAPACK_LIBRARIES) 
    Reason given by package: LAPACK could not be found because dependency BLAS could not be found.

Could NOT find JNI (missing: JAVA_AWT_LIBRARY JAVA_JVM_LIBRARY JAVA_INCLUDE_PATH JAVA_INCLUDE_PATH2 JAVA_AWT_INCLUDE_PATH) 
VTK is not found. Please set -DVTK_DIR in CMake to VTK build directory, or to VTK install subdirectory with VTKConfig.cmake file
ADE: Download: v0.1.1f.zip
Try 1 failed
CMake Warning at cmake/OpenCVDownload.cmake:202 (message):
  ADE: Download failed: 28;"Timeout was reached"

  For details please refer to the download log file:

  C:/Daten/Entwicklung/3rdParty/OpenCV/build/CMakeDownloadLog.txt

Call Stack (most recent call first):
  modules/gapi/cmake/DownloadADE.cmake:5 (ocv_download)
  modules/gapi/cmake/init.cmake:19 (include)
  cmake/OpenCVModule.cmake:298 (include)
  cmake/OpenCVModule.cmake:361 (_add_modules_1)
  cmake/OpenCVModule.cmake:385 (ocv_glob_modules)
  CMakeLists.txt:917 (ocv_register_modules)


OpenCV Python: during development append to PYTHONPATH: C:/Daten/Entwicklung/3rdParty/OpenCV/build/python_loader
Allocator metrics storage type: 'long long'
Registering hook 'INIT_MODULE_SOURCES_opencv_dnn': C:/Daten/Entwicklung/3rdParty/OpenCV/upstream/modules/dnn/cmake/hooks/INIT_MODULE_SOURCES_opencv_dnn.cmake
opencv_dnn: filter out cuda4dnn source code
Excluding from source files list: <BUILD>/modules/dnn/layers/layers_common.rvv.cpp
highgui: using builtin backend: WIN32UI

General configuration for OpenCV 4.5.4 =====================================
  Version control:               4.5.4

  Platform:
    Timestamp:                   2022-08-22T09:01:37Z
    Host:                        Windows 10.0.19044 AMD64
    CMake:                       3.21.0
    CMake generator:             Visual Studio 16 2019
    CMake build tool:            C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/MSBuild/Current/Bin/MSBuild.exe
    MSVC:                        1929
    Configuration:               Debug Release

  CPU/HW features:
    Baseline:                    SSE SSE2 SSE3
      requested:                 SSE3
    Dispatched code generation:  SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX
      requested:                 SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
      SSE4_1 (16 files):         + SSSE3 SSE4_1
      SSE4_2 (2 files):          + SSSE3 SSE4_1 POPCNT SSE4_2
      FP16 (1 files):            + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
      AVX (5 files):             + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
      AVX2 (31 files):           + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2
      AVX512_SKX (8 files):      + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX

  C/C++:
    Built as dynamic libs?:      YES
    C++ standard:                11
    C++ Compiler:                C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe  (ver 19.29.30139.0)
    C++ flags (Release):         /DWIN32 /D_WINDOWS /W4 /GR  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:precise     /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /MP  /MD /O2 /Ob2 /DNDEBUG 
    C++ flags (Debug):           /DWIN32 /D_WINDOWS /W4 /GR  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:precise     /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /MP  /MDd /Zi /Ob0 /Od /RTC1 
    C Compiler:                  C:/Program Files (x86)/Microsoft Visual Studio/2019/Professional/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe
    C flags (Release):           /DWIN32 /D_WINDOWS /W3  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:precise     /MP   /MD /O2 /Ob2 /DNDEBUG 
    C flags (Debug):             /DWIN32 /D_WINDOWS /W3  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:precise     /MP /MDd /Zi /Ob0 /Od /RTC1 
    Linker flags (Release):      /machine:x64  /INCREMENTAL:NO 
    Linker flags (Debug):        /machine:x64  /debug /INCREMENTAL 
    ccache:                      NO
    Precompiled headers:         YES
    Extra dependencies:
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 calib3d core dnn features2d flann highgui imgcodecs imgproc ml objdetect photo stitching ts video videoio
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 gapi java python2 python3
    Applications:                tests perf_tests apps
    Documentation:               NO
    Non-free algorithms:         NO

  Windows RT support:            NO

  GUI:                           WIN32UI
    Win32 UI:                    YES
    VTK support:                 NO

  Media I/O: 
    ZLib:                        build (ver 1.2.11)
    JPEG:                        build-libjpeg-turbo (ver 2.1.0-62)
    WEBP:                        build (ver encoder: 0x020f)
    PNG:                         build (ver 1.6.37)
    TIFF:                        build (ver 42 - 4.2.0)
    JPEG 2000:                   build (ver 2.4.0)
    OpenEXR:                     build (ver 2.3.0)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES (prebuilt binaries)
      avcodec:                   YES (58.134.100)
      avformat:                  YES (58.76.100)
      avutil:                    YES (56.70.100)
      swscale:                   YES (5.9.100)
      avresample:                YES (4.0.0)
    GStreamer:                   NO
    DirectShow:                  YES
    Media Foundation:            YES
      DXVA:                      YES
    Intel Media SDK:             YES (C:/Program Files (x86)/IntelSWTools/Intel(R) Media SDK 2021 R1/Software Development Kit/lib/x64/libmfx_vs2015.lib)

  Parallel framework:            Concurrency

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Intel IPP:                   2020.0.0 Gold [2020.0.0]
           at:                   C:/Daten/Entwicklung/3rdParty/OpenCV/build/3rdparty/ippicv/ippicv_win/icv
    Intel IPP IW:                sources (2020.0.0)
              at:                C:/Daten/Entwicklung/3rdParty/OpenCV/build/3rdparty/ippicv/ippicv_win/iw
    Lapack:                      NO
    Eigen:                       NO
    Custom HAL:                  NO
    Protobuf:                    build (3.5.1)

  OpenCL:                        YES (NVD3D11)
    Include path:                C:/Daten/Entwicklung/3rdParty/OpenCV/upstream/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python (for build):            NO

  Java:                          
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    C:/Daten/Entwicklung/3rdParty/OpenCV/4.5.4/x64
-----------------------------------------------------------------

Configuring done
Generating done

schorsch_76 · September 13, 2022, 1:59pm

I also realized, that i can reencode it with ffmpeg then it is playable in VLC and the Windows Media Player…

cudawarped · September 13, 2022, 3:14pm

If you give it the correct extention, i.e. .h264 vlc may be able to play it. If I label a raw .h264 file as .mp4 then vlc cannot play it either because it has no idea which codec is used.

schorsch_76 · September 13, 2022, 3:23pm

Thanks, i already tried that but no success

schorsch_76 · September 13, 2022, 3:33pm

Maybe I need to upgrade vlc… I got vlc 3.0.x

cudawarped · September 13, 2022, 3:43pm

I doubt it what does ffmpeg list as the codec for the file?

schorsch_76 · September 13, 2022, 4:02pm

Input #0, h264, from 'video.mp4':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x940 [SAR 1:1 DAR 64:47], 29 fps, 29 tbr, 1200k tbn

crackwitz · September 13, 2022, 4:11pm

code please ([mre])

are you calling the release() method?

schorsch_76 · September 13, 2022, 4:15pm

Yes, its is a std::unique_ptr<>. It calls the destructor when the application ends. I can tommorrow supply a MRE.

crackwitz · September 13, 2022, 6:56pm

I’m asking if you call the release() method explicitly, because you must. try it.

and still, without a [mre] nobody can debug your code for you.

schorsch_76 · September 14, 2022, 5:04am

https://docs.opencv.org/4.5.4/dd/d9e/classcv_1_1VideoWriter.html#aea9c6764cf067ff7af2576267d39c76c

The documentation states that the destructor does that but for the MRE i added an explicit call to release().

ffmpeg shows the same message as with my main application:

Input #0, h264, from 'video_0.mp4':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 640x480 [SAR 1:1 DAR 4:3], 30 fps, 30 tbr, 1200k tbn

Here it is.

#include <opencv2/videoio.hpp>

#include <string>
#include <memory>

std::string FileName(size_t idx)
{
	std::string fname = "video_" + std::to_string(idx);	
	return fname + ".mp4";
}

void Run(size_t idx, size_t MaxFrames)
{
	int file_fps = 30;

	auto p_cam = std::make_unique<cv::VideoCapture>();
	auto p_writer = std::make_unique<cv::VideoWriter>();

	while (p_cam && p_writer && MaxFrames > 0)
	{
		if (!p_cam->isOpened())
		{
			p_cam->open(0);
		}

		cv::Mat frame;
		*p_cam >> frame;


		if (!p_writer->isOpened())
		{
			cv::Size frame_size = frame.size();

			p_writer->open(FileName(idx),
				cv::VideoWriter::fourcc('H', '2', '6', '4'),
				file_fps, frame_size
				,
				{
					cv::VIDEOWRITER_PROP_HW_ACCELERATION,  cv::VIDEO_ACCELERATION_MFX,
					cv::VIDEOWRITER_PROP_HW_DEVICE, -1
				});
		}
		if (p_writer->isOpened())
		{
			*p_writer << frame;
			--MaxFrames;
		}
	}

	if (p_writer)
	{
		p_writer->release();
	}
}

int main(int argc, char** argv)
{
	for (size_t i = 0; i < 3; ++i)
	{
		Run(i, 300);
	}
}

Thats the callstack into the VideoWriter_IntelMfx::write_one() function

|>|opencv_videoio454d.dll!VideoWriter_IntelMFX::write_one(const cv::debug_build_guard::_InputArray & bgr) Zeile 192|C++|
|---|---|---|
| |opencv_videoio454d.dll!VideoWriter_IntelMFX::write(const cv::debug_build_guard::_InputArray & input) Zeile 187|C++|
| |opencv_videoio454d.dll!cv::VideoWriter::write(const cv::debug_build_guard::_InputArray & image) Zeile 676|C++|
| |opencv_videoio454d.dll!cv::VideoWriter::operator<<(const cv::Mat & image) Zeile 684|C++|
| |MfxMRE.exe!Run(unsigned __int64 idx, unsigned __int64 MaxFrames) Zeile 46|C++|
| |MfxMRE.exe!main(int argc, char * * argv) Zeile 61|C++|

schorsch_76 · September 14, 2022, 5:13am

Playing h264 folowwing that tutorial doesnt work either

schorsch_76 · September 14, 2022, 6:17am

Just upgraded to OpenCV 4.5.5 … no change …

EDIT:
FYI: I got it working with 4.5.4. The key was to read the tests.

github.com

opencv/opencv/blob/4.x/modules/videoio/test/test_mfx.cpp

// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html

#include "test_precomp.hpp"

namespace opencv_test { namespace {

TEST(videoio_mfx, read_invalid)
{
    if (!videoio_registry::hasBackend(CAP_INTEL_MFX))
        throw SkipTestException("MediaSDK backend was not found");

    VideoCapture cap;
    ASSERT_NO_THROW(cap.open("nonexistent-file", CAP_INTEL_MFX));
    ASSERT_FALSE(cap.isOpened());
    Mat img;
    ASSERT_NO_THROW(cap >> img);
    ASSERT_TRUE(img.empty());
}

This file has been truncated. show original

It has the file ending like .265 and the codec H265. Then VLC could open the file. the H264 i could not get working.

#include <opencv2/videoio.hpp>

#include <string>
#include <memory>
#include <iostream>

std::string FileName(size_t idx)
{
	std::string fname = "video_" + std::to_string(idx);	
	return fname + ".265";
}
int OurFourCC()
{
	return cv::VideoWriter::fourcc('H', '2', '6', '5');
}

void Run(size_t idx, size_t MaxFrames)
{
	int file_fps = 30;

	auto p_cam = std::make_unique<cv::VideoCapture>();
	auto p_writer = std::make_unique<cv::VideoWriter>();

	while (p_cam && p_writer && MaxFrames > 0)
	{
		if (!p_cam->isOpened())
		{
			p_cam->open(0);
		}

		cv::Mat frame;
		*p_cam >> frame;

		if (!p_writer->isOpened())
		{
			cv::Size frame_size = frame.size();

			p_writer->open(FileName(idx), 
				OurFourCC(),
				file_fps, frame_size
				,
				{
					cv::VIDEOWRITER_PROP_HW_ACCELERATION,  cv::VIDEO_ACCELERATION_MFX,
					cv::VIDEOWRITER_PROP_HW_DEVICE, -1
				});

			std::cout << "Writer Backend: " << p_writer->getBackendName() << "\n";
		}
		if (p_writer->isOpened())
		{
			*p_writer << frame;
			--MaxFrames;
		}
	}

	if (p_writer)
	{
		p_writer->release();
	}
}

int main(int argc, char** argv)
{
	for (size_t i = 0; i < 3; ++i)
	{
		Run(i, 300);
	}
}

schorsch_76 · September 14, 2022, 8:47am

But is is also true for the encoder?

cudawarped · September 14, 2022, 10:13am

Whilst I have not tried this myself the documentation

Where fourcc can be one of MPG2, H264, X264, AVC , H265 or HEVC

would imply this, which is why it doesn’t make sense for the output file to have a .mp4 extension. I am supprised that you had to change the output file name to “.h264” instead of simply changing the extension from “.mp4” to “.h264” once the file was written. I guess internally OpenCV is adding something to the file when you specify a “.mp4” extension.

crackwitz · September 14, 2022, 10:52am

ffmpeg is supposed to do all the work, usually, but that usually means CPU-only encode/decode. I don’t know if the ffmpeg backend can do hardware acceleration (maybe). MFX is something I know of but have never actually seen used by OpenCV.

the mp4 container is capable of carrying H.264 and H.265 streams. it does however require finalizing (adding indices, updating headers), like most container formats.

if ffmpeg were used, it wouldn’t have liked the "H264" fourcc. it prefers "avc1".

I would highly recommend stripping those “smart” pointers out of the program entirely. those are local variables.

And you must check isOpened() before doing any reading/writing. and you must NOT assume that open() will always succeed. if you open() and then !isOpened(), that is fatal. and you must not assume that reading a frame will always succeed. you must test whether it’s empty. if it is, all you can do is break the loop.

I am not in a position to use MFX but MSMF (picked by VIDEO_ACCELERATION_ANY), if told to write a “.mp4”, is perfectly happy to do everything… so the code seems mostly okay in principle.

cudawarped · September 14, 2022, 11:30am

Ignore my previous comment I didn’t notice you were using cv::VIDEO_ACCELERATION_MFX and not CAP_INTEL_MFX see post above instead.

schorsch_76 · September 14, 2022, 3:19pm

It was (at least for me) a strange issue … without the look into the source it would have been much more difficult .

Topic		Replies	Views
Build OpenCV with MFX for sandy bridge (linux)	4	1436	December 23, 2020
Can't read video saved with videoWriter C++ videowriter , videoio	6	2195	July 21, 2022
VideoWriter error invalid type C++ videoio	2	219	December 4, 2023
How to use FFMPEG of my System with OpenCV? videowriter , ffmpeg	6	3571	January 21, 2021
Unable to read video files C++ videoio	2	1066	April 21, 2022

VideoWriter_IntelMFX: Written file is not playable

Related Topics