Cross compiling OpenCV with CUDA 11.4 for ARM64

Hey,

I have docker image built on the top of nvidia/cuda:11.4.0-devel-ubuntu20.04 as base and dockcross ARM64/linux.

In the past I have cross compiled OpenCV 4.5.0 with CUDA 10.2 on a X86 for ARM64 successfully. While few dependencies have changed, I have almost managed to do the same with OpenCV 4.5.4 and CUDA 11.2 (as the base image states above).

However, I have encountered the following issue:

[ 79%] Building CXX object modules/video/CMakeFiles/opencv_test_video.dir/test/test_accum.cpp.o
[ 79%] Building CXX object modules/ximgproc/CMakeFiles/opencv_ximgproc.dir/src/anisodiff.cpp.o
[ 79%] Linking CXX executable ../../../../bin/opencv_waldboost_detector
/usr/xcc/aarch64-unknown-linux-gnu/lib/gcc/aarch64-unknown-linux-gnu/9.4.0/../../../../aarch64-unknown-linux-gnu/bin/ld.bfd: warning: libcudart.so.11.0, needed by ../../../../lib/libopencv_core.so.4.5.0, not found (try using -rpath or -rpath-link)

make[2]: *** [modules/xobjdetect/tools/waldboost_detector/CMakeFiles/opencv_waldboost_detector.dir/build.make:114: bin/opencv_waldboost_detector] Error 1
make[1]: *** [CMakeFiles/Makefile2:8565: modules/xobjdetect/tools/waldboost_detector/CMakeFiles/opencv_waldboost_detector.dir/all] Error 2

These are my cmake arguments:

cmake ..  \
    -D CUDA_INC_PATH=/usr/local/cuda/targets/aarch64-linux/include \
    -D CMAKE_LIBRARY_PATH=/usr/local/cuda/targets/aarch64-linux/lib/stubs \
    -D CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda/targets/aarch64-linux \
    -D CMAKE_FIND_ROOT_PATH=/opencv/opencv/cmake  \
    -D CMAKE_TOOLCHAIN_FILE=../cmake_aarch64.toolchain \
    -D OPENCV_EXTRA_MODULES_PATH=/opencv/opencv/opencv_contrib-${OPENCV_VERSION}/modules \
    -D CMAKE_BUILD_TYPE=Release \
    -D CMAKE_INSTALL_PREFIX=/output \
    -D BUILD_TIFF=ON \
    -D WITH_WEBP=OFF \
    -D BUILD_TBB=ON \
    -D BUILD_PNG=ON \
    -D WITH_LAPACK=OFF \
    -D BUILD_NEW_PYTHON_SUPPORT=ON \
    -D BUILD_JPEG=ON \
    -D BUILD_JASPER=ON \
    -D BUILD_ZLIB=ON \
    -D BUILD_EXAMPLES=OFF \
    -D BUILD_JAVA=OFF \
    -D BUILD_opencv_python2=OFF \
    -D BUILD_opencv_python3=ON \
    -D CUDA_NVCC_FLAGS="-D_FORCE_INLINES" \
    -D ENABLE_NEON=ON \
    -D WITH_OPENCL=OFF \
    -D WITH_OPENMP=OFF \
    -D WITH_FFMPEG=ON \
    -D WITH_GSTREAMER=OFF \
    -D WITH_GSTREAMER_0_10=OFF \
    -D WITH_CUDA=ON \
    -D WITH_GTK=ON \
    -D WITH_VTK=OFF \
    -D WITH_TBB=ON \
    -D WITH_1394=OFF \
    -D WITH_OPENEXR=OFF \
    -D CUDA_ARCH_BIN=7.2 \
    -D CUDA_ARCH_PTX="" \
    -D BUILD_EXAMPLES=OFF \
    -D INSTALL_C_EXAMPLES=ON \
    -D INSTALL_TESTS=OFF

And libcudart.so.11.0 exists in the container:

ls -la /usr/local/cuda/targets/aarch64-linux/lib | grep libcudart
lrwxrwxrwx 1 root root        17 Nov 15  2021 libcudart.so -> libcudart.so.11.0
lrwxrwxrwx 1 root root        21 Nov 15  2021 libcudart.so.11.0 -> libcudart.so.11.4.167
-rw-r--r-- 1 root root    670808 Nov 15  2021 libcudart.so.11.4.167
-rw-r--r-- 1 root root   1078022 Nov 15  2021 libcudart_static.a

Do you have any idea why this is happening? At first I thought that it was an issue with OpenCV 4.5.0 and CUDA 11.4, so I changed OpenCV to 4.5.4, but it seems like it does not help.

Thank you
Shaked

Edit:

I have ran the build in verbose mode using make VERBOSE=1 -j$(nproc), then I saw that right after the aforementioned error, there are two files:

collect2: error: ld returned 1 exit status
make[2]: *** [modules/xobjdetect/tools/waldboost_detector/CMakeFiles/opencv_waldboost_detector.dir/build.make:115: bin/opencv_waldboost_detector] Error 1
make[2]: Leaving directory '/opencv/opencv-4.5.4/build'
make[1]: *** [CMakeFiles/Makefile2:8944: modules/xobjdetect/tools/waldboost_detector/CMakeFiles/opencv_waldboost_detector.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Looking at modules/xobjdetect/tools/waldboost_detector/CMakeFiles/opencv_waldboost_detector.dir/build.make in line 115, I saw this command:

cd /opencv/opencv/build/modules/xobjdetect/tools/waldboost_detector && $(CMAKE_COMMAND) -E cmake_link_script CMakeFiles/opencv_waldboost_detector.dir/link.txt --verbose=$(VERBOSE)

Running it obviously generates the same error but gives a bit more information:

$ cmake -E cmake_link_script CMakeFiles/opencv_waldboost_detector.dir/link.txt --verbose=1
cmake -E cmake_link_script CMakeFiles/opencv_waldboost_detector.dir/link.txt --verbose=1
/usr/xcc/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++    -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG   -Wl,--gc-sections -Wl,--as-needed -Wl,-rpath-link,"/opencv/opencv/build/CMakeFiles/" CMakeFiles/opencv_waldboost_detector.dir/waldboost_detector.cpp.o -o ../../../../bin/opencv_waldboost_detector   -L/usr/local/cuda-11.4/targets/aarch64-linux/lib  -L/usr/local/cuda/targets/aarch64-linux/lib/stubs  -Wl,-rpath,/usr/local/cuda-11.4/targets/aarch64-linux/lib:/usr/local/cuda/targets/aarch64-linux/lib/stubs:/opencv/opencv/build/lib: ../../../../lib/libopencv_highgui.so.4.5.4 ../../../../lib/libopencv_xobjdetect.so.4.5.4 ../../../../lib/libopencv_videoio.so.4.5.4 ../../../../lib/libopencv_imgcodecs.so.4.5.4 ../../../../lib/libopencv_objdetect.so.4.5.4 ../../../../lib/libopencv_dnn.so.4.5.4 ../../../../lib/libopencv_calib3d.so.4.5.4 ../../../../lib/libopencv_features2d.so.4.5.4 ../../../../lib/libopencv_imgproc.so.4.5.4 ../../../../lib/libopencv_flann.so.4.5.4 ../../../../lib/libopencv_core.so.4.5.4 ../../../../lib/libopencv_cudev.so.4.5.4 -Wl,-rpath-link,/opencv/opencv/build/lib
/usr/xcc/aarch64-unknown-linux-gnu/lib/gcc/aarch64-unknown-linux-gnu/9.4.0/../../../../aarch64-unknown-linux-gnu/bin/ld.bfd: warning: libcudart.so.11.0, needed by ../../../../lib/libopencv_core.so.4.5.4, not found (try using -rpath or -rpath-link)

I followed the suggestion in the error above:

(try using -rpath or -rpath-link)

And instead of using:

/usr/xcc/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++    -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG   -Wl,--gc-sections -Wl,--as-needed -Wl,-rpath-link,"/opencv/opencv/build/CMakeFiles/" CMakeFiles/opencv_waldboost_detector.dir/waldboost_detector.cpp.o -o ../../../../bin/opencv_waldboost_detector   -L/usr/local/cuda-11.4/targets/aarch64-linux/lib  -L/usr/local/cuda/targets/aarch64-linux/lib/stubs  -Wl,-rpath,/usr/local/cuda-11.4/targets/aarch64-linux/lib:/usr/local/cuda/targets/aarch64-linux/lib/stubs:/opencv/opencv/build/lib: ../../../../lib/libopencv_highgui.so.4.5.4 ../../../../lib/libopencv_xobjdetect.so.4.5.4 ../../../../lib/libopencv_videoio.so.4.5.4 ../../../../lib/libopencv_imgcodecs.so.4.5.4 ../../../../lib/libopencv_objdetect.so.4.5.4 ../../../../lib/libopencv_dnn.so.4.5.4 ../../../../lib/libopencv_calib3d.so.4.5.4 ../../../../lib/libopencv_features2d.so.4.5.4 ../../../../lib/libopencv_imgproc.so.4.5.4 ../../../../lib/libopencv_flann.so.4.5.4 ../../../../lib/libopencv_core.so.4.5.4 ../../../../lib/libopencv_cudev.so.4.5.4 -Wl,-rpath-link,/opencv/opencv/build/lib

I appended /usr/local/cuda-11.4/targets/aarch64-linux/lib to the last -rpath-link, i.e:

/usr/xcc/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++    -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG   -Wl,--gc-sections -Wl,--as-needed -Wl,-rpath-link,"/opencv/opencv/build/CMakeFiles/" CMakeFiles/opencv_waldboost_detector.dir/waldboost_detector.cpp.o -o ../../../../bin/opencv_waldboost_detector   -L/usr/local/cuda-11.4/targets/aarch64-linux/lib  -L/usr/local/cuda/targets/aarch64-linux/lib/stubs  -Wl,-rpath,/usr/local/cuda-11.4/targets/aarch64-linux/lib:/usr/local/cuda/targets/aarch64-linux/lib/stubs:/opencv/opencv/build/lib: ../../../../lib/libopencv_highgui.so.4.5.4 ../../../../lib/libopencv_xobjdetect.so.4.5.4 ../../../../lib/libopencv_videoio.so.4.5.4 ../../../../lib/libopencv_imgcodecs.so.4.5.4 ../../../../lib/libopencv_objdetect.so.4.5.4 ../../../../lib/libopencv_dnn.so.4.5.4 ../../../../lib/libopencv_calib3d.so.4.5.4 ../../../../lib/libopencv_features2d.so.4.5.4 ../../../../lib/libopencv_imgproc.so.4.5.4 ../../../../lib/libopencv_flann.so.4.5.4 ../../../../lib/libopencv_core.so.4.5.4 ../../../../lib/libopencv_cudev.so.4.5.4 -Wl,-rpath-link,/opencv/opencv/build/lib:/usr/local/cuda-11.4/targets/aarch64-linux/lib

This works and running make install afterwards finishes successfuly:

[100%] Built target opencv_waldboost_detector
Install the project...
-- Install configuration: "Release"
-- Installing: /output/bin/opencv_waldboost_detector
-- Set runtime path of "/output/bin/opencv_waldboost_detector" to "/output/lib:/usr/local/cuda-11.4/targets/aarch64-linux/lib:/usr/local/cuda/targets/aarch64-linux/lib/stubs"

Now I only need to understand how to inject this from cmake’s arguments. Any help would be more than welcome, thanks you!

I cannot add the links to the post but this are related:

i’d go radical and disable the waldboost module / sample code:

cmake
 ...
-DBUILD_opencv_xobjdetect=OFF

I actually started with that but I was worried that it won’t end up there.

Currently I managed to build with it by either using:

-D OPENCV_EXTRA_RPATH_LINK_PATH=/usr/local/cuda-11.4/targets/aarch64-linux/lib

OR

-D CUDA_STUB_TARGET_PATH=/opencv/opencv/build/lib:/usr/local/cuda-11.4/targets/aarch64-linux/lib

However, I am curious why the same thing didn’t happen to me while running more or less the same build with CUDA 10.2. Any ideas?

1 Like