Hey,
I have docker image built on the top of nvidia/cuda:11.4.0-devel-ubuntu20.04
as base and dockcross ARM64/linux.
In the past I have cross compiled OpenCV 4.5.0 with CUDA 10.2 on a X86 for ARM64 successfully. While few dependencies have changed, I have almost managed to do the same with OpenCV 4.5.4 and CUDA 11.2 (as the base image states above).
However, I have encountered the following issue:
[ 79%] Building CXX object modules/video/CMakeFiles/opencv_test_video.dir/test/test_accum.cpp.o
[ 79%] Building CXX object modules/ximgproc/CMakeFiles/opencv_ximgproc.dir/src/anisodiff.cpp.o
[ 79%] Linking CXX executable ../../../../bin/opencv_waldboost_detector
/usr/xcc/aarch64-unknown-linux-gnu/lib/gcc/aarch64-unknown-linux-gnu/9.4.0/../../../../aarch64-unknown-linux-gnu/bin/ld.bfd: warning: libcudart.so.11.0, needed by ../../../../lib/libopencv_core.so.4.5.0, not found (try using -rpath or -rpath-link)
make[2]: *** [modules/xobjdetect/tools/waldboost_detector/CMakeFiles/opencv_waldboost_detector.dir/build.make:114: bin/opencv_waldboost_detector] Error 1
make[1]: *** [CMakeFiles/Makefile2:8565: modules/xobjdetect/tools/waldboost_detector/CMakeFiles/opencv_waldboost_detector.dir/all] Error 2
These are my cmake arguments:
cmake .. \
-D CUDA_INC_PATH=/usr/local/cuda/targets/aarch64-linux/include \
-D CMAKE_LIBRARY_PATH=/usr/local/cuda/targets/aarch64-linux/lib/stubs \
-D CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda/targets/aarch64-linux \
-D CMAKE_FIND_ROOT_PATH=/opencv/opencv/cmake \
-D CMAKE_TOOLCHAIN_FILE=../cmake_aarch64.toolchain \
-D OPENCV_EXTRA_MODULES_PATH=/opencv/opencv/opencv_contrib-${OPENCV_VERSION}/modules \
-D CMAKE_BUILD_TYPE=Release \
-D CMAKE_INSTALL_PREFIX=/output \
-D BUILD_TIFF=ON \
-D WITH_WEBP=OFF \
-D BUILD_TBB=ON \
-D BUILD_PNG=ON \
-D WITH_LAPACK=OFF \
-D BUILD_NEW_PYTHON_SUPPORT=ON \
-D BUILD_JPEG=ON \
-D BUILD_JASPER=ON \
-D BUILD_ZLIB=ON \
-D BUILD_EXAMPLES=OFF \
-D BUILD_JAVA=OFF \
-D BUILD_opencv_python2=OFF \
-D BUILD_opencv_python3=ON \
-D CUDA_NVCC_FLAGS="-D_FORCE_INLINES" \
-D ENABLE_NEON=ON \
-D WITH_OPENCL=OFF \
-D WITH_OPENMP=OFF \
-D WITH_FFMPEG=ON \
-D WITH_GSTREAMER=OFF \
-D WITH_GSTREAMER_0_10=OFF \
-D WITH_CUDA=ON \
-D WITH_GTK=ON \
-D WITH_VTK=OFF \
-D WITH_TBB=ON \
-D WITH_1394=OFF \
-D WITH_OPENEXR=OFF \
-D CUDA_ARCH_BIN=7.2 \
-D CUDA_ARCH_PTX="" \
-D BUILD_EXAMPLES=OFF \
-D INSTALL_C_EXAMPLES=ON \
-D INSTALL_TESTS=OFF
And libcudart.so.11.0
exists in the container:
ls -la /usr/local/cuda/targets/aarch64-linux/lib | grep libcudart
lrwxrwxrwx 1 root root 17 Nov 15 2021 libcudart.so -> libcudart.so.11.0
lrwxrwxrwx 1 root root 21 Nov 15 2021 libcudart.so.11.0 -> libcudart.so.11.4.167
-rw-r--r-- 1 root root 670808 Nov 15 2021 libcudart.so.11.4.167
-rw-r--r-- 1 root root 1078022 Nov 15 2021 libcudart_static.a
Do you have any idea why this is happening? At first I thought that it was an issue with OpenCV 4.5.0 and CUDA 11.4, so I changed OpenCV to 4.5.4, but it seems like it does not help.
Thank you
Shaked
Edit:
I have ran the build in verbose mode using make VERBOSE=1 -j$(nproc)
, then I saw that right after the aforementioned error, there are two files:
collect2: error: ld returned 1 exit status
make[2]: *** [modules/xobjdetect/tools/waldboost_detector/CMakeFiles/opencv_waldboost_detector.dir/build.make:115: bin/opencv_waldboost_detector] Error 1
make[2]: Leaving directory '/opencv/opencv-4.5.4/build'
make[1]: *** [CMakeFiles/Makefile2:8944: modules/xobjdetect/tools/waldboost_detector/CMakeFiles/opencv_waldboost_detector.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Looking at modules/xobjdetect/tools/waldboost_detector/CMakeFiles/opencv_waldboost_detector.dir/build.make
in line 115, I saw this command:
cd /opencv/opencv/build/modules/xobjdetect/tools/waldboost_detector && $(CMAKE_COMMAND) -E cmake_link_script CMakeFiles/opencv_waldboost_detector.dir/link.txt --verbose=$(VERBOSE)
Running it obviously generates the same error but gives a bit more information:
$ cmake -E cmake_link_script CMakeFiles/opencv_waldboost_detector.dir/link.txt --verbose=1
cmake -E cmake_link_script CMakeFiles/opencv_waldboost_detector.dir/link.txt --verbose=1
/usr/xcc/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++ -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG -Wl,--gc-sections -Wl,--as-needed -Wl,-rpath-link,"/opencv/opencv/build/CMakeFiles/" CMakeFiles/opencv_waldboost_detector.dir/waldboost_detector.cpp.o -o ../../../../bin/opencv_waldboost_detector -L/usr/local/cuda-11.4/targets/aarch64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib/stubs -Wl,-rpath,/usr/local/cuda-11.4/targets/aarch64-linux/lib:/usr/local/cuda/targets/aarch64-linux/lib/stubs:/opencv/opencv/build/lib: ../../../../lib/libopencv_highgui.so.4.5.4 ../../../../lib/libopencv_xobjdetect.so.4.5.4 ../../../../lib/libopencv_videoio.so.4.5.4 ../../../../lib/libopencv_imgcodecs.so.4.5.4 ../../../../lib/libopencv_objdetect.so.4.5.4 ../../../../lib/libopencv_dnn.so.4.5.4 ../../../../lib/libopencv_calib3d.so.4.5.4 ../../../../lib/libopencv_features2d.so.4.5.4 ../../../../lib/libopencv_imgproc.so.4.5.4 ../../../../lib/libopencv_flann.so.4.5.4 ../../../../lib/libopencv_core.so.4.5.4 ../../../../lib/libopencv_cudev.so.4.5.4 -Wl,-rpath-link,/opencv/opencv/build/lib
/usr/xcc/aarch64-unknown-linux-gnu/lib/gcc/aarch64-unknown-linux-gnu/9.4.0/../../../../aarch64-unknown-linux-gnu/bin/ld.bfd: warning: libcudart.so.11.0, needed by ../../../../lib/libopencv_core.so.4.5.4, not found (try using -rpath or -rpath-link)
I followed the suggestion in the error above:
(try using -rpath or -rpath-link)
And instead of using:
/usr/xcc/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++ -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG -Wl,--gc-sections -Wl,--as-needed -Wl,-rpath-link,"/opencv/opencv/build/CMakeFiles/" CMakeFiles/opencv_waldboost_detector.dir/waldboost_detector.cpp.o -o ../../../../bin/opencv_waldboost_detector -L/usr/local/cuda-11.4/targets/aarch64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib/stubs -Wl,-rpath,/usr/local/cuda-11.4/targets/aarch64-linux/lib:/usr/local/cuda/targets/aarch64-linux/lib/stubs:/opencv/opencv/build/lib: ../../../../lib/libopencv_highgui.so.4.5.4 ../../../../lib/libopencv_xobjdetect.so.4.5.4 ../../../../lib/libopencv_videoio.so.4.5.4 ../../../../lib/libopencv_imgcodecs.so.4.5.4 ../../../../lib/libopencv_objdetect.so.4.5.4 ../../../../lib/libopencv_dnn.so.4.5.4 ../../../../lib/libopencv_calib3d.so.4.5.4 ../../../../lib/libopencv_features2d.so.4.5.4 ../../../../lib/libopencv_imgproc.so.4.5.4 ../../../../lib/libopencv_flann.so.4.5.4 ../../../../lib/libopencv_core.so.4.5.4 ../../../../lib/libopencv_cudev.so.4.5.4 -Wl,-rpath-link,/opencv/opencv/build/lib
I appended /usr/local/cuda-11.4/targets/aarch64-linux/lib
to the last -rpath-link
, i.e:
/usr/xcc/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++ -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG -Wl,--gc-sections -Wl,--as-needed -Wl,-rpath-link,"/opencv/opencv/build/CMakeFiles/" CMakeFiles/opencv_waldboost_detector.dir/waldboost_detector.cpp.o -o ../../../../bin/opencv_waldboost_detector -L/usr/local/cuda-11.4/targets/aarch64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib/stubs -Wl,-rpath,/usr/local/cuda-11.4/targets/aarch64-linux/lib:/usr/local/cuda/targets/aarch64-linux/lib/stubs:/opencv/opencv/build/lib: ../../../../lib/libopencv_highgui.so.4.5.4 ../../../../lib/libopencv_xobjdetect.so.4.5.4 ../../../../lib/libopencv_videoio.so.4.5.4 ../../../../lib/libopencv_imgcodecs.so.4.5.4 ../../../../lib/libopencv_objdetect.so.4.5.4 ../../../../lib/libopencv_dnn.so.4.5.4 ../../../../lib/libopencv_calib3d.so.4.5.4 ../../../../lib/libopencv_features2d.so.4.5.4 ../../../../lib/libopencv_imgproc.so.4.5.4 ../../../../lib/libopencv_flann.so.4.5.4 ../../../../lib/libopencv_core.so.4.5.4 ../../../../lib/libopencv_cudev.so.4.5.4 -Wl,-rpath-link,/opencv/opencv/build/lib:/usr/local/cuda-11.4/targets/aarch64-linux/lib
This works and running make install
afterwards finishes successfuly:
[100%] Built target opencv_waldboost_detector
Install the project...
-- Install configuration: "Release"
-- Installing: /output/bin/opencv_waldboost_detector
-- Set runtime path of "/output/bin/opencv_waldboost_detector" to "/output/lib:/usr/local/cuda-11.4/targets/aarch64-linux/lib:/usr/local/cuda/targets/aarch64-linux/lib/stubs"
Now I only need to understand how to inject this from cmake’s arguments. Any help would be more than welcome, thanks you!