First, the Jetson Nano is a low-end device, don’t expect it to perform miracles.
Anyway, did you check if Yolo is running on the CPU or the GPU? (use nvidia-smi to check the GPU useage)
I suggest to use the original framework instead of OpenCV. Darknet has OpenCV bindings, so you can pass cv::Mat images for processing (just make sure to enable OpenCV when compiling). And it is well optimized for CUDA.