Shadow problem when rendering a 3D object on an Aruco marker using Ovis module

Python
, ,
1

Hello,

I am working on my bachelor thesis about estimating the light source from a single image in context of augmented reality.

The estimation part is already working to a degree and now I wanted to visualize it using Aruco markers and Ovis module from opencv_contrib. I have built latest version of opencv (4.5.4-dev), opencv_contrib (4.5.4-dev) and Ogre3D (v13.1.1) and got it to render a sample 3D object onto an Aruco marker.

The only issue is, the shadow is way too large and it covers the whole scene (that is the reason the whole picture looks dark). I know that because I tried scaling the mesh down and that way the shadow also scales down, but that is not really a solution. I have also tried with opencv 4.4.0 and Ogre3D 1.12.0, but that version of Ogre doesn’t have scene shadows implented yet.

Rendered image: out|500x500

This is my code so far. I followed the documentation, the samples provided in opencv_contrib for ovis module and this tutorial on Ogre3D webisite here. In this tutorial there is a video that shows shadows working perfectly, but the provided code does not show how it was done.

import glob

import cv2
import numpy as np
from cv2 import aruco

from config import (active_camera, aruco_marker_images_dir, calibration_dir,
                    imsize)

image_paths = glob.glob(f'{aruco_marker_images_dir}/{active_camera}/*')

marker_dict = aruco.getPredefinedDictionary(aruco.DICT_6X6_100)

cameraMatrix = np.load(f'{calibration_dir}/{active_camera}/cameraMatrix.npy')
distCoeff = np.load(f'{calibration_dir}/{active_camera}/distCoeff.npy')


for image_path in image_paths:
    img = cv2.imread(image_path)
    img = cv2.resize(img, imsize)
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

    corners, ids, rejectedImgPoints = aruco.detectMarkers(gray, marker_dict)
    aruco.drawDetectedMarkers(img, corners, ids)

    rvecs, tvecs, _objPoints = aruco.estimatePoseSingleMarkers(corners, 0.045, cameraMatrix, distCoeff)

    for rvec, tvec in zip(rvecs, tvecs):
        img = aruco.drawAxis(img, cameraMatrix, distCoeff, rvec, tvec, 0.05)

cv2.ovis.addResourceLocation("packs/Sinbad.zip")

scene_settings = cv2.ovis.SCENE_INTERACTIVE | cv2.ovis.SCENE_AA | cv2.ovis.SCENE_SHADOWS

win = cv2.ovis.createWindow("Scene", imsize, scene_settings)

win.createEntity("figure", "Sinbad.mesh", (0, 0, 5), (1.57, 0, 0))
win.setCameraIntrinsics(cameraMatrix, imsize)
win.setCameraPose(tvec.ravel() * 100, rvec.ravel() * 100, invert=True)
win.createLightEntity("sun", (0, 0, 20))

while True:
    win.setBackground(img)
    win.update()

Questions:

  • Am I missing something about the Ovis scene setup?

  • Is there a better approach to render a 3D object on an Aruco marker with lighting and shadows?