Trouble with getting SolvePnp Output into Unity

I have been working on an object detection problem in opencv on Android using a c++ library inside of Unity. All of my opencv code is in c++ and I’m using version 4.4.0. I have calibrated my camera and I know the camera matrix and distortion parameters.

I use a custom tflite library to detect a series of points on my real world object, then I use solvePnP to find the rotation and translation for the 3D model of that object. In opencv land it all works really well. I can use solvepnp to get the rotation vector, rotation matrix, and translation vector. Then I can use projectPoints to reproject the points and when I do they line up more or less with the detected points (with a RMSE of 50 or less, usually around 26).

My trouble is in translating this success to Unity. I have taken the rotation vector (tried the matrix too), and the translation vector and tried to use it to translate my 3D model in Unity into the position. I cannot seem to get it right :slight_smile: It always seems to be off by some combination of 180,90 or 45 degrees. It’s never off wildly so I suspect I have not translated something properly.

To start off I have a model. I got the model points by placing my model in unity at 0,0,0 and exporting the points we will be detecting. Now Unity has a coordinate system like openCV except the y axis points up (right handed). I do not do anything to change the model into openCV (left handed) but I did once try negating the y values which did not help.

Next I detect the points and make sure they are lined up with the right model points and I run
cv::solvePnP(model_points, image_points, camera_matrix, dist_coeffs, rotation_vector, translation_vector, false, cv::SOLVEPNP_ITERATIVE );

I get the rotation matrix in case I need it

cv::Mat rotation_matrix(3,3,cv::DataType::type);
cv::Rodrigues(rotation_vector, rotation_matrix);

Next I reproject my points. I will see the found points, and reprojected points on the camera frame up in Unity. They will align very closely.
cv::projectPoints(model_points, rotation_vector, translation_vector, camera_matrix, dist_coeffs, projected_points.points);

I calculate the RMSE and check if it is 50 or less. If so I consider this a valid rotation and translation and I pass it to Unity.

Now I know Unity has a coordinate system difference so I will try to account for that. I have read everything I can find on people who have come before me :slight_smile: But I am still failing. Anyway My model starts at 0,0,0. My translation vector from opencv is passed in as the first three floats in an array. I negate the y axis and update the objects position. This part actually seems to work fine.
//yaxis is inverted
transform.position = new Vector3(t[0], -t[1], t[2]);

Now for the rotation. I have done this with both rotation vector and matrix and the results are the same. For rotation vector I understand that it is a vector showing the axis of rotation and its magnitude is the angle of rotation. So I calculate the magnitude in radians:
float theta = (float)(Math.Sqrt(x * x + y * y + z * z) * 180 / Math.PI);

then I negate the x and z components of the axis:
Vector3 axis = new Vector3(-x, y, -z);

Finally I assign the new rotation to the local rotation of my model.
Quaternion rot = Quaternion.AngleAxis(-theta, axis);
transform.localRotation = rot;

I’m always wrong though. I’ve also tried negative theta which does not work either. However it always seems to be off by 45,90,180 degrees. For example in one camera position I can fix it like this:

                                   transform.localEulerAngles.x - 90f,
                                    transform.localEulerAngles.y + 180f,

However when I move my camera to another angle it is wrong again. Anyway I’m stuck so I thought I would ask if anyone had any ideas what I might be doing wrong.

Thanks for any advice you can give.

left vs right hand coord system ?

Yes I think that’s the root of the problem. Maybe both with the starting model and then with interpreting the output. But I’m failing at being able to translate from openCV’s left hand coord system to Unity’s right hand. At least for the rotation.

what makes you think it’s left-handed?

Well I guess I don’t know. I just saw it referred to that way in a lot of posts. I know Unity has the +y axis pointing up to the sky and I though opencv had the +y axis pointing down to the earth like this:

x: thumb
y: index
z: middle

that makes those pictures, left-handed (unity), right-handed (opencv), right-handed (WMR), right-handed (OpenGL).

so I guess handedness is still opposite between Unity and OpenCV, but also opposite to what I’ve read in this thread so far.

a transformation matrix between left and right handedness would just negate Y but leave all the rest. it’s a mirroring operation, not a rotation or anything.