So I am following this #project, the result video is at the end.

I want a way to extract the global 3d coordinates of the nose and the vector pointing away. I understand that we get the translational vector (which is the nose coordinates… I think) and the rotational vector (I think this is the camera angle?). I just want the two points of the vector pointing away from the face in two 3D points.

Sorry, I am still very new to all of this. could you explain how to use cv.perspectiveTransform() and these functions to help understand this all.

Also I dabbled around with the first function and it is throwing an error:
ValueError: could not broadcast input array from shape (3,1) into shape (3,) on line T[:3, 3] = tvec

I’m gonna talk about “markers” purely because I’m kinda stuck in that lingo. it comes from working with AR markers. I imagine if you work with faces, the axes are placed similarly (Z is nose/forward, X to subject’s left, Y up)

the rotation maps (read each column)…

X to +X, with a bit of -Z but not much

Y to -Y (so it points up/far)

Z to -Z (so it points near/up)

which rotates marker-local coordinates/directions into camera-local points/directions. so that means you’re facing the marker. it probably faces a little to your bottom left.

the translation then simply moves all that away/far by 2571, and a little to the top left of the image center.

the matrix transforms object-local points/vectors to camera-local.

you can get a sense for what it does by figuring where the (object-local) axes get mapped to. X is (1,0,0), Y is (0,1,0), Z is (0,0,1). append a 1 for points or a 0 for vectors.

so the +Z vector gets mapped to… the third column.

now, where does a camera-local vector point if that is its value? a little to -X (left), a little to +Y (down), and mostly to -Z, so that’s near. and that is where the face’s +Z vector points, as viewed by the camera.